ReflectionUtils.newInstance(storageHandler.getSerDeClass(), currTaskContext.getJobConf()); try { InternalUtil.initializeOutputSerDe(currSerDe, currTaskContext.getConfiguration(), currTaskContext.getJobConf()); currTaskContext.getJobConf().getOutputCommitter(); currTaskContext.getJobConf(), childPath.toString(), InternalUtil.createReporter(currTaskContext));
@Override public void abortTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_ABORT_FILE_NAME); }
private static Path getOutputPath(TaskAttemptContext context) { JobConf conf = context.getJobConf(); return FileOutputFormat.getOutputPath(conf); }
@Override public void setupTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_SETUP_FILE_NAME); }
@Override public void commitTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_COMMIT_FILE_NAME); }
@Override public void commitTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_COMMIT_FILE_NAME); }
@Override public void abortTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_ABORT_FILE_NAME); }
@Override public void setupTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_SETUP_FILE_NAME); }
@Override public void commitTask(TaskAttemptContext taskContext) throws IOException { writeFile(taskContext.getJobConf(), TASK_COMMIT_FILE_NAME); }
private static Path getOutputPath(TaskAttemptContext context) { JobConf conf = context.getJobConf(); return FileOutputFormat.getOutputPath(conf); }
private static Path getOutputPath(TaskAttemptContext context) { JobConf conf = context.getJobConf(); return FileOutputFormat.getOutputPath(conf); }
public void commitTask(TaskAttemptContext context) throws IOException { Path taskOutputPath = getTempTaskOutputPath(context); TaskAttemptID attemptId = context.getTaskAttemptID(); JobConf job = context.getJobConf(); if (taskOutputPath != null) { FileSystem fs = taskOutputPath.getFileSystem(job); if (fs.exists(taskOutputPath)) { throw new IOException(); } } } }
Path getTempTaskOutputPath(TaskAttemptContext taskContext) throws IOException { JobConf conf = taskContext.getJobConf(); Path outputPath = FileOutputFormat.getOutputPath(conf); if (outputPath != null) { Path p = new Path(outputPath, (FileOutputCommitter.TEMP_DIR_NAME + Path.SEPARATOR + "_" + taskContext.getTaskAttemptID().toString())); FileSystem fs = p.getFileSystem(conf); return p.makeQualified(fs); } return null; }
private Path getTaskAttemptPath(TaskAttemptContext context, Path out) throws IOException { Path workPath = FileOutputFormat.getWorkOutputPath(context.getJobConf()); if(workPath == null && out != null) { return org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter .getTaskAttemptPath(context, out); } return workPath; }
private Path getTaskAttemptPath(TaskAttemptContext context, Path out) throws IOException { Path workPath = FileOutputFormat.getWorkOutputPath(context.getJobConf()); if(workPath == null && out != null) { return org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter .getTaskAttemptPath(context, out); } return workPath; }
public void abortTask(TaskAttemptContext context) throws IOException { Path taskOutputPath = getTempTaskOutputPath(context); if (taskOutputPath != null) { FileSystem fs = taskOutputPath.getFileSystem(context.getJobConf()); context.getProgressible().progress(); fs.delete(taskOutputPath, true); } }
private Path getTaskAttemptPath(TaskAttemptContext context, Path out) throws IOException { Path workPath = FileOutputFormat.getWorkOutputPath(context.getJobConf()); if(workPath == null && out != null) { return org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter .getTaskAttemptPath(context, out); } return workPath; }
private Path getTaskAttemptPath(TaskAttemptContext context, Path out) throws IOException { Path workPath = FileOutputFormat.getWorkOutputPath(context.getJobConf()); if(workPath == null && out != null) { return org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter .getTaskAttemptPath(context, out); } return workPath; }
public void abortTask(TaskAttemptContext context) { Path taskOutputPath = getTempTaskOutputPath(context); try { if (taskOutputPath != null) { FileSystem fs = taskOutputPath.getFileSystem(context.getJobConf()); context.getProgressible().progress(); fs.delete(taskOutputPath, true); } } catch (IOException ie) { LOG.warn("Error discarding output" + StringUtils.stringifyException(ie)); } }
public void commitTask(TaskAttemptContext context) throws IOException { Path taskOutputPath = getTaskAttemptPath(context); TaskAttemptID attemptId = context.getTaskAttemptID(); JobConf job = context.getJobConf(); if (taskOutputPath != null) { FileSystem fs = taskOutputPath.getFileSystem(job); if (fs.exists(taskOutputPath)) { throw new IOException(); } } } }