JobID jobID = new JobID(new SimpleDateFormat("yyyyMMddHHmm").format(new Date()), 0); TaskAttemptID taId = new TaskAttemptID(new TaskID(jobID, true, 0), 0); jobConf.set("mapred.tip.id", taId.getTaskID().toString()); jobConf.set("mapred.task.id", taId.toString()); jobConf.set("mapred.job.id", jobID.toString());
public static TaskID getTaskID(Configuration cfg) { // first try with the attempt since some Hadoop versions mix the two String taskAttemptId = HadoopCfgUtils.getTaskAttemptId(cfg); if (StringUtils.hasText(taskAttemptId)) { try { return TaskAttemptID.forName(taskAttemptId).getTaskID(); } catch (IllegalArgumentException ex) { // the task attempt is invalid (Tez in particular uses the wrong string - see #346) // try to fallback to task id return parseTaskIdFromTaskAttemptId(taskAttemptId); } } String taskIdProp = HadoopCfgUtils.getTaskId(cfg); // double-check task id bug in Hadoop 2.5.x if (StringUtils.hasText(taskIdProp) && !taskIdProp.contains("attempt")) { return TaskID.forName(taskIdProp); } return null; }
String taskId = t.getTaskAttemptId().getTaskID().toString(); String jobId = t.getTaskAttemptId().getJobID().toString(); if (firstError) {
String taskId = t.getTaskAttemptId().getTaskID().toString(); String jobId = t.getTaskAttemptId().getJobID().toString(); if (firstError) {
/** * test TaskAttemptID */ @SuppressWarnings( "deprecation" ) @Test (timeout=5000) public void testTaskAttemptID (){ TaskAttemptID task = new TaskAttemptID("001",2,true,3,4); assertEquals("attempt_001_0002_m_000003_4", TaskAttemptID.getTaskAttemptIDsPattern("001", 2, true, 3, 4)); assertEquals("task_001_0002_m_000003", task.getTaskID().toString()); assertEquals("attempt_001_0001_r_000002_3",TaskAttemptID.getTaskAttemptIDsPattern("001", 1, TaskType.REDUCE, 2, 3)); assertEquals("001_0001_m_000001_2", TaskAttemptID.getTaskAttemptIDsPatternWOPrefix("001",1, TaskType.MAP, 1, 2).toString()); }
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { System.err.println(taskid); this.reporter = reporter; if (TaskAttemptID.forName(taskid).getTaskID().getId() == taskWithCleanup) { if (taskid.endsWith("_0")) { System.exit(-1); } } }
/** Returns the TaskStatus for a particular taskid. */ TaskStatus getTaskStatus(TaskAttemptID taskid) { TaskInProgress tip = getTip(taskid.getTaskID()); return (tip == null ? null : tip.getTaskStatus(taskid)); }
@Override public String generateFileName(JobContext context, String topic, String brokerId, int partitionId, int count, long offset, String encodedPartition) { //adding taskId to filename to avoid name collision int taskId = ((TaskAttemptContext) context).getTaskAttemptID().getTaskID().getId(); return super.generateFileName(context, topic, String.valueOf(System.currentTimeMillis() / 1000), taskId, count, 0, encodedPartition); }
/** Returns the TaskStatus for a particular taskid. */ TaskStatus getTaskStatus(TaskAttemptID taskid) { TaskInProgress tip = getTip(taskid.getTaskID()); return (tip == null ? null : tip.getTaskStatus(taskid)); }
@Override public void configure(JobConf conf) { TaskAttemptID id = TaskAttemptID.forName(conf.get("mapred.task.id")); shouldFail = id.getId() == 0 && id.getTaskID().getId() == 0; } }
public static void setMRProperties( ProcessorContext context, Configuration config, boolean isMapperOutput ) { TaskAttemptID taskAttemptId = org.apache.tez.mapreduce.hadoop.mapreduce.TaskAttemptContextImpl .createMockTaskAttemptID( context.getApplicationId().getClusterTimestamp(), context.getTaskVertexIndex(), context.getApplicationId().getId(), context.getTaskIndex(), context.getTaskAttemptNumber(), isMapperOutput ); config.set( JobContext.TASK_ATTEMPT_ID, taskAttemptId.toString() ); config.set( JobContext.TASK_ID, taskAttemptId.getTaskID().toString() ); config.setBoolean( JobContext.TASK_ISMAP, isMapperOutput ); config.setInt( JobContext.TASK_PARTITION, taskAttemptId.getTaskID().getId() ); } }
public static void setMRProperties( ProcessorContext context, Configuration config, boolean isMapperOutput ) { TaskAttemptID taskAttemptId = org.apache.tez.mapreduce.hadoop.mapreduce.TaskAttemptContextImpl .createMockTaskAttemptID( context.getApplicationId().getClusterTimestamp(), context.getTaskVertexIndex(), context.getApplicationId().getId(), context.getTaskIndex(), context.getTaskAttemptNumber(), isMapperOutput ); config.set( JobContext.TASK_ATTEMPT_ID, taskAttemptId.toString() ); config.set( JobContext.TASK_ID, taskAttemptId.getTaskID().toString() ); config.setBoolean( JobContext.TASK_ISMAP, isMapperOutput ); config.setInt( JobContext.TASK_PARTITION, taskAttemptId.getTaskID().getId() ); } }
/** * Localize the given JobConf to be specific for this task. */ public void localizeConfiguration(JobConf conf) throws IOException { conf.set(JobContext.TASK_ID, taskId.getTaskID().toString()); conf.set(JobContext.TASK_ATTEMPT_ID, taskId.toString()); conf.setBoolean(JobContext.TASK_ISMAP, isMapTask()); conf.setInt(JobContext.TASK_PARTITION, partition); conf.set(JobContext.ID, taskId.getJobID().toString()); }
/** * Localize the given JobConf to be specific for this task. */ public void localizeConfiguration(JobConf conf) throws IOException { conf.set(JobContext.TASK_ID, taskId.getTaskID().toString()); conf.set(JobContext.TASK_ATTEMPT_ID, taskId.toString()); conf.setBoolean(JobContext.TASK_ISMAP, isMapTask()); conf.setInt(JobContext.TASK_PARTITION, partition); conf.set(JobContext.ID, taskId.getJobID().toString()); }
/** * Localize the given JobConf to be specific for this task. */ public void localizeConfiguration(JobConf conf) throws IOException { conf.set(JobContext.TASK_ID, taskId.getTaskID().toString()); conf.set(JobContext.TASK_ATTEMPT_ID, taskId.toString()); conf.setBoolean(JobContext.TASK_ISMAP, isMapTask()); conf.setInt(JobContext.TASK_PARTITION, partition); conf.set(JobContext.ID, taskId.getJobID().toString()); }
/** * Localize the given JobConf to be specific for this task. */ public void localizeConfiguration(JobConf conf) throws IOException { conf.set("mapred.tip.id", taskId.getTaskID().toString()); conf.set("mapred.task.id", taskId.toString()); conf.setBoolean("mapred.task.is.map", isMapTask()); conf.setInt("mapred.task.partition", partition); conf.set("mapred.job.id", taskId.getJobID().toString()); }
@Override public String[] getTaskJobIDs(TaskCompletionEvent t) { TaskID tid = t.getTaskAttemptId().getTaskID(); ret[0] = tid.toString(); ret[1] = tid.getJobID().toString(); return ret; }
public static TaskAttemptId toYarn( org.apache.hadoop.mapred.TaskAttemptID id) { TaskAttemptId taskAttemptId = recordFactory.newRecordInstance(TaskAttemptId.class); taskAttemptId.setTaskId(toYarn(id.getTaskID())); taskAttemptId.setId(id.getId()); return taskAttemptId; }
public static TaskAttemptId toYarn( org.apache.hadoop.mapred.TaskAttemptID id) { TaskAttemptId taskAttemptId = recordFactory.newRecordInstance(TaskAttemptId.class); taskAttemptId.setTaskId(toYarn(id.getTaskID())); taskAttemptId.setId(id.getId()); return taskAttemptId; }
public static TaskAttemptId toYarn( org.apache.hadoop.mapred.TaskAttemptID id) { TaskAttemptId taskAttemptId = recordFactory.newRecordInstance(TaskAttemptId.class); taskAttemptId.setTaskId(toYarn(id.getTaskID())); taskAttemptId.setId(id.getId()); return taskAttemptId; }