TerminateFragmentRequestProto.newBuilder().setQueryIdentifier( constructQueryIdentifierProto( taskAttemptId.getTaskID().getVertexID().getDAGId().getId())) .setFragmentIdentifierString(taskAttemptId.toString()).build(); communicator.sendTerminateFragment(request, nodeId.getHostname(), nodeId.getPort(),
private SubmitWorkRequestProto createRequest(int fragmentNumber, int numSelfAndUpstreamTasks, int numSelfAndUpstreamComplete, int dagStartTime, int attemptStartTime, int withinDagPriority, String dagName) throws IOException { ApplicationId appId = ApplicationId.newInstance(9999, 72); TezDAGID dagId = TezDAGID.getInstance(appId, 1); TezVertexID vId = TezVertexID.getInstance(dagId, 35); return LlapDaemonTestUtils.buildSubmitProtoRequest(fragmentNumber, appId.toString(), dagId.getId(), vId.getId(), dagName, dagStartTime, attemptStartTime, numSelfAndUpstreamTasks, numSelfAndUpstreamComplete, withinDagPriority, new Credentials()); }
super.registerRunningTaskAttempt(containerId, taskSpec, additionalResources, credentials, credentialsChanged, priority); int dagId = taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId(); if (currentQueryIdentifierProto == null || (dagId != currentQueryIdentifierProto.getDagIndex())) {
int dagIdentifier = taskAttemptId.getTaskID().getVertexID().getDAGId().getId();
.setIsGuaranteed(newState).setFragmentIdentifierString(attemptId.toString()) .setQueryIdentifier(constructQueryIdentifierProto( attemptId.getTaskID().getVertexID().getDAGId().getId())).build();
private SubmitWorkRequestProto constructSubmitWorkRequest(ContainerId containerId, TaskSpec taskSpec, FragmentRuntimeInfo fragmentRuntimeInfo, String hiveQueryId) throws IOException { SubmitWorkRequestProto.Builder builder = SubmitWorkRequestProto.newBuilder(); builder.setFragmentNumber(taskSpec.getTaskAttemptID().getTaskID().getId()); builder.setAttemptNumber(taskSpec.getTaskAttemptID().getId()); builder.setContainerIdString(containerId.toString()); builder.setAmHost(getAmHostString()); builder.setAmPort(getAddress().getPort()); Preconditions.checkState(currentQueryIdentifierProto.getDagIndex() == taskSpec.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId()); builder.setCredentialsBinary( getCredentials(getContext().getCurrentDagInfo().getCredentials())); builder.setWorkSpec(VertexOrBinary.newBuilder().setVertex(Converters.constructSignableVertexSpec( taskSpec, currentQueryIdentifierProto, getTokenIdentifier(), user, hiveQueryId)).build()); // Don't call builder.setWorkSpecSignature() - Tez doesn't sign fragments builder.setFragmentRuntimeInfo(fragmentRuntimeInfo); if (scheduler != null) { // May be null in tests // TODO: see javadoc builder.setIsGuaranteed(scheduler.isInitialGuaranteed(taskSpec.getTaskAttemptID())); } return builder.build(); }
.setAppAttemptNumber(333).setDagIndex(tezDagId.getId()).build();
assertEquals("hiveQueryId", vertexProto.getHiveQueryId()); assertEquals(appId.toString(), vertexProto.getQueryIdentifier().getApplicationIdString()); assertEquals(tezDagId.getId(), vertexProto.getQueryIdentifier().getDagIndex()); assertEquals(333, vertexProto.getQueryIdentifier().getAppAttemptNumber()); assertEquals(tezVertexId.getId(), vertexProto.getVertexIndex());
@Override public int getIndex() { return dagId.getId(); }
@Override public int getIdentifier() { return dagId.getId(); }
.setApplicationIdString(appId.toString()) .setAppAttemptNumber(0) .setDagIndex(dagId.getId()) .build()) .setVertexIndex(vId.getId())
@SuppressWarnings("unchecked") @Override public void handle(DAGEvent event) { DAG dag = context.getCurrentDAG(); int eventDagIndex = event.getDAGId().getId(); if (dag == null || eventDagIndex != dag.getID().getId()) { return; // event not relevant any more } ((EventHandler<DAGEvent>)dag).handle(event); } }
public static org.apache.hadoop.mapreduce.JobID toJobID(TezDAGID id) { return new JobID( String.valueOf(id.getApplicationId().getClusterTimestamp()), id.getId()); }
public int getDagIdentifier() { return taskAttemptId.getTaskID().getVertexID().getDAGId().getId(); }
@Override public int getDagIdentifier() { return taskAttemptID.getTaskID().getVertexID().getDAGId().getId(); }
@SuppressWarnings("unchecked") @Override public void handle(VertexEvent event) { DAG dag = context.getCurrentDAG(); int eventDagIndex = event.getVertexId().getDAGId().getId(); if (dag == null || eventDagIndex != dag.getID().getId()) { return; // event not relevant any more } Vertex vertex = dag.getVertex(event.getVertexId()); ((EventHandler<VertexEvent>) vertex).handle(event); } }
DAG checkAndGetDAGFromRequest() { DAG dag = null; int errorCode = HttpServletResponse.SC_OK; String message = null; Exception ex = null; try { int dagID = getQueryParamInt(WebUIService.DAG_ID); dag = appContext.getCurrentDAG(); if (dag == null || dag.getID().getId() != dagID) { errorCode = HttpServletResponse.SC_NOT_FOUND; message = "Not current Dag: " + dagID; } } catch (NumberFormatException e) { errorCode = HttpServletResponse.SC_BAD_REQUEST; message = "Invalid dag id"; ex = e; } if (errorCode != HttpServletResponse.SC_OK) { dag = null; sendErrorResponse(errorCode, message, ex); } return dag; }
public static void processNonFatalServiceErrorReport(String entityString, ServicePluginError servicePluginError, String diagnostics, DagInfo dagInfo, AppContext appContext, String componentName) { String message = "Error reported by " + componentName + " [" + entityString + "][" + servicePluginError + "] " + (diagnostics == null ? "" : diagnostics); if (dagInfo != null) { DAG dag = appContext.getCurrentDAG(); if (dag != null && dag.getID().getId() == dagInfo.getIndex()) { TezDAGID dagId = dag.getID(); // Send a kill message only if it is the same dag. LOG.warn(message + ", Failing dag: [" + dagInfo.getName() + ", " + dagId + "]"); sendEvent(appContext, new DAGEventTerminateDag(dagId, DAGTerminationCause.SERVICE_PLUGIN_ERROR, message)); } } else { LOG.warn("No current dag name provided. Not acting on " + message); } }
@SuppressWarnings("unchecked") @Override public void handle(TaskEvent event) { DAG dag = context.getCurrentDAG(); int eventDagIndex = event.getTaskID().getVertexID().getDAGId().getId(); if (dag == null || eventDagIndex != dag.getID().getId()) { return; // event not relevant any more } Task task = dag.getVertex(event.getTaskID().getVertexID()). getTask(event.getTaskID()); ((EventHandler<TaskEvent>)task).handle(event); } }
@SuppressWarnings("unchecked") @Override public void handle(TaskAttemptEvent event) { DAG dag = context.getCurrentDAG(); int eventDagIndex = event.getTaskAttemptID().getTaskID().getVertexID().getDAGId().getId(); if (dag == null || eventDagIndex != dag.getID().getId()) { return; // event not relevant any more } Task task = dag.getVertex(event.getTaskAttemptID().getTaskID().getVertexID()). getTask(event.getTaskAttemptID().getTaskID()); TaskAttempt attempt = task.getAttempt(event.getTaskAttemptID()); ((EventHandler<TaskAttemptEvent>) attempt).handle(event); } }