doesSessionBelongToUser(sessionHandle, schedulerDAO.getUser(instanceInfo.getJobId())); List<SchedulerJobInstanceRun> runList = instanceInfo.getInstanceRunList(); if (runList.size() == 0) { throw new LensException(LensSchedulerErrorCode.JOB_INSTANCE_IS_NOT_YET_RUN.getLensErrorInfo(), null, instanceHandle.getHandleIdString(), instanceInfo.getJobId().getHandleIdString()); } catch (InvalidStateTransitionException e) { throw new LensException(LensSchedulerErrorCode.INVALID_EVENT_FOR_JOB_INSTANCE.getLensErrorInfo(), e, ON_KILL.name(), latestRun.getInstanceState().name(), instanceInfo.getId().getHandleIdString(), instanceInfo.getJobId().getHandleIdString()); log.info("Killing instance {} for job {} ", instanceInfo.getId(), instanceInfo.getJobId()); return updateInstanceRun(latestRun, state); } else { log.info("Killing instance {} for job {} with query handle {} ", instanceInfo.getId(), instanceInfo.getJobId(), handle);
private void notifyRerun(SchedulerJobInstanceInfo instanceInfo) throws LensException { getEventService().notifyEvent( new SchedulerAlarmEvent(instanceInfo.getJobId(), new DateTime(instanceInfo.getScheduleTime()), SchedulerAlarmEvent.EventType.SCHEDULE, instanceInfo.getId())); }
continue; log.info("Recovering instance {} of job {} ", instanceInfo.getId(), instanceInfo.getJobId()); switch (run.getInstanceState()) { case WAITING: log.info("Re-running instance {} of job {}", instanceInfo.getId(), instanceInfo.getJobId()); } else { log.error("Not able to recover instance {} of job {}", instanceInfo.getId(), instanceInfo.getJobId()); sessionHandle = openSessionAsUser(schedulerDAO.getUser(instanceInfo.getJobId())); if (!checkQueryState(sessionHandle, run)) { log.info("Re-running instance {} of job {}", instanceInfo.getId(), instanceInfo.getJobId()); notifyRerun(instanceInfo);
@Test(priority = 2) public void testKillRunningInstance() throws Exception { long currentTime = System.currentTimeMillis(); XJob job = getTestJob("0/5 * * * * ?", queryString, currentTime, currentTime + 180000); LensSessionHandle sessionHandle = scheduler.openSessionAsUser(user); SchedulerJobHandle jobHandle = scheduler.submitAndScheduleJob(sessionHandle, job); // Let it run Thread.sleep(6000); List<SchedulerJobInstanceInfo> instanceHandleList = scheduler.getSchedulerDAO().getJobInstances(jobHandle); Assert.assertTrue(scheduler.killInstance(sessionHandle, instanceHandleList.get(0).getId())); Thread.sleep(2000); SchedulerJobInstanceInfo info = scheduler.getSchedulerDAO() .getSchedulerJobInstanceInfo(instanceHandleList.get(0).getId()); Assert.assertEquals(info.getInstanceRunList().size(), 1); Assert.assertEquals(info.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.RUNNING); // Query End event eventService.notifyEvent(mockQueryEnded(instanceHandleList.get(0).getId(), QueryStatus.Status.CANCELED)); Thread.sleep(2000); info = scheduler.getSchedulerDAO().getSchedulerJobInstanceInfo(instanceHandleList.get(0).getId()); Assert.assertEquals(info.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.KILLED); scheduler.expireJob(sessionHandle, jobHandle); Assert.assertEquals(scheduler.getSchedulerDAO().getJobState(jobHandle), SchedulerJobState.EXPIRED); scheduler.closeSession(sessionHandle); }
long currentTime = System.currentTimeMillis(); SchedulerJobInstanceHandle instanceHandle = new SchedulerJobInstanceHandle(UUID.randomUUID()); SchedulerJobInstanceInfo firstInstance = new SchedulerJobInstanceInfo(instanceHandle, jobHandle, currentTime, new ArrayList<SchedulerJobInstanceRun>()); SchedulerJobInstanceRun run1 = new SchedulerJobInstanceRun(instanceHandle, 1, new LensSessionHandle(UUID.randomUUID(), UUID.randomUUID()), currentTime, currentTime, "/tmp/", QueryHandle.fromString(UUID.randomUUID().toString()), SchedulerJobInstanceState.WAITING); instances.put(firstInstance.getId(), firstInstance); schedulerDAO.storeJobInstance(firstInstance); schedulerDAO.storeJobInstanceRun(run1); firstInstance.getInstanceRunList().add(run1); SchedulerJobInstanceInfo secondInstance = new SchedulerJobInstanceInfo(instanceHandle, jobHandle, currentTime, new ArrayList<SchedulerJobInstanceRun>()); SchedulerJobInstanceRun run2 = new SchedulerJobInstanceRun(instanceHandle, 1, new LensSessionHandle(UUID.randomUUID(), UUID.randomUUID()), currentTime, currentTime, "/tmp/", QueryHandle.fromString(UUID.randomUUID().toString()), SchedulerJobInstanceState.WAITING); instances.put(secondInstance.getId(), secondInstance); schedulerDAO.storeJobInstance(secondInstance); schedulerDAO.storeJobInstanceRun(run2); secondInstance.getInstanceRunList().add(run2); Assert.assertEquals(instances.get(handleList.get(0).getId()), instance1); Assert.assertEquals(instances.get(handleList.get(1).getId()), instance2);
instance = new SchedulerJobInstanceInfo(instanceHandle, jobHandle, scheduledTimeMillis, new ArrayList<SchedulerJobInstanceRun>()); queryName += "-" + scheduledTimeMillis; run = instance.getInstanceRunList().size() == 0 ? null : instance.getInstanceRunList().get(instance.getInstanceRunList().size() - 1); if (run == null || run.getInstanceState() != SchedulerJobInstanceState.LAUNCHING) { run = new SchedulerJobInstanceRun(instanceHandle, instance.getInstanceRunList().size() + 1, null, currentTime, currentTime, "N/A", null, SchedulerJobInstanceState.LAUNCHING); instance.getInstanceRunList().add(run); if (schedulerDAO.storeJobInstanceRun(run) != 1) { log.error("Exception occurred while storing the instance run for instance handle {} of job {}", instance, schedulerDAO.updateJobInstanceRun(run); } catch (InvalidStateTransitionException e1) { log.error("Can't make transition for instance " + instance.getId() + " of job " + instance.getJobId(), e);
public int storeJobInstance(SchedulerJobInstanceInfo instanceInfo) { try { return store.insertIntoJobInstanceTable(instanceInfo); } catch (SQLException e) { log.error("Error while storing job instance for {}", instanceInfo.getId(), e); return 0; } }
@Test(priority = 3) public void testUpdateJobInstance() { SchedulerJobInstanceHandle handle = instances.keySet().iterator().next(); SchedulerJobInstanceInfo info = instances.get(handle); SchedulerJobInstanceRun run = info.getInstanceRunList().get(0); run.setInstanceState(SchedulerJobInstanceState.LAUNCHING); schedulerDAO.updateJobInstanceRun(run); // Get the instance Assert.assertEquals(schedulerDAO.getSchedulerJobInstanceInfo(handle), info); }
private SchedulerJobInstanceInfo parseSchedulerInstance(Object[] instanceInfo) throws SQLException { SchedulerJobInstanceHandle id = SchedulerJobInstanceHandle.fromString((String) instanceInfo[0]); SchedulerJobHandle jobId = SchedulerJobHandle.fromString((String) instanceInfo[1]); long createdOn = (Long) instanceInfo[2]; // Get the Runs sorted by run id to make sure the last on the list is the latest run. String fetchSQL = "SELECT * FROM " + JOB_INSTANCE_RUN_TABLE + " WHERE " + COLUMN_ID + "=? ORDER BY " + COLUMN_RUN_ID; List<Object[]> instanceRuns = runner.query(fetchSQL, multipleRowsHandler, (String) instanceInfo[0]); List<SchedulerJobInstanceRun> runList = processInstanceRun(instanceRuns); return new SchedulerJobInstanceInfo(id, jobId, createdOn, runList); }
List<SchedulerJobInstanceInfo> instanceHandleList = scheduler.getSchedulerDAO().getJobInstances(jobHandle); eventService.notifyEvent(mockQueryEnded(instanceHandleList.get(0).getId(), QueryStatus.Status.FAILED)); Thread.sleep(1000); SchedulerJobInstanceInfo info = scheduler.getSchedulerDAO() .getSchedulerJobInstanceInfo(instanceHandleList.get(0).getId()); Assert.assertEquals(info.getInstanceRunList().size(), 1); Assert.assertEquals(info.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.FAILED); scheduler.rerunInstance(sessionHandle, instanceHandleList.get(0).getId()); Thread.sleep(5000); eventService.notifyEvent(mockQueryEnded(instanceHandleList.get(0).getId(), QueryStatus.Status.SUCCESSFUL)); Thread.sleep(1000); info = scheduler.getSchedulerDAO().getSchedulerJobInstanceInfo(instanceHandleList.get(0).getId()); Assert.assertEquals(info.getInstanceRunList().size(), 2); Assert.assertEquals(info.getInstanceRunList().get(1).getResultPath(), "/tmp/query1/result"); Assert.assertEquals(info.getInstanceRunList().get(1).getInstanceState(), SchedulerJobInstanceState.SUCCEEDED); scheduler.expireJob(sessionHandle, jobHandle); Assert.assertEquals(scheduler.getSchedulerDAO().getJobState(jobHandle), SchedulerJobState.EXPIRED);
SchedulerJobInstanceInfo instance = new SchedulerJobInstanceInfo(instanceHandle, jobHandle, currentTime, new ArrayList<SchedulerJobInstanceRun>()); SchedulerDAO store = scheduler.getSchedulerDAO(); SchedulerJobInstanceRun run = new SchedulerJobInstanceRun(instanceHandle, instance.getInstanceRunList().size() + 1, null, currentTime, 0, "N/A", null, SchedulerJobInstanceState.WAITING); instance.getInstanceRunList().add(run); store.storeJobInstanceRun(run); Assert.assertEquals(storedInfo.getInstanceRunList().size(), 2); SchedulerJobInstanceInfo previousInstanceInfo = scheduler.getInstanceDetails(instanceInfoList.get(0).getId()); Assert.assertEquals(previousInstanceInfo.getInstanceRunList().size(), 1); Assert.assertEquals(previousInstanceInfo.getInstanceRunList().get(0).getResultPath(), "/tmp/path"); Assert.assertEquals(previousInstanceInfo.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.SUCCEEDED); scheduler.expireJob(sessionHandle, jobHandle);
@Test public void updateInstance() throws Exception { String startDate = Util.modifyDate(Util.getCurrentDate(format), format, Calendar.SECOND, 10); String endDate = Util.modifyDate(startDate, format, Calendar.MINUTE, 3); XJob xJob = scheduleHelper.getXJob("job-update-action", QueryInventory.JDBC_DIM_QUERY, null, startDate, endDate, "0/20 * * * * ?"); String jobHandle = scheduleHelper.submitNScheduleJob(xJob, sessionHandleString); Thread.sleep(20000); List<SchedulerJobInstanceInfo> instanceList = scheduleHelper.getAllInstancesOfJob(jobHandle, "10", sessionHandleString); Thread.sleep(10000); LensAPIResult res = scheduleHelper.updateInstance(instanceList.get(0).getId().getHandleIdString(), "RERUN", sessionHandleString); Assert.assertTrue(res.getData().equals(true)); SchedulerJobInstanceInfo instanceInfo = scheduleHelper.getInstanceDetails(instanceList.get(0).getId() .getHandleIdString(), sessionHandleString).getData(); List<SchedulerJobInstanceRun> runList = instanceInfo.getInstanceRunList(); Assert.assertEquals(runList.size(), 2); Assert.assertEquals(runList.get(1).getRunId(), 2); }
/** * {@inheritDoc} */ @Override public void rerunInstance(LensSessionHandle sessionHandle, SchedulerJobInstanceHandle instanceHandle) throws LensException { SchedulerJobInstanceInfo instanceInfo = schedulerDAO.getSchedulerJobInstanceInfo(instanceHandle); doesSessionBelongToUser(sessionHandle, schedulerDAO.getUser(instanceInfo.getJobId())); SchedulerJobState currentState = schedulerDAO.getJobState(instanceInfo.getJobId()); if (currentState != SchedulerJobState.SCHEDULED) { throw new LensException(LensSchedulerErrorCode.JOB_IS_NOT_SCHEDULED.getLensErrorInfo(), null, instanceInfo.getJobId().getHandleIdString(), currentState); } // Get the latest run. List<SchedulerJobInstanceRun> runList = instanceInfo.getInstanceRunList(); if (runList.size() == 0) { throw new LensException(LensSchedulerErrorCode.JOB_INSTANCE_IS_NOT_YET_RUN.getLensErrorInfo(), null, instanceHandle.getHandleIdString(), instanceInfo.getJobId().getHandleIdString()); } SchedulerJobInstanceRun latestRun = runList.get(runList.size() - 1); try { latestRun.getInstanceState().nextTransition(ON_RERUN); notifyRerun(instanceInfo); log.info("Rerunning the instance with {} for job {} ", instanceHandle, instanceInfo.getJobId()); } catch (InvalidStateTransitionException e) { throw new LensException(LensSchedulerErrorCode.INVALID_EVENT_FOR_JOB_INSTANCE.getLensErrorInfo(), e, ON_RERUN.name(), latestRun.getInstanceState().name(), instanceInfo.getId().getHandleIdString(), instanceInfo.getJobId().getHandleIdString()); } }
/** * Inserts the job instance info object into job instance table * * @param instanceInfo * @return number of rows inserted. * @throws SQLException */ public int insertIntoJobInstanceTable(SchedulerJobInstanceInfo instanceInfo) throws SQLException { String insertSQL = "INSERT INTO " + JOB_INSTANCE_TABLE + " VALUES(?,?,?)"; return runner .update(insertSQL, instanceInfo.getId().getHandleIdString(), instanceInfo.getJobId().getHandleIdString(), instanceInfo.getScheduleTime()); }
@Test(enabled = false) public void testQueryNotFoundCaseOnRestart() throws Exception { String startDate = Util.modifyDate(Util.getCurrentDate(format), format, Calendar.SECOND, 2); String endDate = Util.modifyDate(startDate, format, Calendar.SECOND, 15); XJob xJob = scheduleHelper.getXJob("job-restart", QueryInventory.getSleepQuery("5"), null, startDate, endDate, "0/10 * * * * ?"); String jobHandle = scheduleHelper.submitNScheduleJob(xJob, sessionHandleString); Thread.sleep(10000); List<SchedulerJobInstanceInfo> instanceList = scheduleHelper.getAllInstancesOfJob(jobHandle, "10", sessionHandleString); lens.stop(); Util.runRemoteCommand("hadoop dfs -rmr /tmp/lensserver/query.*"); lens.start(); Thread.sleep(15000); // wait till instance gets killed and new instance is spawned String firstInstanceHandle = instanceList.get(0).getId().getHandleIdString(); SchedulerJobInstanceInfo instanceInfo = scheduleHelper.getInstanceDetails(firstInstanceHandle, sessionHandleString) .getData(); Assert.assertEquals(instanceInfo.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.KILLED); qHelper.waitForCompletion(instanceInfo.getInstanceRunList().get(1).getQueryHandle()); Thread.sleep(3000); instanceInfo = scheduleHelper.getInstanceDetails(firstInstanceHandle, sessionHandleString).getData(); Assert.assertEquals(instanceInfo.getInstanceRunList().get(1).getInstanceState(), SchedulerJobInstanceState.SUCCEEDED); }
List<SchedulerJobInstanceRun> runList = info.getInstanceRunList(); if (runList.size() == 0) { log.error("No instance run for {} with query {}", instanceHandle, queryContext.getQueryHandle()); latestRun.setResultPath(queryContext.getResultSetPath()); if (schedulerDAO.updateJobInstanceRun(latestRun) == 1) { log.info("Updated instance run {} for instance {} for job {} to {}", latestRun.getRunId(), info.getId(), info.getJobId(), state); } else { log.error("Failed to update instance run {} for instance {} for job {} to {}", latestRun.getRunId(), info.getId(), info.getJobId(), state);
@Override public void run() { while (!Thread.currentThread().isInterrupted()) { // Get all the instance runs which are waiting. List<SchedulerJobInstanceRun> instanceRuns = schedulerDAO.getInstanceRuns(SchedulerJobInstanceState.WAITING); for (SchedulerJobInstanceRun run : instanceRuns) { SchedulerJobInstanceInfo instanceInfo = schedulerDAO.getSchedulerJobInstanceInfo(run.getHandle()); try { run.setInstanceState(run.getInstanceState().nextTransition(ON_PREPARE)); run.setEndTime(System.currentTimeMillis()); schedulerDAO.updateJobInstanceRun(run); eventService.notifyEvent( new SchedulerAlarmEvent(instanceInfo.getJobId(), new DateTime(instanceInfo.getScheduleTime()), SchedulerAlarmEvent.EventType.SCHEDULE, instanceInfo.getId())); } catch (LensException e) { log.error("Not able to notify schedule event for job {} and instanceId {}", instanceInfo.getJobId(), instanceInfo.getId()); } catch (InvalidStateTransitionException e) { log.error("Wrong state transition", e); } } } try { Thread.sleep(delay); } catch (InterruptedException e) { log.warn("Thread WaitingInstanceHandler was inerrupted", e); } } }
@Test(priority = 1) public void testScheduler() throws Exception { LensServices.get().init(LensServerConf.getHiveConf()); LensServices.get().start(); scheduler = LensServices.get().getService(SchedulerService.NAME); eventService = LensServices.get().getService(EventServiceImpl.NAME); setupQueryService(scheduler); LensSessionHandle sessionHandle = scheduler.openSessionAsUser(user); long currentTime = System.currentTimeMillis(); XJob job = getTestJob("0/5 * * * * ?", queryString, currentTime, currentTime + 15000); SchedulerJobHandle jobHandle = scheduler.submitAndScheduleJob(sessionHandle, job); Assert.assertNotNull(jobHandle); Assert.assertEquals(scheduler.getSchedulerDAO().getJobState(jobHandle), SchedulerJobState.SCHEDULED); // Wait for job to finish Thread.sleep(30000); List<SchedulerJobInstanceInfo> instanceHandleList = scheduler.getSchedulerDAO().getJobInstances(jobHandle); Assert.assertEquals(instanceHandleList.size() >= 3, true); Assert.assertEquals(scheduler.getSchedulerDAO().getJobState(jobHandle), SchedulerJobState.EXPIRED); // SuccessFul query eventService.notifyEvent(mockQueryEnded(instanceHandleList.get(0).getId(), QueryStatus.Status.SUCCESSFUL)); // Wait, for event to get processed Thread.sleep(2000); // Check the instance value SchedulerJobInstanceInfo info = scheduler.getSchedulerDAO() .getSchedulerJobInstanceInfo(instanceHandleList.get(0).getId()); Assert.assertEquals(info.getInstanceRunList().size(), 1); Assert.assertEquals(info.getInstanceRunList().get(0).getResultPath(), "/tmp/query1/result"); Assert.assertEquals(info.getInstanceRunList().get(0).getInstanceState(), SchedulerJobInstanceState.SUCCEEDED); scheduler.closeSession(sessionHandle); }
@Test public void testRunningInstanceOnRestart() throws Exception { String startDate = Util.modifyDate(Util.getCurrentDate(format), format, Calendar.SECOND, 2); String endDate = Util.modifyDate(startDate, format, Calendar.SECOND, 15); XJob xJob = scheduleHelper.getXJob("job-restart", QueryInventory.getSleepQuery("5"), null, startDate, endDate, "0/10 * * * * ?"); String jobHandle = scheduleHelper.submitNScheduleJob(xJob, sessionHandleString); Thread.sleep(10000); List<SchedulerJobInstanceInfo> instanceList = scheduleHelper.getAllInstancesOfJob(jobHandle, "10", sessionHandleString); lens.restart(); SchedulerJobInstanceInfo instanceInfo = scheduleHelper.getInstanceDetails(instanceList.get(0).getId() .getHandleIdString(), sessionHandleString).getData(); SchedulerJobInstanceRun instanceRun = instanceInfo.getInstanceRunList().get(0); qHelper.waitForCompletion(instanceRun.getQueryHandle()); Assert.assertEquals(instanceRun.getInstanceState(), SchedulerJobInstanceState.SUCCEEDED); }