private boolean rollWriterWithRetries() { for (int i = 0; i < rollRetries && isRunning(); ++i) { if (i > 0) Threads.sleepWithoutInterrupt(waitBeforeRoll * i); try { if (rollWriter()) { return true; } } catch (IOException e) { LOG.warn("Unable to roll the log, attempt=" + (i + 1), e); } } LOG.error(HBaseMarkers.FATAL, "Unable to roll the log"); return false; }
private long syncSlots() throws Throwable { int retry = 0; int logRolled = 0; long totalSynced = 0; do { try { totalSynced = syncSlots(stream, slots, 0, slotIndex); break; } catch (Throwable e) { LOG.warn("unable to sync slots, retry=" + retry); if (++retry >= maxRetriesBeforeRoll) { if (logRolled >= maxSyncFailureRoll && isRunning()) { LOG.error("Sync slots after log roll failed, abort.", e); throw e; } if (!rollWriterWithRetries()) { throw e; } logRolled++; retry = 0; } } } while (isRunning()); return totalSynced; }
private long pushData(final PushType type, final ByteSlot slot, final long procId, final long[] subProcIds) { if (!isRunning()) { throw new RuntimeException("the store must be running before inserting data"); if (!isRunning()) { throw new RuntimeException("store no longer running"); } else if (isSyncAborted()) { while (pushSyncId == syncId.get() && isRunning()) { syncCond.await();
/** * Make sure that the file set are gotten by calling {@link #getLogFiles()}, where we will sort * the file set by log id. * @return Max-LogID of the specified log file set */ private long initOldLogs(FileStatus[] logFiles) throws IOException { if (logFiles == null || logFiles.length == 0) { return 0L; } long maxLogId = 0; for (int i = 0; i < logFiles.length; ++i) { final Path logPath = logFiles[i].getPath(); leaseRecovery.recoverFileLease(fs, logPath); if (!isRunning()) { throw new IOException("wal aborting"); } maxLogId = Math.max(maxLogId, getLogIdFromName(logPath.getName())); ProcedureWALFile log = initOldLog(logFiles[i], this.walArchiveDir); if (log != null) { this.logs.add(log); } } initTrackerFromOldLogs(); return maxLogId; }
private boolean rollWriter() throws IOException { if (!isRunning()) { return false; } // Create new state-log if (!rollWriter(flushLogId + 1)) { LOG.warn("someone else has already created log {}", flushLogId); return false; } // We have the lease on the log, // but we should check if someone else has created new files if (getMaxLogId(getLogFiles()) > flushLogId) { LOG.warn("Someone else created new logs. Expected maxLogId < {}", flushLogId); logs.getLast().removeFile(this.walArchiveDir); return false; } // We have the lease on the log return true; }
LOG.debug("Starting WAL Procedure Store lease recovery"); boolean afterFirstAttempt = false; while (isRunning()) {
/** * If last log's tracker is not null, use it as {@link #storeTracker}. Otherwise, set storeTracker * as partial, and let {@link ProcedureWALFormatReader} rebuild it using entries in the log. */ private void initTrackerFromOldLogs() { if (logs.isEmpty() || !isRunning()) { return; } ProcedureWALFile log = logs.getLast(); if (!log.getTracker().isPartial()) { storeTracker.resetTo(log.getTracker()); } else { storeTracker.reset(); storeTracker.setPartialFlag(true); } }
lock.lock(); try { while (isRunning()) { try {
@Test(expected=RuntimeException.class) public void testWalAbortOnLowReplication() throws Exception { setupDFS(); assertEquals(3, UTIL.getDFSCluster().getDataNodes().size()); LOG.info("Stop DataNode"); UTIL.getDFSCluster().stopDataNode(0); assertEquals(2, UTIL.getDFSCluster().getDataNodes().size()); store.insert(new TestProcedure(1, -1), null); for (long i = 2; store.isRunning(); ++i) { assertEquals(2, UTIL.getDFSCluster().getDataNodes().size()); store.insert(new TestProcedure(i, -1), null); Thread.sleep(100); } assertFalse(store.isRunning()); }
assertFalse(store.isRunning()); assertTrue(reCount.toString(), reCount.get() >= store.getNumThreads() && reCount.get() < thread.length);
@Test public void testWalRollOnLowReplication() throws Exception { UTIL.getConfiguration().setInt("dfs.namenode.replication.min", 1); setupDFS(); int dnCount = 0; store.insert(new TestProcedure(1, -1), null); UTIL.getDFSCluster().restartDataNode(dnCount); for (long i = 2; i < 100; ++i) { store.insert(new TestProcedure(i, -1), null); waitForNumReplicas(3); Thread.sleep(100); if ((i % 30) == 0) { LOG.info("Restart Data Node"); UTIL.getDFSCluster().restartDataNode(++dnCount % 3); } } assertTrue(store.isRunning()); }
private boolean rollWriterWithRetries() { for (int i = 0; i < rollRetries && isRunning(); ++i) { if (i > 0) Threads.sleepWithoutInterrupt(waitBeforeRoll * i); try { if (rollWriter()) { return true; } } catch (IOException e) { LOG.warn("Unable to roll the log, attempt=" + (i + 1), e); } } LOG.error(HBaseMarkers.FATAL, "Unable to roll the log"); return false; }
private boolean rollWriterWithRetries() { for (int i = 0; i < rollRetries && isRunning(); ++i) { if (i > 0) Threads.sleepWithoutInterrupt(waitBeforeRoll * i); try { if (rollWriter()) { return true; } } catch (IOException e) { LOG.warn("Unable to roll the log, attempt=" + (i + 1), e); } } LOG.error(HBaseMarkers.FATAL, "Unable to roll the log"); return false; }
/** * @return Max-LogID of the specified log file set */ private long initOldLogs(final FileStatus[] logFiles) throws IOException { this.logs.clear(); long maxLogId = 0; if (logFiles != null && logFiles.length > 0) { for (int i = 0; i < logFiles.length; ++i) { final Path logPath = logFiles[i].getPath(); leaseRecovery.recoverFileLease(fs, logPath); if (!isRunning()) { throw new IOException("wal aborting"); } maxLogId = Math.max(maxLogId, getLogIdFromName(logPath.getName())); ProcedureWALFile log = initOldLog(logFiles[i], this.walArchiveDir); if (log != null) { this.logs.add(log); } } Collections.sort(this.logs); initTrackerFromOldLogs(); } return maxLogId; }
private boolean rollWriter() throws IOException { if (!isRunning()) return false; // Create new state-log if (!rollWriter(flushLogId + 1)) { LOG.warn("someone else has already created log " + flushLogId); return false; } // We have the lease on the log, // but we should check if someone else has created new files if (getMaxLogId(getLogFiles()) > flushLogId) { LOG.warn("Someone else created new logs. Expected maxLogId < " + flushLogId); logs.getLast().removeFile(this.walArchiveDir); return false; } // We have the lease on the log return true; }
private boolean rollWriter() throws IOException { if (!isRunning()) { return false; } // Create new state-log if (!rollWriter(flushLogId + 1)) { LOG.warn("someone else has already created log {}", flushLogId); return false; } // We have the lease on the log, // but we should check if someone else has created new files if (getMaxLogId(getLogFiles()) > flushLogId) { LOG.warn("Someone else created new logs. Expected maxLogId < {}", flushLogId); logs.getLast().removeFile(this.walArchiveDir); return false; } // We have the lease on the log return true; }
/** * If last log's tracker is not null, use it as {@link #storeTracker}. Otherwise, set storeTracker * as partial, and let {@link ProcedureWALFormatReader} rebuild it using entries in the log. */ private void initTrackerFromOldLogs() { if (logs.isEmpty() || !isRunning()) { return; } ProcedureWALFile log = logs.getLast(); if (!log.getTracker().isPartial()) { storeTracker.resetTo(log.getTracker()); } else { storeTracker.reset(); storeTracker.setPartialFlag(true); } }
/** * If last log's tracker is not null, use it as {@link #storeTracker}. * Otherwise, set storeTracker as partial, and let {@link ProcedureWALFormatReader} rebuild * it using entries in the log. */ private void initTrackerFromOldLogs() { if (logs.isEmpty() || !isRunning()) return; ProcedureWALFile log = logs.getLast(); if (!log.getTracker().isPartial()) { storeTracker.resetTo(log.getTracker()); } else { storeTracker.reset(); storeTracker.setPartialFlag(true); } }
@Test(expected=RuntimeException.class) public void testWalAbortOnLowReplication() throws Exception { setupDFS(); assertEquals(3, UTIL.getDFSCluster().getDataNodes().size()); LOG.info("Stop DataNode"); UTIL.getDFSCluster().stopDataNode(0); assertEquals(2, UTIL.getDFSCluster().getDataNodes().size()); store.insert(new TestProcedure(1, -1), null); for (long i = 2; store.isRunning(); ++i) { assertEquals(2, UTIL.getDFSCluster().getDataNodes().size()); store.insert(new TestProcedure(i, -1), null); Thread.sleep(100); } assertFalse(store.isRunning()); }
@Test public void testWalRollOnLowReplication() throws Exception { UTIL.getConfiguration().setInt("dfs.namenode.replication.min", 1); setupDFS(); int dnCount = 0; store.insert(new TestProcedure(1, -1), null); UTIL.getDFSCluster().restartDataNode(dnCount); for (long i = 2; i < 100; ++i) { store.insert(new TestProcedure(i, -1), null); waitForNumReplicas(3); Thread.sleep(100); if ((i % 30) == 0) { LOG.info("Restart Data Node"); UTIL.getDFSCluster().restartDataNode(++dnCount % 3); } } assertTrue(store.isRunning()); }