private static void accumWhenZero(GraphManager graphManager, int stageId, long duration) { PronghornStage stage = getStage(graphManager, stageId); if ((stage instanceof PipeCleanerStage) || (stage instanceof ReplicatorStage) ) { //these can be very fast and should not be logged. } else { int x = totalZeroDurations.incrementAndGet(); if (Integer.numberOfLeadingZeros(x-1)!=Integer.numberOfLeadingZeros(x)) { if (duration<0) { logger.info("Bad duration {}",duration); } else { logger.trace("Warning: the OS has measured stages taking zero ms {} times. " + "Most recent case is for {}. ", x, stage); } } graphManager.stageRunNS[stageId] += defaultDurationWhenZero; } }
protected void logStageScheduleRates() { int totalStages = GraphManager.countStages(gm); for(int i=1;i<=totalStages;i++) { PronghornStage s = GraphManager.getStage(gm, i); if (null != s) { Object rate = GraphManager.getNota(gm, i, GraphManager.SCHEDULE_RATE, null); if (null == rate) { logger.debug("{} is running without breaks",s); } else { logger.debug("{} is running at rate of {}",s,rate); } } } }
private static int[] buildCountOfStagesForEachThread(GraphManager graphManager, int rootCounter, int[] rootMemberCounter, IntHashTable rootsTable, IntArrayHolder lastKnownRoot) { Arrays.fill(rootMemberCounter, 0); //long start = System.nanoTime(); int countStages = GraphManager.countStages(graphManager); for(int stages=0; stages <= countStages; stages++) { PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { rootMemberCounter[rootId(stage.stageId, rootsTable, lastKnownRoot)]++; } } //System.err.println("performance issue here calling this too often "+countStages+" duration "+(System.nanoTime()-start)); //logger.info("group counts "+Arrays.toString(rootMemberCounter)); return rootMemberCounter; }
private int[] buildCountOfStagesForEachThread(GraphManager graphManager, int rootCounter, int[] rootMemberCounter) { //long start = System.nanoTime(); Arrays.fill(rootMemberCounter, 0); int countStages = GraphManager.countStages(graphManager); for(int stages=1; stages <= countStages; stages++) { PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { int rootId = rootId(stage.stageId, rootsTable); rootMemberCounter[rootId]++; GraphManager.addNota(graphManager, GraphManager.THREAD_GROUP, rootId, stage); } } //System.err.println("performance issue here calling this too often "+countStages+" duration "+(System.nanoTime()-start)); //logger.info("group counts "+Arrays.toString(rootMemberCounter)); return rootMemberCounter; }
PronghornStage stage = GraphManager.getStage(graphManager, i); if (null != stage) {
PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { int id = rootId(stage.stageId, rootsTable, lastKnownRoot);
if (null!=m.stageIdToStage[i]) { if (!isStageTerminated(m, i) ) { PronghornStage stage = getStage(m,i);
if (null!=getStage(gm,i)) { if (GraphManager.monitorAll || (!GraphManager.hasNota(gm, i, GraphManager.MONITOR))) { long elapsedAtPercentile = ElapsedTimeRecorder.elapsedAtPercentile(gm.stageElapsed[i], percentile); if (debug) { logger.info("Stage {} Elap {} ns ",getStage(gm,i).stageId, Appendables.appendNearestTimeUnit(new StringBuilder(), elapsedAtPercentile).toString() ); if (ElapsedTimeRecorder.totalCount(gm.stageElapsed[i])>significantSampleCount) { if (ElapsedTimeRecorder.elapsedAtPercentile(gm.stageElapsed[i], percentile)>limit) { visitor.visit(getStage(gm,i));
public int weight(Pipe p) { if (weights[p.id]==0) { int result = (int)p.config().slabBits(); //returns the max pipe length from this pipe or any of the pipes that feed its producer. //if this value turns out to be large then we should probably not join these two stages. int producerId = GraphManager.getRingProducerId(graphManager, p.id); if (producerId>=0) { PronghornStage producer = GraphManager.getStage(graphManager, producerId); int count = GraphManager.getInputPipeCount(graphManager, producer); while (--count>=0) { Pipe inputPipe = GraphManager.getInputPipe(graphManager, producer, count); result = Math.max(result, inputPipe.config().slabBits()); } } else { //no producer found, an external thread must be pushing data into this, there is nothing to combine it with } weights[p.id] = result; } return weights[p.id]; }
while (--j>=0) { int monitorDataPipe = inputs[j].id; PronghornStage producer = GraphManager.getStage(graphManager, GraphManager.getRingProducerStageId(graphManager, monitorDataPipe)); if (producer instanceof PipeMonitorStage) { PipeMonitorStage p = (PipeMonitorStage)producer;
public int weight(Pipe p) { if (weights[p.id]==0) { int result = (int)p.config().slabBits(); //returns the max pipe length from this pipe or any of the pipes that feed its producer. //if this value turns out to be large then we should probably not join these two stages. int producerId = GraphManager.getRingProducerId(graphManager, p.id); if (producerId>=0) { PronghornStage producer = GraphManager.getStage(graphManager, producerId); int count = GraphManager.getInputPipeCount(graphManager, producer); while (--count>=0) { Pipe inputPipe = GraphManager.getInputPipe(graphManager, producer, count+1); result = Math.max(result, inputPipe.config().slabBits()); } } else { //no producer found, an external thread must be pushing data into this, there is nothing to combine it with } weights[p.id] = result; } return weights[p.id]; }
private void add(PronghornStage[] pronghornStages, PronghornStage stage, final int root, final GraphManager graphManager, final IntHashTable rootsTable) { int i = 0; while (i<pronghornStages.length && pronghornStages[i]!=null) { if (pronghornStages[i]==stage) { return;//already added } i++; } //now add the new stage at index i pronghornStages[i]=stage; //Recursively add the ones under the same root. int outputCount = GraphManager.getOutputPipeCount(graphManager, stage.stageId); for(int r = 1; r<=outputCount; r++) { Pipe outputPipe = GraphManager.getOutputPipe(graphManager, stage, r); int consumerId = GraphManager.getRingConsumerId(graphManager, outputPipe.id); //this exists and has the same root so add it if (consumerId>=0 && rootId(consumerId, rootsTable)==root) { add(pronghornStages, GraphManager.getStage(graphManager, consumerId), root, graphManager, rootsTable); } } }
PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { expectedCount++; PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { PronghornStage stage = GraphManager.getStage(graphManager, stages); if (null!=stage) { while (stages >= 0) { PronghornStage stage = GraphManager.getStage(graphManager, stages--); if (null!=stage) {
count = add(stageArrays, GraphManager.getStage(graphManager, consumerId), root, graphManager, rootsTable, lastKnownRoot, count, rootMemberCounter, log, false, ensureAllPrevious);
PronghornStage stage = GraphManager.getStage(graphManager, stages--); if (null!=stage) {