private void handleEventInt(long now, int partitionIndex, long eventTime) { wmPolicies[partitionIndex].reportEvent(eventTime); markIdleAt[partitionIndex] = now + idleTimeoutNanos; allAreIdle = false; handleNoEventInt(now); }
private static int getLocalParallelism(int defaultLocalParallelism, Vertex v) { int localParallelism = v.getLocalParallelism(); if (localParallelism == -1) { localParallelism = v.getMetaSupplier().preferredLocalParallelism(); if (localParallelism == -1) { localParallelism = defaultLocalParallelism; } } return localParallelism; }
@Override public void close(Throwable error) throws Exception { wrapped.close(error); } }
@Override public void addToDag(Planner p) { String vertexName = p.uniqueVertexName(this.name()); Vertex v1 = p.dag.newVertex(vertexName + FIRST_STAGE_VERTEX_NAME_SUFFIX, distinctP(keyFn)) .localParallelism(localParallelism()); PlannerVertex pv2 = p.addVertex(this, vertexName, localParallelism(), distinctP(keyFn)); p.addEdges(this, v1, (e, ord) -> e.partitioned(keyFn, HASH_CODE)); p.dag.edge(between(v1, pv2.v).distributed().partitioned(keyFn)); }
EdgeDef(Edge edge, EdgeConfig config, int oppositeVertexId, boolean isJobDistributed) { this.oppositeVertexId = oppositeVertexId; this.sourceOrdinal = edge.getSourceOrdinal(); this.destOrdinal = edge.getDestOrdinal(); this.priority = edge.getPriority(); this.isDistributed = isJobDistributed && edge.isDistributed(); this.routingPolicy = edge.getRoutingPolicy(); this.partitioner = edge.getPartitioner(); this.config = config; }
/** * Variant of {@link #preferLocalParallelismOne(ProcessorSupplier)} where * the supplied {@code DistributedSupplier<Processor>} will be * wrapped into a {@link ProcessorSupplier}. */ @Nonnull static ProcessorMetaSupplier preferLocalParallelismOne( @Nonnull DistributedSupplier<? extends Processor> procSupplier ) { return of(ProcessorSupplier.of(procSupplier), 1); }
@Override public boolean complete() { inComplete = true; return closedWindowFlatmapper.tryProcess(COMPLETING_WM); }
void process4(@Nonnull Inbox inbox) throws Exception { for (Object item; (item = inbox.peek()) != null && tryProcess4(item); ) { inbox.remove(); } }
void processAny(int ordinal, @Nonnull Inbox inbox) throws Exception { for (Object item; (item = inbox.peek()) != null && tryProcess(ordinal, item); ) { inbox.remove(); } } }
/** * Returns the policy object used by the Core API processors to drive the * sliding window logic. * */ public SlidingWindowPolicy toSlidingWindowPolicy() { return slidingWinPolicy(windowSize, slideBy); } }
/** * Returns an event time policy that results in no timestamping. Only * useful in jobs with streaming sources that don't do any aggregation. * If there is an aggregation step in the job and you use these parameters, * your job will keep accumulating the data without producing any output. */ public static <T> EventTimePolicy<T> noEventTime() { return eventTimePolicy(i -> Long.MIN_VALUE, noWrapping(), NO_WATERMARKS, 0, 0, -1); }
/** * Returns a partitioner which applies the default Hazelcast partitioning. * It will serialize the item using Hazelcast Serialization, then apply * Hazelcast's {@code MurmurHash}-based algorithm to retrieve the partition * ID. This is quite a bit of work, but has stable results across all JVM * processes, making it a safe default. */ static Partitioner<Object> defaultPartitioner() { return new Default(); }
@Override public void close(Throwable error) throws Exception { wrapped.close(error); } }
@Override public long checkWmHistory() { if (idleMessagePending) { idleMessagePending = false; return IDLE_MESSAGE.timestamp(); } return NO_NEW_WM; }
/** * Converts this definition to one defining a tumbling window of the * same length as this definition's frame. */ public SlidingWindowPolicy toTumblingByFrame() { return new SlidingWindowPolicy(frameSize, frameOffset, 1); }
private boolean emit() { return outputOrdinals != null ? emitFromTraverser(outputOrdinals, outputTraverser) : emitFromTraverser(outputTraverser); } }
@Override public void close() throws Exception { wrapped.close(); } }
/** * Returns true, if snapshots will be saved for this job. */ default boolean snapshottingEnabled() { return processingGuarantee() != ProcessingGuarantee.NONE; }
/** * Returns the definition of a tumbling window of length {@code * windowSize}. The tumbling window is a special case of the sliding * window with {@code slideBy = windowSize}. Given {@code * windowSize = 4}, the generated windows would cover timestamps {@code * ..., [-4, 0), [0..4), [4..8), ...} */ public static SlidingWindowPolicy tumblingWinPolicy(long windowSize) { return slidingWinPolicy(windowSize, windowSize); } }