API providing a variety of ways to access chunks of data (data are typically
elements or binding sets).
TODO This could be much more flexible and should be harmonized to support
high volume operators, GPU operators, etc. probably the right thing to do is
introduce another interface here with a getChunk():IChunk where IChunk let's
you access the chunks data in different ways (and chunks can be both
IBindingSet []s and element[]s so we might need to raise that into
the interfaces and/or generics as well).
TODO It is likely that we can convert to the use of
BlockingQueueinstead of
BlockingBuffer in the operators and then handle the logic
for combining chunks inside of the
QueryEngine. E.g., by scanning
this list for chunks for the same bopId and combining them logically into a
single chunk.
For scale-out, chunk combination will naturally occur when the node on which
the operator will run requests the
ByteBuffers from the source nodes.
Those will get wrapped up logically into a source for processing. For
selective operators, those chunks can be combined before we execute the
operator. For unselective operators, we are going to run over all the data
anyway.
TODO Expose an
IChunkedIterator, which handles both element at a time
and chunk at a time.
TODO Expose a mechanism to visit the direct
ByteBuffer slices in
which the data are stored. For an operator which executes on a GPU, we want
to transfer the data from the direct
ByteBuffer in which it was
received into a direct
ByteBuffer which is a slice onto its VRAM.
(And obviously we need to do the reverse with the outputs of a GPU operator).