How to use
getBlockCount
method
in
org.apache.avro.file.DataFileReader

Best Java code snippets using org.apache.avro.file.DataFileReader.getBlockCount (Showing top 9 results out of 315)

Common ways to obtain DataFileReader

private void myMethod () {
 DataFileReader d =File file;DatumReader reader;new DataFileReader<>(file, reader)
SeekableInput sin;DatumReader reader;new DataFileReader<>(sin, reader)
SeekableInput sin;new DataFileReader<>(sin, new GenericDatumReader<Void>())
Smart code suggestions by Codota
}

static <T> T runOnPreview(byte[] bits, AvroPreviewProcessor<T> processor) throws IOException {
 DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>();
 SeekableByteArrayInput sbai = new SeekableByteArrayInput(bits);
 DataFileReader<GenericRecord> dataFileReader = null;
 try {
  dataFileReader = new DataFileReader<>(sbai, datumReader);
  int headerLen = (int) dataFileReader.previousSync();
  byte[] header = Arrays.copyOf(bits, headerLen);
  if (dataFileReader.hasNext()) {
   GenericRecord gr = dataFileReader.next();
   return processor.process(header, gr, dataFileReader.getBlockCount(), dataFileReader.getBlockSize());
  } else {
   throw new RuntimeException("Empty Avro file - cannot run preview! ");
  }
 } finally {
  try { if (dataFileReader!=null) dataFileReader.close(); } catch (IOException safeToIgnore) {}
 }
}

public boolean next(TetherData data, NullWritable ignore)
 throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 data.buffer(reader.nextBlock());
 data.count((int)reader.getBlockCount());
 return true;
}

blockCount = fileReader.getBlockCount();
blockSize = fileReader.getBlockSize();
numRecords += blockCount;

Log.trace(String.format("Avro: ChunkIdx: %d read %d records, start at %d off, block count: %d, block size: %d", cidx, cnt, din.getChunkDataStart(cidx), dataFileReader.getBlockCount(), dataFileReader.getBlockSize()));

public boolean next(TetherData data, NullWritable ignore)
 throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 data.buffer(reader.nextBlock());
 data.count((int)reader.getBlockCount());
 return true;
}

static <T> T runOnPreview(byte[] bits, AvroPreviewProcessor<T> processor) throws IOException {
 DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>();
 SeekableByteArrayInput sbai = new SeekableByteArrayInput(bits);
 DataFileReader<GenericRecord> dataFileReader = null;
 try {
  dataFileReader = new DataFileReader<>(sbai, datumReader);
  int headerLen = (int) dataFileReader.previousSync();
  byte[] header = Arrays.copyOf(bits, headerLen);
  if (dataFileReader.hasNext()) {
   GenericRecord gr = dataFileReader.next();
   return processor.process(header, gr, dataFileReader.getBlockCount(), dataFileReader.getBlockSize());
  } else {
   throw new RuntimeException("Empty Avro file - cannot run preview! ");
  }
 } finally {
  try { if (dataFileReader!=null) dataFileReader.close(); } catch (IOException safeToIgnore) {}
 }
}

public boolean next(TetherData data, NullWritable ignore)
 throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 data.buffer(reader.nextBlock());
 data.count((int)reader.getBlockCount());
 return true;
}

blockCount = fileReader.getBlockCount();
blockSize = fileReader.getBlockSize();
numRecords += blockCount;

Log.trace(String.format("Avro: ChunkIdx: %d read %d records, start at %d off, block count: %d, block size: %d", cidx, cnt, din.getChunkDataStart(cidx), dataFileReader.getBlockCount(), dataFileReader.getBlockSize()));

Popular methods of DataFileReader

<init>
Construct using a DataFileStream.Header. Does not call #sync(long) or #seek(long).
next
hasNext
close
openReader
Construct a reader for a file at the current position of the input, without reading the header.
getSchema
sync
Move to the next synchronization point after a position. To process a range of file entires, call th
previousSync
Return the last synchronization point before our current position.
seek
Move to a specific, known synchronization point, one returned from DataFileWriter#sync() while writi
pastSync
Return true if past the next synchronization point after a position.
getMeta
getMetaString

Popular in Java

Finding current android device location
getApplicationContext (Context)
getOriginalFilename (MultipartFile)
Return the original filename in the client's filesystem.This may contain path information depending
getContentResolver (Context)
Window (java.awt)
A Window object is a top-level window with no borders and no menubar. The default layout for a windo
FileOutputStream (java.io)
A file output stream is an output stream for writing data to aFile or to a FileDescriptor. Whether
MessageFormat (java.text)
MessageFormat provides a means to produce concatenated messages in language-neutral way. Use this to
SimpleDateFormat (java.text)
Formats and parses dates in a locale-sensitive manner. Formatting turns a Date into a String, and pa
Date (java.util)
A specific moment in time, with millisecond precision. Values typically come from System#currentTime
Vector (java.util)
The Vector class implements a growable array of objects. Like an array, it contains components that

How to use getBlockCountmethodin org.apache.avro.file.DataFileReader

Best Java code snippets using org.apache.avro.file.DataFileReader.getBlockCount (Showing top 9 results out of 315)

How to use
getBlockCount
method
in
org.apache.avro.file.DataFileReader