private static int decompress(Decompressor decompressor, Slice input, int inputOffset, int inputLength, byte[] output, int outputOffset) { byte[] byteArray = (byte[]) input.getBase(); int byteArrayOffset = inputOffset + (int) (input.getAddress() - ARRAY_BYTE_BASE_OFFSET); int size = decompressor.decompress(byteArray, byteArrayOffset, inputLength, output, outputOffset, output.length - outputOffset); return size; } }
@Override public int decompress(byte[] input, int offset, int length, OutputBuffer output) throws OrcCorruptionException { try { byte[] buffer = output.initialize(maxBufferSize); return decompressor.decompress(input, offset, length, buffer, 0, buffer.length); } catch (MalformedInputException e) { throw new OrcCorruptionException(e, orcDataSourceId, "Invalid compressed stream"); } }
public Page deserialize(SerializedPage serializedPage) { checkArgument(serializedPage != null, "serializedPage is null"); if (!decompressor.isPresent() || serializedPage.getCompression() == UNCOMPRESSED) { return readRawPage(serializedPage.getPositionCount(), serializedPage.getSlice().getInput(), blockEncodingSerde); } int uncompressedSize = serializedPage.getUncompressedSizeInBytes(); byte[] decompressed = new byte[uncompressedSize]; int actualUncompressedSize = decompressor.get().decompress(serializedPage.getSlice().getBytes(), 0, serializedPage.getSlice().length(), decompressed, 0, uncompressedSize); checkState(uncompressedSize == actualUncompressedSize); return readRawPage(serializedPage.getPositionCount(), Slices.wrappedBuffer(decompressed, 0, uncompressedSize).getInput(), blockEncodingSerde); } }
@Override public void decompress(ByteBuffer in, ByteBuffer out) throws IOException { int inOffset = in.position(); int uncompressLen = decompressor.decompress(in.array(), in.arrayOffset() + inOffset, in.limit() - inOffset, out.array(), out.arrayOffset() + out.position(), out.remaining()); out.position(uncompressLen + out.position()); out.flip(); }
private static int decompress(Decompressor decompressor, Slice input, int inputOffset, int inputLength, byte[] output, int outputOffset) { byte[] byteArray = (byte[]) input.getBase(); int byteArrayOffset = inputOffset + (int) (input.getAddress() - ARRAY_BYTE_BASE_OFFSET); int size = decompressor.decompress(byteArray, byteArrayOffset, inputLength, output, outputOffset, output.length - outputOffset); return size; } }
@Override public int decompress(byte[] input, int offset, int length, OutputBuffer output) throws OrcCorruptionException { try { byte[] buffer = output.initialize(maxBufferSize); return decompressor.decompress(input, offset, length, buffer, 0, buffer.length); } catch (MalformedInputException e) { throw new OrcCorruptionException(e, orcDataSourceId, "Invalid compressed stream"); } }
@Override public int decompress(byte[] input, int offset, int length, OutputBuffer output) throws OrcCorruptionException { try { byte[] buffer = output.initialize(maxBufferSize); return decompressor.decompress(input, offset, length, buffer, 0, buffer.length); } catch (MalformedInputException e) { throw new OrcCorruptionException(e, orcDataSourceId, "Invalid compressed stream"); } }
@Override public int decompress(byte[] input, int offset, int length, OutputBuffer output) throws OrcCorruptionException { try { byte[] buffer = output.initialize(maxBufferSize); return decompressor.decompress(input, offset, length, buffer, 0, buffer.length); } catch (MalformedInputException e) { throw new OrcCorruptionException(e, orcDataSourceId, "Invalid compressed stream"); } }
private static int decompress(Decompressor decompressor, Slice input, int inputOffset, int inputLength, byte[] output, int outputOffset) { byte[] byteArray = (byte[]) input.getBase(); int byteArrayOffset = inputOffset + (int) (input.getAddress() - ARRAY_BYTE_BASE_OFFSET); int size = decompressor.decompress(byteArray, byteArrayOffset, inputLength, output, outputOffset, output.length - outputOffset); return size; } }
@Benchmark public int decompress(BytesCounter counter) { int written = decompressor.decompress(compressed, 0, compressed.length, uncompressTarget, 0, uncompressTarget.length); counter.bytes += uncompressed.length; return written; }
@Test(expectedExceptions = MalformedInputException.class, expectedExceptionsMessageRegExp = "Input is corrupted: offset=894") public void testInvalidSequenceOffset() throws IOException { byte[] compressed = Resources.toByteArray(getClass().getClassLoader().getResource("data/zstd/offset-before-start.zst")); byte[] output = new byte[compressed.length * 10]; getDecompressor().decompress(compressed, 0, compressed.length, output, 0, output.length); } }
@Test public void testDecompressWithOutputPaddingAndChecksum() throws IOException { int padding = 1021; byte[] compressed = Resources.toByteArray(getClass().getClassLoader().getResource("data/zstd/with-checksum.zst")); byte[] uncompressed = Resources.toByteArray(getClass().getClassLoader().getResource("data/zstd/with-checksum")); byte[] output = new byte[uncompressed.length + padding * 2]; // pre + post padding int decompressedSize = getDecompressor().decompress(compressed, 0, compressed.length, output, padding, output.length); assertByteArraysEqual(uncompressed, 0, uncompressed.length, output, padding, decompressedSize); }
@Test public void testConcatenatedFrames() throws IOException { byte[] compressed = Resources.toByteArray(getClass().getClassLoader().getResource("data/zstd/multiple-frames.zst")); byte[] uncompressed = Resources.toByteArray(getClass().getClassLoader().getResource("data/zstd/multiple-frames")); byte[] output = new byte[uncompressed.length]; getDecompressor().decompress(compressed, 0, compressed.length, output, 0, output.length); assertByteArraysEqual(uncompressed, 0, uncompressed.length, output, 0, output.length); }
private void verifyCompressedData(byte[] originalUncompressed, byte[] compressed, int compressedLength) { byte[] uncompressed = new byte[originalUncompressed.length]; int uncompressedSize = getVerifyDecompressor().decompress(compressed, 0, compressedLength, uncompressed, 0, uncompressed.length); assertByteArraysEqual(uncompressed, 0, uncompressedSize, originalUncompressed, 0, originalUncompressed.length); }
public Page deserialize(SerializedPage serializedPage) { checkArgument(serializedPage != null, "serializedPage is null"); if (!decompressor.isPresent() || serializedPage.getCompression() == UNCOMPRESSED) { return readRawPage(serializedPage.getPositionCount(), serializedPage.getSlice().getInput(), blockEncodingSerde); } int uncompressedSize = serializedPage.getUncompressedSizeInBytes(); byte[] decompressed = new byte[uncompressedSize]; int actualUncompressedSize = decompressor.get().decompress(serializedPage.getSlice().getBytes(), 0, serializedPage.getSlice().length(), decompressed, 0, uncompressedSize); checkState(uncompressedSize == actualUncompressedSize); return readRawPage(serializedPage.getPositionCount(), Slices.wrappedBuffer(decompressed, 0, uncompressedSize).getInput(), blockEncodingSerde); } }
@Test(dataProvider = "data") public void testDecompressByteBufferHeapToDirect(DataSet dataSet) throws Exception { if (!isByteBufferSupported()) { return; } byte[] uncompressedOriginal = dataSet.getUncompressed(); ByteBuffer compressed = ByteBuffer.wrap(prepareCompressedData(uncompressedOriginal)); ByteBuffer uncompressed = ByteBuffer.allocateDirect(uncompressedOriginal.length); getDecompressor().decompress(compressed, uncompressed); uncompressed.flip(); assertByteBufferEqual(ByteBuffer.wrap(uncompressedOriginal), uncompressed); }
@Test(dataProvider = "data") public void testDecompressByteBufferHeapToHeap(DataSet dataSet) throws Exception { if (!isByteBufferSupported()) { return; } byte[] uncompressedOriginal = dataSet.getUncompressed(); ByteBuffer compressed = ByteBuffer.wrap(prepareCompressedData(uncompressedOriginal)); ByteBuffer uncompressed = ByteBuffer.allocate(uncompressedOriginal.length); getDecompressor().decompress(compressed, uncompressed); uncompressed.flip(); assertByteBufferEqual(ByteBuffer.wrap(uncompressedOriginal), uncompressed); }
public Page deserialize(SerializedPage serializedPage) { checkArgument(serializedPage != null, "serializedPage is null"); if (!decompressor.isPresent() || serializedPage.getCompression() == UNCOMPRESSED) { return readRawPage(serializedPage.getPositionCount(), serializedPage.getSlice().getInput(), blockEncodingSerde); } int uncompressedSize = serializedPage.getUncompressedSizeInBytes(); byte[] decompressed = new byte[uncompressedSize]; int actualUncompressedSize = decompressor.get().decompress(serializedPage.getSlice().getBytes(), 0, serializedPage.getSlice().length(), decompressed, 0, uncompressedSize); checkState(uncompressedSize == actualUncompressedSize); return readRawPage(serializedPage.getPositionCount(), Slices.wrappedBuffer(decompressed, 0, uncompressedSize).getInput(), blockEncodingSerde); } }
@Test(dataProvider = "data") public void testDecompressByteBufferDirectToDirect(DataSet dataSet) throws Exception { if (!isByteBufferSupported()) { return; } byte[] uncompressedOriginal = dataSet.getUncompressed(); ByteBuffer compressed = toDirectBuffer(prepareCompressedData(uncompressedOriginal)); ByteBuffer uncompressed = ByteBuffer.allocateDirect(uncompressedOriginal.length); getDecompressor().decompress(compressed, uncompressed); uncompressed.flip(); assertByteBufferEqual(ByteBuffer.wrap(uncompressedOriginal), uncompressed); }
@Test(dataProvider = "data") public void testDecompress(DataSet dataSet) throws Exception { byte[] uncompressedOriginal = dataSet.getUncompressed(); byte[] compressed = prepareCompressedData(uncompressedOriginal); byte[] uncompressed = new byte[uncompressedOriginal.length]; Decompressor decompressor = getDecompressor(); int uncompressedSize = decompressor.decompress( compressed, 0, compressed.length, uncompressed, 0, uncompressed.length); assertByteArraysEqual(uncompressed, 0, uncompressedSize, uncompressedOriginal, 0, uncompressedOriginal.length); }