private void writeNullForMissingFieldsAtCurrentLevel() { int currentFieldsCount = ((GroupColumnIO)currentColumnIO).getChildrenCount(); for (int i = 0; i < currentFieldsCount; i++) { if (!fieldsWritten[currentLevel].isWritten(i)) { try { ColumnIO undefinedField = ((GroupColumnIO)currentColumnIO).getChild(i); int d = currentColumnIO.getDefinitionLevel(); if (DEBUG) log(Arrays.toString(undefinedField.getFieldPath()) + ".writeNull(" + r[currentLevel] + "," + d + ")"); writeNull(undefinedField, r[currentLevel], d); } catch (RuntimeException e) { throw new ParquetEncodingException("error while writing nulls for fields of indexes " + i + " . current index: " + fieldsWritten[currentLevel], e); } } } }
private void setRepetitionLevel() { r[currentLevel] = currentColumnIO.getRepetitionLevel(); if (DEBUG) log("r: "+r[currentLevel]); }
private void writeNull(ColumnIO undefinedField, int r, int d) { if (undefinedField.getType().isPrimitive()) { columnWriter[((PrimitiveColumnIO)undefinedField).getId()].writeNull(r, d); } else { GroupColumnIO groupColumnIO = (GroupColumnIO)undefinedField; int childrenCount = groupColumnIO.getChildrenCount(); for (int i = 0; i < childrenCount; i++) { writeNull(groupColumnIO.getChild(i), r, d); } } }
private void setRepetitionLevel() { r[currentLevel] = currentColumnIO.getRepetitionLevel(); if (DEBUG) log("r: " + r[currentLevel]); }
@Override public void endMessage() { writeNullForMissingFields(((GroupColumnIO)currentColumnIO).getChildrenCount() - 1); if (DEBUG) log("< MESSAGE END >"); if (DEBUG) printState(); }
public void printState() { log(currentLevel + ", " + fieldsWritten[currentLevel] + ": " + Arrays.toString(currentColumnIO.getFieldPath()) + " r:" + r[currentLevel]); if (r[currentLevel] > currentColumnIO.getRepetitionLevel()) { // sanity check throw new InvalidRecordException(r[currentLevel] + "(r) > " + currentColumnIO.getRepetitionLevel() + " ( schema r)"); } }
public RecordConsumer getRecordWriter(ColumnWriteStore columns) { RecordConsumer recordWriter = new MessageColumnIORecordConsumer(columns); if (DEBUG) recordWriter = new RecordConsumerLoggingWrapper(recordWriter); return validating ? new ValidatingRecordConsumer(recordWriter, getType()) : recordWriter; }
public void printState() { log(currentLevel+", "+currentIndex[currentLevel]+": "+Arrays.toString(currentColumnIO.getFieldPath())+" r:"+r[currentLevel]); if (r[currentLevel] > currentColumnIO.getRepetitionLevel()) { // sanity check throw new InvalidRecordException(r[currentLevel]+"(r) > "+currentColumnIO.getRepetitionLevel()+" ( schema r)"); } }
public RecordConsumer getRecordWriter(ColumnWriteStore columns) { RecordConsumer recordWriter = new MessageColumnIORecordConsumer(columns); if (DEBUG) recordWriter = new RecordConsumerLoggingWrapper(recordWriter); return validating ? new ValidatingRecordConsumer(recordWriter, getType()) : recordWriter; }
@Override public void endGroup() { if (DEBUG) log("endGroup()"); int lastIndex = ((GroupColumnIO)currentColumnIO).getChildrenCount() - 1; writeNullForMissingFields(lastIndex); -- currentLevel; setRepetitionLevel(); if (DEBUG) printState(); }
private void writeNullForMissingFields(final int to) { final int from = currentIndex[currentLevel]; for (;currentIndex[currentLevel]<=to; ++currentIndex[currentLevel]) { try { ColumnIO undefinedField = ((GroupColumnIO)currentColumnIO).getChild(currentIndex[currentLevel]); int d = currentColumnIO.getDefinitionLevel(); if (DEBUG) log(Arrays.toString(undefinedField.getFieldPath())+".writeNull("+r[currentLevel]+","+d+")"); writeNull(undefinedField, r[currentLevel], d); } catch (RuntimeException e) { throw new ParquetEncodingException("error while writing nulls from " + from + " to " + to + ". current index: "+currentIndex[currentLevel], e); } } }
@Override public void startField(String field, int index) { try { if (DEBUG) log("startField(" + field + ", " + index + ")"); currentColumnIO = ((GroupColumnIO)currentColumnIO).getChild(index); emptyField = true; if (DEBUG) printState(); } catch (RuntimeException e) { throw new ParquetEncodingException("error starting field " + field + " at " + index, e); } }
@Override public void endField(String field, int index) { if (DEBUG) log("endField("+field+", "+index+")"); currentColumnIO = currentColumnIO.getParent(); currentIndex[currentLevel] = index + 1; r[currentLevel] = currentLevel == 0 ? 0 : r[currentLevel - 1]; if (DEBUG) printState(); }
@Override public void endMessage() { writeNullForMissingFieldsAtCurrentLevel(); columns.endRecord(); if (DEBUG) log("< MESSAGE END >"); if (DEBUG) printState(); }
@Override public void startGroup() { if (DEBUG) log("startGroup()"); ++ currentLevel; r[currentLevel] = r[currentLevel - 1]; currentIndex[currentLevel] = 0; if (DEBUG) printState(); }
private void writeNull(ColumnIO undefinedField, int r, int d) { if (undefinedField.getType().isPrimitive()) { columnWriter[((PrimitiveColumnIO)undefinedField).getId()].writeNull(r, d); } else { GroupColumnIO groupColumnIO = (GroupColumnIO)undefinedField; int childrenCount = groupColumnIO.getChildrenCount(); for (int i = 0; i < childrenCount; i++) { writeNull(groupColumnIO.getChild(i), r, d); } } }
@Override public void addBoolean(boolean value) { if (DEBUG) log("addBoolean(" + value + ")"); emptyField = false; getColumnWriter().write(value, r[currentLevel], currentColumnIO.getDefinitionLevel()); setRepetitionLevel(); if (DEBUG) printState(); }
@Override public void addLong(long value) { if (DEBUG) log("addLong(" + value + ")"); emptyField = false; getColumnWriter().write(value, r[currentLevel], currentColumnIO.getDefinitionLevel()); setRepetitionLevel(); if (DEBUG) printState(); }
@Override public void addLong(long value) { if (DEBUG) log("addLong("+value+")"); getColumnWriter().write(value, r[currentLevel], currentColumnIO.getDefinitionLevel()); setRepetitionLevel(); if (DEBUG) printState(); }
@Override public void startGroup() { if (DEBUG) log("startGroup()"); ++ currentLevel; r[currentLevel] = r[currentLevel - 1]; int fieldsCount = ((GroupColumnIO)currentColumnIO).getChildrenCount(); fieldsWritten[currentLevel].reset(fieldsCount); if (DEBUG) printState(); }