Javadoc
Flatten Record field, and compute a list of flattened fields
Note: Lineage represents the source path from root for the flattened field. For. eg. If the original schema is:
{
"type" : "record",
"name" : "parentRecordName",
"fields" : [ {
"name" : "parentFieldRecord",
"type" : {
"type" : "record",
"name" : "nestedRecordName",
"fields" : [ {
"name" : "nestedFieldString",
"type" : "string"
}, {
"name" : "nestedFieldInt",
"type" : "int"
} ]
}
}]
}
The expected output schema is:
{
"type" : "record",
"name" : "parentRecordName",
"fields" : [ {
"name" : "parentFieldRecord__nestedFieldString",
"type" : "string",
"flatten_source" : "parentFieldRecord.nestedFieldString"
}, {
"name" : "parentFieldRecord__nestedFieldInt",
"type" : "int",
"flatten_source" : "parentFieldRecord.nestedFieldInt"
}, {
"name" : "parentFieldInt",
"type" : "int"
} ]
}
Here, 'flatten_source' and field 'name' has also been modified to represent their origination from nested schema
lineage helps to determine that