Javadoc
Flattens the given sub-array of symbols into an sub-array of symbols. Every
Sequence in the input are replaced by its production recursively.
Non-Sequence symbols, they internally have other symbols
those internal symbols also get flattened.
When flattening is done, the only place there might be Sequence
symbols is in the productions of a Repeater, Alternative, or the
symToParse and symToSkip in a UnionAdjustAction or SkipAction.
Why is this done? We want our parsers to be fast. If we left
the grammars unflattened, then the parser would be constantly
copying the contents of nested Sequence productions onto the
parsing stack. Instead, because of flattening, we have a long
top-level production with no Sequences unless the Sequence is
absolutely needed, e.g., in the case of a Repeater or an
Alterantive.
Well, this is not exactly true when recursion is involved. Where
there is a recursive record, that record will be "inlined" once,
but any internal (ie, recursive) references to that record will
be a Sequence for the record. That Sequence will not further
inline itself -- it will refer to itself as a Sequence. The same
is true for any records nested in this outer recursive record.
Recursion is rare, and we want things to be fast in the typical
case, which is why we do the flattening optimization.
The algorithm does a few tricks to handle recursive symbol definitions.
In order to avoid infinite recursion with recursive symbols, we have a map
of Symbol->Symbol. Before fully constructing a flattened symbol for a
Sequence we insert an empty output symbol into the map and then
start filling the production for the Sequence. If the same
Sequence is encountered due to recursion, we simply return the
(empty) output Sequence from the map. Then we actually fill out
the production for the Sequence.
As part of the flattening process we copy the production of
Sequences into larger arrays. If the original Sequence
has not not be fully constructed yet, we copy a bunch of nulls.
Fix-up remembers all those null patches. The fix-ups gets finally
filled when we know the symbols to occupy those patches.