Converts an incoming graph token stream, such as one from
SynonymGraphFilter, into a flat form so that
all nodes form a single linear chain with no side paths. Every
path through the graph touches every node. This is necessary
when indexing a graph token stream, because the index does not
save
PositionLengthAttribute and so it cannot
preserve the graph structure. However, at search time,
query parsers can correctly handle the graph and this token
filter should
not be used.
If the graph was not already flat to start, this
is likely a lossy process, i.e. it will often cause the
graph to accept token sequences it should not, and to
reject token sequences it should not.
However, when applying synonyms during indexing, this
is necessary because Lucene already does not index a graph
and so the indexing process is already lossy
(it ignores the
PositionLengthAttribute).