Javadoc
This differs from MRI in a few ways. This version does not apply value to a separate token buffer.
It is for use when we know we will not be omitting or including ant non-syntactical characters. Use
tokadd_mbchar(int, ByteList) if the string differs from actual source. Secondly, this returns a boolean
instead of the first byte passed. MRI only used the return value as a success/failure code to return
EOF.
Because this version does not use a separate token buffer we only just increment lex_p. When we reach
end of the token it will just get the bytes directly from source directly.