Repository navigation
Fix atEnd edge case - #169
Merged
Merged
Conversation
- atEnd before an ignored byte order mark that ends the input - position after peek of a multi-byte character - UTF-8 encoded byte count and overlong form of U+FFFF
…U+FFFF - atEnd decodes one element ahead, without sending peek - position subtracts the encoded size of the peeked element - UTF-8 counts U+FFFF as 3 bytes and rejects its 4-byte form - upToEnd temporary no longer shadows the encoder variable
Owner
|
OK, good. Thx. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi @svenvc, here is a fix for a problem that breaks STON with that mythical unicode point. please take a look.
the llm's summary:
STON's reader loops on [ readStream atEnd not and: [ readStream peek isSeparator ] ] (and on isDigit and similar while parsing numbers and symbols), and it reports parse errors at readStream position. Both broke on a ZnCharacterReadStream. ZnEncodedReadStream>>#atEnd used to check only whether bytes remained, so when the input ended in a byte order mark that the decoder ignores, atEnd answered false while peek answered nil. STON then sent nil isSeparator and failed: STON fromStream: on '[1,2]' followed by U+FEFF raised a MessageNotUnderstood instead of answering #(1 2). On fix-bom-atend, atEnd decodes one element ahead into peeked, so it answers true when only an ignored mark is left, and a peek after atEnd answers false always yields a character. Separately, position used to subtract one byte for a peeked character whatever its size, so after peeking a multi-byte character STON reported the wrong error position (5 instead of 3 for '[1,€]'). It now subtracts the peeked character's actual encoded size, and the UTF-8 byte count for U+FFFF is corrected to 3.