Fix whitespace tokenisation in Go lexer #1122
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in #1097, the Go lexer tokenises whitespace using the
Other
token. This causes issues when rendering in HTML the code parsed by Rouge because:Other
will result in the whitespace being wrapped in separate HTML tags, which in turn causesThe
Other
token's intended use is described as follows:This is not a correct characterisation of whitespace in Go and is not consistent with other lexers. This commit changes the token to
Text
, the token generally used by other lexers. It fixes #1097.