Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix whitespace tokenisation in Go lexer #1122

Merged
merged 1 commit into from
May 27, 2019

Conversation

pyrmont
Copy link
Contributor

@pyrmont pyrmont commented May 20, 2019

As discussed in #1097, the Go lexer tokenises whitespace using the Other token. This causes issues when rendering in HTML the code parsed by Rouge because:

  1. the use of Other will result in the whitespace being wrapped in separate HTML tags, which in turn causes
  2. all newlines to be collapsed (due to CSS rules that expect newlines not to be wrapped in seperate HTML tags).

The Other token's intended use is described as follows:

Token for data not matched by a parser (e.g. HTML markup in PHP code)

This is not a correct characterisation of whitespace in Go and is not consistent with other lexers. This commit changes the token to Text, the token generally used by other lexers. It fixes #1097.

As discussed in rouge-ruby#1097, the Go lexer tokenises whitespace using the
`Other` token. This causes issues when rendering in HTML the code parsed
by Rouge because:

1. the use of `Other` will result in the whitespace being wrapped in
   separate HTML tags, which in turn causes
2. all newlines to be collapsed (due to CSS rules that expect newlines
   not to be wrapped in seperate HTML tags).

The `Other` token's intended use is described as follows:

> Token for data not matched by a parser (e.g. HTML markup in PHP code)

This is not a correct characterisation of whitespace in Go and is not
consistent with other lexers. This commit changes the token to `Text`,
the token generally used by other lexers. It fixes rouge-ruby#1097.
@pyrmont pyrmont merged commit af62def into rouge-ruby:master May 27, 2019
@pyrmont pyrmont deleted the bugfix.go-whitespace-token branch January 8, 2020 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Golang syntax highlighting ignores newlines
1 participant