Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract f-string prefix as a separate token #1635

Merged
merged 2 commits into from
Dec 5, 2020
Merged

Extract f-string prefix as a separate token #1635

merged 2 commits into from
Dec 5, 2020

Conversation

tancnle
Copy link
Collaborator

@tancnle tancnle commented Nov 26, 2020 •
edited by pyrmont
Loading

The f-string prefix was extracted together with the text as one token. This makes syntax highlight less flexible.

This change will:

  • Extract f prefix as String::Affix token, similar to how it is done in Pygments.
  • Add styles for String::Affix in all themes (mirror the style of Keyword)

This fixes #1634.

Before 
[1] pry(main)> require './lib/rouge'
true
[2] pry(main)> lexer = Rouge::Lexers::Python.new
#<Rouge::Lexers::Python:0x00007fadc66979b0 @options={}, @debug=false>
[3] pry(main)> lexer.lex("foo = f\"test: {t}\"\n").to_a
[
    [0] [
        [0] <Token Name> < Rouge::Token,
        [1] "foo"
    ],
    [1] [
        [0] <Token Text> < Rouge::Token,
        [1] " "
    ],
    [2] [
        [0] <Token Operator> < Rouge::Token,
        [1] "="
    ],
    [3] [
        [0] <Token Text> < Rouge::Token,
        [1] " "
    ],
    [4] [
        [0] <Token Literal.String> < Rouge::Token::Tokens::Literal,
        [1] "f\"test: "
    ],
    [5] [
        [0] <Token Literal.String.Interpol> < Rouge::Token::Tokens::Literal::String,
        [1] "{"
    ],
    [6] [
        [0] <Token Name> < Rouge::Token,
        [1] "t"
    ],
    [7] [
        [0] <Token Literal.String.Interpol> < Rouge::Token::Tokens::Literal::String,
        [1] "}"
    ],
    [8] [
        [0] <Token Literal.String> < Rouge::Token::Tokens::Literal,
        [1] "\""
    ],
    [9] [
        [0] <Token Text> < Rouge::Token,
        [1] "\n"
    ]
]
After 
[1] pry(main)> require './lib/rouge'
true
[2] pry(main)> lexer = Rouge::Lexers::Python.new
#<Rouge::Lexers::Python:0x00007fe355697bf8 @options={}, @debug=false>
[3] pry(main)> lexer.lex("foo = f\"test: {t}\"\n").to_a
[
    [ 0] [
        [0] <Token Name> < Rouge::Token,
        [1] "foo"
    ],
    [ 1] [
        [0] <Token Text> < Rouge::Token,
        [1] " "
    ],
    [ 2] [
        [0] <Token Operator> < Rouge::Token,
        [1] "="
    ],
    [ 3] [
        [0] <Token Text> < Rouge::Token,
        [1] " "
    ],
    [ 4] [
        [0] <Token Literal.String.Affix> < Rouge::Token::Tokens::Literal::String,
        [1] "f"
    ],
    [ 5] [
        [0] <Token Literal.String> < Rouge::Token::Tokens::Literal,
        [1] "\"test: "
    ],
    [ 6] [
        [0] <Token Literal.String.Interpol> < Rouge::Token::Tokens::Literal::String,
        [1] "{"
    ],
    [ 7] [
        [0] <Token Name> < Rouge::Token,
        [1] "t"
    ],
    [ 8] [
        [0] <Token Literal.String.Interpol> < Rouge::Token::Tokens::Literal::String,
        [1] "}"
    ],
    [ 9] [
        [0] <Token Literal.String> < Rouge::Token::Tokens::Literal,
        [1] "\""
    ],
    [10] [
        [0] <Token Text> < Rouge::Token,
        [1] "\n"
    ]
]
Before (base16-dark) After (base16-dark)
Screen Shot 2020-11-26 at 7 35 30 pm Screen Shot 2020-11-26 at 7 34 43 pm

Extract f-string prefix as a separate token
64d64fe 
The f-string prefix was extracted together with the text as one
token. This makes syntax highlight less flexible. This change will
extract `f` prefix as `String::Affix` token.
@tancnle tancnle marked this pull request as ready for review November 26, 2020 06:37
@tancnle
Copy link
Collaborator Author

tancnle commented Nov 26, 2020

It seems like we already have the f-string samples.

@tancnle
Copy link
Collaborator Author

tancnle commented Nov 26, 2020

I think we need to add the new style to themes as well. 🤔

@pyrmont pyrmont merged commit 16ffecb into rouge-ruby:master Dec 5, 2020
@pyrmont
Copy link
Contributor

pyrmont commented Dec 5, 2020

@tancnle Thanks for adding this! Looks good!

@tancnle tancnle deleted the python-string-affix-token branch December 5, 2020 05:51
mattt pushed a commit to NSHipster/rouge that referenced this pull request May 19, 2021
@tancnle @mattt
Use String::Affix token for string prefixes in Python lexer (rouge-ru…
17015d7 
…by#1635)

String prefixes were given the same token (`Str`) at the string itself.
This commit uses the `String::Affix` token for the prefix character.
This is consistent with the tokens used by Pygments. New styles for
`String::Affix` are also added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pyrmont pyrmont pyrmont approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Python f-string formatting isn't correct
2 participants
@tancnle @pyrmont