Boost in tokenizer performance with 3 liner #896
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested this; seems legit. Of course, this means we can't consider whitespace to be part of token starts. I don't think any language currently does this, so this is probably fine. And given that you're right, this would absolutely improve performance by a significant degree, I think this is worth it. Any objections? Otherwise, let's merge. We can always take it out later if there is some language that this conflicts with (but I can't think of any off hand). |
|
Without this change opening the plugins readme as shown on #885 was like this: highlight-no-lazy-mode.mp4and with this change it is now like this: with-match-spaces-rule.mp4 |
|
Oh wow, that is quite noticeable! We'll merge tomorrow if no one has any further comments. |
jgmdev
added a commit
to jgmdev/lite-xl
that referenced
this pull request
Mar 29, 2022
* mainly the language_md got affected which has some exotic rules * some other languages are also using spaces at start of pattern and even if not affected this change tackles that
jgmdev
added a commit
that referenced
this pull request
Mar 29, 2022
syntax: fix conflicts introduced with #896
adamharrison
pushed a commit
to adamharrison/lite-xl
that referenced
this pull request
Apr 1, 2022
adamharrison
pushed a commit
to adamharrison/lite-xl
that referenced
this pull request
Apr 1, 2022
* mainly the language_md got affected which has some exotic rules * some other languages are also using spaces at start of pattern and even if not affected this change tackles that
sprainbrains
pushed a commit
to sprainbrains/lite-xl
that referenced
this pull request
Apr 1, 2022
* mainly the language_md got affected which has some exotic rules * some other languages are also using spaces at start of pattern and even if not affected this change tackles that
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on improving responsiveness of lite-xl while highlithing on #885 I noticed that the slow downs in tokenization process where occurring on lines with long amounts of consecutive spaces. It seems the tokenizer was trying to apply each of its rules to each of the spaces found which slowed things a lot.
Adding a rule of %s+ that matches to "normal" in the beginning of every syntax table, will hugely improve the performance of the tokenizer basically for free! I haven't measured yet the gains but it is noticeable how tokenization is much more faster now.