[cs] CPD: Replace C# tokenizer by an Antlr-based one #2280
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When filtering tokens, the analyzeToken method can be overriden to access the current token. This can then be used to implement isLanguageSpecificDiscarding. However, it may be desirable to "look ahead" and base the decision of whether to filter or not on multiple tokens. In order to support this new use case, a new extension point analyzeTokens is provided, which not only has access to the current token, but can also iterate over the upcoming tokens. The functionality of iterating over remaining tokens uses Guava for its implementation. Since pmd-core targets Java 7, the Android flavour of Guava is used. In order to stay consistent with pmd-apex-jorje, this has also been adjusted to the Android flavour. For PMD 7.0, the jre flavour can be used instead.
This is based on the Antlr grammar from https://github.com/antlr/grammars-v4/tree/master/csharp. This adds column information for C# and fixes pmd#2139.
Generated by 🚫 Danger |
adangel
changed the title
adangel
reviewed
Feb 13, 2020
pmd-core/src/test/java/net/sourceforge/pmd/cpd/token/internal/BaseTokenFilterTest.java
Outdated
Show resolved
Hide resolved
pmd-cs/src/main/antlr4/net/sourceforge/pmd/lang/cs/antlr4/CSharpLexer.g4
Show resolved
Hide resolved
adangel
added a commit
that referenced
this pull request
Feb 29, 2020
This was referenced Mar 6, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting a PR, please check that:
master. The PMD team will merge back to support branches as needed../mvnw clean verifypasses. This will build and test PMD, execute PMD and checkstyle rules. Check this for more infoPR Description:
This replaces the C# tokenizer by one based on Antlr, in line to how it's done for many other languages already.
This has a couple of advantages, including adding column information and fixing #2139
In order to correctly filter using directives as the previous tokenizer did, I had to extend the
BaseTokenFilterclass. Existing subclasses should not be impacted by this, but the new functionality might be useful for other languages as well.Fixes #2139