Hey @MalloD12 ! Thanks for picking this up for review. I think this is all set, let me know if you need anything. I think this is a pretty big deal, so would love to try to get this in for the next release if possible!

Hi @jasonlyle88,

I started looking into this on Friday. I'm not a grammar expert, but your rule changes look good to me. Also, previous tests added in #5108 keep working as expected. For debugging the different rules for example, S_IDENTIFIER, SPECIAL_CHARS, etc; did you use SimpleSqlGrammarTest? Before moving ahead with this I would like to make sure we are not breaking any of these rules.

Thanks,
Daniel.

I did use the SimpleSqlGrammarTest.groovy file while developing this, and all the tests are still in that file unmodified (other than a formatting change that was bugging me that doesn't change the functionality of the test scenario). I also added more test cases to cover the issues I was seeing.

Additionally, I added some output to the SimpleSqlGrammarTest.groovy that outputs a separator, the current test string, and then for each token: the type of identifier and the string that is identified for that token. This helped me make sure the identifiers didn't change across my changes: all existing strings stayed the same kind of identifier as far as I could find

Made me second guess myself since I just did a manual checking of the output across tests for the token type and text for each token. Added my output code to my local master, compiled, captured and saved the output. Then switched to this branch, compiled, captured and saved the output. Did a diff of the two files and the results are identical (other than my branch had the additional tests). Attaching the files here in case anyone wants to see the results
lb_grammar_test_results_branch.txt
lb_grammar_test_results_master.txt

Made me second guess myself since I just did a manual checking of the output across tests for the token type and text for each token. Added my output code to my local master, compiled, captured and saved the output. Then switched to this branch, compiled, captured and saved the output. Did a diff of the two files and the results are identical (other than my branch had the additional tests). Attaching the files here in case anyone wants to see the results lb_grammar_test_results_branch.txt lb_grammar_test_results_master.txt

Hey @jasonlyle88,

Nice, I liked that manual testing check. I didn't get how you generated those output files. Could you please detail those steps you mentioned for me? That could be very useful for me when having to review any other grammar issue next time.

Other than that, this PR looks good to me. I just made a formatting change to the test class and added two of the scenarios you listed in the description of #5674, I think they are already covered by the ones you already added but like of those random mixed symbols 😄.

Awesome, sounds good! Thanks.

As for how I generated them: I'm using IntilliJ Idea. When I have the SimpleSqlGrammarTest.groovy file open, I have icons in my line gutter allowing me to run an individual test or all tests in the current class (only 1 in the case of this file). When I run the test, it open a result panel with a tree of the test results (class results at the root, test results at the next level node, and individual test cases at the next level node). By selecting the root (or the individual test) node, I get to see all the output from the test. I just copied and pasted that output to make the file.

I'm sure you can do this when running the test through the CLI as well... but I don't know how to run individual tests via the CLI (letting my Java noob show through here 🤣).

No problem, always happy to contribute!

Awesome, sounds good! Thanks.

As for how I generated them: I'm using IntilliJ Idea. When I have the SimpleSqlGrammarTest.groovy file open, I have icons in my line gutter allowing me to run an individual test or all tests in the current class (only 1 in the case of this file). When I run the test, it open a result panel with a tree of the test results (class results at the root, test results at the next level node, and individual test cases at the next level node). By selecting the root (or the individual test) node, I get to see all the output from the test. I just copied and pasted that output to make the file.

I'm sure you can do this when running the test through the CLI as well... but I don't know how to run individual tests via the CLI (letting my Java noob show through here 🤣).

Thanks @jasonlyle88! I thought you had a trick to run these tests and grab the output from all of them automatically.

Thanks everyone for the quick work here! Appreciate it!