Hello, I have rewritten the lexer and added the missing demo file.

@alexlihengwang Thanks :) Will try to get to this over the weekend!

@alexlihengwang Have you had a chance to look at the comments I made in December?

Also pinging @ph03 given you filed the original issue. Perhaps you can answer some of the questions above.

Having BibTeX support in rouge would allow for syntax highlighting in GitLab, which led me to this PR.

I know nothing of Ruby, little about bib(la)tex but a thing or two about LaTeX in general. Therefore, I won't be able to help with code, but maybe still give you some useful pointers.

The following is based on biblatex, whose documentation in PDF format is here, the source of which is on GitHub. The "official" bib example file is probably a good body for testing.

biblatex uses biber as its back-end processing tool, and here it says that biber in turn relies on btparse, a C library. The latter is documented really well here. biblatex is the de-facto current standard, so relying on what they are relying on should be safe. The Synopsis of bt_language has this to say:

# Lexical grammar, mode 1: top-level
AT                    \@
NEWLINE               \n
COMMENT               \%~[\n]*\n
WHITESPACE            [\ \r\t]+
JUNK                  ~[\@\n\ \r\t]+

# Lexical grammar, mode 2: in-entry
NEWLINE               \n
COMMENT               \%~[\n]*\n
WHITESPACE            [\ \r\t]+
NUMBER                [0-9]+
NAME                  [a-z0-9\!\$\&\*\+\-\.\/\:\;\<\>\?\[\]\^\_\`\|]+
LBRACE                \{
RBRACE                \}
LPAREN                \(
RPAREN                \)
EQUALS                =
HASH                  \#
COMMA                 ,
QUOTE                 \"

Regarding your comments for the PR, @pyrmont , two things are shown there: numbers are indeed only [0-9]+, or [\d]+ (like in pygments, as referenced in the original issue).

As a side-note, it also says:

One of the problems with BibTeX is that there is no formal specification of the language. This means that users exploring the arcane corners of the language are largely on their own, and programmers implementing their own parsers are completely on their own---except for observing the behaviour of the original implementation.

Ouch... anyway.

Escaping quotes with \" does not seem to be how it is done, see here. In the example file linked above, \" occurs to create German umlauts, aka for turning {\"U}ber into Über in the output (when there's no UTF8...). In bt_language.pod as linked above, it says:

" always starts a string, regardless of context.

I think this is good to merge. I'll leave it for a couple more days in case there is any additional feedback.

Thanks a lot for providing the information you did, @alexpovel. That was very helpful!

Oh that was a pleasant resolve then, thanks @pyrmont ! Glad I could clear things up, after @alexlihengwang implemented everything.

@alexlihengwang @alexpovel Thanks to both of you for your help on this. This will be part of the next version of Rouge, v3.20.0. That will be pushed to RubyGems on Tuesday 9 June 🎉 Thanks again!

Thanks for the help @alexpovel @pyrmont