@goodhoko Thanks for the PR! I realise it introduces a bit of duplication, but I thought from a maintainability perspective, it's a little bit easier to parse (as a human) the new rules if they're separated out. This is also more consistent with most other lexers and has the bonus benefit of allowing more targeted tokens.

@pyrmont Cool! Thanks. I added one more commit that replaces the generi Num rule with more specific Num::Float. This is ready to me merged from my side now. .)

@goodhoko I think we need to put the Num::Float rule last because the decimal point and exponent are all optional, right? So integers will match that rule if it comes first. Let me know what you think.

@pyrmont Exactly because the . and e parts of the Float and [bB] or [xX] parts of the Bin and Hex are optional the rules needs to come first. If the integer were first, it would 'eat up' the integer part of the float or hex literals breaking the whole number apart.

For example.
1e10 -> ["1", Int], ["e", Text], ["10", Int]

Whoops—you're completely right. What do you want to do? You can either make a float rule that will ensure there's at least a decimal point or an exponent or we can leave it as-is and just tokenise most integers as floats.

@pyrmont ah good catch! I didn't notice that the Float regexp would match integers too.

Since correctness is probably not the ultimate goal of Rouge I propose to revert back to b01aa87 and just accept we won't parse reals as floats but rather as general Number (along with some Integers too.)

But pursuing correctness is possible. Here's the correct regexp for reals:

((((([\d][\d_]*[\d]|[\d])?\.([\d][\d_]*[\d]|[\d])([eE][+-]?([\d][\d_]*[\d]|[\d]))?)|(([\d][\d_]*[\d]|[\d])([eE][+-]?([\d][\d_]*[\d]|[\d]))))[fF])|(([\d][\d_]*[\d]|[\d])[fF]))|(((([\d][\d_]*[\d]|[\d])?\.([\d][\d_]*[\d]|[\d])([eE][+-]?([\d][\d_]*[\d]|[\d]))?)|(([\d][\d_]*[\d]|[\d])([eE][+-]?([\d][\d_]*[\d]|[\d])))))

Constructed recursively from the Grammar. Matches only Doubles or Floats.

To reduce the overwhelming complexity, we could break the regexp into parts (just as on the grammar page) and then use these parts in the rule definitions. For example

      decDigit =  %r"[0-9]"
      decDigitOrSeparator = %r"#{decDigit}|_"
      decDigits = %r"(#{decDigit}(#{decDigitOrSeparator})*#{decDigit})|#{decDigit}"
      doubleExponent = %r"[eE][+-]?(#{decDigits})"
      doubleLiteral = %r"((#{decDigits})?\.#{decDigits}(#{doubleExponent})?)|#{decDigits}#{doubleExponent}"
      floatLiteral = %r"(#{doubleLiteral}[fF])|(#{decDigits}[fF])"
      ...
      rule %r"(#{floatLiteral})|(#{doubleLiteral})", Num::Float
      # and so on for other Num literals

I'd gladly do that, I'm just not sure if this level of granularity is wanted here in Rouge. .)

The Rust lexer has arguably a decent compromise:

We could do that here, too? Otherwise I don't mind going back to b01aa87.

@pyrmont cool. Gimme some time I'll try to rewrite this into something sane. .)

@pyrmont I reworked this, please check it out.

The rules for numbers are now correct - they reflect the grammar with no shortcomings.
I believe the code complexity is reasonable. Note that Kotlin grammar is quite relaxed compared to Rust. Kotlin allows more forms of number literals which in turn requires longer rules.

I also moved the number rules above the Punctuation rules. Otherwise the dot of the dot-starting number literals (.99) would be parsed as Punctuation.