This needs to be updated with further information. I think it would also be helpful to submit a separate PR changing the YARD settings.

I haven't finished reading this but I promise to do that during the weekend and add feedback :) This is very useful stuff

Thanks @miparnisari :) I hope it's useful. I've merged this into master but it's intended to be a draft so please feel free to contribute feedback (either via GitHub or as a PR).

I don't know how github works anymore so I forgot to hit the "Submit Review" button ><.

@jneen That's cool! Thanks for the feedback. I'll get it updated as soon as I can (probably later tonight) :)

Well I read this, very very well written and I actually learned a few things.

• Can we add information on how to debug while developing? What I do is run rackup >> debug.txt and then load localhost:9292/json?debug which prints something like

lexer: json
stack: [:root]
stream: "\"firstName\": \"John\","
  trying #<Rule /\s+/m>
  trying #<Rule /"/>
    got "\""
    yielding Literal.String.Double, "\""
    pushing :string
lexer: json
stack: [:root, :string]
stream: "firstName\": \"John\",\n"
  trying #<Rule /[^\\"]+/>
    got "firstName"
    yielding Literal.String.Double, "firstName"

• A question which I don't have an answer for but I sure wondered while developing. How many characters are evaluated at any given time? It seems like the number is 24 characters but I can't find this constant in the code.
• Can we incorporate what was written in Add a wiki page with Docker instructions for development #947 ?
• Also this Document options #76
• And this document lexer tags, aliases, and file associations #50

I'll reply to the other points a bit later tonight but regarding the way the source is consumed, @miparnisari wrote:

A question which I don't have an answer for but I sure wondered while developing. How many characters are evaluated at any given time? It seems like the number is 24 characters but I can't find this constant in the code.

Rouge uses StringScanner from the Ruby standard library. The actual method called to advance through the string is #skip:

I'm not sure how #scan #skip is implemented but I'd assume it's doing one character (or possibly one Unicode code point) at a time.

I'm not sure how #scan is implemented

From the docs for StringScanner#scan, it looks like the pointer advances till the end of match. For example:

s = StringScanner.new('test string')
#  test string
# ^
s.scan(/\w+/)   # => test
#   string
# ^
s.scan(/\w+/)   # nil
#   string
# ^
s.scan(/\s+/)   # => " "
#  string
# ^
s.scan(/\w+/)   # => string
# EOS
#    ^

Gah, I said #scan but meant #skip. Sorry for the confusion!

but meant #skip

😃 From the same docs :

#skip is similar to #scan, but without returning the matched string.

@miparnisari The other answers to your questions:

1. (Debug Documentation) I've created Add debug documentation #1144 regarding debug documentation.

2. (Magic Constants) I've now looked at (some) of the C source for StringScanner and I don't see any reason you should be seeing this behaviour. It looks to me like it should be evaluating a character at a time.

3. (Docker Documentation) I'd like to include documentation about setting up a Ruby environment for development. We have some information on the wiki but I'd like to move that to docs/ so that, like the lexer development guide, it will show up on RubyDoc.info.

4. (Option Documentation) Speaking of RubyDoc.info, code comments that are in the appropriate form for YARD, automatically generate documentation on RubyDoc and should provide the documentation requested in Document options #76. However, as we haven't released an update to the gem yet, the current docs don't pick up the changes that were made recently to include documentation attached to protected methods. You can, however, see that in the docs generated from master. To the extent method parameters, options, etc aren't documented, we really should be updating the comments in the source code and having it generate documentation on RubyDoc.

5. (DSL Documentation) My understanding is that YARD can also be taught how to read DSLs and generate documentation from that. That might not produce the list that document lexer tags, aliases, and file associations #50 is about but it might be 80% of what's required. I'm kind of loathe to have too much documentation that has to be updated manually. Things like the lexer development guide are one thing but having a master list of all the settings for each lexer is prone to fall out of sync.