Skip to content

Commit

Permalink
Improve Markdown code block tokens (microsoft#17591)
Browse files Browse the repository at this point in the history
* Improve Markdown code block tokens

Code blocks without a language weren't tokenized.
Code blocks didn't have their ending ``` punctuation tokenized.
Code blocks used to only have one token. Now each block has the following tokens available for syntax highlighters:
- Starting and ending ``` punctuations
- Code block's language setting
- Code snippet

* Variable whitespace for MD code block ``` token

Allow for variable amount of whitespacing before ``` code blocks

* Reorder raw blocks

Raw blocks were preventing tokenizing as languaged blocks. Putting them on bottom resolves this.

* Fix MD block detection when following paragraph

Used to require a new line inbetween ``` code blocks and preceding paragraph text.

* Prevent broken language grammar leaks in MD fences

Prevents leaks in MD code fences while also capturing the closing fence punctuations.

* Update Markdown tokenizer test file
  • Loading branch information
joshpeng authored and mjbvz committed Jan 3, 2017
1 parent 7b663a4 commit b9a362a
Show file tree
Hide file tree
Showing 2 changed files with 1,660 additions and 309 deletions.
Loading

0 comments on commit b9a362a

Please sign in to comment.