Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve Markdown code block tokens (microsoft#17591)
* Improve Markdown code block tokens Code blocks without a language weren't tokenized. Code blocks didn't have their ending ``` punctuation tokenized. Code blocks used to only have one token. Now each block has the following tokens available for syntax highlighters: - Starting and ending ``` punctuations - Code block's language setting - Code snippet * Variable whitespace for MD code block ``` token Allow for variable amount of whitespacing before ``` code blocks * Reorder raw blocks Raw blocks were preventing tokenizing as languaged blocks. Putting them on bottom resolves this. * Fix MD block detection when following paragraph Used to require a new line inbetween ``` code blocks and preceding paragraph text. * Prevent broken language grammar leaks in MD fences Prevents leaks in MD code fences while also capturing the closing fence punctuations. * Update Markdown tokenizer test file
- Loading branch information