Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON rewrite #1029

Merged
merged 9 commits into from
Jul 19, 2019
Merged

JSON rewrite #1029

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d9cf996
JSON rewrite
miparnisari Nov 3, 2018
ecc25e4
JSON: add test, minor fixes
miparnisari May 30, 2019
22eb192
Fix rubocop warnings "Ambiguous regexp literal."
miparnisari Jul 9, 2019
fc8223c
Change object tokens, support bare key-values
pyrmont Jul 10, 2019
abe2d92
Improve bare key-value pair lexing
pyrmont Jul 10, 2019
5a6295b
Simplify rules
pyrmont Jul 12, 2019
6b3de9a
Update HOCON lexer to work with new JSON lexer
pyrmont Jul 12, 2019
578cb56
Fix typo in HOCON lexer
pyrmont Jul 12, 2019
94a024d
Move pathological case to visual sample
pyrmont Jul 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change object tokens, support bare key-values 
This commit makes several changes to the proposed lexer.

First, it changes the tokens used in JavaScript objects. It was proposed
that keys be tokenised as `Str::Double` and that string values be
tokenised as `Name::Tag`. The token `Name::Label` seems a more
appropriate token for keys while `Str::Double` should be used for
string values.

Second, it remoeves the tokenisation of escape sequences inside object
keys. While this is correct insofar as the keys are merely strings, it
fits awkwardly with the tokenisation of keys with a non-string token.
This commit tokenises all characters within the key as `Name::Label`.

Third, it adds support for 'bare' key-value pairs, that is text of the
form <string>: <value>. These 'bare' key-value pairs are not tokenised
the same way as they would be within objects. Rather, keys are tokenised
as ordinary strings. This preserves backwards compatability with how the
JSON lexer previously worked.
  • Loading branch information
@pyrmont
pyrmont committed Jul 12, 2019
commit fc8223caca8e1af2311bf3226f2a0b58181235da
48 changes: 28 additions & 20 deletions lib/rouge/lexers/json.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,58 +12,66 @@ class JSON < RegexLexer
'application/hal+json', 'application/problem+json',
'application/schema+json'

state :root do
state :whitespace do
rule %r/\s+/, Text::Whitespace
end

state :root do
mixin :whitespace
rule %r/{/, Punctuation, :object
rule %r/\[/, Punctuation, :array

rule(%r//) { push :value }
end

state :object do
rule %r/\s+/, Text::Whitespace
rule %r/"/, Str::Double, :name
mixin :whitespace
rule %r/"/, Name::Label, :name
rule %r/:/, Punctuation, :value
rule %r/,/, Punctuation
rule %r/}/, Punctuation, :pop!
end

pyrmont marked this conversation as resolved.
Show resolved Hide resolved
state :value do
rule %r/"/, Name::Tag, :stringvalue
mixin :whitespace
rule %r/"/, Str::Double, :string_value
mixin :constants
rule %r/}/ do
token Punctuation
pop! 2 # pop both this state and the :object one below it
end
rule %r/\[/, Punctuation, :array
rule %r/{/, Punctuation, :object
rule %r/}/ do
token Punctuation
pop! 2 # pop both this state and the :object one below it
if stack[-2].name == :object
token Punctuation
pop! 2 # pop both this state and the :object one below it
else
token Error
pop!
end
end
rule %r/,/, Punctuation, :pop!
rule %r/\s+/, Text::Whitespace
rule %r/:/, Punctuation
end

state :name do
rule %r/[^\\"]+/, Str::Double
rule %r/\\./, Str::Escape
rule %r/"/, Str::Double, :pop!
rule %r/[^\\"]+/, Name::Label
rule %r/\\./, Name::Label
rule %r/"/, Name::Label, :pop!
end

state :stringvalue do
rule %r/[^\\"]+/, Name::Tag
state :string_value do
rule %r/[^\\"]+/, Str::Double
rule %r/\\./, Str::Escape
rule %r/"/, Name::Tag, :pop!
rule %r/"/, Str::Double, :pop!
end

state :array do
rule %r/\]/, Punctuation, :pop!
rule %r/"/, Name::Tag, :stringvalue
rule %r/"/, Str::Double, :string_value
rule %r/,/, Punctuation
mixin :constants
mixin :root
end

state :constants do
state :constants do
rule %r/(?:true|false|null)/, Keyword::Constant
rule %r/-?(?:0|[1-9]\d*)\.\d+(?:e[+-]?\d+)?/i, Num::Float
rule %r/-?(?:0|[1-9]\d*)(?:e[+-]?\d+)?/i, Num::Integer
Expand Down