Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

some updates to the PHP lexer #1397

Merged
merged 15 commits into from
Apr 7, 2020
Merged

some updates to the PHP lexer #1397

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
4a21ac0
* fix case insensitivity of (accordingly to the language):
julp Jan 18, 2020
060295a
downcase null/true/false/stdclass for now
julp Jan 18, 2020
0c64839
fix heredoc: the syntax with `"` was not recognized
julp Jan 18, 2020
00911c6
to handle class keyword of an anonymous class
julp Jan 18, 2020
9090523
fix regexpes (syntax) and make as keyword case insensitive in use-list
julp Jan 22, 2020
f441f10
pull out `true`, `false` and `null` from keywords:
julp Jan 22, 2020
2d98e5a
Recognize stdClass as a class instead of a keyword
julp Apr 4, 2020
ef6fc5e
cleanup PHP keywords:
julp Apr 4, 2020
2aa5634
recognizes E_* and PHP_* constants (as `(E|PHP)(_[[:upper:]]+)+`)
julp Apr 4, 2020
775cd71
add modifier `i` to those rules since keywords are case insensitive i…
julp Apr 4, 2020
a12604d
Remove trailing whitespace
pyrmont Apr 5, 2020
ca22a6c
Simplify numeric lexing rules
pyrmont Apr 5, 2020
5c80d72
simplify rules for numbers
julp Apr 5, 2020
c93f333
Merge branch 'php-7.4' of github.com:julp/rouge into php-7.4
julp Apr 5, 2020
67d3bb5
fix tokenization of nullable types (eg `?int`): the rule handling
julp Apr 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
* fix case insensitivity of (accordingly to the language): 
  + `<?php`
  + keywords
  + function/method names
* updates to the language:
  + support `_` in (binary, decimal, hexadecimal, ...) numbers (7.4.0)
  + support for binary numbers (`0b...`) (5.4.0)
  + Unicode codepoints escape syntax (`\u{...}`) (7.0.0)
  + add some missing keywords:
    * `fn` keyword (7.4.0)
    * type declarations (PHP 7, including `void` and nullable types from PHP 7.1)
    * 7.0.0: `class` (anonymous classes), `yield from`
    * 5.4.0: `callable`, `insteadof`, `trait`, `__TRAIT__`
    * 5.3.0: `goto`, `__NAMESPACE__`, `__DIR__`
    * others: casts, `instanceof`, `__CLASS__`, `__FUNCTION__`, `__METHOD__`, `__halt_compiler`
    * `self` even if it's not really a reserved word
  • Loading branch information
@julp
julp committed Apr 4, 2020
commit 4a21ac004616eb44d81d942728dce2d2ec4c4834
33 changes: 23 additions & 10 deletions lib/rouge/lexers/php.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ def builtins
end
end

LNUM = '\d+(?:_\d+)*'
DNUM = '(?:(?:#{LNUM}?\.#{LNUM})|(?:#{LNUM}\.#{LNUM}?))'
WHITESPACE = '[ \n\r\t]+'
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
# source: http://php.net/manual/en/language.variables.basics.php
# the given regex is invalid utf8, so... we're using the unicode
# "Letter" property instead.
Expand All @@ -69,10 +72,13 @@ def self.keywords
print for require continue foreach require_once declare return
default static do switch die stdClass echo else TRUE elseif
var empty if xor enddeclare include virtual endfor include_once
while endforeach global __FILE__ endif list __LINE__ endswitch
while endforeach global __file__ endif list __line__ endswitch
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
new __sleep endwhile not array __wakeup E_ALL NULL final
php_user_filter interface implements public private protected
abstract clone try catch finally throw this use namespace yield
fn callable insteadof trait __trait__ goto __namespace__ __dir__
instanceof __class__ __function__ __method__ __halt_compiler
self
)
end

Expand All @@ -85,13 +91,13 @@ def self.detect?(text)
state :root do
# some extremely rough heuristics to decide whether to start inline or not
rule(/\s*(?=<)/m) { delegate parent; push :template }
rule(/[^$]+(?=<\?(php|=))/) { delegate parent; push :template }
rule(/[^$]+(?=<\?(php|=))/i) { delegate parent; push :template }

rule(//) { push :template; push :php }
end

state :template do
rule %r/<\?(php|=)?/, Comment::Preproc, :php
rule %r/<\?(php|=)?/i, Comment::Preproc, :php
rule(/.*?(?=<\?)|.*/m) { delegate parent }
end

Expand Down Expand Up @@ -137,12 +143,17 @@ def self.detect?(text)
end

rule %r/(true|false|null)\b/, Keyword::Constant
rule /(?:void|\??(?:int|float|bool|string|iterable))\b/i, Keyword::Type
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
rule /\??(?:self|callable)\b/i, Keyword::Type
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
rule %r/\$\{\$+#{id}\}/i, Name::Variable
rule %r/\$+#{id}/i, Name::Variable
rule /(yield)(#{WHITESPACE})(from)/io do
groups Keyword, Text, Keyword
end

# may be intercepted for builtin highlighting
rule %r/\\?#{nsid}/i do |m|
name = m[0]
name = m[0].downcase
pyrmont marked this conversation as resolved.
Show resolved Hide resolved

if self.class.keywords.include? name
token Keyword
Expand All @@ -153,11 +164,12 @@ def self.detect?(text)
end
end

rule %r/(\d+\.\d*|\d*\.\d+)(e[+-]?\d+)?/i, Num::Float
rule %r/\d+e[+-]?\d+/i, Num::Float
rule %r/0[0-7]+/, Num::Oct
rule %r/0x[a-f0-9]+/i, Num::Hex
rule %r/\d+/, Num::Integer
rule %r/(?:(?:#{LNUM}|#{DNUM})[eE][+-]?#{LNUM})/o, Num::Float
rule %r/#{DNUM}/o, Num::Float
rule %r/0[0-7]+(?:_[0-7]+)*/, Num::Oct
rule %r/0[bB][01]+(?:_[01]+)*/, Num::Bin
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
rule %r/0x[a-f0-9]+(?:_[a-f0-9]+)*/i, Num::Hex
rule %r/#{LNUM}/o, Num::Integer
rule %r/'([^'\\]*(?:\\.[^'\\]*)*)'/, Str::Single
rule %r/`([^`\\]*(?:\\.[^`\\]*)*)`/, Str::Backtick
rule %r/"/, Str::Double, :string
Expand Down Expand Up @@ -189,7 +201,8 @@ def self.detect?(text)
state :string do
rule %r/"/, Str::Double, :pop!
rule %r/[^\\{$"]+/, Str::Double
rule %r/\\([nrt\"$\\]|[0-7]{1,3}|x[0-9A-Fa-f]{1,2})/,
rule /\\u\{[0-9a-fA-F]+\}/, Str::Escape
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
rule %r/\\([efrntv\"$\\]|[0-7]{1,3}|x[0-9A-Fa-f]{1,2})/,
Str::Escape
rule %r/\$#{id}(\[\S+\]|->#{id})?/, Name::Variable

Expand Down