Just a related discussion. Some syntax like meson has only a small set of predefined valid function names and you cannot define your own.

The current syntax file highlights any function call whether it exists or not. In other words about the known function names. In other kind of syntax the may be something other than functions, for example in shell syntax they may be the known internal commands.

It could be nice to assist the user with which function is valid or not with highlighting and complete suggestions.

On the other side I didn't find any obvious way to include that information in the syntax file and also have it proposed for auto-completion.

I would be glad to have suggestions to include this mechanism in current syntax system or to propose an enhancement to it to include the known function names.

This is really nice, I was going to do this and totally forgot, Thank you!

Just a related discussion. Some syntax like meson has only a small set of predefined valid function names and you cannot define your own.

The current syntax file highlights any function call whether it exists or not. In other words about the known function names. In other kind of syntax the may be something other than functions, for example in shell syntax they may be the known internal commands.

It could be nice to assist the user with which function is valid or not with highlighting and complete suggestions.

On the other side I didn't find any obvious way to include that information in the syntax file and also have it proposed for auto-completion.

What do you think about besides having the "symbols" array also introducing a new element named "functions"? This new element would be used to list the language core functions and the color applied to them could be the same used for the operators or a different type introduced just for this with different styling. Here a screenshot of what vscode does:

vscode at least on this language uses operator color for core functions and the regular functions color for every other user defined function.

Also what about also using the "info" property to display the item type? For symbols it should be "Keyword" and for core functions "Function", also we should maybe also integrate these item kinds somehow https://github.com/lite-xl/lite-xl-lsp/blob/master/server.lua#L137-L143 and the icons support that @TorchedSammy worked here https://github.com/TorchedSammy/lite-xl-lspkind

It could be nice to assist the user with which function is valid or not with highlighting and complete suggestions.

On the other side I didn't find any obvious way to include that information in the syntax file and also have it proposed for auto-completion.

On the highlighting side, wouldn't it be enough to add the function names to the symbols table?
So setting the functions patterns to highlight with the normal token type, then in the symbols table we match the function name to the function token type?

On the autocomplete suggestions side, I think this should be implemented as a different plugin (something like autocomplete_meson), which could also offer suggestions on the arguments.
Anything more complex is probably in LSP territory.

Thank you for your comments. For the known functions' names there seems to be no easy approach that fit naturally within the current syntax system. Maybe we will have a good idea at some moment.

Wouldn't something like

@@ -12,7 +12,7 @@ syntax.add {
     { pattern = "0x[%da-fA-F]+",          type = "number"   },
     { pattern = "-?%d+%d*",               type = "number"   },
     { pattern = "[%+%-=/%%%*!]",          type = "operator" },
-    { pattern = "[%a_][%w_]*%f[(]",       type = "function" },
+    { pattern = "[%a_][%w_]*%f[(]",       type = "normal"   },
     { pattern = "[%a_][%w_]*",            type = "symbol"   },
   },
   symbols = {
@@ -31,6 +31,30 @@ syntax.add {
     ["in"]         = "keyword",
     ["true"]       = "literal",
     ["false"]      = "literal",
+    ["project"]            = "function",
+    ["executable"]         = "function",
+    ["test"]               = "function",
+    ["get_option"]         = "function",
+    ["project_version"]    = "function",
+    ["current_build_dir"]  = "function",
+    ["current_source_dir"] = "function",
+    ["set"]                = "function",
+    ["system"]             = "function",
+    ["get_id"]             = "function",
+    ["find_library"]       = "function",
+    ["dependency"]         = "function",
+    ["found"]              = "function",
+    ["install_data"]       = "function",
+    ["install_subdir"]     = "function",
+    ["configure_file"]     = "function",
+    ["subdir"]             = "function",
+    ["find_program"]       = "function",
+    ["run_command"]        = "function",
+    ["stdout"]             = "function",
+    ["strip"]              = "function",
+    ["substring"]          = "function",
+    ["configuration_data"] = "function",
+    ["get_compiler"]       = "function",
   },
 }

help a bit?

@Guldoman you are perfectly right, they can be declared like "function" is the "symbols" list. It is a perfectly good solution.

I got blinded to this solution because I wanted to have them recognized like functions only when they are followed by a "(" which is not possible with the "symbols" approach AFAICT. On the other side the fact to use the "(" to validate a function pattern if not terribly important so probably we may go for this solution to improve some syntax like Meson, thank you.

I mean, if we get creative enough, using the power of regexes we can match actual functions:

@@ -12,7 +12,32 @@ syntax.add {
     { pattern = "0x[%da-fA-F]+",          type = "number"   },
     { pattern = "-?%d+%d*",               type = "number"   },
     { pattern = "[%+%-=/%%%*!]",          type = "operator" },
-    { pattern = "[%a_][%w_]*%f[(]",       type = "function" },
+    { regex   = "("..
+    "project|"..
+    "test|"..
+    "get_option|"..
+    "meson\\.project_version|"..
+    "meson\\.current_build_dir|"..
+    "meson\\.current_source_dir|"..
+    "set|"..
+    "host_machine\\.system|"..
+    "cc\\.get_id|"..
+    "cc\\.find_library|"..
+    "dependency|"..
+    "found|"..
+    "install_data|"..
+    "install_subdir|"..
+    "configure_file|"..
+    "subdir|"..
+    "find_program|"..
+    "run_command|"..
+    "stdout|"..
+    "strip|"..
+    "substring|"..
+    "configuration_data|"..
+    "meson\\.get_compiler"..
+    ")(?= *\\()",                           type = "function"   },
+    { pattern = "[%a_][%w_]*%f[(]",       type = "normal"   },
     { pattern = "[%a_][%w_]*",            type = "symbol"   },
   },
   symbols = {

I thought about this solution but it disqualifies for feeding the new auto-complete mechanism. In addition we are somewhat abusing the pattern matching mechanism.

I would like a new mechanism that marries in a simple way the declaration of a list of plain words with a pattern matching expression.

For example something like ::symbol:function::%f[(] the we should manage to use this pattern to match the symbol part with any of the declared word.

If one of you like the idea and want to tackle a possible implementation it would be great.

We may end up with more simple, declarative syntax that works well with auto-complete.

I thought about this solution but it disqualifies for feeding the new auto-complete mechanism. In addition we are somewhat abusing the pattern matching mechanism.

Yeah, fair enough.

I would like a new mechanism that marries in a simple way the declaration of a list of plain words with a pattern matching expression.

For example something like ::symbol:function::%f[(] the we should manage to use this pattern to match the symbol part with any of the declared word.

If one of you like the idea and want to tackle a possible implementation it would be great.

We may end up with more simple, declarative syntax that works well with auto-complete.

What do you think about something like:

syntax.add {
  ...
  patterns = {
    ...
    {
      pattern = "[%a_][%w_]*%f[(]",
      type = "function",
      one_of = {
        "project",
        "executable",
        "get_option",
        ...
      }
    },
    ...
  }
}

This would work basically like our current symbols table, but to actually match, the text must be present inside the one_of table.

We could even do something like:

syntax.add {
  ...
  patterns = {
    ...
    {
      pattern = "[%a_][%w_]*()%.[%a_][%w_]*%f[(]",
      type = { "keyword", "function" },
      one_of = {
        {
          "meson",
          "cc"
        },
        {
          ".project",
          ".executable",
          ".get_option",
          ...
        },
      }
    },
    ...
  }
}

to support multi-part matches.
In this case, the first part of the match must be included in the first table of one_of, while the second part of the match in the second table.
If we were to support this, the autocomplete functionality could get a bit more complex, tho.

We could also add the missing completion_item_kind (https://github.com/lite-xl/lite-xl-lsp/blob/69935e4a395ce1a9bb540a79aa14abb907673500/server.lua#L137-L143) as possible token types, but this could mean changing many of our language plugins and themes.

I like this idea of one_of. I would like to test this approach in a feature branch.

I would limit the one_of mechanism only for patterns that match plain words. For multi-part match restrict each part of the match to be a plain word and provide a flat list of acceptable tuples like:

one_of = {
  {"meson", "get_compiler"},
  {"meson", "something"},
  {"*", "foo"},
  ...
}

the special entry "*" would mean "match anything" and would be ignored for auto-completion.

We may let each part of the tuple be a pattern but the auto-complete will have an hard time to recognize patterns versus plain words.

I would limit the one_of mechanism only for patterns that match plain words.

What do you mean? How would you limit the scope of the one_of? For example how would you make it so it applies only to functions?

For multi-part match restrict each part of the match to be a plain word and provide a flat list of acceptable tuples like:

one_of = {
  {"meson", "get_compiler"},
  {"meson", "something"},
  {"*", "foo"},
  ...
}

the special entry "*" would mean "match anything" and would be ignored for auto-completion.

This approach seems more complex than the one I proposed.
It also has the disadvantage of having to repeat words (like meson in your example).
But the main problem is that it would force us to go through the whole one_of table to find a satisfying match, so we'd have a (not terrible, but still) linear complexity.

Adapting your example using my approach (in my case an empty table has the same meaning of your *):

{
  pattern = "[%a_][%w_]*()%.()[%a_][%w_]*%f[(]",
  type = { "keyword", "normal", "function" },
  one_of = {
    {
      "meson",
    },
    { },
    {
      "project_version",
      "current_build_dir",
      "current_source_dir",
      "get_compiler",
      "something",
      ...
    },
  }
},
{
  pattern = "[%a_][%w_]*()%.()[%a_][%w_]*%f[(]",
  type = { "keyword", "normal", "function" },
  one_of = {
    { },
    { },
    {
      "foo",
      "bar",
      "baz",
      ...
    },
  }
},

This approach has the possibility to reuse tables for different one_ofs.
Also the lookup time is O(1) for each part.

We may let each part of the tuple be a pattern but the auto-complete will have an hard time to recognize patterns versus plain words.

Yeah, let's leave them as plain words.

I think the way we give the list for the multi-pattern is more of a detail, not terribly important right now.

We can go the way you suggest and provide a first implementation. Once we get started with some concrete syntax example we may choose on some details like the best approach for the multi-pattern list.

I suggest also that we may call the "one_of" "values" but this is a matter of preference.

I think the way we give the list for the multi-pattern is more of a detail, not terribly important right now.

Yeah, that's fair.

We can go the way you suggest and provide a first implementation. Once we get started with some concrete syntax example we may choose on some details like the best approach for the multi-pattern list.

This is not a priority for me at the moment, but I'll see what I can do.

I suggest also that we may call the "one_of" "values" but this is a matter of preference.

Yeah, we need a more descriptive term for this.