Adding some development notes

pulsar-edit · confused-Techie · Sep 4, 2022 · Jun 25, 2022 · Jun 26, 2022 · Jun 26, 2022
commit 59c0795c8a25fd23b03297a524ce0a28b1aa7bba
diff --git a/docs/dev/README.md b/docs/dev/README.md
@@ -0,0 +1,5 @@
+# Development README
+
+On this directory, we can include things that we found out how they work, and how do we want to handle that in the future
+
+- [Tree Sitter](tree-sitter.md), the tokenizer for the Atom Text Editor
diff --git a/docs/dev/tree-sitter.md b/docs/dev/tree-sitter.md
@@ -0,0 +1,103 @@
+# Tree Sitter in Pulsar
+
+Tree-sitter is a tokenizer that uses native modules. The idea is that a language generates an AST of the source code and then Pulsar will tokenize these on the editor with some rules on CSON files (that kind of resemble CSS selectors)
+
+## Debugging a Grammar
+
+Inside Pulsar's source code is possible to require Tree-Sitter and try to parse some grammar. To do this, run this code on Devtools:
+
+```js
+const Parser = require('tree-sitter');
+const Java = require('tree-sitter-java');
+
+const parser = new Parser();
+parser.setLanguage(Java);
+
+tree = parser.parse(`
+class A {
+  void func() {
+    obj.func2("arg");
+  }
+}
+`);
+console.log(tree.rootNode.toString());
+```
+
+This will create a parser, set its language to Java, and try to parse the source code that we sent. This specific fragment of code will print:
+
+```
+(program
+  (class_declaration
+    name: (identifier)
+    body: (class_body
+      (method_declaration
+        type: (void_type)
+        name: (identifier)
+        parameters: (formal_parameters)
+        body: (block
+          (expression_statement
+            (method_invocation
+              object: (identifier)
+              name: (identifier)
+              arguments: (argument_list
+                (string_literal)))))))))
+```
+
+I did the pretty-print manually. Basically, it says that the "root node" is a `program` that contains a `class_declaration`. Following that, comes the class's name, then its body, etc etc.
+
+## Modern tree-sitter
+
+If you look at the AST above, you'll see that there are things inside parenthesis and things like `name: ` and `body: `. This second one is what Tree-Sitter now calls "field name", and Pulsar is not yet using this anywhere. This is problematic for multiple reasons, but the main one is that tokenization gets wrong: for example, in the code above, we want to tokenize `obj.func2("arg")` by marking `func2` as a function that's being called, but the AST for that fragment is:
+
+```
+(method_invocation
+  object: (identifier)
+  name: (identifier)
+  arguments: (argument_list (string_literal)))
+```
+
+What disambiguates the method name from other things is the field name: `obj` have field name `object`, and `func2` have field name `name`. As Pulsar is not parsing this, the closest match we can get is:
+
+```cson
+  'method_invocation > identifier': 'entity.name.function'
+```
+
+But unfortunately, this does not solve the issue - both `obj` and `func2` are tokenized as functions in this case.
+
+### Fixing this
+
+`src/tree-sitter-language-mode.js` is where the sintax tree is walked to generate tokens. It basically have methods like `seek`, `_moveDown`, etc that `.push` some token into `containingNodeTypes` and other local fields. Later, these are tokenized via `_currentScopeId` that basically tries to match the rule we're in inside `this.languageLayer.grammar.scopeMap` data structure.
+
+This data structure is defined in `src/syntax-scope-map.js`, and contains `anonymousScopeTable` (that is, AFAIK, a list of words that are tokenized always the same - think like "keywords" on the language) and a `namedScopeTable` (which, surprisingly, does not treat the "field name" even though it has `name` on it). This structure is basically a "leaf first" structure. So, tokenizing `obj.func("a string")`, we would get:
+
+1. `method_invocation`, that gets `push`ed into `containingNodeTypes` then we "move down"
+1. `identifier` (for `obj`), that also gets `push`ed into `identifier`
+1. We check the tokenID then "move right"
+1. `identifier` (for `func`), replaces the sibling's `identifier` that was pushed before, and we check tokenID, and "move right" again
+1. Repeate the process from the beginning, but for `argument_list` instead of `method_invocation` (replace the sibling's `identifier` with `argument_list`, then move down to push `string_literal`)
+1. Finally "move up", `pop`ing the `string_literal`, then `argument_list`, and finally `method_invocation`, and continue walking the rest of the AST
+
+To get the Token ID, we walk though the data structure, checking things as we go. So for example, in this case, after `push`ing things for `obj`, we have inside `containingNodeTypes`: `['method_invocation', 'identifier']`. We have this same structure for `func`.
+
+If we look at the `scopeMap` structure, inside `namedScopeTable`, we'll see something like:
+
+```js
+identifier: {
+  parents: {
+    method_invocation: {
+      result: ["entity.name.function", ...]
+    }
+  }
+}
+```
+
+And this is how the tokenizer is done. Is also how the bug appears: both `func` and `obj` have the same `containingNodeTypes`.
+
+
+### Possible solution
+
+To make `src/syntax-scope-map.js` aware of "named fields" (we can do that by checking the `cursor.currentFieldName` or by `push`ing the `this.treeCursor.currentFieldName`), then match things correctly.
+
+We will also need to decide on a syntax on the CSON file to this format, and also parse this format inside the `namedScopeTable`.
+
+Finally, we'll need to change the `get` method of the SyntaxScopeMap` to match things correctly and get tokenization for things filtered by the field name.
diff --git a/package.json b/package.json
@@ -7,7 +7,7 @@
     "id": "pulsar",
     "name": "Pulsar",
     "urlWeb": "https://atom.io/",
-    "urlGH": "https://github.com/pulsar-edit",
+    "urlGH": "https://github.com/pulsar-edit"
   },
   "main": "./src/main-process/main.js",
   "repository": {