I think it might be a good idea to step back for a second, clarify the current status quo and check what we can improve more structurally

Right now we have the following setup:

• parsing normal source files was lex the source string into tokens, then parse that into our rowan AST.
• parsing macro expansions, we expand the macro which yields a TokenTree, then that is being parsed into the AST via syntax bridge (which basically attempts to retokenize)
• macro expansion, we turn the AST token tree of its input(s) into a TokenTree via syntax bridge
  • what we do differs a bit dependong on the kind of macro
    • for MBE macros we then do the matching and transcribing against this TokenTree
    • for proc-macros we serialize this format into json, throw that at the proc-macro server, deserialize it again and transform that to the proc-macro types on demand (as required by the bridge)
    • for builtin macros it depends, most stuff just works on the TokenTree stuff directly as most builtins are simple, except for derives which re-parse the TokenTrees into the AST
      • noticable eager macros parse the TokenTree back fully, then descends the tree to find any new macro call, expands that, transforms that back into the AST and patches that into its AST until the AST is fully expanded turning it back into a TokenTree
  • that leaves us with a final TokenTree

That means we currently have two types of syntax trees, our AST (or well, CST) which is what the lowering and the IDE works with and the TokenTree model which is what macro expansion work with.

We should ideally lay out the requirements here to see if we actually need the TokenTree (my gut tells me that I think we do, just that its shape is not ideal). I haven't given this much thought otherwise right now, and likely won't the coming week but to clarify I'd like to have a proper plan laid out with reasoning behind it before we do any major change here then as this is touching on a fundamental system in rust-analyzer.

Originally posted by @Veykril in #18764