Looking into this a bit more it seems like you can reproduce this more or less reliably if you try to do other stuff while the server is starting up. My suspicion is that something is off in ghcide wtr to how it handles cancellations and restarts and somewhere along the way the codelens request gets lost but I have not yet verified that.

So here is a fairly useful log https://gist.github.com/cocreature/04734faeef09af452c7504c62eea0314

What you can see is the following:

1. A codelens requests gets sent my vscode [Trace - 12:44:11 PM] Sending request 'textDocument/codeLens - (2)'.
2. This request never seems to make it to the place where we log that request in the language server.
3. Instead you can see that we process a hover request Hover request at position 16:21 in file: /home/moritz/quickstart/daml/Main.daml
4. At some point vscode decides that it has had enough and cancels the codelens request. I’m not entirely sure what triggers that.

[Trace - 12:44:14 PM] Sending notification '$/cancelRequest'. Params: {    "id": 2 }

5. VSCode quickly sends another codelens request

[Trace - 12:44:14 PM] Sending request 'textDocument/codeLens - (6)'.

6. Once again this doesn’t make it to our language server. Instead we get another hover request

2021-10-21 10:44:17.43 [INFO]  [LanguageServer]  [] 
Hover request at position 17:20 in file: /home/moritz/quickstart/daml/Main.daml

7. There is a long break until vscode cancels the codelens request

[Trace - 12:44:50 PM] Sending notification '$/cancelRequest'.
Params: {
    "id": 6
}

8. And VSCode sends another one [Trace - 12:44:50 PM] Sending request 'textDocument/codeLens - (8)'.
9. This one actually makes it to our language server `[Trace - 12:44:50 PM] Sending request 'textDocument/codeLens - (8)'.
10. But apparently we fail to send a response because another request comes in. A bit later we get cancelled again

[Trace - 12:45:04 PM] Sending notification '$/cancelRequest'.
Params: {
    "id": 8
}

11. This repeats a few more times until eventually we get no other request in between and things work for once.

I finally understand the issue. What is happening is the following:

1. We receive a codelens request. This results in us ending up in https://github.com/digital-asset/daml-ghcide/blob/e28a78b9a450fa7970320e78175b8dd382ce0fbd/src/Development/IDE/LSP/LanguageServer.hs#L143 spawning a new thread to process the request.
2. The request gets up to

   daml/compiler/damlc/daml-ide/src/DA/Daml/LanguageServer/CodeLens.hs

   Line 36 in 355352f

   (mbModMapping, DamlEnv{..}) <- runAction ide $

   , i.e., we start the runAction.
3. Then we get another request, e.g. a hover (that’s why you can reproduce that by hovering like crazy).
4. Hover also ends up calling runAction. The way runAction is implemented this kills the existing request with AsyncCancelled.
5. Now you might be looking at the code at https://github.com/digital-asset/daml-ghcide/blob/e28a78b9a450fa7970320e78175b8dd382ce0fbd/src/Development/IDE/LSP/LanguageServer.hs#L164 and thinking “huh surely we catch the exception and print a sensible log but there is no log. What is going on?“ Great question! The answer is that catch here is from Control.Exception.Safe which does not catch asynchronous exceptions. AsyncCancelled is an asynchronous exception so we don’t catch it and the thread silently dies. Changing it to Control.exception.catch shows the error as expected.

As for fixing the issue, just dropping the async doesn’t work because that would mean cancel requests would have no effect.
I think we need to have special processing for cancel requests and treat other requests synchronously.