related: #11312

Problem

There's no builtin abstraction for:

• Representing a task.
  • Example: vim.system() returns its own ad-hoc "task" that can be "awaited" via :wait().
• Orchestrating "pipelines" (quasi monads?) of work ("tasks") and handling errors.
  • Example: shell-like task chains: vim.fs.files():filter(..):map(function(f) vim.fs.delete(f) end)
  • See also vim.iter.

Expected behavior

• "Task" abstraction:
  • Maximally leveraging Lua coroutines + libuv. Only add concepts ("task", "promise") if absolutely needed.
  • Coroutines (or tasks that wrap coroutines) can be nested. (ref)
  • Util to create an awaitable task from "normal" functions (cf. "promisify"?).
    • Don't want to call await everywhere, that is a horrible consequence of the JS async/await model. Instead consider Go's go for opt-in concurrency.
    • Can e.g. vim.system() be "promisified" without its knowledge? Or could it handle differently when it detects that it's in a coroutine? So vim.system() would be synchronous normally, but vim.async(vim.system, ...):wait() would be asynchronous.
      • See cb_to_co() from this article.
  • Document (or generalize) "coroutine to callback".
• Structured concurrency:
  • await_all, await_any (pseudo-names). See JS Promise.all().
  • Tasks can be canceled.
  • Results (and failures) can be aggregated. (Can't do this with jobwait()!)
  • Failures/errors can be handled (possibly canceling the rest of the task tree).

Implementations

• https://github.com/lewis6991/async.nvim
• https://github.com/nvim-neotest/nvim-nio "enables async I/O through higher level abstractions and utilities around coroutines"
• https://github.com/kevinhwang91/promise-async
• https://github.com/ms-jpq/lua-async-await

Related

• https://github.com/sourcegraph/conc
• https://github.com/bitfield/script (example of "pipeline" interface)
• https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/

notes from reddit discussion

Briefly speaking, without cancellability, a structured concurrency API can and should be based on fire-and-forget coroutine functions (fafcf). Anything else would likely either be reinventing the wheel or running into a non-composable mess that is currently Plenary’s async.

With pure fafcfs, you can have most of your requirements: ability to chain, start concurrent computations, wait for completion of arbitrary “fafcfs.” It’d be quite a expressive and elegant system.

As you notice, what’s missing is cancellability. That would require creating a special protocol that fafcfs need to conform to.

After briefly looking at async.nvim, it looks like it’s trying to build concurrency by stepping through chunks, which is kind of what Plenary’s async is trying to do.

Ideally, I’d like to try to achieve cancellability without resorting to reimplementing an event loop and a scheduler in Lua. Perhaps fafcfs with a special protocol for indicating that it shouldn’t proceed would be enough.

I believe that Neovim/Lua can take design hints from Python. In this regard, Lua fafcfs are Python coroutines. They can be nested freely. If you want to a cancellable computation, you need to use Tasks, which builds upon coroutines to provide a richer interface.