This PR started with great intentions but kinda languished in the end. I've ported most of the argument handling especially you Windows to Lua, and hoped that it would reduce the complexity of the C code and make it easier to debug. Midway through I was distracted by the arena allocator and focused some effort into it.

The boring stuff: moving argument parsing into Lua

The Windows code is a mess of spaghetti with encoding conversion at various places, and nightmares with memory allocation (especially when dealing with the luaL_check* functions, where it could jump out of the function, skipping memory management).

I moved the majority of the code into Lua, making process.start() (at least the unwrapped, C version) accept only what they need: A command line string on Windows, and the environment variables concatenated into a single string. This allows the arguments to be passed directly to CreateProcessW with a bit of string manipulation on the Unix side.

The more interesting stuff: arena_allocator

This popped up mid-conversation with @Guldoman, but it is possible to use Lua userdata pushed to a table to control their lifetime. PUC Lua itself allows this (not sure about LuaJIT); even when this is not allowed, you can simply add all allocation to a Lua table, assign a finalizer for the said table, and free memory in the finalizer. There is no finer-grain control over this yet (at least I am not aware of, looking at to-be-closed variables here), but this GREATLY simplifies memory handling inside a C Lua API function. Return anywhere, and your memory is freed. It is semi-similar to golang arenas.

This arena allocator is implemented as a Lua table, with a userdata (the lxl_arena) in it. When lxl_arena_malloc is called, we get the Lua state, allocate some userdata and store it in the table. This userdata is now "pinned" to the table and freed when the table gets freed. You can optionally free memory in an arena with lxl_arena_free, but its absolutely optional and might be useful for implementing memory checking. lxl_arena_init pushes the table onto a stack, and this means that unless you accidentally pop this item off the stack, it will be valid for the duration of the C function. You can also pin this table to the Lua Registry for some long-running processes.

The limit of this system is that there's a fixed limit to the userdata size, ((size_t)(~(size_t)0)-2) (maximum value of size_t - 2), but that's honestly a pretty respectable limit.