Skip to content

Request for jq and json2csv #131

Open
@hhhuff

Description

I would love to see if someone could add jq and json2csv as I have a repeated workflow that include these tools. Thank you.

Activity

fprimex

fprimex commented on Jun 3, 2022

@fprimex

I have jq working in the simulator and am trying to finish up getting it added. I'm not familiar with json2csv. Do you have a link to it?

hhhuff

hhhuff commented on Jun 4, 2022

@hhhuff
Author
fprimex

fprimex commented on Jun 4, 2022

@fprimex

I had a quick look and the difficulty here is that the program is written in Go. This language toolset is not yet working with ios_system. I compiled it and it produces neither output nor files.

GOOS=js GOARCH=wasm go build -o json2csv.wasm

Go can natively produce both wasm and ios/arm64 formats, however the wasm SDK is not the one needed for to produce compatibly ios_system output.

The ios/arm64 output would work, but Go does not support outputting dynamic, shared libraries on ios/arm64. I tried combinations of -buildmode=shared, buildmode=c-shared, and -linkshared but none were accepted.

Honestly, your best bet for getting json2csv is to port it to python or jq. jq has an @csv output target. I would like to see Go work with ios_system at some point, but I feel like that is probably a long ways off.

fprimex

fprimex commented on Jun 4, 2022

@fprimex

I would be interested in decompiling the resulting Go wasm, replacing the needed SDK calls, and then reassembling it. That would enable Go programs in ios_system. I'll try that later.

fprimex

fprimex commented on Jun 4, 2022

@fprimex

Used wabt to convert the wasm to c. It was over 32MB and 1M lines long 😅. It must have the entire Go stdlib in it. I tried compiling it, but killed it when it had consumed 18GB of RAM on my system.

holzschu

holzschu commented on Jun 5, 2022

@holzschu
Owner

I've been thinking. WASM should be portable (it is designed for that). The only potentially non-portable parts are when it tries to access the file system, which is usually encapsulated into JavaScript commands. What kind of errors do you get with json2csv.wasm?

fprimex

fprimex commented on Jun 5, 2022

@fprimex

IIRC it just doesn't work. No output, no errors, no files.

holzschu

holzschu commented on Jun 5, 2022

@holzschu
Owner

That doesn't make a lot of clues, I'll grant you that. But normally, a webAssembly command that does not execute produces an error, so "no errors" could be a positive sign. Since webAssembly does not have access to the file system, I would try with stdin and stdout (json2csv < input.json > output.csv).

fprimex

fprimex commented on Jun 5, 2022

@fprimex

This similarly produces no output.

fprimex

fprimex commented on Jun 6, 2022

@fprimex

Go wasm does not provide wasi and therefore cannot be used in the context ios_system / a-shell wants. The tinygo provider, however, does and compiling with it works fine. Unfortunately, tinygo does not support everything that Go does, and the resulting wasm "panics" with a reflect.NumMethod not implemented error. This is apparently common for tinygo programs using encoding/json.

I'm looking at one other option and then I think I'm out of ideas. The good news is that using jq to do this would be relatively simple.

holzschu

holzschu commented on Jun 6, 2022

@holzschu
Owner

I haven't looked at the Go documentation, but if their WebAssembly version comes with a JavaScript container, maybe we can run this JS container rather than ours? jsc lets you run JavaScript commands, and the code a-Shell uses to run WebAssembly is in wasm.js. If it works, we will have to design a way to switch between JS containers depending on the WebAssembly flavour.

fprimex

fprimex commented on Jun 8, 2022

@fprimex

Here is the working wasm file for json2csv to place in Documents/bin. GH made me zip it so I could upload it 🙄 , so it'll need to be unzipped as well.

json2csv.wasm.zip

holzschu

holzschu commented on Jun 9, 2022

@holzschu
Owner

Thanks for compiling this command.
I've made it available on https://github.com/holzschu/a-Shell-commands/ but when I test, I get blank output every time. I might be using it wrong, though.

fprimex

fprimex commented on Jun 9, 2022

@fprimex

I think that version is old and broken. I have attached a tar.gz with the wasm and some tests that can be run that are from the project's acceptance tests and docs. See the tests.txt file for what to run and the expected output (there will be some newline representation differences because of how I copied the tests over most likely).

json2csv_wasm.tar.gz

6 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    fix committedA fix for this issue has been committed. Will soon be closed.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Request for jq and json2csv · Issue #131 · holzschu/ios_system