Skip to content
/ jkparse Public

JSON parser for shell scripts that utilizes the (associative) array capabilities of bash, ksh, zsh, and similar shells. Uses the json-c library.

License

Notifications You must be signed in to change notification settings

jacre8/jkparse

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

jkparse

A JSON parser for shell scripts that utilizes the (associative) array capabilities of bash, ksh, zsh, and similar shells. Programmed in C. Uses the json-c library.

The companion shell script library, libjkparse.sh, contains functions to help read, modify, and output JSON using this tool.

Why another JSON parser?

An associative array seems like the most natural approach for transversing a JSON object in a shell script. Relative to alternatives, this implementation offers strict bash and zsh compatibility, a simple interface, strong type detection, and permits arbitrary characters in keys.

Bare POSIX/ash shells are not targeted by this implementation; there are alternatives that already work well within that constraint. This utilizes gcc and glibc extensions which are not universally recognized.

Exampe Use

$ jkparse --help
Typical usage: . <(jkparse [OPTIONS...] [JSON])
  Parse JSON and return shell code for variable initialization based on the
JSON contents.  This will read the JSON to parse either from the first non-
option argument or, if one is not present, from stdin.  The returned shell code
can be processed by bash v4+, ksh93, or zsh v5.5+.  Two variable declarations
are output:
  JSON_TYPE - this is a single character describing the detected type of the
JSON argument.  This character is the first character for one of the following
types: null, boolean, int, double, string, array, or object.  The type will be
null if JSON cannot be represented by one of the other types.
  JSON_OBJ - this is the parsed result.  It is formatted based on JSON_TYPE as
one of the following:
 null - empty string
 boolean - string containing either 'true' or 'false'
 int, double - decimal string value
 string - string value without quotes or JSON escapes
 array - array containing a string representation of each member
 object - associative array containing a string representation of each member
  Output values whose type is neither string nor null can always be fed back
through this program, without modification, for further processing.  String and
null typed output values can also be fed back through, without modification, if
the --quote-strings option is specified when the output is generated.
  There is no special handling for duplicated keys in objects.  When there are
duplicate keys, multiple assignments will be output in the order that the keys
appear in the original JSON.
  This does not stream process the input when reading from stdin; if ever input
stream processing were implemented, this may output the variable declarations
twice.

OPTIONS:
 -a, --array-var=JSON_OBJ_TYPES
    When JSON_OBJ is either an array or object type, declare a third variable,
  named JSON_OBJ_TYPES, that contains an array or associative array containing
  characters corresponding to the types of the array or object's members,
  respectively.  The characters in this array are the same characters used in
  JSON_TYPE.  When JSON_OBJ_TYPES is an empty string, which is the default,
  this variable declaration is omitted from the output
 -e, --empty-key=EMPTY_KEY
    Empty keys are valid in JSON but not in shell script arrays.  Specify a
  string to replace empty keys with.  The default is $'\1'.  This value must
  be suitable for shell use.  No verification or substitution in the output is
  made for a non-empty value that is specified here.  If EMPTY_KEY is an empty
  string, object values with empty keys will be excluded from the output
 -l, --local-declarations
    Declare variables using the local keyword rather than the default, typeset
 -o, --obj-var=JSON_OBJ
    Specify a variable name for JSON_OBJ other than the default, JSON_OBJ.
  If blank, the object and array variables will be omitted from the output
 -q, --quote-strings
    Include quotations around output string values, and escape as necessary to
  generate valid JSON, so that they can be fed back through this program with
  corresponding type detection.  For the sake of subsequent encoding, the type
  indicator for strings will be 'q' with this option instead of 's'.  With this
  option, null values will also be explictily output as null, rather than as
  empty strings
 -s, --stringify
    Take the input and output it escaped as a JSON string, without surrounding
  quotes, whitespace, or shell escapes.  This is a formatting-only function
  that is intended for use in constructing JSON text.  The only other option
  that this may be logically combined with is -q, which only adds surrounding
  quotes in the output when combined
 -t, --type-var=JSON_TYPE
    Specify a variable name for JSON_TYPE other than the default, JSON_TYPE.
  If blank, the type variable will be omitted from the output
 -u, --unset-vars
    Output commands to unset JSON_OBJ and, if defined, JSON_OBJ_TYPES, before
  outputting their new declarations.  This permits re-using the same variable
  names, and using JSON_OBJ for both input and output, while transversing an
  object
 -v, --short-version
    Output just the version number and exit
 -V, --verbose
    If there is a parse error, output a descriptive message to stderr
 --help
    This help screen
 --version
    Output version, copyright, and build options, then exit
  Any non-empty variable name specified via an option will appear verbatim in
the output without additional verification.  Additional options for variable
declaration may be specified in the -a, -o, and -t option arguments.  E.g.,
-o '-g JSON_OBJ' will promote the scope of the object's declaration in BASH.

$ . <(jkparse -a OBJ_TYPES '{"number":5, "need":"input", "end":null}')
$ echo $JSON_TYPE
o
$ echo ${JSON_OBJ[number]}
5
$ echo ${OBJ_TYPES[need]}
s
$ echo ${JSON_OBJ[need]}
input
$ echo ${OBJ_TYPES[end]}
n

$ printf '["string", "with\na newline", 5]' > /tmp/jarray
$ . <(jkparse -q < /tmp/jarray)
$ # This would also work: . <(jkparse -q "$(< /tmp/jarray)")
$ echo $JSON_TYPE
a
$ echo "${JSON_OBJ[1]}" # assuming ksh indexing...
"with
a newline"

$ echo 42 | jkparse -o '$OBJ' -t '$TYPE'
typeset $TYPE=i;typeset $OBJ=42

$ jkparse -l {23:}
local JSON_TYPE=n;local JSON_OBJ=

$ eval "$(jkparse -q '{"inventory":[{"container":{"type":"can",
"label":"beans","weight":12.7}},{"container":{"type":"bag",
"label":null,"weight":2.3}}],"leftHand":"flashlight"}')"
$ . <(jkparse -qu "${JSON_OBJ[inventory]}")
$ . <(jkparse -qu "${JSON_OBJ[1]}") // Assuming ksh indexing...
$ . <(jkparse -ua OBJ_TYPES "${JSON_OBJ[container]}")
$ echo ${JSON_TYPE}
o
$ echo ${OBJ_TYPES[weight]}
d
$ echo ${JSON_OBJ[type]}
bag

Building and Installing

Second to the json-c library, the biggest dependency is for a printf command that supports the %q format option. Either a standalone printf binary or a shell supporting this format option must be available. The coreutils version of printf works. Without an override, the make target will search the build host for a viable printf function. If that is okay, the following may be used for building:

make

For specifying a different location for a printf binary:

make PRINTF_EXECUTABLE=/bin/printf

If using a shell's built-in printf function:

make USE_SHELL_PRINTF=/bin/ksh

Once compiled, install the output executable in a suitable location as root. The default install target places it in /usr/local/bin:

sudo make install

Alternatives

JSON parsers for shell use (some of these, like jq, generate JSON as well):
jsoncvt can also directly output a ksh associative array. It has less dependencies, human readable output, and supports generating multidimensional arrays.
jq, a de-facto standard. Powerful, with a bit of a learning curve.
jshon
JSON.sh
json.sh
TickTick
libshell has a lot of capabilities, including JSON parsing.
posix-awk-shell-jq
json-to-sh
jwalk
jsawk

JSON generators for shell use:
jsoncat lightweight, formats JSON in a human readable format with optional color, and supports special characters in keys.
jo
json_pp

About

JSON parser for shell scripts that utilizes the (associative) array capabilities of bash, ksh, zsh, and similar shells. Uses the json-c library.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published