In brief, @base64d has a documented limitation:

Note: If the decoded string is not UTF-8, the results are undefined.

This could be restated as follows:

 Let $B be an arbitrary base64 string, then `$B | @base64d` is undefined
 if `base64 -D <<< $B` is not a valid UTF-8 string.  

Here base64 refers to the command-line program of that name, and I'm assuming that base64 -D acts as the inverse of base64.

Using the moreutils program isutf8 to test this condition, we see
that the given base64 string does NOT satisfy it:

$ S="V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU="
$ base64 -D <<< "$S" | xxd -p
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d
1955
$ base64 -D <<< "$S" | isutf8
(standard input): line 1, char 8, byte 8: Expecting bytes in the following ranges: 00..7F C2..F4.

Fair enough. I'd make the suggestion to document that the results of @base64d is incorrect instead of merely undefined when decoding non UTF-8 strings.

Having put this to rest, is there any plans to have a way to "shell out" to call external utilities to aid in processing this edge cases?

@mterron - I believe the author or authors of this particular section of the official manual did not want to commit to any particular behavior at the time of writing, no doubt because, as you say, there is much to be said for raising an error condition.

As for shelling out -- yes, there are plans to support this (see e.g. #147 and #1614), but it won't help much in the present case, for the reason already stated.

@pkoppstein give a way to shell out I'll make it work :)

Thanks!

@mterron wrote:

I'll make it work :)

It appears you are not quite grasping the fact that the implementation of the shell-out function will (of necessity) be designed to prevent what I understand you want to do.

The "j" in jq can be understood as a commitment that, with two exceptions, every jq filter should produce strictly valid JSON, the exceptions being the values for NaN and Infinite, but even these two values are ultimately converted to valid JSON on output (e.g. echo Nan | jq . #=> null).

This is not to say that every version of jq is guaranteed to reject non-JSON strings, but that's not for want of trying :-)

Oh I understand very well. xxd will convert the binary back to an hexadecimal string that is valid json

@mterron - Excellent. You might want to mention your use case on one of the shell-out tracking issues.

I think this is worth fixing and it does not break compatibility. It is useful if we can decode the image data from aws ec2 get-console-screenshot with jq only.

Congratulations to @itchyny on gojq's @base64d, which passes the test using https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html with flying colors. With B as a local copy of this file:

$ base64 < B > B.base64
$ diff <(gojq -Rrj '@base64d' B.base64) B
$

@pkoppstein does that mean that the Go version of jq is now beating the 'original' jq at its own game? I have made a very simple test, and gojq surprisingly seemed to be much faster at decoding 118K of base64-encoded raw binary data, compared to jq (which, of course, will produce garbage).

Now I'd really love to see a benchmark comparing the two :) [note: I'm well aware that gojq does not fully implement everything that jq does...]

Just tested it on macOS. gojq worked beautifully.

@GwynethLlewelyn - gojq is unquestionably better than jq in several important respects --
not least that it is being actively maintained -- but unfortunately
gojq has one major intrinsic drawback compared to jq, namely that in
many cases that jq can handle, it simply runs out of memory. My
understanding is that this is, in effect, by design -- that is, the
problem apparently cannot be addressed within the scope of gojq's
current design.

My experience has been that, in general, gojq tends to trade memory
for speed, at least compared to jq, and the following statistics
suggest that might be the case for @base64d too:

/usr/bin/time -lp gojq -Rrj '@base64d'  B.base64 > /dev/null
             3,436,544  maximum resident set size
             1,499,136  peak memory footprint

/usr/bin/time -lp jq -Rrj '@base64d'  B.base64 > /dev/null
             1,937,408  maximum resident set size
             1,204,224  peak memory footprint

There is also the matter of retaining the order of keys within
objects, which for some users and applications is unimportant, but
that is not always the case.

As a user of jq I'm sorely disappointed that a simple note on the limitation of this function isn't in the manpage with over 5 years of lead time.

I spent a good couple hours debugging why my base64 decoded strings were getting corrupted only to find that it's a known limitation of jq.

There is a note in the manpage about how interpolation behaves and nothing else.

@hachi - In the section on @base64d, the man page (https://jqlang.github.io/jq/manual/) says:

The inverse of @base64, input is decoded as specified by RFC 4648. Note: If the decoded string is not UTF-8, the results are undefined.