I really want to keep the streaming as we want to process the objects as the come in. See below how each token in the stream is handled as it come in with the handler.process() method. This change here would first consume all the stream and only then tokenize it. It should be done while the stream comes in as it can take quite some time e.g. for an S2I build to be finished, and the stream contains progress messages, too.

In looking at it, I understand and see what you mean, I'll see what if anything can be done with vaadin.

@rhuss Do you have any test cases around this that I can look at? The current set passed so I felt like I was good.

I'll say I've looked at both json-lib and json-simple and neither of them seem to have the feature to read the stream for just a single JSON block like the existing library so I wonder if you'd have to roll your own?

That said, I might have missed it so if you see one that does it in a manner similar to the current one

References:

• https://github.com/fangyidong/json-simple/blob/master/src/main/java/org/json/simple/parser/JSONParser.java
• http://json-lib.sourceforge.net/apidocs/jdk15/net/sf/json/util/JSONTokener.html

Unfortunately, I don't have a test case for checking that a response is parsed in a streamed fashion. Tbh, I consider that this kind of testing should be done by the json library itself, as I'm only using it here.

You can easily see, whether the streaming processing is done properly when pulling an image from a registry with the plugin. The progress bar for the individual layers is served directly from the Docker daemon.

If it wouldn't be handled as a stream you just would see nothing for a long time and then the whole process flow all at once within a second. Haven't tried that but that should be the case with your PR.

I'm afraid that you are right, that json-simple doesn't have a streaming parser, too. I don't know json-lib, so can't comment on this lib.

Actually rolling one's own implementation shouldn't be that super hard:

• Read the first chunk of the stream
• Check for a "{" or a "[". Set a counter to 1.
• Continue to read in chunks, increase the counter for each "{" (or "[" if this was the first char). Decrease the counter for each "}" (or "]")
• If the counter is 0, take the bytes so far and create a JSON object/array for that, feed it into the handler.
• Take the rest of the current chunk, strip whitespace from the beginning, look for a "{" or "[" again and start over.

I think its safe to assume that no literally JSON types (string, number, boolean) are included in isolation in the stream.

I believe that we can use https://github.com/fangyidong/json-simple/blob/a8b94b79b697df64b23428c27d65d6a187e9ebf9/src/main/java/org/json/simple/parser/JSONParser.java#L91 here.

This method will consume the next JSON token from the stream, and that is what we need here. We would then feed the object obtained from this method to the handler here and then read the next object from the stream until the stream is done.

Wdyt ?

I know, switching to json-simple would be a quite intrusive change, but as mentioned I made excellent experiences with it (I'm using it since 2008 in Jolokia without any issue. Also it's super small.)

@rdifrango good point haven't verified this but will do so. I thought it would really parse until the end of an object/array, not until the end of the stream. I'm pretty sure that the Docker daemon doesn't send EOF markers within the stream after each object/array.

For me, from https://github.com/fangyidong/json-simple/blob/a8b94b79b697df64b23428c27d65d6a187e9ebf9/src/main/java/org/json/simple/parser/JSONParser.java#L270-L271 it looks like for me that TYPE_EOF is more like an internal marker, not directly related to the stream itself. But have to check this ..

Nope, doesn't seem to work ;-(

public class JsonStreamParsingTest {

    public static void main(String[] args) throws IOException, ParseException {
        Reader testReader = new StringReader("{ \"name\": \"first\", \"value\": 10 }\n\n" +
                                             "{ \"name\": \"second\", \"value\": 20 }");
        JSONParser parser = new JSONParser();
        int i = 0;
        while (testReader.ready()) {
            System.out.println(i++ + ": " + parser.parse(testReader));
        }
    }
}

leads to

Exception in thread "main" Unexpected token LEFT BRACE({) at position 34.
	at org.json.simple.parser.JSONParser.parse(JSONParser.java:146)
	at org.json.simple.parser.JSONParser.parse(JSONParser.java:92)
	at org.jolokia.util.JsonStreamParsingTest.main(JsonStreamParsingTest.java:22)

which means, that JSONParser also tries to slurp in everything to parse (and doesn't stop when it picked up the first object)

Wdyt about switching to gson ?

I tried this and it seems to work. gson might be a bit of an overkill as it's a full JSON mapping framework, but we might look into this if we don't find a better streaming parsing alternative

public class JsonStreamParsingTest {

    public static void main(String[] args) throws IOException {
        Reader testReader = new StringReader("{ \"name\": \"first\", \"value\": 10 }\n" +
                                             "{ \"name\": \"second\", \"value\": 20 }");
        JsonParser parser = new JsonParser();

        int i = 0;
        JsonReader json = new JsonReader(testReader);
        json.setLenient(true);
        while (json.peek() != JsonToken.END_DOCUMENT) {
            JsonElement element = parser.parse(json);
            System.out.println(i++ + ": " + element);
        }
    }
}

yields

0: {"name":"first","value":10}
1: {"name":"second","value":20}

@rhuss Agreed, it might be a little bit of overkill, but I'll take a second stab switching this out to GSON as I think it more closely aligns to your goals on the streaming side.

@rhuss just as a note, I choose optString as it doesn't toss a checked exception, instead it returns a sensible/ aka default empty, value.