An incremental applicative-style JSON parser, suitable for high performance memory efficient stream parsing.

The parser is using Data.Aeson types and FromJSON instance, it can be easily combined with aeson monadic parsing instances when appropriate.

>>> parseByteString value "[1,2,3]" :: [[Int]]
[[1,2,3]]

The value parser matches any FromJSON value. The above command is essentially identical to the aeson decode function; the parsing process can generate more objects, therefore the results is [a].

Example of json-stream style parsing:

>>> parseByteString (arrayOf integer) "[1,2,3]" :: [Int]
[1,2,3]

Parsers can be combinated using <*> and <|> operators. The parsers are run in parallel and return combinations of the parsed values.

>>> let text = "[{\"name\": \"John\", \"age\": 20}, {\"age\": 30, \"name\": \"Frank\"} ]"
>>> let parser = arrayOf $ (,) <$> "name" .: string <*> "age"  .: integer
>>> parseByteString  parser text :: [(T.Text,Int)]
[("John",20),("Frank",30)]

When parsing larger values, it is advisable to use lazy ByteStrings. The parsing is then more memory efficient as less lexical state is needed to be held in memory for parallel parsers.

More examples are available on https://github.com/ondrap/json-stream.

The parser tries to do the least amount of work to get the job done, skipping over items that are not required. General guidelines to get best performance:

Do not use the value parser for the whole object if the object is big. Do not use json-stream applicative parsing for creating objects if they have lots of records, unless you are skipping large part of the structure. Every <*> causes parallel parsing, too many parallel parsers kill performance.

arrayOf value :: Parser MyStructure -- MyStructure with FromJSON instance

will probably behave better than

arrayOf $ MyStructure <$> "field1" .: string <*> "field2" .: integer <*> .... <*> "field20" .: string

and also better (at least memory-wise) than

value :: Parser [MyStructure]

unless the structure has hundreths of fields and you are parsing only a substructure.

The integer parser was optimized in such a way that the integer numbers skip the conversion to Scientific, resulting in a slightly faster speed.

It is possible to use the *> operator to filter objects based on a condition, e.g.:

arrayOf $ id <$> "error" .: number
              *> "name" .: string

This will return all objects that contain attribute error with number content. The parser will skip trying to decode the name attribute if error is not found.

Constant space decoding is possible if the grammar does not specify non-constant operations. The non-constant operations are value, string, many and in some instances <*>.

The value parser works by creating an aeson AST and passing it to the parseJSON method. The AST can consume a lot of memory before it is rejected in parseJSON. To achieve constant space the parsers safeString, number, integer, real and bool must be used; these parsers reject and do not parse data if it does not match the type.

The object key length is limited to ~64K. Object records with longer key are ignored and unparsed.

Numbers are limited to 200.000 digits. Longer numbers will make the parsing fail.

The many parser works by accumulating all matched values. Obviously, number of such values influences the amount of used memory.

The <*> operator runs both parsers in parallel and when they are both done, it produces combinations of the received values. It is constant-space as long as the number of element produced by child parsers is limited by a constant. This can be achieved by using .! and .: functions combined with constant space parsers or limiting the number of returned elements with takeI.

If the source object contains an object with multiple keys with a same name, json-stream matches the key multiple times. The only exception is objectWithKey (.: and .:?) that return at most one value for a given key.

The parser uses internally Data.Aeson types, so that the FromJSON instances are directly usable with the value parser. It may be more convenient to parse the outer structure with json-stream and the inner objects with aeson as long as constant-space decoding is not required.

Json-stream defines the object-access operators .:, .:? but in a slightly different albeit more natural way. New operators are .! for array access and .| to handle missing values.

>>> let test = "[{\"name\": \"test1\", \"value\": 1}, {\"name\": \"test2\", \"value\": null}, {\"name\": \"test3\"}]"
>>> let person = (,) <$> "name" .: string <*> "value" .: integer .| (-1)
>>> let people = arrayOf person
>>> parseByteString people test :: [(T.Text, Int)]
[("test1",1),("test2",-1),("test3",-1)]