在Shell脚本中解析、验证和打印JSON

1. Introduction

The JavaScript Object Notation (JSON) format standard has been around since 2013. One of the reasons for its adoption is the ease with which a human can process it visually. To make that even easier, there are a number of ways to prettify and structure JSON output.

In this tutorial, we’ll delve into many ways to parse, validate, and tidy up a JSON object for human consumption. First, we start with a discussion of the format itself. Next, we look at native shell implementations for handling JSON. After that, we check out several tools for the same purpose. Then, we’ll devote some time to the main language that the format was developed for. Finally, we explore other programming languages and how they can handle JSON objects.

We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.

2. JavaScript Object Notation (JSON)

JavaScript Object Notation (JSON) is a text format created to ease humans when reading and creating objects passed between applications while enabling the portable representation of data. It consists of two main elements:

collections of name-value pairs like “type”:”binary”
ordered lists of values like [“txt”, “gz”, “tar”]

For example, we can create a simple JSON to represent a directory tree:

{
  "path":"/home/user",
  "files": [
    ".bash_history",
    ".bashrc",
    ".profile"
  ],
  "directories": {
    ".ssh": {
      "files": [
        "id_rsa",
        "id_rsa.pub"
      ],
      "directories": {}
    }
  }
}

Notably, this prettified (or beautified) version should be easier to consume for humans. Yet, for machine processing, we can remove all non-quoted whitespace from a JSON to save on memory:

{"path":"/home/user","files":[".bash_history",".bashrc",".profile"],"directories":{".ssh":{"files":["id_rsa","id_rsa.pub"],"directories": {}}}}

Now, let’s decipher the example. In the primary object {}, which starts at the beginning of the JSON string, we have a path key with its value representing the root of the directory tree. Next, on the same level, files is an array of file names in the path. Finally, directories is an object that contains key-value pairs of subdirectory names and complex object values.

Obviously, this idea can transcend JavaScript, so JSON is applicable in many other programming languages.

Of course, one of those is the Linux shell.

3. Native Shell JSON

The Linux shell, be it sh, Ash, Bash, Dash, Zsh, or any other, is a programming language.

Consequently, there are JSON parsers implemented natively in this language, but also some that rely on standard POSIX tools usually called from the shell.

3.1. Simple grep and awk JSON Formatter

As a quick solution to beautify JSON objects, we can pipe their contents to a chain of commands:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' |
grep -Eo '"[^"]*" *(: *([0-9]*|"[^"]*")[^{}\["]*|,)?|[^"\]\[\}\{]*|\{|\},?|\[|\],?|[0-9 ]*,?' |
awk '{if ($0 ~ /^[}\]]/ ) offset-=4; printf "%*c%s\n", offset, " ", $0; if ($0 ~ /^[{\[]/) offset+=4}'
 {
    "field":"data",
    "array":
    [
        "i1",

        "i2"
    ],

    "object":
    {
        "subfield":"subdata"
    }
 }

In doing so, we employ a grep regular expression to match locations and amounts for indentation and apply formatting with awk.

Thus, we create a more visually appealing version of the one-liner JSON data without validity checks. Also, this solution depends on the two POSIX tools and is obviously not ideal due to the complex commands. Let’s continue with some other options.

3.2. JSON.sh

JSON.sh is a native shell project that claims compatibility with Ash, Bash, Dash, and Zsh, while arguably producing results that are easier to parse. To use JSON.sh, we can clone the Git repository or simply download it as an archive.

Next, we just pipe some JSON data to JSON.sh:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ./JSON.sh
["field"]       "data"
["array",0]     "i1"
["array",1]     "i2"
["array"]       ["i1","i2"]
["object","subfield"]   "subdata"
["object"]      {"subfield":"subdata"}
[]      {"field":"data","array":["i1","i2"],"object":{"subfield":"subdata"}}

Consequently, JSON.sh performs validation, and if the object is correct, we see a dissection of each element, line by line. In addition, there are options to [-p] run empty fields, show only [-l]eafs, and skip [-n]o-path elements.

3.3. jwalk

Another script with functionality similar to JSON.sh, but a much richer feature set, is jwalk. Again, we can either clone or directly download it as an archive.

At this point, we can invoke jwalk just like we did JSON.sh:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ./jwalk
object
field   string  data
array   array
array   0       string  i1
array   1       string  i2
object  object
object  subfield        string  subdata

Unlike the sh-native JSON.sh, jwalk uses awk and sed, which, despite being part of POSIX, are still extras to the standard sh.

However, these tools enable jwalk to not only validate and output JSON elements but also use complex [-p]atterns (–pattern) and filter by their paths and names:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ./jwalk --pattern 'ar*'
array   array
array   0       string  i1
array   1       string  i2
$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ./jwalk --pattern object.subfield
object  subfield        string  subdata

Here, we use a wildcard match for an element, as well as a specific path of object.subfield.

In any case, parsing the tab-delimited results from either of the above implementations should be easier in the shell.

4. Shell JSON With Tools

As with other tasks, we can invoke third-party tools from the shell to handle JSON. Let’s explore some popular choices for our purposes.

4.1. jq

The well-documented and open-source processor jq is a lightweight way to parse, validate, and generally deal with the JSON format.

It accepts both file and stdin as its input and always performs validations. Further, jq can prettify (default), [-c]ompact, [-C]olorize (default), and [-S]ort JSON objects:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | jq
{
  "field": "data",
  "array": [
    "i1",
    "i2"
  ],
  "object": {
    "subfield": "subdata"
  }
}
$ echo 'INVALID' | jq
parse error: Invalid numeric literal at line 2, column 0$

Also, similar to jwalk, jq can follow paths, starting from the . root:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | jq .object.subfield
"subdata"

Finally, jq supports more complex filters like array indexing, slicing, and iteration, as well as other functions.

4.2. jj

Similar to jq, but arguably not as famous, jj is a JSON stream editor. To install jj, we can download prebuilt binaries or build it manually.

It accepts file and stdin inputs and generally performs best-effort corruption avoidance. In fact, it’s not a good way to validate a given JSON.

Still, jj provides keypath support to select JSON elements based on complex criteria:

$ cat file.json
{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}
$ cat file.json | ./jj array
["i1", "i2"]
$ cat file.json | ./jj array.1
i2
$ cat file.json | ./jj object.subfield
subdata

Just like jq, jj uses colored syntax highlighting for the output by default and can [-p]rettify a JSON:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ./jj -p
{
  "field": "data",
  "array": ["i1", "i2"],
  "object": {
    "subfield": "subdata"
  }
}

However, validity checks aren’t strict for this tool. Finally, options to make the output [-u]gly (compact), change [-v]alues, [-D]elete values, and send data to an [-o]utput file are also available.

5. Shell JSON Through NodeJS

Naturally, the node (NodeJS) JavaScript engine and its Node Package Manager (npm) can provide most of the functionality we need.

Here, we explore two JSON processing and validation npm packages that have several functions in common:

receive input via stdin or a file
validate
prettify
replace tabular delimiters
replace a file with its pretty version in-place

Now, let’s go into examples of each individual package.

5.1. jsonlint (NodeJS)

The jsonlint npm package is based on the service at jsonlint.com and makes pretty prints and validations of JSON in the shell trivial.

Let’s install it for global use:

$ npm install -g jsonlint
$ jsonlint --help

Usage: jsonlint [file] [options]

file     file to parse; otherwise uses stdin
[...]

The extra functions jsonlint offers on top of the main ones we already discussed are key sorting and JSON schema selection.

We can prettify a JSON by simply passing it to the stdin of the jsonlint tool:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | jsonlint
{
  "field": "data",
  "array": [
    "i1",
    "i2"
  ],
  "object": {
    "subfield": "subdata"
  }
}

Finally, validation is also part of the process:

$ echo 'INVALID' | jsonlint
Error: Parse error on line 1:
INVALID
^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
    at Object.parseError (/usr/local/lib/node_modules/jsonlint/lib/jsonlint.js:55:11)
    at Object.parse (/usr/local/lib/node_modules/jsonlint/lib/jsonlint.js:132:22)
    at parse (/usr/local/lib/node_modules/jsonlint/lib/cli.js:82:14)
    at Socket. (/usr/local/lib/node_modules/jsonlint/lib/cli.js:149:41)
    at Socket.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

While jsonlint is helpful, it’s not very versatile, unlike the next package.

5.2. json (NodeJS)

A classic npm package for JSON processing is json.

Let’s install it globally:

$ npm install -g json
$ json --help
Usage:
   | json [OPTIONS] [LOOKUPS...]
  json -f FILE [OPTIONS] [LOOKUPS...]
[...]

Importantly, json has many more features than jsonlint:

select values via JSON paths
merge objects
itemize objects as {“key”: , “value”: }
output only object keys
output in different modes, allowing indent changes and special formats

Let’s see how json makes a JSON object pretty:

$ echo '{field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | json
{
  "field": "data",
  "array": [
    "i1",
    "i2"
  ],
  "object": {
    "subfield": "subdata"
  }
}

Of course, if we supply an invalid object, json will let us know:

$ echo 'INVALID' | json
json: error: input is not JSON: Unexpected 'I' at line 1, column 1:
        INVALID
        ^
INVALID

Still, json also supports streaming, filtering, and in-place file editing.

6. Shell JSON With Interpreters

As usual, interpreters provide the most flexible, albeit usually more complex, way of processing JSON.

After looking at node, let’s see some other options. Although we don’t look at too much raw code, as usual, one-liners are also an option.

6.1. Perl JSON

The main method to handle JSON with perl from the shell is via modules:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | json_pp
{
   "array" : [
      "i1",
      "i2"
   ],
   "field" : "data",
   "object" : {
      "subfield" : "subdata"
   }
}
$ echo 'INVALID' | json_pp
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "INVALID\n") at /usr/bin/json_pp line 59.

As the example shows, the json_pp tool that comes with the JSON::PP Perl core module can prettify and validate a JSON object.

Alternatively, JSON::XS is a cpan module with the same functionality and provides the json_xs tool.

6.2. Python JSON

The python interpreter comes with the json.tool module, usable via the -m switch directly in the shell:

$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | python -m json.tool
{
   "array" : [
      "i1",
      "i2"
   ],
   "field" : "data",
   "object" : {
      "subfield" : "subdata"
   }
}
$ echo 'INVALID' | python -m json.tool
No JSON object could be decoded

While it doesn’t provide comprehensive error output when validating, the Python json.tool can prettify input from stdin or a file (argument).

A pip module with similar functionality is pjson:

$ pip install pjson
[...]
$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | pjson
{
  "field": "data",
  "array": [
    "i1",
    "i2"
  ],
  "object": {
    "subfield": "subdata"
  }
}
$ echo 'INVALID' | pjson
Expecting value: line 1 column 1 (char 0)

Despite not being very versatile, pjson adds color to the output.

6.3. Ruby JSON

In ruby, working with JSON is usually done via the json gem:

$ gem install json
[...]
$ echo '{"field":"data", "array": ["i1", "i2"], "object":{"subfield":"subdata"}}' | ruby -r json -e 'jj JSON.parse gets'
{
  "field": "data",
  "array": [
    "i1",
    "i2"
  ],
  "object": {
    "subfield": "subdata"
  }
}
$ echo 'INVALID' | ruby -r json -e 'jj JSON.parse gets'
Traceback (most recent call last):
        2: from -e:1:in `<main>'
        1: from /var/lib/gems/2.7.0/gems/json-2.6.3/lib/json/common.rb:216:in `parse'
/var/lib/gems/2.7.0/gems/json-2.6.3/lib/json/common.rb:216:in `parse': unexpected token at 'INVALID (JSON::ParserError) '

Although with a syntax that’s a bit more complex, Ruby can still prettify and validate JSON objects via its json gem.

7. Summary

In this article, we explored many ways to handle JSON in the shell.

In conclusion, while there are native shell implementations, we can also leverage external tools and interpreters when validating, prettifying, and dealing with the JSON format.

Persistence

REST

Security