1. Overview
jq is a de facto standard utility for working with JSON data in Linux.
In this tutorial, we’ll learn how to make the jq program reusable by passing Bash variables to it.
2. Using the –arg Option
We can pass Bash variables to a jq program as a pre-defined named variable using the –arg option:
$ jq --arg jq_var ${bash_var} [options...] filter [files ...]
First, let’s add sample JSON data in the fruits_template.json file:
$ cat fruits_template.json
{
"fruit": "Apple",
"size": "Large",
"color": "Red"
}
We must note that we plan to reuse the fruits_template.json file throughout this tutorial.
Further, let’s assume that we want to override the value of the fruit field with a user-defined value that’s available through the bash_fruit_var Bash variable:
$ bash_fruit_var=Banana
Next, let’s go ahead and write a jq program using the –arg option to display the fruit field from the bash_fruit_var variable and all other field values from the fruits_template.json file using the .field operator:
$ jq \
--arg jq_fruit_var $bash_fruit_var \
'{"fruit": $jq_fruit_var, "color": .color, "size": .size}' \
fruits_template.json
{
"fruit": "Banana",
"color": "Red",
"size": "Large"
}
We must note that we’ve added the bash_ and jq_ prefixes to the variable names to indicate that the former is a Bash variable while the latter is a jq variable. Further, we shall remember that we need to reference a jq variable by adding the $ prefix.
Finally, let’s also see an alternate approach to accessing the jq_fruit_var through $ARGS.named:
$ jq \
--arg jq_fruit_var $bash_fruit_var \
'{"fruit": $ARGS.named.jq_fruit_var, "color": .color, "size": .size}' \
fruits_template.json
<same output as earlier>
Great! We got this right. Further, we must note $ARGS.named only holds the named argument variables, as applicable in this case.
3. Using $ENV or the env Function
We can also pass the Bash variables as environment variables to the jq program. Subsequently, we can reference them within the jq program using either the $ENV variable or the env function.
Let’s start by using the $ENV variable to access the bash_fruit_var from the environment:
$ bash_fruit_var=Banana \
jq '{"fruit": $ENV.bash_fruit_var, "color": .color, "size": .size}' fruits_template.json
{
"fruit": "Banana",
"color": "Red",
"size": "Large"
}
We must note that we could also export the bash_fruit_var to pass it as an environment variable to the jq program.
Next, let’s use the env function to access the bash_fruit_var from the environment:
$ bash_fruit_var=Banana \
jq '{"fruit": env.bash_fruit_var, "color": .color, "size": .size}' fruits_template.json
<same output as earlier>
Great! We’ve got the same result. Further, we must note that $ENV as an object represents the environment variables set when the jq program started, whereas env outputs an object representing jq‘s current environment.
4. Using the –args Option
Alternatively, we can also pass the Bash variables as positional arguments to the jq program using the –args option:
$ jq [options...] filter [files ...] --args ${bash_var1} ${bash_var2}
We must note that jq will treat all the arguments after –args as positional arguments, so we must place it after the filename.
Let’s go ahead and define two Bash variables, namely bash_fruit_var and bash_color_var:
$ bash_fruit_var=Grapes bash_color_var=Green
Moving on, let’s pass bash_fruit_var and bash_color_var variables to the jq program to override the fruit and color fields in the fruits_template.json file:
$ jq \
'{"fruit": $ARGS.positional[0], "color": $ARGS.positional[1], "size": .size}' \
fruits_template.json \
--args $bash_fruit_var $bash_color_var
{
"fruit": "Grapes",
"color": "Green",
"size": "Large"
}
We must note that $ARGS.positional only holds positional argument variables with zero-based indexing.
5. Using the –argjson Option
We can also pass JSON-encoded Bash variables to the jq program by using the –argjson option:
$ jq --argjson jq_json_var ${bash_jsonvar} [options...] filter [files ...]
Let’s define the bash_fruit_json Bash variable to hold a JSON object with two fields, namely fruit and color:
$ bash_fruit_json='{"fruit":"Orange", "color":"orange"}'
In continuation, let’s pass the Bash variable bash_fruit_json as the jq variable jq_fruit_json and use it to override the values of fruit and color fields in the fruits_template.json file:
$ jq --argjson jq_fruit_json "$bash_fruit_json" \
'{"fruit":$jq_fruit_json.fruit,"color":$jq_fruit_json.color,"size":.size}' \
fruits_template.json
{
"fruit": "Orange",
"color": "orange",
"size": "Large"
}
Finally, let’s rewrite our program to access the jq_fruit_json variable through $ARGS.named:
$ jq --argjson jq_fruit_json "$bash_fruit_json" \
'{"fruit":$ARGS.named.jq_fruit_json.fruit,"color":$ARGS.named.jq_fruit_json.color,"size":.size}' \
fruits_template.json
<same output as earlier>
Perfect! The result is as we expected. Further, we must note that enclosing the Bash variable within double quotes is mandatory when it contains JSON-encoded values.
6. Using the –jsonargs Option
We can use the –jsonargs option for passing JSON-encoded Bash variables as positional arguments:
$ jq [options...] filter [files ...] --jsonargs "${bash_json_var1}" "${bash_json_var2}"
Similar to the position of the –args option, we must remember to place the –jsonargs after the filename because jq will interpret all the values after –jsonargs as positional arguments.
Let’s go ahead and pass bash_fruit_json as a positional argument to the jq program:
$ jq \
'{"fruit":$ARGS.positional[0].fruit,"color":$ARGS.positional[0].color,"size":.size}' \
fruits_template.json \
--jsonargs "$bash_fruit_json"
{
"fruit": "Orange",
"color": "orange",
"size": "Large"
}
The output looks correct. Further, we must note that we used the $ARGS.positional object to access the JSON-encoded data passed to the program.
7. Using the –rawfile Option
We can use the –rawfile option to pass the contents of a file as a variable to the jq program:
$ jq --rawfile jq_file_contents "$bash_path_var" [options...] filter [files ...]
We must note that while $bash_path_var is a Bash variable representing a valid file path, jq_file_contents is a jq variable containing the contents from that file.
Let’s start by taking a look at the raw_fruits_data file containing the name and color information for multiple fruits:
$ cat raw_fruits_data
Banana|Yellow
Orange|Orange
Grapes|Green
Next, let’s define the bash_fruits_file Bash variable and initialize it to the raw_fruits_data file path:
bash_fruits_file=raw_fruits_data
We must note that storing the file path in a Bash variable allows us to reuse the same jq program in a Bash script.
Moving on, let’s write a jq program to transform the raw data available in the raw_fruits_data file into multiple JSON fruit objects by passing its content in the jq_raw variable:
$ jq --rawfile jq_raw "$bash_fruits_file" \
'. as $fruit_json | $jq_raw | split("\n") | .[] | select(. != "") | [{"fruit":.|split("|")[0], "color": .|split("|")[1], "size":$fruit_json.size}]' \
fruits_template.json
{
"fruit": "Banana",
"color": "Yellow",
"size": "Large"
}
{
"fruit": "Orange",
"color": "Orange",
"size": "Large"
}
{
"fruit": "Grapes",
"color": "Green",
"size": "Large"
}
We must note that we stored the content of the fruits_template.json file in the fruit_json variable and applied the split function over the jq_raw variable to get an array of strings. Further, we used the iterator operator (.[]) and the select function to filter out the empty values.
Generally speaking, using the –rawfile option, we can pass non-JSON data directly from a file to the jq program.
8. Using the –slurpfile Option
We can use the –slurpfile option to pass the contents of a file containing JSON-encoded data as an array to the jq program:
$ jq --slurpfile jq_json_arr "$bash_json_path_var" [options...] filter [files ...]
Let’s take a look at the my_fruit_jsons file containing multiple JSON objects:
$ cat my_fruit_jsons
{
"fruit": "Grape",
"color": "Green"
}
{
"fruit": "Banana",
"color": "Yellow"
}
{
"fruit": "Orange",
"color": "Orange"
}
We must note that the my_fruit_jsons file itself isn’t a valid JSON array object.
Next, let’s define the bash_fruits_json_file Bash variable and initialize it to the my_fruit_jsons file path:
bash_fruits_json_file=my_fruit_jsons
We must note that storing the file path in a Bash variable allows us to reuse the same jq program in a script.
Next, let’s see this in action by binding the JSON objects to the my_fruits array object in the jq program:
$ jq --slurpfile my_fruits "$bash_fruits_json_file" '$my_fruits' fruits_template.json
[
{
"fruit": "Grape",
"color": "Green"
},
{
"fruit": "Banana",
"color": "Yellow"
},
{
"fruit": "Orange",
"color": "Orange"
}
]
We can notice that the output is a valid JSON object now. Further, we must note that the size field is missing, as it’s not present in the individual JSON objects.
Moving on, let’s write a jq program to transform the JSON objects available in the my_fruit_jsons file into raw text separated by a | character:
$ jq --raw-output \
> --slurpfile my_fruits "$bash_fruits_json_file" \
> '. as $fruit_json | $my_fruits |
> reduce .[] as $my_fruit (""; . + $my_fruit.fruit + "|" + $my_fruit.color + "|" + $fruit_json.size + "\n")
> | split("\n") | .[] | select(. != "")' \
> fruits_template.json
Grape|Green|Large
Banana|Yellow|Large
Orange|Orange|Large
We must note the content of the fruits_template.json file is available in the $fruit_json variable. Moreover, we used the reduce function to generate pipe-separated values, each separated with the newline character (“\n”).
Later, we used the split function followed by the select function to filter out the empty line by iterating over individual string values. Additionally, we used the –raw-output option to show the output as raw text.
9. Using Variables Within Filter
Let’s start by defining the Bash variable bash_field_var and using it intuitively to get a specific field from the JSON object in fruits_template.json:
$ bash_field_var=fruit
$ jq --arg jq_field $bash_field_var '.$jq_field' fruits_template.json
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$jq_field
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$jq_field
jq: 2 compile errors
We can notice that we get a syntax error. However, the error also suggests the right way to access the field using the variable.
Next, let’s use [$jq_field] with the .field operator to access the fields represented by the bash_field_var Bash variable:
$ bash_field_var=fruit
$ jq --arg field $bash_field_var '.[$field]' fruits_template.json
"Apple"
$ bash_field_var=color
$ jq --arg field $bash_field_var '.[$field]' fruits_template.json
"Red"
Great! We got the correct result this time.
10. Conclusion
In this tutorial, we learned multiple ways to pass Bash variables to a jq program. Additionally, we explored different options with the jq command, such as –arg, –args, –argjson, –jsonargs, –rawfile, and –slurpfile, along with a few functions such as split and reduce.