JSON Data Transformation: jq, JS, and Python Recipes

TL;DR

• jq is the fastest way to transform JSON on the command line. Learn 10 patterns and you can handle 90% of data transformation tasks
• JavaScript's map, filter, reduce, and destructuring cover most in-app JSON reshaping needs
• Python's json module with dictionary comprehensions handles batch processing and data pipeline tasks
• Always format and validate your JSON before transformation. Transforming invalid JSON produces unpredictable results

Table of Contents

• jq: The Command-Line JSON Processor
• 10 Essential jq Recipes
• JavaScript Transformation Patterns
• Python JSON Processing
• Reshaping API Responses for Frontend
• Merging and Diffing JSON Documents
• Converting Between Formats
• Frequently Asked Questions

jq: The Command-Line JSON Processor

jq is a lightweight command-line JSON processor. It takes JSON input, applies transformations, and produces JSON (or text) output. Install it on any platform:

# macOS
brew install jq

# Ubuntu/Debian
sudo apt-get install jq

# Windows (via chocolatey)
choco install jq

Basic usage:

# Pretty-print minified JSON
echo '{"name":"dev","age":25}' | jq '.'

# Output:
# {
#   "name": "dev",
#   "age": 25
# }

The . filter is the identity filter. It returns the input unchanged but pretty-printed. This alone makes jq useful as a command-line JSON formatter.

10 Essential jq Recipes

1. Extract a single field

echo '{"user": {"name": "Alice", "age": 30}}' | jq '.user.name'
# "Alice"

2. Extract from an array

echo '[{"name": "Alice"}, {"name": "Bob"}]' | jq '.[].name'
# "Alice"
# "Bob"

3. Filter array items

echo '[{"name": "Alice", "age": 30}, {"name": "Bob", "age": 20}]' | jq '[.[] | select(.age > 25)]'
# [{"name": "Alice", "age": 30}]

4. Transform object shape

echo '{"first_name": "Alice", "last_name": "Smith", "email": "alice@test.com"}' | jq '{name: (.first_name + " " + .last_name), contact: .email}'
# {"name": "Alice Smith", "contact": "alice@test.com"}

5. Count array items

echo '[1, 2, 3, 4, 5]' | jq 'length'
# 5

6. Sort by field

echo '[{"name": "Charlie"}, {"name": "Alice"}, {"name": "Bob"}]' | jq 'sort_by(.name)'

7. Group by field

echo '[{"dept": "eng", "name": "Alice"}, {"dept": "sales", "name": "Bob"}, {"dept": "eng", "name": "Charlie"}]' | jq 'group_by(.dept)'

8. Unique values

echo '[{"status": "active"}, {"status": "inactive"}, {"status": "active"}]' | jq '[.[].status] | unique'
# ["active", "inactive"]

9. Add a computed field

echo '{"price": 100, "tax_rate": 0.2}' | jq '. + {total: (.price * (1 + .tax_rate))}'
# {"price": 100, "tax_rate": 0.2, "total": 120}

10. Convert to CSV

echo '[{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]' | jq -r '.[] | [.name, .age] | @csv'
# "Alice",30
# "Bob",25

Before running jq transformations, validate your source JSON with our JSON Formatter to make sure the structure matches your expectations.

JavaScript Transformation Patterns

Reshape an API response

// API returns:
const apiResponse = {
  data: {
    users: [
      { id: 1, firstName: 'Alice', lastName: 'Smith', dept: { name: 'Engineering' } },
      { id: 2, firstName: 'Bob', lastName: 'Jones', dept: { name: 'Sales' } },
    ]
  }
};

// Frontend needs:
const users = apiResponse.data.users.map(({ id, firstName, lastName, dept }) => ({
  id,
  fullName: `${firstName} ${lastName}`,
  department: dept.name,
}));

// Result:
// [
//   { id: 1, fullName: "Alice Smith", department: "Engineering" },
//   { id: 2, fullName: "Bob Jones", department: "Sales" }
// ]

Filter and aggregate

const orders = [
  { product: 'Widget', amount: 25.00, status: 'completed' },
  { product: 'Gadget', amount: 75.50, status: 'completed' },
  { product: 'Widget', amount: 25.00, status: 'cancelled' },
  { product: 'Gizmo', amount: 120.00, status: 'completed' },
];

// Total revenue from completed orders
const revenue = orders
  .filter(o => o.status === 'completed')
  .reduce((sum, o) => sum + o.amount, 0);
// 220.50

// Group by product
const byProduct = orders.reduce((groups, order) => {
  const key = order.product;
  (groups[key] = groups[key] || []).push(order);
  return groups;
}, {} as Record<string, typeof orders>);

Flatten nested structures

// Nested category tree
const categories = {
  name: 'Root',
  children: [
    { name: 'Electronics', children: [
      { name: 'Phones', children: [] },
      { name: 'Laptops', children: [] },
    ]},
    { name: 'Books', children: [] },
  ]
};

// Flatten to a list
function flattenTree(node: Category, depth = 0): Array<{ name: string; depth: number }> {
  return [
    { name: node.name, depth },
    ...node.children.flatMap(child => flattenTree(child, depth + 1)),
  ];
}

// Result: [{ name: "Root", depth: 0 }, { name: "Electronics", depth: 1 }, ...]

Python JSON Processing

Python's json module with list/dict comprehensions handles transformation concisely:

Load, transform, save

import json
from pathlib import Path

# Load
data = json.loads(Path('users.json').read_text())

# Transform: extract active users with selected fields
active_users = [
    {
        'name': f"{user['first_name']} {user['last_name']}",
        'email': user['email'],
    }
    for user in data['users']
    if user['status'] == 'active'
]

# Save
Path('active_users.json').write_text(
    json.dumps(active_users, indent=2, ensure_ascii=False)
)

Batch process multiple files

import json
from pathlib import Path

results = []
for file in Path('data/').glob('*.json'):
    data = json.loads(file.read_text())
    summary = {
        'file': file.name,
        'records': len(data),
        'total': sum(item['amount'] for item in data),
    }
    results.append(summary)

print(json.dumps(results, indent=2))

Merge JSON objects with conflict resolution

def deep_merge(base: dict, override: dict) -> dict:
    """Deep merge two dicts. Override values take precedence."""
    result = base.copy()
    for key, value in override.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = deep_merge(result[key], value)
        else:
            result[key] = value
    return result

config = deep_merge(default_config, env_config)

Reshaping API Responses for Frontend

A common transformation pattern: the API returns a flat list with IDs, and the frontend needs a nested tree structure.

// API returns flat data with parent references
const items = [
  { id: 1, name: 'Root', parentId: null },
  { id: 2, name: 'Child A', parentId: 1 },
  { id: 3, name: 'Child B', parentId: 1 },
  { id: 4, name: 'Grandchild', parentId: 2 },
];

// Frontend needs a tree
function buildTree(items: Item[], parentId: number | null = null): TreeNode[] {
  return items
    .filter(item => item.parentId === parentId)
    .map(item => ({
      ...item,
      children: buildTree(items, item.id),
    }));
}

const tree = buildTree(items);

Merging and Diffing JSON Documents

Shallow merge (spread operator):

const merged = { ...defaults, ...userPreferences };

Deep merge (with a library):

import deepmerge from 'deepmerge';
const merged = deepmerge(defaults, userPreferences);

JSON diff (finding what changed):

function jsonDiff(a: any, b: any, path = ''): string[] {
  const diffs: string[] = [];
  
  const allKeys = new Set([...Object.keys(a || {}), ...Object.keys(b || {})]);
  for (const key of allKeys) {
    const fullPath = path ? `${path}.${key}` : key;
    if (!(key in (a || {}))) diffs.push(`Added: ${fullPath}`);
    else if (!(key in (b || {}))) diffs.push(`Removed: ${fullPath}`);
    else if (typeof a[key] === 'object' && typeof b[key] === 'object') {
      diffs.push(...jsonDiff(a[key], b[key], fullPath));
    } else if (a[key] !== b[key]) {
      diffs.push(`Changed: ${fullPath}: ${a[key]} -> ${b[key]}`);
    }
  }
  return diffs;
}

Converting Between Formats

JSON to CSV:

function jsonToCsv(data) {
  const headers = Object.keys(data[0]);
  const rows = data.map(row => 
    headers.map(h => JSON.stringify(row[h] ?? '')).join(',')
  );
  return [headers.join(','), ...rows].join('\n');
}

JSON to TypeScript interfaces: Use our JSON to TypeScript converter for instant conversion. For Java classes, use our JSON to Java converter. For C# models, use our JSON to C# converter.

For the complete JSON developer reference, see our pillar guide: The Complete JSON Guide for Web Developers.

For JSON parsing performance optimisation, see: Boost JSON Parsing Speed in JavaScript.

The Debuggers builds data processing pipelines and API integration layers for web and mobile applications.

Frequently Asked Questions

Is jq faster than JavaScript for JSON transformation?

For processing files from the command line, jq is faster because it starts instantly (no Node.js startup overhead) and is optimised for streaming. For in-application transformation where the JSON is already in memory, JavaScript is faster because there is no serialisation/deserialisation overhead. Use jq for scripts and command-line workflows; use JavaScript for application logic.

How do I handle null values during transformation?

Use optional chaining (?.) and nullish coalescing (??) in JavaScript. In jq, use the alternative operator (//): .name // "Unknown" returns "Unknown" if .name is null. In Python, use .get('name', 'Unknown'). Always define explicit default values for fields that might be null to prevent downstream errors.

What is the best way to transform large JSON files (1GB+)?

Do not load the entire file into memory. Use streaming: jq handles streaming natively, Node.js has JSONStream and stream-json packages, and Python has ijson for iterative parsing. These tools process JSON incrementally, keeping memory usage constant regardless of file size.

How do I validate JSON before transformation?

Always validate first. Paste the JSON into our JSON Formatter to check syntax, then verify the structure matches your expectations. For automated pipelines, add a JSON Schema validation step before the transformation step. Invalid input produces unpredictable output.

---

Format and validate before transforming

Use our free JSON Formatter to validate and beautify JSON data before applying transformations. Convert results to typed code with our JSON to TypeScript converter.

Need data transformation pipelines? The Debuggers builds backend systems and data processing workflows for modern applications.