Bulk ingest into Upstash Redis: /pipeline batches under 10MB per request

TL;DR: Bulk-load data with /pipeline batches sized under 10MB per request. One HTTP call per batch, results in order. Verify Count keys before and after (DBSIZE or a SCAN sample), spot-check values, and confirm the console command graph shows the expected ingest rate without throttling errors.

The fix

Bulk ingest into Upstash Redis with /pipeline

The shape

POST batches of independent writes to /pipeline. Each batch is one HTTP request with a 2D JSON array body:

curl -X POST https://YOURRESTURL/pipeline \ -H "your auth header \ -d '[ ["SET", "k1", "v1"], ["SET", "k2", "v2"] ]'

Results come back in order. A failed command shows as an error entry in its position and does not stop the rest.

Size the batches

• Stay under the 10MB max request size per batch. JSON and base64 inflate payloads: measure serialized bytes.

• Hundreds of commands per batch is a good starting point. Tune by watching for timeouts.

• Bigger batches mean fewer HTTP round trips but each commands latency equals the batch latency. For ingest, throughput is what matters.

Throughput math

Tier limit is 10K commands per second. Size batch count and parallelism to stay under it with headroom. Back off and retry on 429s: byte-identical retries are safe for SET-style idempotent writes.

SDK path

In @upstash/redis, build the batch with Promise.all over independent commands and let auto-pipelining merge them, or use redis.pipeline() explicitly for full control.

Verify

Count keys before and after (DBSIZE or a SCAN sample), spot-check values, and confirm the console command graph shows the expected ingest rate without throttling errors.

When to use this

• This covers exactly what the title says: Bulk ingest into Upstash Redis.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with Bulk; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Compatibility

• Not pinned to a specific version; follows current Bulk behavior.

Verified against Upstash REST API docs: /pipeline takes a 2D JSON array per HTTP request with ordered results; max request size 10MB; pipelining is for throughput when commands are independent. Source: https://upstash.com/docs/redis/features/restapi