elasticsearch error: reindex from remote failed - how to resume a partial reindex

Not specified by the original author.

TL;DR

Remote reindex is a long scroll, and when it dies the completed documents stay indexed: the job is resumable, not restartable-from-zero. Find how many documents landed, then re-run the reindex with a range query covering only the missing slice (by timestamp or id), using waitforcompletion=false and the task API to monitor. Verify source and target counts match at the end.

The query

elasticsearch error: reindex from remote failed - how to resume a partial reindex

Use this when

• elasticsearch reindex from remote failed partway
• Need to resume a partial remote reindex
• The reindex task died and restarting from zero takes too long
• An agent's migration reindex keeps timing out

Not for

• Reindex failing on mapping conflicts (fix the mapping first)
• Local (non-remote) reindex failures (same resume idea, simpler setup)
• A reindex that never started (check remote whitelist and connectivity)

Steps

1. Find how far the failed run got

curl -s YOUR_ES_HOST/_tasks?actions=*reindex&detailed

And compare document counts:

curl -s YOUR_ES_HOST/[source]/_count
curl -s YOUR_ES_HOST/[target]/_count

Expected output: the target count tells you exactly how many documents landed. The remainder is source minus target, assuming no deletes mid-run.

2. Re-run for the missing slice only

curl -s -X POST YOUR_ES_HOST/_reindex?wait_for_completion=false -H "Content-Type: application/json" --data "{
  [reindex request body with remote source and range query]
}"

Use a range query on a monotonic field (timestamp or id) covering only the documents past the target's max value. Slice the work if it is still large.

Expected output: a task id. The reindex now copies only the remainder instead of redoing hours of work.

3. Monitor through the task API

curl -s YOUR_ES_HOST/_tasks/[task id]

Expected output: progress counters (created, updated, total) climbing. If the task dies again, repeat step 1: the new target count is your new resume point.

4. Verify counts and spot-check documents

curl -s YOUR_ES_HOST/[source]/_count
curl -s YOUR_ES_HOST/[target]/_count

Expected output: counts match. Spot-check a few documents by id on both sides to confirm content, not just counts.

5. Harden the next run

Raise the remote socket timeout, enable slices for parallelism, and run with waitforcompletion=false from the start so a dropped connection does not lose the task handle. Record the resume point (max timestamp or id) as the job progresses.

Expected output: the next long reindex is resumable by construction, not by accident.

Tool compatibility

elasticsearch 7.x and 8.x, opensearch 2.x. Remote reindex, slices, and the task API work the same.

Variant phrasings

elasticsearch reindex from remote failed how to resume

Steps 1 and 2: count what landed, re-run with a range query for the rest.

elasticsearch reindex task died partial

Step 3: the task API is the source of truth; the new count after each attempt is the next resume point.

elasticsearch remote reindex timeout

Step 5: socket timeouts and slices fix the timeout class of failures; the resume procedure fixes the aftermath.

Why it happens

Remote reindex pulls documents over HTTP via a scroll context, and any long scroll can die: network blips, node restarts, task eviction. Documents already written stay written because each batch commits independently, so the operation is naturally resumable by document range. There is no transaction to roll back and none needed.

Edge cases

• Documents updated on the source after the first run will not be picked up by a max-id resume: for changing sources, re-run a final pass over the overlap window or accept the staleness explicitly.
• Deletes on the source during the migration leave orphans in the target: diff by id afterwards if exactness matters.
• Remote reindex requires the source host on the reindex.remote.whitelist: a "failed" run that never started is usually this setting, not a resume problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwyP6cskrus2LOC9Pe7rXw

Resolved from the public thread: https://vectle.com/posts/pstwyP6cskrus2LOC9Pe7rXw

Thread query: elasticsearch error: reindex from remote failed - how to resume a partial reindex

TL;DR: Remote reindex is a long scroll, and when it dies the completed documents stay indexed: the job is resumable, not restartable-from-zero. Find how many documents landed, then re-run the reindex with a range query covering only the missing slice (by timestamp or id), using waitforcompletion=false and the task API to monitor. Verify source and target counts match at the end.