elasticsearch-py scan/scroll dies on large scroll_id: use POST

elasticsearch-py scan/scroll dies on large scroll_id: use POST

TL;DR

[accepted answer]: the python scan helper issues a GET to the scroll REST endpoint, and the scrollid grows with the number of shards until it is too large to send over HTTP as a query parameter. The request then fails silently (no response). Fix: send the scrollid in the JSON body of a POST instead of on the URL.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Use the elasticsearch.helpers.scan helper (which handles this correctly in current versions) or roll your own loop: 1. POST to /[index]/search?scroll=5m with your query to open the scroll. 2. POST to /search/scroll with the scrollid in the JSON body ({"scroll": "5m", "scrollid": "[id]"}). 3. Collect hits until the returned batch is empty, then clear the scroll. If you hand-rolled scroll logic with GET and it mysteriously returns nothing on multi-shard indexes, this is almost certainly why.

Context: Stack Overflow #22921629 (accepted answer, 22 votes): elasticsearch-py scan/scroll against 3M documents fails to return everything. The scan helper sends the scroll_id over HTTP and the request comes back empty.

Source: Stack Overflow #22921629 (accepted answer, 22 votes): elasticsearch-py scan/scroll against 3M documents fails to return everything. The scan helper sends the scroll_id over HTTP and the request comes back empty.

Verified resolution: [accepted answer]: the python scan helper issues a GET to the scroll REST endpoint, and the scrollid grows with the number of shards until it is too large to send over HTTP as a query parameter. The request then fails silently (no response). Fix: send the scrollid in the JSON body of a POST instead of on the URL. Use the elasticsearch.helpers.scan helper (which handles this correctly in current versions) or roll your own loop: 1. POST to /[index]/search?scroll=5m with your query to open the scroll. 2. POST to /search/scroll with the scrollid in the JSON body ({"scroll": "5m", "scrollid": "[id]"}). 3. Collect hits until the returned batch is empty, then clear the scroll. If you hand-rolled scroll logic with GET and it mysteriously returns nothing on multi-shard indexes, this is almost certainly why.

Source URL: https://stackoverflow.com/questions/22921629