Non-determinism when concurrently deleting cross references

TL;DR

There is no API to delete multiple references in one request. Workaround: delete them sequentially, or fetch all current references, filter out the ones to remove, and set the rest with a single collection.data.reference_replace() call.

Steps

1. [jeroiraz (commenter)]: Concurrently deleting cross-references from the same object leads to non-deterministic failures (read-repair and consistency errors). There is no API to delete multiple references in one request. Workaround: delete them sequentially, or fetch all current references, filter out the ones to remove, and set the rest with a single collection.data.reference_replace() call.

When to use

You are seeing this: Weaviate - I'm not sure whether this is specifically a client issue, but I seem to be encountering a weird condition where concurrently deleting cross references from a collection (via asyncio.gather) results in non-deterministic/incorrect behavior, and... Use this skill when you run into "Non-determinism when concurrently deleting cross references".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/weaviate/weaviate/issues/7879

In gh:weaviate/weaviate#7879 (https://github.com/weaviate/weaviate/issues/7879), the issue reported: I'm not sure whether this is specifically a client issue, but I seem to be encountering a weird condition where concurrently deleting cross references from a collection (via asyncio.gather) results in non-deterministic/incorrect behavior, and more often than not seems to result in unexpected data when running in a multi-node setup.

Specifically, an operation like the following won't actually trigger any "Failed to delete" errors, but the assertion will fail because the number of cross references will be some random number more than 0 (what it should be).

# bulk delete all cross... The verified answer: [jeroiraz (commenter)]: Concurrently deleting cross-references from the same object leads to non-deterministic failures (read-repair and consistency errors). There is no API to delete multiple references in one request. Workaround: delete them sequentially, or fetch all current references, filter out the ones to remove, and set the rest with a single collection.data.reference_replace() call.