weaviate batch import 'object already exists' error

TL;DR

Weaviate rejects batch objects whose UUIDs already exist in the class, which happens on re-imports or retried batches. Fix it by generating deterministic UUIDs from your record ids (so retries are idempotent) and either skipping existing objects or upserting over them deliberately.

weaviate batch import 'object already exists' error

Use this when

• A Weaviate batch import fails with object already exists
• Re-running an import script errors instead of being a no-op
• An agent retries a failed batch and collides with the partial first attempt

Not for this skill when

• The class schema rejects the object (thats a schema error)
• The vectorizer fails (thats the model)
• The connection drops mid-batch (thats network)

Steps

1. Confirm the collision is on UUID, not on your business key:

# check one of the failing ids directly
obj = client.data_object.get_by_id(failing_uuid, class_name="Article")
print(obj is not None)

Expected output: True, the object is already there. The import isnt broken; it is just not idempotent.

2. Generate deterministic UUIDs from your record ids so retries target the same objects:

import uuid
def stable_id(record_id):
    return str(uuid.uuid5(uuid.NAMESPACE_DNS, record_id))

Expected output: the same record id always maps to the same UUID. Random UUIDs per run guarantee collisions on retry.

3. Decide the re-import policy: skip existing or overwrite. For skip, check-then-write:

with client.batch as batch:
    for rec in records:
        uid = stable_id(rec["id"])
        if client.data_object.get_by_id(uid, class_name="Article") is None:
            batch.add_data_object(rec, "Article", uuid=uid)

Expected output: the batch imports only new objects. For overwrite semantics, update the existing object instead of skipping.

4. For full re-imports, consider truncating the class first:

# delete all objects in the class, then import fresh

Expected output: a clean import with no collisions. Only do this when the class holds nothing you need to keep.

Variant phrasings

first import works, second fails

Classic non-idempotent import. Deterministic UUIDs plus skip-or-overwrite (steps 2-3) make the second run clean.

error on a subset of a large batch

A partial earlier run already wrote those objects. The failing subset is exactly the overlap; skip-existing handles it.

Why it happens

Weaviate treats UUID as the primary key and refuses duplicates within a class. Import scripts that mint fresh random UUIDs on every run can never be retried safely, and agents love retrying failed batches. Deterministic UUIDs turn "already exists" from an error into a signal you can branch on.

Edge cases

• Batch error objects include the failed UUID; log them instead of failing the whole batch so one duplicate doesnt kill ten thousand inserts.
• If two different records map to one UUID (bad id function), you get silent overwrites; make the id function inject the class name too.
• Weaviate's batch is eventually consistent; a get right after a batch may miss a just-written object, so skip-checks can race. Add a small delay on retry paths.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-k0hCctJgbBnLV9bH2ngZA