ScyllaDB: Running CQL query on different nodes in the Scylla cluster gives different results

TL;DR

You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster.

When to use

You are seeing this: ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. Use this skill when you run into "ScyllaDB: Running CQL query on different nodes in the Scylla cluster gives different results".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 4.6.3. If you are on something much newer or older, the details may have shifted.

You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster. ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. If you used CL=ONE for your writes, than only 1 replica needs to ACK for the application to consider this successful. The replication to the 2nd replica will be done a-synchronically (and can also fail for various reasons). Here comes the anti-entropy mechanism, which you can read more about them here: https://docs.scylladb.com/architecture/anti-entropy/ You must make sure your cluster completed cluster wide repair before the table's gc_grace_seconds value (default 10 days) You should have also fully repaired your cluster before adding the 2nd DC, or at least done it after you've added the 2nd DC. This is also written in our docs.

Context: Stack Overflow #72934006 (accepted answer, 7 votes): I am using Scylla version 4.6.3. I have a cluster of 3 nodes in a single DC. I have followed the instructions given in https://docs.scylladb.com/operating-scylla/procedures/cluster-management/add-dc-to-existing-dc/ to add 3 new nodes in a new DC. After the nodes in the new DC are started, I check nodetool status and ensure all are up and running. Now since all nodes are part of the same cluster, I assume the query results should be the same irrespective of which node I run the cql query on, isn't it? But I see that the data is different when the query is run on different nodes. In fact the...

Source: Stack Overflow #72934006 (accepted answer, 7 votes): I am using Scylla version 4.6.3. I have a cluster of 3 nodes in a single DC. I have followed the instructions given in https://docs.scylladb.com/operating-scylla/procedures/cluster-management/add-dc-to-existing-dc/ to add 3 new nodes in a new DC. After the nodes in the new DC are started, I check nodetool status and ensure all are up and running. Now since all nodes are part of the same cluster, I assume the query results should be the same irrespective of which node I run the cql query on, isn't it? But I see that the data is different when the query is run on different nodes. In fact the...

Verified resolution: You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster. ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. If you used CL=ONE for your writes, than only 1 replica needs to ACK for the application to consider this successful. The replication to the 2nd replica will be done a-synchronically (and can also fail for various reasons). Here comes the anti-entropy mechanism, which you can read more about them here: https://docs.scylladb.com/architecture/anti-entropy/ You must make sure your cluster completed cluster wide repair before the table's gc_grace_seconds value (default 10 days) You should have also fully repaired your cluster before adding the 2nd DC, or at least done it after you've added the 2nd DC. This is also written in our docs.

Source URL: https://stackoverflow.com/questions/72934006/scylladb-running-cql-query-on-different-nodes-in-the-scylla-cluster-gives-diffe