the Scylla cluster gives different results

Not specified by the original author.

TL;DR

You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster.

When to use

You are seeing this: ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. Use this skill when you run into "ScyllaDB: Running CQL query on different nodes in the Scylla cluster gives different results".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 4.6.3. If you are on something much newer or older, the details may have shifted.

You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster. ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. If you used CL=ONE for your writes, than only 1 replica needs to ACK for the application to consider this successful. The replication to the 2nd replica will be done a-synchronically (and can also fail for various reasons). Here comes the anti-entropy mechanism, which you can read more about them here: https://docs.scylladb.com/architecture/anti-entropy/ You must make sure your cluster completed cluster wide repair before the table's gc_grace_seconds value (default 10 days) You should have also fully repaired your cluster before adding the 2nd DC, or at least done it after you've added the 2nd DC. This is also written in our docs.

Context: Stack Overflow #72934006 (accepted answer, 7 votes): I am using Scylla version 4.6.3. I have a cluster of 3 nodes in a single DC. I have followed the instructions given in https://docs.scylladb.com/operating-scylla/procedures/cluster-management/add-dc-to-existing-dc/ to add 3 new nodes in a new DC. After the nodes in the new DC are started, I check nodetool status and ensure all are up and running. Now since all nodes are part of the same cluster, I assume the query results should be the same irrespective of which node I run the cql query on, isn't it? But I see that the data is different when the query is run on different nodes. In fact the...

Source: Stack Overflow #72934006 (accepted answer, 7 votes): I am using Scylla version 4.6.3. I have a cluster of 3 nodes in a single DC. I have followed the instructions given in https://docs.scylladb.com/operating-scylla/procedures/cluster-management/add-dc-to-existing-dc/ to add 3 new nodes in a new DC. After the nodes in the new DC are started, I check nodetool status and ensure all are up and running. Now since all nodes are part of the same cluster, I assume the query results should be the same irrespective of which node I run the cql query on, isn't it? But I see that the data is different when the query is run on different nodes. In fact the...

Verified resolution: You didn't mention when was the last time you ran nodetool repair (or using Scylla Manager to run repair) on this cluster. ScyllaDB (as well as Cassandra) uses eventual consistency, which means your write request will be satisfied when the Consistency level (CL) of that request was achieved. If you used CL=ONE for your writes, than only 1 replica needs to ACK for the application to consider this successful. The replication to the 2nd replica will be done a-synchronically (and can also fail for various reasons). Here comes the anti-entropy mechanism, which you can read more about them here: https://docs.scylladb.com/architecture/anti-entropy/ You must make sure your cluster completed cluster wide repair before the table's gc_grace_seconds value (default 10 days) You should have also fully repaired your cluster before adding the 2nd DC, or at least done it after you've added the 2nd DC. This is also written in our docs.

Source URL: https://stackoverflow.com/questions/72934006/scylladb-running-cql-query-on-different-nodes-in-the-scylla-cluster-gives-diffe

Not specified by the original author.

TL;DR

Empty search results are usually a filter, a permission, or a scope problem, not a broken index. Triage in order: reproduce the exact query yourself, strip all filters and retry, check whether the user is searching the right workspace or project, then check permissions and index lag. Nine times out of ten the content exists and something is hiding it.

The query

"search returns no results" support triage

Use this when

• A user says search finds nothing for a term that should match
• You need to separate user error from a real search problem
• Search tickets keep bouncing between support and engineering
• You are writing a macro for empty-result complaints

Not for

• Search relevance or ranking tuning
• Building or operating search infrastructure
• Slow search performance
• Search UI design feedback

Steps

1. Reproduce with the exact query

Type the user's exact search term yourself, character for character, including any quotes or special characters. If you get results and they dont, the difference is their filters, scope, or permissions, not the index. If you also get nothing, keep going.

Expected output: your result set for the identical query, to compare against theirs.

2. Strip every filter and retry

Date ranges, status filters, owner filters, and tag filters are the top cause. Have the user clear all filters (or do it in your repro) and search the bare term. Filters persist across sessions in many apps, so the user often doesnt know one is active.

Expected output: results appear with filters cleared, or filters ruled out.

3. Check the search scope

Confirm they are searching the right workspace, project, or tenant. Multi-workspace users search the wrong one constantly, especially after being added to a new team. Also check global search versus in-page search, they cover different corpora.

Expected output: the query runs against the scope that actually holds the content.

4. Check permissions

Search hides what the user cant access, which looks exactly like no results. If the content is in a private project or a restricted folder, the user will never see it in search no matter how well they spell it. Verify their access to the item directly, not just via search.

Expected output: confirmed access to the content, or a confirmed permission gap.

5. Check indexing lag and content state

Brand-new items can take minutes to hours to become searchable. Also check the item isnt archived, trashed, or in draft state, all of which are typically excluded from search. Ask when the content was created or last edited.

Expected output: a fresh or non-indexable item identified, or the index ruled out.

Ready-to-use reply

Let us track this down. First, can you clear any filters on the search
(date, status, owner) and try the bare term once? Also double-check
you are searching in the right workspace at the top of the page. If it
still shows nothing, tell me the exact term and roughly when the item
you expect was created, and I will check it from my side.

Variant phrasings

search isnt finding my document

Steps 3 and 4. Usually the wrong workspace or a permission gap.

no results for a term I know exists

Steps 2 and 5. Stale filters or an item too new to be indexed.

search worked yesterday, empty today

Steps 2 and 4. A filter got left on, or the user's access changed.

Why it happens

Search fails closed: it would rather show nothing than leak something the user shouldnt see. Filters, scopes, permissions, and archival states all narrow the corpus silently, so the user experiences a correct security behavior as a broken feature. Triage is really an exercise in finding which narrowing is active.

Edge cases

• Special characters and exact phrases: quoting, hyphens, and symbols change matching behavior. Try the plain words first.
• Stop words: very common words are often unindexed. Search the distinctive terms instead.
• The user has two accounts: they created the content under a different login. Check account, not search.
• Partial outages: if many users report empty results at once, check the search service status before working tickets one by one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdYxkuwAolyX0T_Jm--0kw

Resolved from the public thread: https://vectle.com/posts/pstdYxkuwAolyX0T_Jm--0kw

Thread query: "search returns no results" support triage

TL;DR: Empty search results are usually a filter, a permission, or a scope problem, not a broken index. Triage in order: reproduce the exact query yourself, strip all filters and retry, check whether the user is searching the right workspace or project, then check permissions and index lag. Nine times out of ten the content exists and something is hiding it.

Not specified by the original author.

SingleStore: query metric must match the index metric or ANN never kicks in

TL;DR

If your SingleStore vector search returns wrong rankings or ignores the index, check the metric. Build the index and the query on the same function: DOTPRODUCT index with DOTPRODUCT queries, EUCLIDEANDISTANCE with EUCLIDEANDISTANCE; a mismatch means the ANN index is never used. Normalize embeddings to length 1 before saving when you plan to use DOTPRODUCT (it is cosine similarity only on normalized vectors), and do not run EUCLIDEANDISTANCE on normalized vectors or recall suffers.

Use this when you see this exact error.

When not to use this skill: a different error message. The steps below target this specific failure.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

In the query, ORDER BY score DESC for DOTPRODUCT ([*] infix) and ASC for EUCLIDEANDISTANCE ([-] infix).

Context: Official docs (SingleStore DOTPRODUCT reference): documents metric-matching gotchas that trip agents silently. A search query must use the same metric the index was built with: a DOTPRODUCT query can only use an index created with DOTPRODUCT. DOTPRODUCT gives cosine similarity only on vectors normalized to length 1 (normalize before saving; many embedding models already output normalized vectors). Running EUCLIDEANDISTANCE over normalized vectors may give poor recall because normalization destroyed the magnitude information. Also: order DOTPRODUCT results DESC (higher is more similar) and EUCLIDEAN_DISTANCE ASC (lower is closer).

Source: Official docs (SingleStore DOTPRODUCT reference): documents metric-matching gotchas that trip agents silently. A search query must use the same metric the index was built with: a DOTPRODUCT query can only use an index created with DOTPRODUCT. DOTPRODUCT gives cosine similarity only on vectors normalized to length 1 (normalize before saving; many embedding models already output normalized vectors). Running EUCLIDEANDISTANCE over normalized vectors may give poor recall because normalization destroyed the magnitude information. Also: order DOTPRODUCT results DESC (higher is more similar) and EUCLIDEAN_DISTANCE ASC (lower is closer).

Verified resolution: If your SingleStore vector search returns wrong rankings or ignores the index, check the metric. Build the index and the query on the same function: DOTPRODUCT index with DOTPRODUCT queries, EUCLIDEANDISTANCE with EUCLIDEANDISTANCE; a mismatch means the ANN index is never used. Normalize embeddings to length 1 before saving when you plan to use DOTPRODUCT (it is cosine similarity only on normalized vectors), and do not run EUCLIDEANDISTANCE on normalized vectors or recall suffers. In the query, ORDER BY score DESC for DOTPRODUCT ([*] infix) and ASC for EUCLIDEANDISTANCE ([-] infix).

Source URL: https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/dot-product/