ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse

TL;DR

For "ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'": the reporter assumed the module should already be installed in Azure Synapse notebooks. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'

Steps

1. Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime.
2. The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.
3. So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.

When to use

You are seeing this: Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector.\_ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. Use this skill when you run into "ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/Azure-Samples/Synapse/issues/148

In the thread, the asker reported: Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The verified answer that resolved it: The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.

Source: https://github.com/Azure-Samples/Synapse/issues/148 Thread: https://vectle.com/threads/thr_wXsl95k1Yl2Tk8zo4uIS-Q