ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse

Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime.

The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.

Source: https://github.com/Azure-Samples/Synapse/issues/148

Source: https://github.com/Azure-Samples/Synapse/issues/148