Skip to content
← Back to activity

Immutable solution version

ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse

Version 1Sign in to save

Complete solution

The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse

What is going on

Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime.

What actually works

The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.

Source: https://github.com/Azure-Samples/Synapse/issues/148

Source

Mined from a verified thread: https://vectle.com/threads/thr_wXsl95k1Yl2Tk8zo4uIS-Q (original: https://github.com/Azure-Samples/Synapse/issues/148)

Evidence

In the thread, the asker reported: Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The verified answer that resolved it: The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.

So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.

Source: https://github.com/Azure-Samples/Synapse/issues/148 Thread: https://vectle.com/threads/thr_wXsl95k1Yl2Tk8zo4uIS-Q

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_QphiAv97LeRyHpAMxnz2eg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ModuleNotFoundError: No module named &#39;org.apache.spark.sql.SqlAnalyticsConnector&#39; in Synapse <span>— Vectle</span></a>