Immutable solution version
ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse
Complete solution
The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.
ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse
What is going on
Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime.
What actually works
The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.
So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.
Source: https://github.com/Azure-Samples/Synapse/issues/148
Source
Mined from a verified thread: https://vectle.com/threads/thr_wXsl95k1Yl2Tk8zo4uIS-Q (original: https://github.com/Azure-Samples/Synapse/issues/148)
Evidence
In the thread, the asker reported: Following the Synapse PySpark notebook sample for reading a SQL pool table, the cell import org.apache.spark.sql.SqlAnalyticsConnector._ failed with ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector'. The reporter assumed the module should already be installed in Azure Synapse notebooks. It turns out the connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The verified answer that resolved it: The SQL analytics connector is a Scala library, so that import only works when the notebook is running the Scala runtime. The reporter was in a PySpark (Python) session, hence the ModuleNotFoundError, and confirmed switching the notebook to the Scala runtime fixed it.
So if you hit this error, check your notebook's language setting: the spark.read.sqlanalytics(...) samples need Scala, not PySpark.
Source: https://github.com/Azure-Samples/Synapse/issues/148 Thread: https://vectle.com/threads/thr_wXsl95k1Yl2Tk8zo4uIS-Q
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_QphiAv97LeRyHpAMxnz2eg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ModuleNotFoundError: No module named 'org.apache.spark.sql.SqlAnalyticsConnector' in Synapse <span>— Vectle</span></a>