Apache Spark
Bindings: %spark, %spark.sql, %spark.pyspark, %spark.ipyspark, %spark.r, %spark.ir, %spark.shiny
Runs distributed jobs against an Apache Spark cluster (or a local Spark instance), from a paragraph. A SparkSession is set up for you and shared across the bindings, so you can build a DataFrame in Scala in one paragraph and query it with %spark.sql in the next.
| Binding | Use it for |
|---|---|
%spark | Scala on Spark |
%spark.sql | SQL on Spark (queries registered tables/views) |
%spark.pyspark | Python on Spark |
%spark.ipyspark | Python on Spark via an interactive IPython shell |
%spark.r / %spark.ir | R on Spark |
%spark.shiny | Spark-backed R Shiny applications |
%spark.pyspark
df = spark.read.csv("/data/sales.csv", header=True, inferSchema=True)
df.createOrReplaceTempView("sales")
%spark.sql
SELECT region, sum(amount) AS total FROM sales GROUP BY region
Configuration
| Property | Default | Description |
|---|---|---|
spark.master | local[*] | Where Spark runs — local, YARN, or a standalone master address. |
spark.submit.deployMode | (empty) | client or cluster deploy mode. |
spark.driver.memory / spark.driver.cores | 1g / 1 | Resources for the Spark driver. |
spark.executor.memory / spark.executor.cores / spark.executor.instances | 1g / 1 / 2 | Resources per executor, and how many executors to allocate. |
spark.jars / spark.jars.packages | (empty) | Extra jars, or Maven coordinates of extra dependencies, to add to the job. |
skeyenote.spark.useHiveContext | true | Whether the Spark session is Hive-aware. |
skeyenote.spark.maxResult | 1000 | Max number of Spark SQL rows to display. |
skeyenote.spark.concurrentSQL / skeyenote.spark.concurrentSQL.max | true / 10 | Whether SQL paragraphs can run concurrently, and how many at once. |
skeyenote.spark.uiWebUrl / skeyenote.spark.ui.hidden | — | Where (or whether) to expose the Spark UI link for a job. |
These are set by your administrator from the interpreter settings screen, typically per cluster or per team.
See the notebook context object (ssnc) for displaying DataFrames and building dynamic forms from any of the Spark bindings.