> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nx1cloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark in NexusOne

> Overview of Apache Spark, a primary distributed processing engine for data workloads in NexusOne.

Apache Spark is a distributed analytics engine and framework for large-scale data processing.
The engine executes high-performance data transformations and analytics by leveraging in-memory computing
for processing and falls back to disk when required.
The framework provides unified APIs for SQL, streaming, machine learning, and graph processing.

NexusOne deploys Spark as the primary execution engine for data platform workloads spanning ingestion
pipelines, complex ETL transformations, interactive analytics, and ad-hoc reporting.

## Deployment context

Within NexusOne, Spark jobs can execute in the following environments:

* **Jupyter Notebooks**: An interactive REPL environment with a pre-initialized
  [SparkSession](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.html).
  Optimal for exploratory data analysis, prototyping, and collaborative development. If you are using the NexusOne portal, you can launch
  a Jupyter Notebook using the NexusOne [Hosted Apps](/documentation/platform/tasks/launch-a-hosted-app) feature.
* **Apache Airflow**: Provides production orchestration via Apache Airflow using
  [PySpark decorators](https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/decorators/pyspark.html).

  * **Batch scheduling**: Jobs submitted run as scheduled cron tasks, monitored continuously, and have alerting configured.
  * **Execution method**: Jobs run as containerized Spark apps in Kubernetes pods with full lifecycle management.

  If you are using the NexusOne portal, you can launch Apache Airflow using the NexusOne [Monitor](/documentation/platform/overview/monitor) feature.
* **Command line via Kyuubi**: Provides command-line submission for external job execution. This is available for jobs submitted from outside the Kubernetes cluster.

## Current version information

* Python version: `3.12`
* Spark version: `3.5.6`

## Supported Spark interfaces

* **PySpark via the Python API**: Programmatic access to DataFrame and SQL APIs for pipeline development and data
  engineering workflows.
* **Spark SQL**: Write standard SQL queries that execute on Spark. Perfect for analysts and data warehouse teams
  who prefer SQL syntax for querying and transforming data.

## Supported library and table format

Apache Iceberg: `1.8`

The Apache Iceberg table format provides snapshot isolation and schema evolution.

## Default Spark configuration

```python theme={null}
spark.dynamicAllocation.enabled=true
spark.executor.memory=8g
spark.driver.memory=12g
spark.sql.catalog.iceberg.type=hive
```

## Additional resources

* To learn practical ways to use Spark in the NexusOne environment, refer to the
  [Spark hands-on examples](./spark-hands-on-examples) page.
* To learn about best practices when using Spark, refer to the
  [Spark best practices](./spark-best-practices) page.
* For more details about Spark, refer to the
  [Spark](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/index.html)
  and [Iceberg](https://iceberg.apache.org/docs/latest/spark-writes/) official documentation.