Skip to main content
Kyuubi-submit is a Python-based command-line interface wrapper tool designed to simplify the submission and monitoring of Spark jobs through Kyuubi. It allows you to run Spark SQL queries from outside the Kubernetes cluster. The Kyuubi-submit tool performs the following:
  • Simplifies job submission by providing an intuitive command-line interface
  • Automates monitoring with real-time status updates and progress tracking
  • Provides comprehensive logging with automatic log retrieval
  • Supports configuration management through YAML files for reusable job templates
  • Integrates seamlessly with Spark History Server and YuniKorn queue management
The following sequence describes the architectural flow behind how a submission moves through the system:
  1. You enter an input through the command-line tool or a YAML file.
  2. Kyuubi-submit merges that input with its configuration.
  3. The request goes to the Kyuubi REST API.
  4. Kyuubi submits the job to Spark on Kubernetes.
  5. You monitor the run and review the logs.

​
kyuubi-submit.py vs nx1 kyuubi

The core difference between kyuubi-submit.py and the nx1 kyuubi command is the execution environment and the resources it can access.
  • kyuubi-submit.py runs outside the cluster and interacts with Kyuubi remotely through the REST API. It requires an external network access to the Kyuubi server running in NexusOne, and handles the uploading of Kyuubi resources, such as JARs, files, or data from your local machine to the cluster.
  • nx1 kyuubi runs inside the Kubernetes cluster, specifically within the NexusOne JupyterHub app. It communicates with Kyuubi over internal cluster networking and assumes that all Kyuubi resources are already accessible within the cluster.

​
Additional resources