Authentication

Only two NexusOne command-line tool requires setting up authentication manually, and that’s the nx1 and the Kyuubi batch submit tool.

`nx1`

The nx1 tool supports three authentication methods. If you configure more than one, then NexusOne applies priority in the following order:

Command-line options
Environment variables
Profile

Command-line options

The command-line options authentication method passes authentication values directly to the nx1 command at runtime. The names of the command-line options include:

--api_key: API key
--host: API host

To use this method, take the following steps:

Get your API key.

Run the nx1 command and include both options.

# Example
nx1 command_name \
--api_key "your-psk-key" \
--host "https://aiapi.example.nx1cloud.com"

Environment variables

The environment variables authentication method uses variables configured on the command line. The required variables include:

NX1_API_KEY: API key
NX1_HOST: API host

To use this method, take the following steps:

Linux
Windows

Get your API key.

Set the environment variables.

# Example
export NX1_API_KEY="your-psk-key"
export NX1_HOST="https://aiapi.example.nx1cloud.com"

Run the nx1 command.
# Example nx1 command_name

Get your API key.

Set the environment variables.

:: Example
set NX1_API_KEY=your-psk-key
set NX1_HOST=https://aiapi.example.nx1cloud.com

Run the nx1 command.
# Example nx1 command

Profile

The profile authentication method uses a configuration file stored in a default directory. The file contains details of different profiles. Profiles can represent different environments, such as development, staging, and production. They allow you to store multiple credentials and switch between them easily without re-entering your credentials. The default NexusOne directory is the following:

~/.nx1/profiles

The file contents are in YAML format and follow this structure.

# ~/.nx1/profiles
default:
  host: https://api.production.nx1cloud.com
  api_key: your-psk-key
  verify_ssl: true
  timeout: 30

To use this method, take the following steps:

Get your API key.
Create the new directory and file.
mkdir -p ~/.nx1 touch ~/.nx1/profiles

Add a default profile.

# Example
nx1 profile add \
--name default \
--api-key "your-psk-key" \
--host "https://aiapi.example.nx1cloud.com"

This adds your credentials to the ~/.nx1/profiles file.

Kyuubi batch submit

The Kyuubi batch submit tool supports three authentication methods. If you configure more than one, then NexusOne applies priority in the following order:

Command-line options
YAML config
Environment variable
Interactive prompt

Command-line options

The command-line options authentication method passes authentication values at runtime. The names of the command-line options include:

--username: Kyuubi username
--password: Kyuubi password

To use this option, take the following steps:

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Run the kyuubi_submit.py script and include both options.

# Example
python kyuubi_submit.py \
  --username your-username \
  --password your-password \
  --server https://kyuubi.example.com \
  --resource ./my_pyspark_job.py \
  --name "My-PySpark-Job"

YAML config

The YAML config authentication method uses a configuration file. The file contains details about your credentials, including additional details you would normally pass to the command line, such as:

Resource
Classname
Args
Jars

The following code shows a full sample of the additional details:

server: https://kyuubi.example.com
username: your-username
# password: optional (will prompt if not set)

resource: ./my-app.jar
classname: com.example.MainClass
name: My-Batch-Job

# args supports shell-style quoting (e.g., values with spaces)
args: "--input data.csv --output results/"

jars:
  - ./lib/mysql-connector.jar
  - s3a://bucket/common-utils.jar

pyfiles:
  - ./utils.py

files:
  - s3a://bucket/config.json # Remote recommended for files

conf:
  spark.executor.memory: 4g
  spark.executor.instances: 10
  spark.dynamicAllocation.enabled: "true"

queue: my-team-queue
history_server: spark-history.example.com
show_logs: true
debug: false

To use this option, take the following steps:

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py
Create the config file.
touch ob-config.yaml

Run the Kyuubi script and include the config file.

python kyuubi_submit.py --config-file job-config.yaml

To optionally override YAML values, pass the options.

python kyuubi_submit.py --config-file job-config.yaml --conf "spark.executor.memory=8g"

Environment variable

The environment variable authentication method uses a variable configured on the command line. It requires the following variable:

KYUUBI_SUBMIT_PASSWORD: Kyuubi password

To use this option, take the following steps:

Linux
Windows

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Set the environment variable.

# Example
export KYUUBI_SUBMIT_PASSWORD="your-password"

Run the Kyuubi script command.

# Example
python kyuubi_submit.py \
  --username your-username \
  --server https://kyuubi.example.com \
  --resource ./my_pyspark_job.py \
  --name "My-PySpark-Job"

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Set the environment variable.

:: Example
set KYUUBI_SUBMIT_PASSWORD="your-password"

Run the Kyuubi script command.

# Example
python kyuubi_submit.py \
  --username your-username \
  --server https://kyuubi.example.com \
  --resource ./my_pyspark_job.py \
  --name "My-PySpark-Job"

Interactive prompt

The interactive prompt authentication method expects you to always enter your Kyuubi password each time you run a command. To use this option, take the following steps:

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Run the kyuubi_submit.py script without including the --password option.

# Example
python kyuubi_submit.py \
  --username your-username \
  --server https://kyuubi.example.com \
  --resource ./my_pyspark_job.py \
  --name "My-PySpark-Job"

When prompted to enter a password, enter your password.

Using the CLIs

CLI tools

`nx1`

Command-line options

Environment variables

Profile

Kyuubi batch submit

Command-line options

YAML config

Environment variable

Interactive prompt

​nx1

​Command-line options

​Environment variables

​Profile

​Kyuubi batch submit

​Command-line options

​YAML config

​Environment variable

​Interactive prompt

`nx1`

Command-line options

Environment variables

Profile

Kyuubi batch submit

Command-line options

YAML config

Environment variable

Interactive prompt