Authentication

NexusOne command-line tools, such as core and modules deployer, Kyuubi batch submit tool, and nx1 require setting up authentication manually.

Core and modules deployer

The core and modules deployer needs access to your Kubernetes cluster’s API server so it can deploy the core and modules tiers of NexusOne.

Kubeconfig

The shared_service_deployer.py script resolves your cluster credentials in the following order:

The --kubeconfig command-line option, if you pass one.
In-cluster credentials, if the script itself happens to be running inside a pod.
Your default kubeconfig, usually ~/.kube/config, if neither of the preceding options applies.

Authenticating to your cluster involves the following two different identities:

kubeconfig’s identity: The shared_service_deployer.py script uses your kubeconfig’s identity to talk to the Kubernetes API directly. It creates a target namespace, a terraform-deployer ServiceAccount, and a Kubernetes Job. To do all of that, the identity your kubeconfig authenticates as needs permission to create the following:
- Namespaces
- ServiceAccounts, Roles, and RoleBindings
- ClusterRoles and ClusterRoleBindings
- Jobs
terraform-deployer ServiceAccount’s identity: This is what the Kubernetes Job’s pod runs as once created. Terraform uses this identity to create and manage every resource inside a target namespace.

If a NexusOne deployment fails with a permissions error, then check your kubeconfig and the terraform-deployer ServiceAccount separately. Each needs its own permissions, and the fix depends on which one is missing them.

`nx1`

The nx1 tool supports three authentication methods. If you configure more than one, then NexusOne applies priority in the following order:

Command-line options
Environment variables
Profile

Command-line options

The command-line options authentication method passes authentication values directly to the nx1 command at runtime. The names of the command-line options include:

--api_key: API key
--host: API host

To use this method, take the following steps:

Get your API key.

Run the nx1 command and include both options.

# Example
nx1 command_name \
--api_key "your-psk-key" \
--host "https://aiapi.example.nx1cloud.com"

Environment variables

The environment variables authentication method uses variables configured on the command line. The required variables include:

NX1_API_KEY: API key
NX1_HOST: API host

To use this method, take the following steps:

Linux
Windows

Get your API key.

Set the environment variables.

# Example
export NX1_API_KEY="your-psk-key"
export NX1_HOST="https://aiapi.example.nx1cloud.com"

Run the nx1 command.
# Example nx1 command_name

Get your API key.

Set the environment variables.

:: Example
set NX1_API_KEY=your-psk-key
set NX1_HOST=https://aiapi.example.nx1cloud.com

Run the nx1 command.
# Example nx1 command

Profile

The profile authentication method uses a configuration file stored in a default directory. The file contains details of different profiles. Profiles can represent different environments, such as development, staging, and production. They allow you to store multiple credentials and switch between them easily without re-entering your credentials. The default NexusOne directory is the following:

~/.nx1/profiles

The file contents are in YAML format and follow this structure.

# ~/.nx1/profiles
default:
  host: https://api.production.nx1cloud.com
  api_key: your-psk-key
  verify_ssl: true
  timeout: 30

To use this method, take the following steps:

Get your API key.
Create the new directory and file.
mkdir -p ~/.nx1 touch ~/.nx1/profiles

Add a default profile.

# Example
nx1 profile add \
--name default \
--api-key "your-psk-key" \
--host "https://aiapi.example.nx1cloud.com"

This adds your credentials to the ~/.nx1/profiles file.

Kyuubi batch submit

The Kyuubi batch submit tool supports three authentication methods. If you configure more than one, then NexusOne applies priority in the following order:

Command-line options
YAML config
Environment variable
Interactive prompt

Command-line options

The command-line options authentication method passes authentication values at runtime. The names of the command-line options include:

--username: Kyuubi username
--password: Kyuubi password
--token: Keycloak bearer token

If you provide both --username/--password and --token, then --username/--password takes priority.

To use this option, take the following steps:

Username/password
Token

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Run the kyuubi_submit.py script and include both options.

# Example
python kyuubi_submit.py \
  --username "your-username" \
  --password "your-password" \
  --server "https://kyuubi.example.com" \
  --resource "./my_pyspark_job.py" \
  --name "My-PySpark-Job"

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Request a token from Keycloak. The client_id must be kyuubi-cli.

# Example
curl -X POST https://sso-your-realm.example.nx1cloud.com/realms/your-realm/protocol/openid-connect/token \
  -d "client_id=kyuubi-cli" \
  -d "grant_type=password" \
  -d "username=your-username" \
  -d "password=your-password"

The curl command returns an access_token value.

Run the kyuubi_submit.py script with the previously requested token.

# Example
python kyuubi_submit.py \
  --token "your-requested-token" \
  --server "https://kyuubi.example.com" \
  --resource "./my_pyspark_job.py" \
  --name "My-PySpark-Job"

YAML config

The YAML config authentication method uses a configuration file. The file contains details about your credentials, including additional details you would normally pass to the command line, such as:

Resource
Classname
Args
Jars

The following code shows a full sample of the additional details:

server: https://kyuubi.example.com
username: your-username
# password: optional (will prompt if not set)

resource: ./my-app.jar
classname: com.example.MainClass
name: My-Batch-Job

# args supports shell-style quoting (e.g., values with spaces)
args: "--input data.csv --output results/"

jars:
  - ./lib/mysql-connector.jar
  - s3a://bucket/common-utils.jar

pyfiles:
  - ./utils.py

files:
  - s3a://bucket/config.json # Remote recommended for files

conf:
  spark.executor.memory: 4g
  spark.executor.instances: 10
  spark.dynamicAllocation.enabled: "true"

queue: my-team-queue
history_server: spark-history.example.com
show_logs: true
debug: false

To use this option, take the following steps:

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py
Create the config file.
touch job-config.yaml

Run the Kyuubi script and include the config file.

python kyuubi_submit.py --config-file "job-config.yaml"

To optionally override YAML values, pass the options.

python kyuubi_submit.py --config-file "job-config.yaml" --conf "spark.executor.memory=8g"

Environment variable

The environment variable authentication method uses a variable configured on the command line. It requires the following variable:

KYUUBI_SUBMIT_PASSWORD: Kyuubi password

To use this option, take the following steps:

Linux
Windows

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Set the environment variable.

# Example
export KYUUBI_SUBMIT_PASSWORD="your-password"

Run the Kyuubi script command.

# Example
python kyuubi_submit.py \
  --username "your-username" \
  --server "https://kyuubi.example.com" \
  --resource "./my_pyspark_job.py" \
  --name "My-PySpark-Job"

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Set the environment variable.

:: Example
set KYUUBI_SUBMIT_PASSWORD="your-password"

Run the Kyuubi script command.

# Example
python kyuubi_submit.py \
  --username "your-username" \
  --server "https://kyuubi.example.com" \
  --resource "./my_pyspark_job.py" \
  --name "My-PySpark-Job"

Interactive prompt

The interactive prompt authentication method expects you to always enter your Kyuubi password each time you run a command. To use this option, take the following steps:

Clone the kyuubi-submit GitHub repository.

git clone https://github.com/nexuscognitive/kyuubi-submit.git

Install dependencies and requirements.

cd kyuubi-submit/python/
pip install requests pyyaml urllib3

Make the Kyuubi script executable.
chmod +x kyuubi-submit.py

Run the kyuubi_submit.py script without including the --password option.

# Example
python kyuubi_submit.py \
  --username "your-username" \
  --server "https://kyuubi.example.com" \
  --resource "./my_pyspark_job.py" \
  --name "My-PySpark-Job"

When prompted to enter a password, enter your password.

Using the CLIs

CLI tools

Core and modules deployer

Kubeconfig

`nx1`

Command-line options

Environment variables

Profile

Kyuubi batch submit

Command-line options

YAML config

Environment variable

Interactive prompt

Using the CLIs

CLI tools

​Core and modules deployer

​Kubeconfig

​nx1

​Command-line options

​Environment variables

​Profile

​Kyuubi batch submit

​Command-line options

​YAML config

​Environment variable

​Interactive prompt

Core and modules deployer

Kubeconfig

`nx1`

Command-line options

Environment variables

Profile

Kyuubi batch submit

Command-line options

YAML config

Environment variable

Interactive prompt