> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nx1cloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Non-web portal users

> nx1-deployer `v1.16.0` release notes for non-web portal users, covering disaster recovery, Keycloak, S3 Gateway, autoscaling, and new API endpoints.

<Note>nx1-deployer image `v1.16.0`</Note>

The August 2026 release for non-web portal users adds new API endpoints across the AI router, code scanning, S3 credentials, and
data quality. It also adds disaster recovery support and fixes multi-site domain and cache issues in Keycloak. AI API,
Metastore, and Keycloak also gain autoscaling, and Kyuubi and S3 Gateway get reliability fixes.

## New features

New features recently added to the NexusOne platform.

### AI router classifier management

The NX1 LLM router now supports classifiers that tag incoming requests with a DataHub domain, so the router's decision rules
can act on that domain. You can manage those classifiers and publish them to the router's live config through the following
new API endpoints:

* [Compile enabled classifiers into the live router config](/api-reference/endpoints/ai-router/compile-enabled-classifiers-into-the-live-router-config)
* [Create a heuristic classifier](/api-reference/endpoints/ai-router/create-a-heuristic-classifier)
* [Delete a classifier](/api-reference/endpoints/ai-router/delete-a-classifier)
* [Get a single classifier](/api-reference/endpoints/ai-router/get-a-single-classifier)
* [List governance domains](/api-reference/endpoints/ai-router/list-governance-domains-classifier-targets-decision-rules)
* [List heuristic classifiers](/api-reference/endpoints/ai-router/list-heuristic-classifiers-keywordembedding)
* [Update a classifier](/api-reference/endpoints/ai-router/update-a-classifier)

### AutoLLM fine-tuning

AutoLLM is a new pipeline for fine-tuning your own large language models. A CrewAI run first assembles a DataHub domain's knowledge
into a Q/A training dataset, which you review and approve before it trains. Its new API endpoints include the following:

* [Approve a dataset proposal and finalize it](/api-reference/endpoints/autollm/approve-a-dataset-proposal-and-finalize-it)
* [Create a fine-tune training job](/api-reference/endpoints/autollm/create-a-fine-tune-training-job)
* [Create a fine-tuning dataset proposal](/api-reference/endpoints/autollm/create-a-fine-tuning-dataset-proposal)
* [Delete a fine-tune training job](/api-reference/endpoints/autollm/delete-a-fine-tune-training-job)
* [Delete a fine-tuning dataset job](/api-reference/endpoints/autollm/delete-a-fine-tuning-dataset-job)
* [Get a fine-tune training job](/api-reference/endpoints/autollm/get-a-fine-tune-training-job)
* [Get a fine-tuning dataset proposal](/api-reference/endpoints/autollm/get-a-fine-tuning-dataset-proposal)
* [Get the full example set for a dataset job](/api-reference/endpoints/autollm/get-the-full-example-set-for-a-dataset-job)
* [List base models available for fine-tuning](/api-reference/endpoints/autollm/list-base-models-available-for-fine-tuning)
* [List fine-tune training jobs](/api-reference/endpoints/autollm/list-fine-tune-training-jobs)
* [List fine-tuning dataset jobs](/api-reference/endpoints/autollm/list-fine-tuning-dataset-jobs)
* [Manually trigger a training DAG](/api-reference/endpoints/autollm/manually-trigger-a-training-dag)
* [Re-run the crew on a failed dataset job](/api-reference/endpoints/autollm/re-run-the-crew-on-a-failed-dataset-job)
* [Trainer status writeback](/api-reference/endpoints/autollm/trainer-status-writeback)

### Code scanner

The code scanner scans a Spark codebase for changes needed to migrate it to NexusOne, and flags each issue by severity. Its
new API endpoints include the following:

* [Create a code scan run](/api-reference/endpoints/code-scanner/create-a-code-scan-run)
* [Delete a code scan run](/api-reference/endpoints/code-scanner/delete-a-code-scan-run)
* [Get a code scan run](/api-reference/endpoints/code-scanner/get-a-code-scan-run)
* [Get the code scan report](/api-reference/endpoints/code-scanner/get-the-code-scan-report)
* [Lifetime code scan statistics](/api-reference/endpoints/code-scanner/lifetime-code-scan-statistics)
* [List code scan runs](/api-reference/endpoints/code-scanner/list-code-scan-runs)
* [Poll code scan run status](/api-reference/endpoints/code-scanner/poll-code-scan-run-status)
* [Resume a paused scan](/api-reference/endpoints/code-scanner/resume-a-paused-scan)
* [Stop a running scan](/api-reference/endpoints/code-scanner/stop-a-running-scan)

### Data migration

A new [Count data migration runs matching filters](/api-reference/endpoints/data-migration/count-data-migration-runs-matching-filters)
endpoint returns a count of migration runs matching a filter, without listing them all.

### Data quality rule suggestions

CrewAI already suggests data quality rules from a table's sample data. You can now preview a suggestion's matching data
before promoting it to a real rule, or create rules in bulk. The new API endpoints include the following:

* [Bulk create data quality rules](/api-reference/endpoints/data-quality/bulk-create-data-quality-rules)
* [Create a suggested data quality rule](/api-reference/endpoints/data-quality/create-a-suggested-data-quality-rule)
* [Preview rule data](/api-reference/endpoints/data-quality/preview-rule-data)

### Inferencing

Deploying a fine-tuned AutoLLM model with vLLM applies its Kubernetes manifests and registers it with the NX1 LLM router
automatically. You can do this through the following new API endpoints:

* [Delete a stopped vLLM deployment](/api-reference/endpoints/inferencing/delete-archive-a-stopped-vllm-deployment)
* [Deploy a fine-tuned model with vLLM](/api-reference/endpoints/inferencing/deploy-a-fine-tuned-model-with-vllm)
* [Get a vLLM deployment](/api-reference/endpoints/inferencing/get-a-vllm-deployment)
* [Get Kubernetes events for a vLLM deployment](/api-reference/endpoints/inferencing/get-k8s-events-for-a-vllm-deployment)
* [List vLLM deployments](/api-reference/endpoints/inferencing/list-vllm-deployments)
* [Stop a vLLM deployment](/api-reference/endpoints/inferencing/stop-a-vllm-deployment)

### S3 credential vending

You can now generate, view, and delete your own S3 access credentials, and vend credentials for a specific bucket or the
platform default. The new API endpoints include the following:

* [Delete the current user's S3 credential](/api-reference/endpoints/s3-credentials/delete-the-current-users-s3-credential)
* [Generate or regenerate the current user's S3 credential](/api-reference/endpoints/s3-credentials/generate-or-regenerate-the-current-users-s3-credential)
* [Get the current user's S3 credential information](/api-reference/endpoints/s3-credentials/get-the-current-users-s3-credential-information)
* [Resolve an S3 access key to its owner and secret](/api-reference/endpoints/s3-credentials/resolve-an-s3-access-key-to-its-owner-and-secret-gateway-only)
* [Vend credentials for a bucket](/api-reference/endpoints/s3/vend-credentials-for-a-bucket)
* [Vend the default credentials](/api-reference/endpoints/s3/vend-the-default-credentials)

## Bug fixes

Fixes to issues affecting apps or features on the NexusOne platform.

### AI API no longer blocks on Trino during a fresh tenant deployment

AI API is NexusOne's natural language SQL and data engineering service. Its pod mounts two Trino-managed Kubernetes volumes at
startup:

* **`trino-s3`**: S3 access config that Trino also uses
* **`trino-gateway-routing-rules`**: Routing config that Trino Gateway uses to reach the right Trino cluster.

AI API reads from those volumes whenever it calls Trino to run the SQL it generates.

In previous releases, Terraform defined those volumes inside the Trino and Trino Gateway modules. But portal, which deploys AI
API, had no dependency on either. That meant AI API's pod could start on a fresh tenant deployment before the volumes even
existed, and it would just sit pending.

In this release, Terraform moves the two volumes out of the Trino and Trino Gateway modules into standalone top-level resources.
The Trino, Trino Gateway, and portal modules now all explicitly depend on them. AI API's pod no longer waits on Trino during a
fresh deployment.

### Data shares catalog URL now uses a global domain

The data shares credential feature lets one NexusOne tenant share Iceberg datasets with another, through a Gravitino Iceberg REST
Catalog (IRC) server. In previous releases, NX1 Portal showed that server's URL using only the local `domain` Terraform variable.
A consumer tenant on a different cluster can't reach a local-only URL.

The portal now shows the IRC server's URL at a `global_domain` Terraform variable by default. The portal still shows the `domain`
URL too, but only as a fallback when `global_domain` and `domain` are different.

### DR support

DataHub's OpenSearch previously had no way to keep a standby site's data current. There was no multi-site snapshot support, no
security plugin protecting the search API, and no way to point DataHub at the active site. JupyterHub's database schema also had
no cluster namespacing, so primary and DR sites sharing one database would collide.

DR support is now built into the shared services core and modules layers, covering the following:

* **OpenSearch snapshot scheduling**: The main site snapshots OpenSearch to S3 hourly now, not every 6 hours. This affects standby and demoted sites as follows:
  * Standby site auto-restores the main site's latest snapshot.
  * Standby site can use the main site's snapshot bucket directly, instead of its own.
  * Demoted site now deletes its own snapshot policy instead of pausing it, so gaps don't appear silently after promotion.
* **OpenSearch security**: A new security plugin adds basic auth and transport TLS, and an optional external route lets DataHub follow whichever site is active.
* **Per-cluster JupyterHub schema**: JupyterHub's schema name now includes a new `cluster_name` Terraform variable. Setting a distinct value per site gives each one its own schema within the same shared database, instead of colliding.
* **Per-cluster Kyuubi schema**: Kyuubi's schema name now includes that same `cluster_name` Terraform variable, so it gets its own schema within the same shared database too. It falls back to the workspace name automatically when `cluster_name` isn't set.

### Keycloak

This release includes Keycloak fixes for multi-site domain routing, cache staleness on standby-to-main site promotion, and Active
Directory sync.

#### Domain routing fixes

NexusOne supports multi-site DR pairs, a main site and a standby site. Some components need a domain that resolves to whichever
site is currently active, so users don't have to know which one is live.

In previous releases, a few components used the `alternate_domain` Terraform variable to provide that active-site domain.
Airflow stayed hardcoded to the local domain instead, breaking access through the second domain.

This release renames `alternate_domain` to `global_domain`. Airflow's webserver URL and Trino connections now follow that
variable when it's set.

#### Cache staleness on site promotion

The main and standby NexusOne sites each run their own Keycloak, but the two share a database and previously ran separate caches.
Promoting a standby site to main could leave stale realm and user data. Users could see outdated group or role assignments until
those cache entries expired on their own.

In this release, a new cache-clear job runs on site promotion and clears stale user, realm, and keys caches. That job needs to run
in Keycloak's own image, which was previously hardcoded. The image tag is now the `keycloak_tag` Terraform variable, default
`26.5.6.1`, and the `keycloak_image` output tracks it automatically for the job.

#### Faster and more frequent Active Directory sync

Keycloak previously checked Active Directory for group and role changes only once a day. In this release, Keycloak checks
every 6 hours instead, so those changes reach Keycloak faster.

### S3 Gateway now enforces Ranger policy consistently

The S3 Gateway had the following two authorization issues in previous releases:

* Ranger policy and the bucket's own legacy policy or Access Control List (ACL) ran separately for every request. They could
  give conflicting answers for the same request.
* The copy-source read that happens during a copy operation doesn't route through Ranger's central authorization hook.

In this release, S3 Gateway makes Ranger the single source of truth for authorization. The gateway no longer checks the legacy
bucket policy or ACL path, though read-only and object-lock checks still apply separately. The copy-source read is now explicitly
authorized through Ranger too.

This applies consistently across AI API and Keycloak-session accounts.

### Session and job recovery for Kyuubi database failover

When you connect to Spark through Kyuubi, Kyuubi creates a session for each connection. Each session gets a Spark engine to run
its jobs. MetadataManager stores that session, engine, and job metadata in a managed database, and reads it back when needed.

In previous releases, that database could fail over and cause MetadataManager's reads to fail instead of retrying. This caused a
Kyuubi session's jobs to restart or drop instead of continuing.

In this release, Kyuubi now retries failed MetadataManager reads a few times, while waiting longer between each try. When a
database failover happens, it no longer interrupts a session or its running jobs.

MetadataManager writes are also now protected from concurrent-write conflicts, so job history stays consistent across a failover.
Lastly, Kyuubi stops logging MetadataManager errors during failover, since the previous retry already resolves them.

### Superset's Trino connection now uses a global domain

Superset's Trino connection config lives in a database that primary and DR sites share. In previous releases, that config's Trino
host value used the local `domain` Terraform variable.

Two sites writing their own local Trino host value to the same shared row meant whichever site's init job ran last would win. The
other site was then left with a config pointing at a Trino host it can't reach.

Superset's Trino host now uses a `global_domain` Terraform variable instead. That name resolves to whichever site is currently
active, so the config stays valid from either cluster.

## Enhancements

Enhancements to existing app features on the NexusOne platform.

### AI API now scales automatically with load

AI API previously ran a fixed 2 replicas, with hardcoded CPU and memory limits. That worked exactly as configured, but it couldn't
grow under heavy load or shrink when idle.

In this release, a Kubernetes HorizontalPodAutoscaler scales AI API automatically, using the following new Terraform variables:

* Scales between the `hpa_min_replicas` and `hpa_max_replicas` variables, 2 and 6 by default.
* Adds or removes pods when average CPU usage crosses the target percentage in the `hpa_target_cpu` variable.
* The `hpa_memory_enabled` variable adds pods when memory crosses the `hpa_memory_average_value` variable, independent of CPU.
  It never triggers a scale-down, since memory usage doesn't reliably drop even when idle.
* Scale-up is immediate. Scale-down waits the `hpa_scaledown_stabilization` variable's seconds. It removes one pod at a time.
* CPU and memory requests/limits are now configured via the `cpu_request`, `cpu_limit`, `memory_request`, and `memory_limit` variables.

### DataHub editor and reader roles now available through Keycloak

DataHub has three built-in access levels, Admin, Editor, and Reader. In previous releases, Keycloak could only grant the Admin
role. Assigning Editor or Reader access required configuring it manually inside DataHub instead.

In this release, two new Keycloak realm roles, `datahub-editor` and `datahub-reader`, grant DataHub's Editor and Reader roles the
same way Admin already worked. All three DataHub access levels are now assignable directly through Keycloak.

### `kyuubi-submit` can now authenticate with a Keycloak token

The `kyuubi-submit` command-line tool already supported a `--token` flag for Bearer-token authentication, as an alternative to
`--username`/`--password`. In previous releases, there was no way to actually obtain a token, since no Keycloak client existed for
it.

In this release, users can request a token from Keycloak directly by specifying `client_id=kyuubi-cli`. That token carries the
user's realm roles as a `groups` claim, for downstream authorization. Users can then pass that generated token to `kyuubi-submit --token` instead of a username and password.

### Spark History Server URL now also shows running jobs

Spark's built-in job UI shows live progress, stages, and executors. It lives on the job's own driver pod, though, at a port that
changes with every job.

The Spark History Server serves a saved copy of that page once a job finishes, at a fixed, familiar URL. But visiting that URL for
a still-running job didn't work, since History Server only has saved copies.

In this release, the History Server's URL routes through a new `spark-web-proxy` service instead. It checks whether a job is still
running and forwards to that job's live page. If the job already finished, then it forwards to the History Server's saved copy
like before.

### T-shirt sizing extends to Metastore and Keycloak

Kyuubi already supported t-shirt sizing, with `small`, `medium`, `large`, and `xlarge` tiers, each setting replicas, CPU, memory,
and JVM heap together. Metastore and Keycloak didn't. Their sizing stayed hardcoded regardless of tenant size.

In this release, Metastore reuses the existing `kyuubi_sizing` variable to size itself alongside Kyuubi, since the two scale
together. Keycloak gets its own new tiers through the existing `datahub_sizing` variable instead. Both changes cover the
following:

* Metastore's replica count now varies by tier, from 2 to 6.
* Keycloak gains real autoscaling for the first time, based on CPU usage.
* Keycloak's minimum replicas never drops below 2, so its session cache always keeps more than one copy.
* Both now scale their JVM heap size with the tier too. Previously, the heap stayed fixed even after raising the memory limit,
  so the extra memory went unused.

## Upgrades

Version upgrades to existing apps on the NexusOne platform.

### JupyterHub `5.4-nx1.10.1` upgrade

Upgraded JupyterHub from `5.4-nx1.6-slim` to `5.4-nx1.10.1`. The `5.4-nx1.9` and `5.4-nx1.10` upgrades happened in the middle of
this release.

### Metastore `3.1.7` upgrade

Upgraded Metastore from `3.1.4` to `3.1.7`. The `3.1.5` and `3.1.6` upgrades happened in the middle of this release.

### S3 gateway `v0.17` upgrade

Upgraded S3 Gateway from `v0.15` to `v0.17`. The `v0.16` upgrade happened in the middle of this release.
