Non-web portal users

The July 2026 release for non-web portal users improves stability and performance across the platform. Kyuubi picks up autoscaling and configurable T-shirt sizing, along with fixes for pod crashes, stale hostnames, and lost session state on failover. S3 Gateway and Tenant manager both make fewer round trips now and ship with new metrics dashboards. Spark’s multipart uploads are safer, and Iceberg commits retry automatically instead of failing outright. Kafka now cleans up old log data on its own and starts up more reliably after restarts. On the sign-in side, users who have no email set in Keycloak can now log in to Ranger, YuniKorn, and MLflow. There’s also a new alternate domain option for testing disaster recovery, plus version upgrades for Kyuubi, Metastore, and Jupyter.

​

Bug fixes

Fixes to issues affecting apps or features on the NexusOne platform.

​

Jupyter Spark home directory mounted read-only

Jupyter mounts a read-write /utils volume into your environment, and saves $SPARK_HOME inside it. That gave you write access to the Spark install, meaning you could accidentally modify or delete files that Spark depends on. SPARK_HOME now also gets its own read-only mount from the same volume. The rest of /utils stays read-write. You can no longer accidentally change the Spark installation.

​

Kafka

This release includes Kafka fixes for disk usage and pod startup stability.

​

Kafka volumes filling up with old log segments

Kafka stores messages as log segments on disk and keeps them until a retention policy removes them. Without explicit retention settings, Kafka kept old segments indefinitely, so its persistent volumes grew until they ran out of space. The broker now sets explicit retention and replication defaults:

• Keeps data for 5 days
• Keeps up to 2 GB per partition
• Checks for old segments to clean up every 5 minutes
• Defaults new topics to two replicas

Kafka clears out aged log data on its own now, so disk usage stays bounded.

​

Kafka pods restarting before brokers finish starting up

Kubernetes uses a liveness probe to check whether a container is healthy, and restarts it when the probe keeps failing. Kafka brokers can be slow to start, especially during restarts and topic recovery, so the old probe timings marked healthy brokers as unhealthy and restarted them early. The probe now gives brokers more time to come up, and lets them shut down cleanly:

• Initial delay: 120 seconds, up from 60
• Check interval: 30 seconds, up from 20
• Failure threshold: 6 failed checks before a restart
• Termination grace period: 120 seconds

​

Keycloak

This release includes Keycloak fixes for user login and Terraform applies.

​

Login failing for users without an email in Keycloak

Ranger, YuniKorn, and MLflow sit behind OAuth 2.0 Proxy, which required an email attribute in the user’s Keycloak profile. Users without an email set couldn’t log in and hit an Internal Server Error. OAuth 2.0 Proxy now does the following:

• Falls back to preferred_username as the user ID when no email is present
• Widens its OIDC scope to openid profile email so it can read that claim

Users without an email in Keycloak can now log in to Ranger, YuniKorn, and MLflow.

​

Terraform apply failing on transient Keycloak errors

Terraform manages your Keycloak realms, clients, and roles by calling the Keycloak API directly. During an apply, when Terraform creates a realm, a follow-up GET request for that realm can briefly return a 403 error until the client refreshes its access token. Without retry logic, that temporary error alone could make the apply fail, even though nothing was actually wrong. Terraform now retries automatically and refreshes its credentials when it hits a 403 error, so your apply goes through instead of failing on it.

​

Kyuubi

This release includes several Kyuubi stability fixes and operational improvements.

​

Health checks causing unnecessary pod restarts

Kubernetes uses liveness and readiness probes to decide whether a pod is healthy enough to keep serving traffic. For Kyuubi pods, both probes used to run an exec-based kyuubi status check inside the container, which could misreport a perfectly healthy pod as unhealthy. This triggered unnecessary restarts that interrupted any sessions running on that pod. Both probes now run REST-based checks against /api/v1/ping instead, with a longer check interval and timeout. This gives Kubernetes a more reliable read on pod health and cuts down on false-positive restarts.

​

Kyuubi pods getting OOMKilled under heavy load

In Kubernetes, OOMKilled occurs when a container exceeds its memory limit and gets terminated. Under sustained heavy load, the Spark submit launcher inside a Kyuubi pod can run out of memory and get OOMKilled. This eventually interrupts all active sessions. The launcher’s heap size is now larger, giving it enough headroom to handle concurrent job submissions without crashing.

​

Lost session state on failover

Running Kyuubi with high availability means another replica can pick up in-flight work if one goes down. Each replica used to keep its own local work directory, so a failover could lose any session state that hadn’t yet persisted elsewhere. All replicas now mount a shared work directory, so session state survives a failover, and Kyuubi can support high availability the way it’s meant to.

​

Stale hostnames after a Kyuubi restart

Kyuubi pods advertise a hostname so clients can reconnect to the exact pod handling their session. When a pod restarts, it could keep advertising its previous hostname instead of picking up the new one. This becomes a problem because clients trying to reconnect end up getting pointed at a stale endpoint that’s no longer running. This release upgrades Kyuubi to v1.11.11 to fix this, and pods now advertise their current hostname immediately after a restart.

​

Metastore locking and startup failures

Iceberg tables backed by the Hive Metastore need working table locking so concurrent writers don’t step on each other. Metastore’s container startup command also failed if the database schema was already initialized once. Metastore’s startup command now safely upgrades the schema if needed, instead of only trying to create it from scratch, so it no longer breaks on an already-initialized database.

​

Pod cleanup job now runs under the tenant’s service account

YuniKorn is a scheduler that places pods into a tenant’s queue in a cluster. It decides which queue a job belongs to based on the job’s owner. The pod cleanup job didn’t set an owner, so YuniKorn had no way to schedule it. The pod cleanup job now reads the name of its service account and sets that as its owner.

​

Ranger UI login failing with a large request header

Ranger’s Tomcat connector limits how large an incoming HTTP header can be. Users with many assigned groups or long OIDC tokens exceeded that limit, so logging in to the Ranger UI failed with a Request header is too large error. The maximum header size is now raised to 256 KB, so Tomcat accepts these requests and login succeeds.

​

Spark

This release includes Spark fixes for multipart uploads, Iceberg commits, and cloud storage connectivity.

​

Fixed multipart upload properties that were silently disabled

Spark uses S3A’s “magic committer” to make multipart uploads to S3 safe and atomic, configured through a handful of properties. Two property names contained typos, so Spark silently ignored them and the protection was never actually active. Fixing those typos means the multipart upload safety properties actually take effect.

​

Iceberg commits failing outright instead of retrying

Iceberg tables use optimistic concurrency control. Two conflicting writes can both try to commit, and one has to retry. Without retry settings, a conflicting commit could just fail outright instead of retrying. Spark’s primary catalog now retries a failed commit up to 50 times, waiting between 50 ms and 2 seconds between attempts, instead of relying on Hive’s locking.

​

Spark jobs failing without Hadoop cloud JARs

Spark jobs that read from or write to cloud storage need the right Hadoop cloud connector JARs bundled into the image. Missing ones can make a job fail outright when it tries to talk to cloud storage. The Spark image is now bumped to 3.5.6-nx1.26, which adds the Hadoop cloud connector JARs.

​

Enhancements

Enhancements to existing app features on the NexusOne platform.

​

An alternate domain and ingress stack for DR verification

Testing a disaster recovery setup means proving a second domain and its certificates work, without touching the primary domain still serving production traffic. Setting the new alternate_domain deployment variable now stands up a complete second set of SSL certificates. It also creates an ingress stack for every route alongside your primary domain. Airflow’s own configuration still points at the primary domain, so it doesn’t work correctly over the alternate one and redirects back.

​

Kyuubi

Kyuubi picks up autoscaling, preset sizing options, and several UI and API improvements in this release.

​

Autoscaling for Kyuubi

Kyuubi used to run a fixed number of replicas. You had to size for peak load, even though usage was often much lower. In this release, Kyuubi scales its replica count automatically based on memory usage. It stays within the min and max range set by your chosen T-shirt size. Kyuubi waits for about 35 minutes after load drops before scaling down. This keeps short drops in usage from disrupting active sessions.

​

Other Kyuubi improvements

• Kyuubi UI:
  • Shows a quick summary of session and operation counts, such as total, active, idle, and error, right at the top of the page.
  • Lets you filter your list of sessions and operations.
• Kyuubi UI/API:
  • Shows which Spark driver pod each session is using, and whether it’s healthy.
  • Keeps a batch listed as Reconnecting if its instance goes down, instead of just dropping it from the list.
• Batch jobs can now mix local files and S3 for multipart upload.
• If a Kyuubi instance dies, then its running and pending batches hand off gracefully to another instance instead of getting abandoned.
• Spark now comes pre-baked into the image, so it starts up faster.
• Terminated Spark driver pods now get cleaned up automatically instead of sticking around.

​

T-shirt sizing for Kyuubi deployments

Kyuubi uses T-shirt sizing, a set of preset deployment size profiles, to provide enough replicas, CPU, and memory for your tenant’s workload. Tuning each of those settings manually could lead to under- or over-provisioning. Kyuubi now ships with four preset sizes to choose from:

• Small
  • Replicas: 1-10
  • CPU: 1-2
  • Memory: 6Gi-16Gi
  • Heap: 2g
  • Submitter threads: 10
• Medium
  • Replicas: 2-10
  • CPU: 2-4
  • Memory: 10Gi-32Gi
  • Heap: 4g
  • Submitter threads: 25
• Large
  • Replicas: 3-10
  • CPU: 2-8
  • Memory: 16Gi-64Gi
  • Heap: 8g
  • Submitter threads: 50
• XL
  • Replicas: 4-10
  • CPU: 4-16
  • Memory: 48Gi-256Gi
  • Heap: 24g
  • Submitter threads: 150

In your tenant Terraform variables, set the new kyuubi_sizing deployment variable to one of the following values:

• small
• medium
• large
• xlarge

It defaults to large if left unset.

​

View Spark driver logs from the batch API

Every batch job submitted through Kyuubi runs its Spark driver on its own pod. If a job failed or stalled, then checking that driver’s logs meant leaving Kyuubi and digging through Kubernetes or the Spark UI to find the right pod. Kyuubi can read Spark driver pod logs directly now, through a new /batches/{id}/driverLog endpoint you can call to pull them up without ever leaving Kyuubi.

​

S3 Gateway and Authz API

S3 Gateway listing is faster, and Authz now exposes metrics for Prometheus scraping.

​

Fewer round trips for S3 listing and authorization

S3 Gateway used to check authorization against both Ranger and legacy bucket policies. It also made a separate call for every object and prefix when listing. Ranger is now the only thing that decides access, so listing costs one round trip instead of many.

​

Metrics endpoint for Authz scraping

Prometheus discovers what to scrape using annotations on a pod’s deployment. Without them, a deployment’s metrics go uncollected even if it exposes a metrics endpoint. Authz’s deployment didn’t have these annotations set. Authz’s deployment now includes Prometheus scrape annotations, so Prometheus collects its metrics the same way it collects metrics from other services.

​

Tenant manager

This release improves Tenant manager performance, observability, and monitoring.

​

Fewer database and Keycloak round trips

Tenant manager used to call Keycloak to verify every request’s identity, even after verifying the same token. Each tenant runs on a cluster, and each deployment belongs to a tenant. When Tenant manager listed tenants or deployments, it also ran a separate query per row to fetch that related cluster or tenant. With this fix, Tenant manager now caches verified identities briefly instead of being re-checked against Keycloak every time. The tenant and deployment lists now fetch related records in one query instead of many.

​

Metrics and a default Grafana dashboard

Tenant manager didn’t expose any metrics. There was no way to see request volume, auth cache performance, or deployment trends without digging through logs. Tenant manager exposes a /metrics endpoint now. It covers the following:

• HTTP request counts and latency
• Auth cache hit and miss rates
• Deployment monitor poll duration and status transitions

A new NX1 Tenant manager API Grafana dashboard visualizes them. Prometheus scrapes only the tenant-api pod. The UI pod exposes no metrics endpoint.

​

Upgrades

Version upgrades to existing apps on the NexusOne platform.

​

Iceberg v1.11.0 upgrade

Upgraded Iceberg to v1.11.0.

​

Jupyter v5.4-nx1.6-slim upgrade

Upgraded the Jupyter single-user image from v5.4-nx1.4-slim to v5.4-nx1.6-slim to resolve known vulnerabilities.

​

Kyuubi v1.11.11 upgrade

Upgraded Kyuubi from v1.11.5 to v1.11.11.

​

Metastore v3.1.4 upgrade

Upgraded Metastore from v3.0.13 through v3.1.3 to v3.1.4.