Skip to main content
You deploy a fine-tuned model to start using it for inferencing. NexusOne starts the model so it can answer requests. When the model is ready, NexusOne adds it to an LLM Router.

Prerequisites

Before starting this task, ensure you have the following:
  • Required roles: nx1_finetune_admin, nx1_finetune_user, or nx1_engineer.
  • A previously created training job with the complete status.

Deploy a fine-tuned model

Perform these steps to deploy a fine-tuned model:
  1. Log in to NexusOne.
  2. On the top navigation bar, hover your mouse over AI and then select AutoML & AutoLLM.
  3. Click Fine-tune LLMs (AutoLLM).
  4. Click Training.
  5. Click the three dots at the end of the row for the training job with the complete status.
  6. Select Deploy for inference.
  7. In the Deployment name field, enter a name for the deployment. Use up to 52 lowercase letters, digits, or hyphens, and start and end with a letter or digit.
  8. Optional: In the Served model name field, enter the model name that appears on the LLM Router. If you leave it empty, then NexusOne uses the model’s registered name from MLflow.
  9. Optional: In the GPU count field, enter how many GPUs the model uses. If you leave it empty, then NexusOne uses 1.
  10. Optional: In the Max model length field, enter the most tokens the model can handle in one request.
  11. Optional: In the Min replicas field, enter the fewest running copies of the model server NexusOne keeps.
  12. Optional: In the Max replicas field, enter the most running copies of the model server NexusOne runs. If you leave it empty, then NexusOne uses 4.
  13. Click Deploy.
The new deployment appears in the Inferencing table with the deploying status. When the model is ready, the status changes to active and NexusOne adds the model to an LLM Router. To follow the deployment, refer to View an inferencing deployment.

Additional resources

For a high-level overview of inferencing and how it’s used in NexusOne, refer to Fine-tune LLM.