End-to-end flow of the app manager feature
Each created app works like a project with version control built in. Creating one just sets up an empty project. Nothing runs yet. The next step is adding files. But you don’t add them straight to the app. You add them to an app version instead. Every app can have multiple versions and only one version can be active at a time. The rest stay inactive. When you create an app version, you add the files it needs, such as DAGs and notebooks. NexusOne tracks each version separately, so your changes stay isolated to that version. Lastly, you make the app version active. So, what does making an app version active do? NexusOne copies the app version files to the app’s folder namedcurrent to replace what was there. That’s the point where the app ships and
your changes actually go live.
An app isn’t only about files, though. It also controls who can use it through
roles, and stores any credentials it needs as secrets.
App structure
An app comprises components, roles, and secrets.Components
Components are files that belong to an app. You add them to an app version. Components have several types:- Artifacts: Files such as
.csv,.jar,.py, or other file formats - DAGs: Apache Airflow DAG files
- Notebooks: Jupyter notebook files
- Serving configs: YAML or JSON files that define how an app serves model predictions
Roles
Roles define who can use an app. Create the role in the Govern feature before you define it in an app.Secrets
Secrets are sensitive values scoped to an app, which the app’s DAGs and notebook can read at run time.Where components run
As previously described, Apache Airflow runs the app manager pipelines. However, not every component runs in a pipeline. Components fall into two groups, based on whether they run in a pipeline or not:- Runs in a pipeline: The supported components that run in a pipeline include the following:
- DAGs: The DAG component runs directly as Airflow DAGs.
- Notebooks: The notebook component runs as a task inside a DAG, if you build it with the Elyra pipeline editor in JupyterHub.
- Doesn’t run in a pipeline: The components that don’t run in a pipeline include the following:
- Artifacts: The artifacts component skips the pipeline entirely. Artifact files support a DAG, notebook, or serving config component, such as a helper script, a config file, or a model file.
- Serving configs: The serving configs component skips the pipeline for a different reason. It only describes how to serve a model.
What an app ships and its respective component
As previously described, an app is a package that you ship to NexusOne. When you make an app version active, NexusOne delivers its files so they run. But first, you need to decide what the app ships when you create it, and each choice supports specific component types. The following sections describe the supported shipping options.Model service
Keeps a trained model running and answers requests at an endpoint. The supported component types can be any of the following:- Artifact: Files the model needs.
- Serving config: A file that defines how to serve the model.
Notebook pipeline
Runs notebooks from your JupyterHub workspace as a scheduled job. You build the DAG that chains them with the Elyra pipeline editor. The supported component types can be any of the following:- Artifact: Files the DAG or notebooks need.
- DAG: The DAG you build with the Elyra pipeline editor.
- Notebook: The notebooks the DAG runs.
Pipeline
Runs a scheduled job you write yourself as an Airflow DAG file. The supported component types can be any of the following:- Artifact: Files the DAG needs.
- DAG: The Airflow DAG file you write.
Pipeline + service
Combines a notebook pipeline with a model service. DAGs and notebooks prepare the data, then a serving config serves the model. The supported component types can be any of the following:- Artifact: Files the DAG, notebooks, or model need.
- DAG: The DAG that chains the notebooks.
- Notebook: The notebooks that prepare the data.
- Serving config: A file that defines how to serve the model.
Default component locations
You can access files from all component types in Jupyter Notebook. They’re stored in unique directories depending on the app version. When working with a Notebook component type, there might be times when you want an app version to use an existing notebook file in JupyterHub. In that scenario, you attach the notebook file to the app version. This means the app version points to the notebook file’s location instead of storing a copy of it. Whether you upload or attach a component file, where you find it depends on the app version:-
For an active app version, you can find the
components in the following paths:
- Artifacts:
<bucket_name>/deps/<app_name>/current/ - DAGs:
<bucket_name>/deps/<app_name>/current/dags/ - Notebooks:
<bucket_name>/deps/<app_name>/current/notebooks/, or<bucket_name>/users/<username>/if attached - Serving configs:
<bucket_name>/deps/<app_name>/current/serving/
- Artifacts:
-
For all versions of an app, including the active version, you can find the
components in the following paths:
- Artifacts:
<bucket_name>/deps/<app_name>/<version_name>/ - DAGs:
<bucket_name>/deps/<app_name>/<version_name>/dags/ - Notebooks:
<bucket_name>/deps/<app_name>/<version_name>/notebooks/, or<bucket_name>/users/<username>/if attached - Serving configs:
<bucket_name>/deps/<app_name>/<version_name>/serving/
- Artifacts:
Attaching a notebook doesn’t copy it into the default notebook paths, such as
<bucket_name>/deps/<app_name>/current/notebooks/ for an active app version and
<bucket_name>/deps/<app_name>/<version_name>/notebooks/ for all app versions.When you attach a notebook file, the app version records its location and a hash of its content.
If you edit the file again, then re-attach it to update that hash.Use cases
These examples show how different industries can use NexusOne’s app manager:- Financial services: Version and manage assets in a trading or reporting pipeline. These assets include notebooks for model training and serving configs to deploy a fraud-detection model.
- Healthcare: Manage clinical pipeline assets as versioned apps that are explicitly promoted to production when you make updates to the assets.