Skip to main content
Ingesting a database into the NexusOne platform means copying a dataset from an existing database into the NexusOne platform. Once ingested, the dataset becomes available for your data pipeline workflow.

​
Prerequisites

Before starting this task, ensure you have the following:
  • Required roles: nx1_ingest.
  • Ensure you are ingesting a database that NexusOne currently supports.
  • A Ranger policy that your platform administrator creates for you. It should be on the schema you ingest into, in the iceberg catalog, and grant the alter, create, drop, insert, select, and update permissions.
  • When you create the Ranger policy, assign it to the nx1_ingest role.

​
Add datasets from a database

You add the connection details and the table or query to read, so NexusOne can copy the dataset from the database.
  1. Log in to NexusOne.
  2. On the top navigation bar, hover your mouse over Data, and then select Ingest.
  3. Click New pipeline.
  4. In the Source step, select Database.
  5. Enter the connection details:
    • Engine: Select the type of database. NexusOne fills in the default port for that type.
    • Host: Enter the database host.
    • Port: Enter the database port.
    • Database: Enter the database name.
    • Username: Enter a username.
    • Password: Enter a password. NexusOne stores it as a secret and never shows it again.
  6. Select what to copy:
    • A table: Click A table, and then enter the schema and table in the Table field. For example, public.orders.
    • A query: Click A query, and then enter an SQL query in the Query field.
  7. Click Next.
  8. Optional: In the Preview and schema step, NexusOne shows the columns after the first run. To change a column, click Add a column change, enter a column name, and then select what to do with it:
    • Load as: Select the data type to load the column as.
    • Rename: Enter the new column name.
    • Encrypt: Optionally, enter an encryption key name.
    • Drop: Leave the column out of the new table.
  9. Click Next.
  10. In the Destination step, add ingest details containing information about the new table:
    Ensure that the values added to your Schema and Table adhere to Apache Spark’s identifiers.
    • Schema: Enter a new schema name.
    • Table: Enter a new table name.
    • Mode: Select how to store incoming records at the destination table.
      • Append: Add new records to the existing dataset.
      • Merge: Add or update existing records where applicable.
      • Overwrite: Replace all existing records.
    • Pipeline name: Enter a unique name for this pipeline. This name appears on the Ingest page and in Airflow’s portal for tracking.
    • Schedule: Set how often the pipeline should run. Schedule options include Once, Hourly, Daily at…, Weekly…, and Custom cron.
    When you are done, click Next.
  11. In the Governance step, add optional details about the data, and then click Next:
    • Domain: Select a DataHub domain. For example, Company, Product, Sales. This is only applicable if you have one previously created on DataHub via the Govern feature on NexusOne.
    • Tags: Select or create one or more tags to label this pipeline.
  12. In the Review step, review your choices, and then click Create pipeline to finish.

​
Monitor, trigger, or delete a job

When you schedule an ingested database, it runs as a job in Apache Airflow. You can monitor, trigger, or delete the job.
  1. When you create a pipeline, NexusOne opens the Ingest page, which lists your pipelines in a table.
  2. Click the All tab to show pipelines with any status.
  3. Select Mine to show only your pipelines.
  4. Click your pipeline’s name to open its page.
  5. Click the Runs tab to track each run. The Result column shows the status of a run.
  6. Use the buttons at the top of the page to manage the pipeline:
    • Run now: Triggers a run outside its schedule.
    • Pause or Resume: Stops or restarts the pipeline’s schedule.
    • Open in Airflow: Opens the pipeline’s DAG details in Airflow’s portal.
    • The three dots ... menu, then Delete pipeline: Deletes the pipeline.

​
Additional resources