Skip to main content
In NexusOne, a lakehouse refers to an internal database with a catalog and table format. Ingesting a lakehouse means copying previously ingested data from a table in that internal database into another table. This preserves the table format and catalog metadata so that downstream apps like Trino can query it. This guide walks you through how to ingest a lakehouse.

​
Prerequisites

Before starting this task, ensure you have the following:
  • Required roles: nx1_ingest.
  • Previously ingested a dataset.
  • Ranger policies that your platform administrator creates for you:
    • A policy on the schema that holds the source table. It should grant the select permission.
    • A policy on the schema you ingest into, in the iceberg catalog. It should grant the alter, create, drop, insert, select, and update permissions.
  • When you create both Ranger policies, assign them to the nx1_ingest role.

​
Ingest a lakehouse

Specify a table containing a previously ingested dataset, so NexusOne can copy it into another table.
  1. Log in to NexusOne.
  2. On the top navigation bar, hover your mouse over Data, and then select Ingest.
  3. Click New pipeline.
  4. In the Source step, select Lakehouse table.
  5. Select the table that contains the previously ingested data:
    • Catalog: Select the catalog that holds the table.
    • Schema: Select the lakehouse schema.
    • Table: Select the table.
  6. Optional: To copy only some rows, click Add condition under Only rows where, and then select a column, a comparison, and a value. For example, select region, =, and EU to copy only rows where region is EU.
  7. Click Next.
  8. Optional: In the Preview and schema step, click Add a column change, enter a column name, and then select what to do with the column:
    • Load as: Select the data type to load the column as.
    • Rename: Enter the new column name.
    • Encrypt: Optionally, enter an encryption key name.
    • Drop: Leave the column out of the new table.
  9. Click Next.
  10. In the Destination step, add ingest details containing information about the new table:
    Ensure that the values added to your Schema and Table adhere to Apache Spark’s identifiers.
    • Schema: Enter a new schema name.
    • Table: Enter a new table name.
    • Mode: Select how to store incoming records at the destination table.
      • Append: Add new records to the existing dataset.
      • Merge: Add or update existing records where applicable.
      • Overwrite: Replace all existing records.
    • Pipeline name: Enter a unique name for this pipeline. This name appears on the Ingest page and in Airflow’s portal for tracking.
    • Schedule: Set how often the pipeline should run. Schedule options include Once, Hourly, Daily at…, Weekly…, and Custom cron.
    When you are done, click Next.
  11. In the Governance step, add optional details about the data, and then click Next:
    • Domain: Select a DataHub domain. For example, Company, Product, Sales. This is only applicable if you have one previously created on DataHub via the Govern feature on NexusOne.
    • Tags: Select or create one or more tags to label this pipeline.
  12. In the Review step, review your choices, and then click Create pipeline to finish.

​
Monitor, trigger, or delete a job

When you schedule an ingested lakehouse, it runs as a job in Apache Airflow. You can monitor, trigger, or delete the job.
  1. When you create a pipeline, NexusOne opens the Ingest page, which lists your pipelines in a table.
  2. Click the All tab to show pipelines with any status.
  3. Select Mine to show only your pipelines.
  4. Click your pipeline’s name to open its page.
  5. Click the Runs tab to track each run. The Result column shows the status of a run.
  6. Use the buttons at the top of the page to manage the pipeline:
    • Run now: Triggers a run outside its schedule.
    • Pause or Resume: Stops or restarts the pipeline’s schedule.
    • Open in Airflow: Opens the pipeline’s DAG details in Airflow’s portal.
    • The three dots ... menu, then Delete pipeline: Deletes the pipeline.

​
Additional resources