Ingest a lakehouse

This tutorial walks you through how to copy a table you already ingested into a new table in the NexusOne platform.

​

Prerequisite

Before starting this tutorial, ensure you have the following:

• Required roles: nx1_ingest and superset_user.
• A table from the Add a CSV file from a public URL tutorial. This tutorial uses its csv_url_schema.csv_url_table table as the source.
• While configuring variables to deploy NexusOne, your ranger_repo_mode variable has the value shared. This ensures you have the nx1-unifiedsql service name. NexusOne supports other values too, but this tutorial uses shared.
• Two separate Ranger policies that your platform administrator creates for you:
  • A policy on the csv_url_schema schema. It should grant the select permission.
  • A policy on the schema you ingest into, such as csv_schema_dest in the iceberg catalog. It should grant the alter, create, drop, insert, select, and update permissions.
• When you create both Ranger policies, assign them to the nx1_ingest or superset_user role.

​

Select the source table

1. Log in to NexusOne.
2. On the top navigation bar, hover your mouse over Data, and then select Ingest.
3. Click New pipeline.
4. In the Source step, select Lakehouse table.
5. Select the table you want to copy:
   • Catalog: iceberg
   • Schema: csv_url_schema
   • Table: csv_url_table
6. Click Next.

​

Add ingest details

1. In the Preview and schema step, don’t add any column changes, click Next.
2. In the Destination step, enter the following information in the fields:
   • Schema: csv_schema_dest
   • Table: csv_table_dest
   • Mode: Append
   • Pipeline name: lakehouse_copy
   • Schedule: Once
3. Click Next.
4. In the Governance step, don’t add any tags or domain, click Next.
5. In the Review step, click Create pipeline.

​

Monitor the pipeline

When you create the pipeline, NexusOne creates an Airflow job and opens the Ingest page, which lists your pipelines in a table. Since you set the pipeline to run once, as soon as it’s created, you can monitor the run using the following steps:

1. On the Ingest page, click the All tab to show pipelines with any status.
2. Select Mine to show only your pipelines.
3. In the table, find your pipeline, lakehouse_copy, and click it.
4. On the pipeline’s page, click the Runs tab.
5. Find the latest run in the list, and watch its status in the Result column.
6. Wait for a few minutes and refresh your browser until the status in the Result column changes to Succeeded.

​

Visualize your dataset

Use the following steps to visualize your dataset:

1. On the top navigation bar, hover your mouse over Platform, and then select Tools.
2. On the Superset card, click SQL Lab to launch Superset’s SQL Lab.
3. Enter the following command in the query box:
   SELECT * FROM csv_schema_dest.csv_table_dest
   

​

Additional resources

• To get an overview of what ingestion is, refer to Data ingestion overview.
• For general instructions about how to ingest a lakehouse in NexusOne, refer to Ingest a lakehouse.
• For more information about roles or permissions, refer to Govern Overview.