Skip to main content
Ingesting a file into the NexusOne platform means uploading or adding a public URL to the file so the platform can process and analyze it. Once ingested, the file is automatically integrated into your data pipeline workflow. This guide walks you through how to ingest a file.

​
Prerequisites

  • Appropriate permission: nx1_ingest, nx1_monitor, nx1_s3_admin, airflow_user, superset_user, spark_sql, and trino_admin
  • Ensure you are ingesting the files NexusOne currently supports.

​
Add or upload a file

You add or upload a file so NexusOne can process and analyze it. This automatically integrates the file into your data pipeline workflow for later use.
  1. Log in to NexusOne.
  2. On the top navigation bar, hover your mouse over Data Pipeline and then select Ingest.
  3. Click File.
  4. Add a file
    • If you are uploading a file, then click Choose File and select the file that you want to upload.
    • If you are adding a public URL to a file, then click Public File URL and enter the public URL to the file you’d like to upload. This file might be in an S3 bucket and exposed over HTTPS.
  5. Add ingest details. These fields define how the NexusOne ingests and accesses the data through a schema and table.
    Ensure that the values added to your Schema and Table adhere to Apache Spark’s identifiers.
    • Enter a unique name for this ingest job. This name appears in the Monitor tab and Airflow’s portal for tracking.
    • Choose an existing database schema or enter a new schema name to create one.
    • Choose an existing table or enter a new table name.
    • Optional: Set how often the ingestion job should run. Schedule options include Run Once, Every 3 hours, Daily, Weekly, Monthly, and Quarterly.
    • Select a mode for how to store incoming records at the destination table.
      • Append: Add new records to the existing dataset.
      • Merge: Add or update existing records where applicable.
      • Overwrite: Replace all existing records.
    • Optional: Select a DataHub domain. For example, Company, Product, Sales. This is only applicable if you have one previously. created on DataHub via the Govern feature on NexusOne.
    • Optional: Select or create one or more tags to label this job.
  6. Optional: Column transformations:
    • Click Add Transformation, then select any of the following column transformation types:
      • Cast: Converts a column’s data type during ingestion. To use this transformation type, enter the column name in the Column field, and then select a target type in the Target Type.
      • Drop: Removes a column from the dataset. To use this transformation type, enter the column name in the Column field.
      • Encrypt: Makes the data unreadable using an encryption key. To use this transformation type, enter the column name in the Column field, and then optionally enter a key name in the Key Name field.
      • Rename: Changes a column’s name. To use this transformation type, enter the column name in the Column field, and then enter a new column name in the New Name field.
    • Repeat until you have added all the transformations necessary for your use case.
  7. After configuring all fields, click Ingest to submit the job.

​
Monitor, trigger, or delete a job

When you schedule an ingested file, it runs as a job in Apache Airflow. You can monitor, trigger, or delete the job.
  1. When you create a job, a success message and a View Jobs button appear.
  2. Track the job status by clicking View Jobs or navigating to the NexusOne homepage and clicking Monitor.
  3. Use the three dots ... menu to trigger or delete a job.
  4. If you clicked Trigger job, then click the job’s name to open its DAG details in Airflow’s portal.

​
Additional resources