Skip to main content
Creating an Engineer API grants you the ability to transform data within your desired app. It leverages the existing data transformation and preparation rules, which modify data to meet specific goals. This could include a rule to summarize the data, remove duplicate rows, or convert the data format.

​
Prerequisites

  • Appropriate permission: nx1_engineer
  • An existing domain created using DataHub
  • A previously ingested dataset within a DataHub domain

​
Select a dataset and describe a transformation rule

The catalog contains metadata about the schema and table. The schema describes the data structure, such as the table name, column names, and data types. The table contains the data.
  1. Log in to NexusOne.
  2. On the NexusOne homepage, click Engineer.
  3. If you are working with multiple catalogs, then click Federated > Select catalogs and choose at least two catalogs. Else, click Lakehouse > Select catalogs and choose the Iceberg catalog.
  4. Select schemas from each catalog.
  5. Select tables from each catalog.
You describe a rule using natural language so that NexusOne can generate an SQL query.
  1. Enter a transformation job name.
  2. Enter a natural language transformation prompt.
  3. Optional: Select the Show preview? checkbox if you’d like a preview of your data after running the generated query from the prompt input you have provided.
  4. Click Transform. If you selected the preview checkbox in the previous step, then also click Finalize to proceed to the next step.

​
Schedule a transformation rule

Scheduling a rule allows you to run the rule at specific time intervals using Apache AirFlow.
  1. Select a destination schema.
  2. Select a destination table.
  3. Select a DataHub domain.
  4. Optional: Select or create one or more tags to label this rule.
  5. Select a time interval for how often the transformation rule should run. Schedule options include None, Every 3 hours, Daily, Weekly, Monthly, and Quarterly. On the first schedule, the job/DAG on Apache Airflow automatically runs. Recurrent runs depend on your selected schedule option.
  6. Select a mode for how to store incoming records at the destination table.
    • Append: Add new records.
    • Merge: Add or update existing records where applicable.
    • Overwrite: Replace all existing records.

​
Create an Engineer API

You create an API so you can transform data within your desired app.
  1. Select the previously chosen DataHub domain.
  2. Enter a name for the API.
  3. Select the nx1_engineer role.
  4. Click Create.
  5. View your Data API and API key:
    • On the NexusOne homepage, navigate to Connect > Data API to view your Data API.
    • Click API Keys from the navigation sidebar to view your API key. If you have a long list of keys present, use the filter by description or key field to find the API key.

​
Additional resources