> ## Documentation Index > Fetch the complete documentation index at: https://docs.nx1cloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Create PyAirbyte ingestion job > Create a new PyAirbyte ingestion job as an Airflow DAG that reads from a specified source and writes to Iceberg tables. ## OpenAPI ````yaml post /api/airbyte/ingest openapi: 3.1.0 info: title: Nx1 AI API description: | AI API for Nx1 Data Platform Management and Automated Data Tasks. Authentication is required via PSK in Authorization header. Default PSK is | [ask a friend] | version: 0.10.2 servers: [] security: [] paths: /api/airbyte/ingest: post: tags: - PyAirbyte ingestion summary: Create PyAirbyte ingestion job description: |- Create a new PyAirbyte ingestion job as an Airflow DAG that reads from a specified source and writes to Iceberg tables. operationId: create_ingest_api_airbyte_ingest_post requestBody: content: application/json: schema: $ref: '#/components/schemas/AirbyteIngestRequest' required: true responses: '200': description: Ingestion job created successfully. content: application/json: schema: $ref: '#/components/schemas/IngestResponse' '400': description: Invalid request. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '500': description: Server error. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' security: - OAuth2AuthorizationCodeBearer: [] - APIKeyHeader: [] components: schemas: AirbyteIngestRequest: properties: name: type: string title: Name description: Name of the ingestion job. source_name: type: string title: Source Name description: PyAirbyte source identifier. For example, 'source-github'. source_config: additionalProperties: type: string type: object title: Source Config description: Source configuration key-value pairs, including secrets. schema_name: type: string title: Schema Name description: Target Iceberg schema where the job creates tables. streams: anyOf: - items: type: string type: array - type: 'null' title: Streams description: >- Optional list of streams to sync. If empty, NexusOne syncs all streams. mode: $ref: '#/components/schemas/IngestModeEnum' description: Write mode. merge: anyOf: - type: string - type: 'null' title: Merge description: >- Comma-separated merge key columns. Defaults to Airbyte primary keys in merge mode. schedule: anyOf: - type: string - type: 'null' title: Schedule description: Cron expression for scheduled runs. domain: anyOf: - type: string - type: 'null' title: Domain description: DataHub domain to assign to ingested tables. tags: anyOf: - items: type: string type: array - type: 'null' title: Tags description: DataHub tags to assign to ingested tables. owner_id: anyOf: - type: string - type: 'null' title: Owner Id description: Owner user ID. column_transformations: anyOf: - items: $ref: '#/components/schemas/ColumnTransformation' type: array - type: 'null' title: Column Transformations description: >- Optional per-stream column transformations. Use the 'stream' field on each transformation to target a specific stream. archive_staging: type: boolean title: Archive Staging description: >- Archive staging files to archive/{run_id}/ instead of deleting them after ingestion. default: true add_ingestion_metadata: type: boolean title: Add Ingestion Metadata description: >- Add ingestion metadata columns to output tables: `ingestion_time`, `ingestion_run_id`, `ingestion_mode`, and `dag_job_id`. default: true type: object required: - name - source_name - source_config - schema_name - mode title: AirbyteIngestRequest description: Request model for creating an Airbyte ingestion job. IngestResponse: properties: job_id: anyOf: - type: string format: uuid - type: 'null' title: Job Id description: A unique ID of the created ingestion job. flow_url: type: string title: Flow Url description: Airflow URL to access the ingestion flow. error: anyOf: - type: string - type: 'null' title: Error description: Error message if the ingestion fails due to any issue. type: object required: - flow_url title: IngestResponse ErrorResponse: properties: error: type: string title: Error description: A brief description of the error that occurred. code: type: integer title: Code description: The HTTP status code associated with the error. default: 500 type: object required: - error title: ErrorResponse HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError IngestModeEnum: type: string enum: - append - overwrite - merge title: IngestModeEnum description: Defines how the job writes data to a target table. ColumnTransformation: properties: stream: anyOf: - type: string - type: 'null' title: Stream description: >- Target stream name. For Airbyte multi-stream ingestion only. The field has no effect on single-table ingestion. column: type: string title: Column description: The source column name to transform. transformation_type: $ref: '#/components/schemas/ColumnTransformationType' description: >- Type of transformation to apply, such as `cast`, `rename`, or `encrypt`. target_type: anyOf: - $ref: '#/components/schemas/SparkDataType' - type: 'null' description: >- Target data type for cast transformation. Required when `transformation_type` is `cast`. new_name: anyOf: - type: string - type: 'null' title: New Name description: >- New column name for rename transformation. Required when `transformation_type` is `rename`. encryption_key_name: anyOf: - type: string - type: 'null' title: Encryption Key Name description: >- Optional encryption key name when using the `vault_encrypt` User Defined Function (UDF). If not provided, uses the default key. type: object required: - column - transformation_type title: ColumnTransformation description: |- Defines a transformation to apply to a column during data ingestion. Supported transformations: - cast: Convert column to a different data type - rename: Rename the column - encrypt: Encrypt the column value - drop: Remove the column from the output ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError ColumnTransformationType: type: string enum: - cast - rename - encrypt - drop title: ColumnTransformationType description: Type of column transformation to apply during ingestion. SparkDataType: type: string enum: - string - int - integer - long - bigint - short - smallint - byte - tinyint - float - double - decimal - boolean - date - timestamp - binary title: SparkDataType description: Supported Spark SQL data types for casting. securitySchemes: OAuth2AuthorizationCodeBearer: type: oauth2 flows: authorizationCode: scopes: {} authorizationUrl: >- https://sso-rapid.rapid.nx1cloud.com/realms/rapid/protocol/openid-connect/auth tokenUrl: >- https://sso-rapid.rapid.nx1cloud.com/realms/rapid/protocol/openid-connect/token APIKeyHeader: type: apiKey in: header name: Authorization-PSK ````