> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nx1cloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Debezium server in NexusOne

> Overview of Debezium Server, a streaming server that streams CDC events into Apache Iceberg tables in NexusOne.

Debezium server is a lightweight, standalone runtime for Change Data Captures (CDC).
It streams database changes from relational databases, NoSQL stores, and message
systems directly to a target sink. In this case, the sink is a lakehouse implemented
using Apache Iceberg tables.

With the Debezium server, you have no need for a full Kafka Connect stack because the
server bundles all components into a single unit. The NexusOne implementation of this
server includes a specialized sink that writes CDC events into Apache Iceberg tables
that make up the lakehouse. The implementation uses the
[Debezium Iceberg Consumer](https://github.com/memiiso/debezium-server-iceberg).

## Environments and runtime specifications

This section describes the NexusOne environments you can deploy Debezium server instances,
and the runtime specifications of the Debezium server, such as the software versions,
supported connectors, and default configurations.

### Deployment context

Within NexusOne, you can deploy Debezium server instances in the following environments:

* **The NexusOne portal**: When using the [Ingest feature](/documentation/data-pipeline/overview/ingest)
  within the portal, it includes a data mirroring sub-feature for capturing
  streams of database changes. This sub-feature deploys a Debezium server when it's used.
* **Public cloud or on-premises**: When you purchase NexusOne to deploy into your public
  cloud or on-premises, it comes pre-packaged with a Debezium server.

### Current version information

* Debezium server: `3.1.1`
* Debezium Iceberg Extension: `0.9.0`
* Apache Iceberg: `1.8.1`

### Default configurations

By default, within NexusOne, the Debezium server writes CDC events to an
Iceberg-compliant lakehouse composed of a Hive metastore and S3-compatible object
storage. This configuration provides the following:

* ACID transaction guarantees through the Iceberg table format
* Schema evolution support for handling source schema changes
* Time travel capabilities for historical data access
* Automatic metadata management via Hive metastore
* Scalable storage on S3-compatible infrastructure

### Platform-managed default settings

NexusOne manages the following default configurations for the Debezium
Server and Iceberg sink:

* **Snapshot mode**: `initial`
  <Note> The `initial` mode performs a full table snapshot followed by streaming changes</Note>
* **Batch processing**: Optimized batch sizes for lakehouse ingestion
* **Connection pooling**: Pre-configured for typical workloads
* **Error handling**: Automatic retry mechanisms with exponential backoff

### Supported source connectors

NexusOne's Debezium server deployment supports the following database connectors:

* **PostgreSQL**: Captures row-level changes using logical replication
* **SQL Server**: Streams changes via SQL Server CDC or Change Tracking
* **MySQL**: Monitors binary logs for data modification events
* **Oracle**: Tracks changes through LogMiner or Oracle GoldenGate

Each connector provides full support for `INSERT`, `UPDATE`, and `DELETE`
operations with before or after state capture, where applicable.

## Source database configuration

Debezium server relies on a source database to produce change events in a format it
can consume. Proper configuration of each database ensures that each insert, update,
and delete is actually captured reliably and delivered to the lakehouse.

Prerequisites and setup procedures vary by database vendor, and they include enabling
CDC features, configuring replication slots, and setting appropriate permissions.

To ensure that Debezium server can capture changes reliably, each database vendor
requires specific configuration and user permissions. The following outlines the
required settings for both aspects.

* **Database configuration**: Enable CDC mechanisms specific to your database platform.
  Use any of the following:
  * **PostgreSQL**: Requires logical replication configuration
  * **SQL Server**: Needs CDC or Change Tracking enabled
  * **MySQL**: Requires binary logging with `ROW` format
  * **Oracle**: Requires supplemental logging configuration
* **User permissions**: Grant appropriate privileges for CDC operations.
  Use the following:
  * Replication permissions for reading change streams
  * Metadata access for schema discovery
  * Connection privileges for establishing monitoring sessions

The following sections provide detailed, step-by-step configuration instructions for
each database vendor to implement the previously listed requirements.

### PostgreSQL configuration

PostgreSQL uses logical replication to expose change events.

* Set `wal_level` to `logical` in the `postgresql.conf` file.
* Create a replication slot for Debezium to consume changes.
* Install the `pgoutput` logical decoding plugin. The plugin is available by default
  in PostgreSQL 10+.
* Grant the following replication permissions to the Debezium user:

  ```sql theme={null}
  CREATE ROLE debezium_user WITH REPLICATION LOGIN PASSWORD 'password';
  GRANT SELECT ON ALL TABLES IN SCHEMA schema_name TO debezium_user;
  ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name GRANT SELECT ON TABLES TO debezium_user;
  ```

### SQL server configuration

SQL Server requires that you enable CDC at both the database and table levels.

* Enable CDC at the database level:

  ```sql theme={null}
  USE DatabaseName;
  EXEC sys.sp_cdc_enable_db;
  ```

* Enable CDC for specific tables:

  ```sql theme={null}
  EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name = N'TableName',
    @role_name = NULL;
  ```

* Grant appropriate permissions to the Debezium user:

  ```sql theme={null}
  CREATE LOGIN debezium_user WITH PASSWORD = 'password';
  CREATE USER debezium_user FOR LOGIN debezium_user;
  EXEC sp_addrolemember 'db_owner', 'debezium_user';
  ```

### MySQL configuration

MySQL uses binary logging to track changes.

* Enable binary logging in `my.cnf`:

  ```bash theme={null}
  server-id = 1
  log_bin = mysql-bin
  binlog_format = ROW
  binlog_row_image = FULL
  expire_logs_days = 10
  ```

* Grant replication permissions:

  ```sql theme={null}
  CREATE USER 'debezium_user'@'%' IDENTIFIED BY 'password';
  GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT 
  ON *.* TO 'debezium_user'@'%';
  FLUSH PRIVILEGES;
  ```

### Oracle configuration

Oracle requires supplemental logging and appropriate permissions.

* Enable database-level supplemental logging:

  ```sql theme={null}
  ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
  ```

* Enable table-level supplemental logging for specific tables:

  ```sql theme={null}
  ALTER TABLE schema.table ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
  ```

* Grant required permissions:

  ```sql theme={null}
  GRANT CREATE SESSION TO debezium_user;
  GRANT SELECT ON V_$DATABASE TO debezium_user;
  GRANT SELECT_CATALOG_ROLE TO debezium_user;
  GRANT EXECUTE_CATALOG_ROLE TO debezium_user;
  GRANT SELECT ANY TRANSACTION TO debezium_user;
  GRANT LOGMINING TO debezium_user;
  ```

For more comprehensive setup instructions, troubleshooting guidance,
and platform-specific considerations, refer to the
[Debezium connector documentation](https://debezium.io/documentation/reference/stable/connectors/index.html).

## Deployment in the NexusOne portal

As [previously described](#deployment-context), you can deploy and configure Debezium
Server instances by using the NexusOne [Ingest feature](/documentation/data-pipeline/overview/ingest)
within the portal, which includes a data mirroring sub-feature with customizable
connector settings, transformation rules, and sink destinations.

The following image shows the user interface of the data mirroring sub-feature
in the NexusOne portal.

<Card img="https://mintcdn.com/nexusone-4c77570d/vrZm_KEZD20WN8df/images/platform-components/debezium-server/01-data-mirroring.png?fit=max&auto=format&n=vrZm_KEZD20WN8df&q=85&s=bdae7619a32e39bc8c2ac0e4adeca2f4" width="1926" height="1690" data-path="images/platform-components/debezium-server/01-data-mirroring.png">
  Data mirroring in the NexusOne portal
</Card>

You can do the following when interacting with this feature:

* Configure source connector settings:
  * Select a database type such as PostgreSQL, SQL Server, MySQL, or Oracle.
  * Provide connection details such as the hostname, port, or database name.
  * Configure authentication credentials.
  * Specify schema and table filters.
* Configure sink destination:
  * Use the default pre-populated Iceberg lakehouse configuration.
  * Customize target schema and table names.
  * Set up a partition strategy for large tables.
* Configure optional settings:
  * Select a snapshot mode for initial data load.
  * Set a heartbeat interval for connection monitoring.
  * Set a schema evolution handling.
  * Set an error handling and retry policy.
* Review the configuration summary and deploy.
* Monitor deployment status and verify a successful connection.

## Understanding the CDC event structure

The Debezium server produces structured change events that contain the before
and after states of data, along with metadata about change operations.

### Event structure components

Each Debezium CDC event contains components that describe the change and provide
context for downstream consumers. The following components include:

* **Envelope**: Wraps the CDC event in a standard structure. The structure
  comprises the following:
  * `before`: Row state before the change, for `INSERT` operations, this field is `null`
  * `after`: Row state after the change, for `DELETE` operations, this field is `null`
  * `op`: Operation type: `c` indicates create, `u` indicates update, `d` indicates delete,
    and `r` indicates a read operation for snapshot events.
  * `ts_ms`: Timestamp of the change in milliseconds
  * `source`: Metadata about the source system and position
* **Source metadata**: Provides traceability for each event. It includes the following:
  * Database name and schema
  * Table name
  * Transaction ID or Log Sequence Number (LSN)
  * Timestamp from source system
  * Connector name and version
* **Schema information**: Describes the structure of the before or after values.
  It includes the following:
  * Field names and data types
  * Optional/required indicators
  * Default values were applicable

### Iceberg sink transformation

When streaming events into Iceberg tables in a lakehouse, the event undergoes a
transformation that converts raw events into a structured table format by:

* Using the `op` field to determine the type of operation, such as an `INSERT`, `UPDATE`, or `DELETE`
* Flattening the `before` and `after` values into columns
* Optionally including metadata columns for audit and lineage tracking
* Storing Debezium coordinates for idempotency
* Extracting partition keys for efficient query performance

## Additional resources

* To learn about best practices when using Debezium, refer to the
  [Debezium best practices](./debezium-server-best-practices) page.
* To learn how to troubleshoot common Debezium server issues in the NexusOne environment, refer to the
  [Troubleshoot common Debezium server issues](./trouble-shoot-common-debezium-server-issues.mdx) page.
* For more details, refer to the official [Debezium documentation](https://debezium.io/documentation/)
  and [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/). The Debezium community
  also provides an extensive guide on
  [source connector configurations](https://debezium.io/documentation/reference/stable/connectors/index.html).
* If you are using the NexusOne portal and want to learn how to use the Debezium server, refer to the
  [Ingest](/documentation/data-pipeline/overview/ingest) page.