open to work

case study · event streaming · 2026

Change data capture pipeline

Terraform infrastructure that turns PostgreSQL row changes into Kafka events with Debezium on MSK Connect—without application polling or a NAT gateway.

read the source ↗

problem

Publish database changes without changing application code

Polling adds delay and repeated database work. This pipeline reads PostgreSQL's write-ahead log instead, producing one event for each insert, update or delete while normal SQL remains unchanged.

The infrastructure also needed to be reproducible and cost-conscious, with the connector plugin available privately and no fixed NAT gateway charge.

architecture

RDS WAL to Debezium to Amazon MSK

CDC pipeline architecture: RDS PostgreSQL write-ahead log streamed by Debezium on MSK Connect into Amazon MSK topics
RDS PostgreSQL exposes logical replication; Debezium on MSK Connect reads the WAL and publishes table events to Amazon MSK. S3 stores the custom connector plugin and CloudWatch receives worker logs.

A dedicated VPC spans two availability zones. One self-referencing security group allows RDS, MSK and MSK Connect to communicate, while an S3 gateway endpoint lets Connect retrieve the plugin without internet egress.

decisions and tradeoffs

Key design decisions

Log-based CDC instead of polling
WAL events preserve operation type and before/after values with lower source-database overhead. The tradeoff is managing replication slots and WAL retention.
MSK Connect instead of a self-managed worker
A managed connector removes host maintenance, but gives less runtime control and requires AWS-specific plugin packaging.
S3 gateway endpoint instead of NAT
The connector retrieves its custom plugin privately without the hourly NAT cost. This path only solves S3 access, not arbitrary outbound traffic.
Connector-created topics
Broker auto-creation stays disabled; explicit connector settings create captured-table topics with the intended partitions and replication factor.

what broke

Logical replication is not enabled by default

Debezium cannot create a replication slot against a standard RDS PostgreSQL configuration. A custom parameter group sets rds.logical_replication = 1, and the instance must reboot before the setting takes effect.

The plugin also cannot be uploaded as the downloaded tarball. The apply step fetches Debezium, repacks its JARs into the flat ZIP expected by MSK Connect, uploads it to S3 and registers that object as a custom plugin.

scope

What the pipeline provisions