Skip to content

refactor: decouple sink type dispatch from PG source OIDs - #98

Merged
dariomazzitellireplik-coder merged 2 commits into
mainfrom
refactor/decouple-sinks-from-pg-source-types
May 16, 2026
Merged

dariomazzitellireplik-coder merged 2 commits into
mainfrom
refactor/decouple-sinks-from-pg-source-types

Conversation

@dariomazzitellireplik-coder

Copy link
Copy Markdown
Collaborator

Summary

Sinks now dispatch primarily on DataType (source-agnostic) instead of pg_type_id. PostgreSQL OIDs become an optional refinement carried by Postgres sources only; non-PG sources (MySQL today) populate None. The previous magic-zero (pg_type_id == 0) sentinel coupled every sink to PG OID semantics and silently broke MySQL → Snowflake type fidelity.

Tagging this v2.4.0 at merge.

OpenSpec change: decouple-sinks-from-pg-source-types.

Changes

Shape change

  • SourceColumn::pg_type_id: u32 → Option<u32>. PG sources populate Some(oid), MySQL populates None. No more magic zero.

Sink-side dispatch

  • Snowflake: rewrote variant_extract_expr(col_name, &DataType, Option<u32>). DataType is the primary axis; OID is consulted only for cases the PG→DataType boundary collapses (arrays, money).
  • StarRocks: was already DataType-primary in production; documented pg_type_to_starrocks / pg_text_to_json as legacy refinement-only.
  • PostgreSQL: rewrote column_type(&SourceColumn) to dispatch on DataType, with OID refinement only for String/Text (varchar/inet/cidr fidelity). Renamed mysql_data_type_to_pg → data_type_to_pg.

Bugs fixed

  • Value::Timestamp in PG sink raw table now serializes as ISO-8601 string (was raw int64). Parity with SR/SF sinks; unblocks MySQL TIMESTAMP→PG once MySQL emits Value::Timestamp.
  • pg_type_to_data_type now maps 1082 (date), 1083/1266 (time/timetz) to DataType::Date/Time instead of collapsing to String. Sinks dispatching on DataType produce accurate DDL without consulting OIDs.

Migration

  • _dbmazz._schema_tracking auto-migrates from v1 to v2 on first daemon start: adds nullable data_type JSONB, drops NOT NULL on pg_type_id, backfills data_type for existing rows. Idempotent. Forward-compatible — old binaries still read the v2 schema (NULL data_type falls back to pg_type_to_data_type(pg_type_id)).

Verification

Local matrix via ez-cdc verify:

Combo Result Notes
PG → PG 17/1/0 C11 fails on created_at TZ display (pre-existing CLI test bug — column was timestamp with time zone on source)
PG → StarRocks 17/0/1 All green; C11 7/7 types pass
MySQL → PG 17/1/0 C11 fails because MySQL source doesn't emit SchemaChange events (known BETA limitation)
MySQL → StarRocks 17/0/1 All green; C11 7/7 types pass — validates the Value::Timestamp ISO fix and Date/Time DataType mapping

Zero regressions from this refactor. Both C11 failures are pre-existing issues outside scope.

Static checks:

  • cargo fmt --all -- --check clean
  • cargo clippy --features mysql-source -- -D warnings clean
  • cargo clippy -- -D warnings clean
  • cargo test --features mysql-source → 248 passed / 0 failed
  • cargo test → 158 passed / 0 failed

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --features mysql-source -- -D warnings
  • cargo clippy -- -D warnings
  • cargo test --features mysql-source
  • cargo test
  • ez-cdc verify --source demo-pg --sink demo-pg-target
  • ez-cdc verify --source demo-pg --sink demo-starrocks
  • ez-cdc verify --source local-mysql --sink demo-pg-target
  • ez-cdc verify --source local-mysql --sink demo-starrocks
  • ez-cdc verify against MySQL → Snowflake (deferred — no SF creds in dev-stack)
  • Legacy v1 _schema_tracking migration tested against a synthetic v1 table (deferred to follow-up integration test)

Sinks now dispatch on DataType (source-agnostic) instead of pg_type_id.
SourceColumn::pg_type_id becomes Option<u32>, populated by Postgres sources
only; non-PG sources (MySQL today) populate None. The previous magic-zero
sentinel coupled every sink to PG OID semantics and silently broke
MySQL-Snowflake type fidelity.

Also fixes Value::Timestamp serialization in the PG sink raw table
(was raw int64, now ISO-8601 string, parity with SR/SF sinks), and
extends pg_type_to_data_type to map 1082/1083/1266 to Date/Time so
non-PG sources can stop falling back to OID-only refinement.

_dbmazz._schema_tracking schema auto-migrates from v1 to v2 on first
daemon start: adds nullable data_type JSONB column, drops NOT NULL on
pg_type_id, backfills data_type for existing rows. Idempotent and
forward-compatible.

Verify matrix run: PG-PG / PG-SR / MySQL-PG / MySQL-SR — 0 regressions.
Two C11 failures are pre-existing CLI/MySQL-beta limitations outside
this change's scope.

Renames internal connectors::sinks::postgres::types::mysql_data_type_to_pg
to data_type_to_pg (never was MySQL-specific).
@dariomazzitellireplik-coder
dariomazzitellireplik-coder merged commit ae2b669 into main May 16, 2026
2 checks passed
@dariomazzitellireplik-coder
dariomazzitellireplik-coder deleted the refactor/decouple-sinks-from-pg-source-types branch May 16, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants