NT
  • About
  • Skills
  • Work
  • Experience
  • Achievements
  • Contact
zz
← All work

Case study · 02

Warehouse migration MCP tool

An MCP server that automates moving a data warehouse, starting with Snowflake to Databricks.

Role
Builder · Nuthan Technologies
Status
Shipped
Read
1 min
Stack
PythonFastMCPFastAPIasyncioSQLiteReactSnowflakeDatabricks

2–3 days

of manual scripting replaced per migration

Jump toProblemApproachResultsNext

Problem

Every cross-warehouse migration meant two to three days of hand-written scripting, repeated for each project.

Approach

  1. 0110 MCP tools over stateless Streamable HTTP. The React UI is itself an MCP client, and any AI client can drive the same tools.
  2. 02No LLM in the migration path: every move is deterministic and auditable.
  3. 03An async job engine: submitting a migration returns a job ID immediately, a background task runs it, and a SQLite (WAL) job store feeds live logs over server-sent events.
  4. 04A streaming path moves 100k-row Arrow and Pandas chunks through a bounded queue for backpressure. An optional S3-Parquet staging path moves data warehouse to warehouse with no rows passing through Python.
  5. 05Per-warehouse work: Snowflake Arrow batches instead of LIMIT/OFFSET rescans, ROW_NUMBER paging for Synapse, Databricks COPY INTO with a fallback, and a type mapper across 5 sources and 4 destinations.
No LLM in the migration path: every move is deterministic and auditable.

Results

  • Replaces 2–3 days of manual scripting per migration.
  • Covers Snowflake, Databricks, BigQuery, Redshift and Synapse.

What I'd do next

  • →Add dry-run validation reports before any data moves.

← Previous

Enterprise healthcare analytics platform

Next →

ChartHaven