Migrating to Databricks: A Practical Guide from Snowflake, Teradata, Hadoop, and Legacy Data Warehouses

If your data engineering team is migrating from Snowflake, Teradata, or Hadoop to Databricks this year, the case for the lakehouse is already made. What you need now is a clear view of what breaks during the move, how long it realistically takes, and which pitfalls are specific to the platform you’re leaving.

This is written for a data engineering lead or architect who already has a name for this project internally. It is not for someone still building the initial business case.

This guide skips the platform tour. It opens with a shared decision framework that applies no matter where you are migrating from. After that, it breaks down the concrete pitfalls for the three source platforms with the highest migration volume: Snowflake, Teradata, and Hadoop.

Oracle, Redshift, and Informatica get a lighter treatment further down. The underlying decisions are the same across all five paths, even though the specific mechanics differ by platform. Because this guide goes deeper than a typical single-topic post, treat the sections below as modular.

Read the framework, then the platform section that matches your migration, plus the FAQ.

What Changes When You Move to Databricks

Every Databricks migration, regardless of source platform, forces the same set of decisions. Getting these right before diving into platform-specific pitfalls saves time later. Skipping any one of them tends to resurface as a fire drill well after go-live, once real workloads hit the gaps.

• Compute and storage separate

Legacy warehouses and MPP appliances typically bundle compute and storage into one unit. Databricks splits them: Delta Lake sits on cloud object storage as the fixed point, while compute scales independently through clusters and SQL warehouses.

Teams used to a fixed-capacity appliance often underestimate how much cluster and warehouse sizing becomes an ongoing tuning exercise rather than a one-time provisioning decision. In practice, this means cluster policies and auto-scaling settings become part of ongoing platform operations, not a line item you finish during the migration itself. This split is also why Databricks pricing conversations feel unfamiliar at first. You are pricing compute and storage independently, sometimes across several compute types in the same workspace, rather than one bundled capacity number.

• Catalog and semantics move to Unity Catalog

Whatever governs access and lineage today, a Teradata data dictionary, Snowflake role-based access control, or a Hive metastore, does not map cleanly onto Unity Catalog. A Databricks Unity Catalog migration is fundamentally a governance redesign project. Plan the catalog structure, metastores, catalogs, schemas, and grants, as its own workstream, separate from the data move itself.

Unity Catalog’s three-level namespace, catalog, schema, table, replaces whatever two-level or platform-specific hierarchy you use today. Renaming everything to fit is mechanical. Deciding how many catalogs to create, and along what boundary, environment, business unit, sensitivity level, is the actual governance decision. It is worth making deliberately rather than defaulting to a single catalog for convenience.

• Security and network architecture change too

Legacy platforms usually authenticate through local users, on-premises Active Directory, or network-level firewalls. Databricks relies on IAM roles or service principals, workspace-level network configuration, and private connectivity options like PrivateLink. Mapping the old model onto the new one is a security exercise that usually needs compliance sign-off alongside the technical work.

• Observability and monitoring need rebuilding

Legacy platforms usually ship built-in monitoring, Teradata Viewpoint, Snowflake’s query history and resource monitors, that teams have relied on for years. Databricks monitoring is assembled from cluster event logs, Unity Catalog system tables, and job run history, and it usually needs deliberate setup rather than arriving pre-configured.

• CI/CD and deployment pipelines shift as well

Stored procedures and SQL scripts typically deploy through change-control processes built around the legacy platform’s own tooling. Databricks work usually deploys through notebooks, Databricks Asset Bundles, or a CI/CD pipeline built around version-controlled code. That is a genuinely different release process, and teams should budget time to rebuild it rather than assume it carries over.

• Team skills shift from SQL-only to Spark-aware

Teams that have spent years writing stored procedures or dialect-specific SQL often need real ramp-up time on PySpark, Spark SQL, and notebook-based development. This is a training and hiring question as much as a technical one. Plan for it explicitly rather than assuming the team will pick it up mid-migration.

• The cost model shifts, and DBUs are only part of it

Whatever you currently pay for compute, storage, and licensing does not translate directly into a Databricks bill. For the full cost model breakdown, see our companion piece, the Databricks Cost Optimization Guide.

• Dependencies need mapping before cutover, not during it

Every report, downstream job, and scheduled export that reads from your current platform is a dependency that can break silently after cutover. A dashboard nobody has opened in two years is still a dependency until someone confirms otherwise.

Most Databricks migration tool options fall into two camps. Code converters translate SQL and stored procedures automatically. Profilers map dependencies and estimate effort up front. Neither replaces a manual review of the business logic in your riskiest workloads, which matters most for the Hadoop path covered below.

The practical starting point is a full inventory of every table, job, report, and downstream consumer on your current platform. Tag each one by how confident anyone is in what it actually does. Workloads with high confidence move through the standard migration process. Workloads with low confidence go through the reconstruction step described in the Hadoop section below, regardless of which platform they actually run on today.

Validation methodology deserves its own plan, not a single yes/no check at the end. It usually combines row count reconciliation, checksum or hash comparison on sampled data, and a defined user acceptance period where business users sign off before cutover. Teams that treat validation as a checklist item rather than a workstream are the ones who run into trouble later. See the section on where migrations break, further down in this guide.

Whether you tackle this in-house or bring in Databricks migration services depends largely on capacity. How much dependency mapping and code translation can your team absorb without disrupting other work? Most organizations underestimate this by a wide margin.

Choosing a Migration Approach: Sequencing and Scope

Two decisions shape a Databricks migration project more than the source platform does: how much you redesign versus port as-is, and which layer you migrate first.

• Lift-and-shift versus modernization

A lift-and-shift moves your existing data models and code with minimal changes, which is faster to scope and automate but carries forward any existing inefficiencies. Modernizing redesigns the platform as you migrate, which takes longer but avoids re-doing the work later. Most enterprise migrations end up as a hybrid: lift-and-shift for stable, well-understood workloads, and modernization for the ones being redesigned anyway. Databricks covers this trade-off in more depth in its own migration approaches guidance.

• ETL-first versus BI-first

An ETL-first sequence builds the full Bronze, Silver, Gold pipeline before cutting over reporting, which is thorough but delays visible business value. A BI-first sequence modernizes reporting and dashboards early, using Lakehouse Federation to query the legacy platform while the underlying pipelines are still being migrated.

Databricks documents both sequencing patterns, along with a federate-then-migrate and replicate-then-migrate variant, in its own migration architecture guidance.

A short checklist helps decide which approach fits:

In practice, Snowflake and Teradata migrations tend to suit a BI-first sequence. The source platform can keep serving reports through Lakehouse Federation while pipelines migrate underneath. Hadoop migrations more often need an ETL-first sequence, since the reconstruction work described below has to happen before anything downstream can be trusted.

For most of the five paths in this guide, a BI-first, hybrid approach front-loads the workload triage described above. It shows value early while the higher-risk pieces, undocumented Hadoop logic, Teradata stored procedures, get the deeper treatment they need without holding up the whole project.

Snowflake to Databricks Migration

Snowflake and Databricks both market themselves as cloud-native, which makes it tempting to treat a Snowflake to Databricks migration as a lift-and-shift with new connection strings. It is a bigger change than that. Snowflake is a SQL-native cloud warehouse. Databricks is a lakehouse with a Spark engine underneath, and that difference shows up the moment you migrate anything beyond flat SQL queries.

Timelines for this path vary widely by pipeline count and how much custom SQL and UDF logic needs translating. A narrow BI-only migration moves considerably faster than one with heavy ELT.

The cost model shift needs its own planning pass early in the project timeline. Snowflake bills per second on a fixed warehouse size, using credits. Databricks bills in DBUs, which vary by compute type, all-purpose, jobs, or serverless SQL, and instance family. Mapping credits to DBUs one-to-one on paper is a reasonable starting estimate, but it rarely survives contact with real workload patterns.

Snowflake’s ecosystem features do not always have a like-for-like Databricks equivalent either. Snowpipe’s continuous ingestion maps loosely onto Databricks Auto Loader. Streams and Tasks map loosely onto Delta Live Tables or Lakeflow pipelines. The semantics differ enough that a direct port usually needs adjustment, not a straight rename.

Snowpark workloads, where teams already write Python or Scala against Snowflake, are the exception. They port more directly to PySpark than pure SQL pipelines do, since the underlying programming model is already closer.

Historical data migration itself is usually the easy part. Bulk exports move cleanly into Delta Lake tables. The harder question is what happens to semi-structured VARIANT columns, which need an explicit decision: flatten into typed columns or keep them as JSON strings.

Validation tooling matters here as much as translation tooling. Data diffing tools that compare row counts, checksums, and sample records between Snowflake and Databricks catch parity gaps well before a business user does. Skipping this step is exactly the validation gap covered later in this guide.

Snowflake’s network policies and Databricks’ workspace-level controls, IP access lists, PrivateLink, and Unity Catalog permissions, are not a direct swap either. Budget a short security review alongside the technical migration, ideally with whoever owns network and access policy today.

Teradata to Databricks Migration

Teradata environments are usually the oldest and most entrenched of the three main paths in this guide. That changes the shape of a Teradata to Databricks migration well beyond dialect differences. Timelines here tend to run longer than for Snowflake, driven mainly by stored procedure translation volume and compliance sign-off cycles rather than data volume itself.

• Stored procedure and BTEQ translation is the bulk of the work

Teradata’s BTEQ scripts and stored procedures encode years of business logic in a syntax with no direct Spark SQL equivalent. Automated converters handle a meaningful share of straightforward SQL. Procedural logic, loops, conditional branching, and temporary table chains, usually needs a person to confirm the translated version behaves correctly under the same edge cases.

• Data type mapping deserves its own checklist too

Teradata’s numeric and date-handling types do not always map one-to-one onto Spark SQL types. Silent truncation or rounding differences are the kind of bug that surfaces months after cutover, often in a downstream finance report rather than during testing.

Ingestion tooling has matured enough that most teams do not write custom JDBC extraction code from scratch. Databricks’ Lakeflow Connect and various partner connectors handle bulk and incremental extraction from Teradata, shifting effort from plumbing toward validating what gets extracted.

• MPP tuning assumptions do not carry over cleanly

Teradata’s performance model relies on its massively parallel processing architecture and specific indexing strategies, including primary indexes and join indexes. Spark’s execution model differs enough that a query tuned for years on Teradata can run correctly but slowly on Databricks. It usually needs re-tuning for partitioning and caching instead of indexing.

• Governance carries real weight

Teradata migrations skew toward banking, telecom, and other regulated industries, where the data is often subject to audit requirements that predate the migration decision itself. Whatever access controls, retention rules, and audit logging your compliance team relies on today need an explicit, verified mapping to Unity Catalog. Common examples include data retention schedules tied to regulatory reporting cycles, and row-level access restrictions on customer financial data. Both need an explicit owner assigned for the migration itself.

Regulated environments usually require a formal test plan with sign-off from compliance as well as engineering. Building that sign-off step into the project timeline from the start avoids a late surprise. Legal or risk teams will eventually ask for evidence the migration preserved data integrity.

Hadoop to Databricks Migration

A Hadoop to Databricks migration usually looks different before you even open a single file. Snowflake and Teradata migrations deal with systems that are actively maintained and reasonably well understood by current staff. Hadoop environments are frequently the opposite: years-old MapReduce jobs, Hive queries, and Oozie workflows written by people who left the company long ago. Many of them are still running in production today.

Timelines here are the least predictable of the three, since the range depends entirely on how much undocumented logic the inventory step uncovers.

Hadoop environments rarely consist of just MapReduce and Hive. Sqoop jobs handle ingestion from relational sources, and Pig scripts often run alongside Hive queries. HBase or Impala sometimes sit underneath reporting layers nobody has touched in years. Each of these needs its own inventory entry, not just the obvious MapReduce and Hive jobs.

The technical side of a Hadoop migration is fairly well trodden. HDFS data moves to cloud object storage, and Hive tables map onto Unity Catalog external or managed tables. Most Oozie scheduling logic has a reasonably direct Databricks Jobs equivalent. None of that is what makes Hadoop migrations hard.

Security migration adds its own layer of complexity. Kerberos-based authentication, common in on-premises Hadoop clusters, has no direct Databricks equivalent. Access control usually needs to be rebuilt against Unity Catalog rather than migrated.

Orchestration migrates too. Oozie workflows need mapping onto Databricks Workflows or an external orchestrator like Airflow. The dependency chains between jobs are exactly what gets lost if the earlier inventory step is rushed.

Cost dynamics differ here too. Many Hadoop environments run on fully depreciated, self-managed hardware. The relevant comparison is on-premises total cost of ownership versus cloud consumption. That comparison usually favors Databricks over the long run, but it needs its own analysis rather than borrowing the Snowflake or Teradata framing.

This is the path where undocumented business logic is the real risk, more than technical translation. A job that has run nightly for eight years, with nobody left who fully remembers what it does, is a common example. Its logic needs to be reconstructed before it can be safely reproduced on Databricks.

Teams that skip the reconstruction step often discover the gap during user acceptance testing. That is when a report stops matching what the business expects. By then, the fix costs far more than doing the reconstruction work up front would have.

This is exactly the gap the VELOX Application Modernization Accelerator is built for. Instead of migrating code blind, VELOX reads the existing MapReduce, Hive, and Oozie logic and reconstructs the underlying business requirements before migration touches it. That reconstructed requirement set becomes the spec for what gets rebuilt on Databricks, instead of a best-effort translation of code nobody can fully explain anymore.

In practice, VELOX’s output is a structured requirements document rather than annotated code alone. That document becomes the artifact business stakeholders review and sign off on, closing the same gap that undocumented logic created in the first place.

KMS has published a related account of exactly this scenario: an Azure Databricks migration where the original authors of a 1990s monolith were long gone before the project even started. The lessons there generalize well beyond that one project.

Other Common Migration Paths: Oracle, Redshift, and Informatica

These three paths carry less migration volume than Snowflake, Teradata, and Hadoop. Search volume and project frequency are simply lower than for the top three, and the risk profile per platform tends to be narrower. The underlying decisions are identical: compute and storage separation, Unity Catalog mapping, and dependency triage. Here is what is specific to each platform.

Oracle

An Oracle to Databricks migration inherits Oracle’s dependence on PL/SQL stored procedures, which need manual review rather than pure automated conversion, similar to Teradata’s procedural logic. Licensing is the other factor worth planning around early.

Oracle’s per-core licensing model means the cost conversation starts before the technical migration does. Teams often use that licensing pressure as part of the business case funding the broader project. Migration tooling for PL/SQL is also less mature than for Snowflake or Teradata dialects, so budget more manual review time per line of procedural code.

Redshift

A Redshift to Databricks migration has to account for Redshift’s distribution styles and sort keys, which have no direct equivalent in Delta Lake’s file-based storage model. Workloads tuned around a specific distribution key often need a different partitioning strategy built from scratch. Query patterns built around Redshift’s columnar compression also deserve a review, since Delta Lake handles file layout and compaction differently. Redshift’s workload management queues also need rethinking, since Databricks handles concurrency through cluster and SQL warehouse sizing instead of static queue configuration.

Informatica

An Informatica to Databricks migration is really an ETL mapping migration. Informatica’s visual mapping logic needs translation into code, typically onto Databricks Lakeflow. Translation quality depends heavily on how much custom transformation logic is buried in individual mappings versus how much follows standard patterns.

Mappings with heavy custom scripting or unusual connector chains take the longest to validate, since their logic rarely lives anywhere outside the mapping itself. Some teams use the migration as a natural point to retire Informatica entirely rather than port every mapping, consolidating tooling in the process.

None of these three paths carries Hadoop’s discovery risk. PL/SQL, Redshift SQL, and Informatica mappings are usually still actively maintained by someone who understands them. Oracle and Redshift paths carry similar timeline drivers to Teradata: procedural code translation and licensing negotiations, more than raw data volume. Informatica-only migrations, when the underlying database is already elsewhere, can be considerably shorter, since there is no data platform to move at all.

Where Do Migrations Actually Break?

Across all five paths, two patterns cause more missed timelines than any platform-specific pitfall.

The first is cutting over without parity validation. Teams migrate the data and the pipeline, confirm it runs, and call the migration done. Whether the output actually matches the legacy system, row for row and edge case for edge case, is the real test. That check often only happens after someone downstream notices a discrepancy in a report. Building validation into the plan from day one, instead of treating it as a final sign-off task, catches these gaps while still cheap to fix.

The second is migrating everything instead of triaging by value. Not every workload on your legacy platform is worth migrating as-is. Some should be retired. Some should be redesigned rather than ported. Only the rest should go through the migration process described above. A workload nobody has queried in over a year is usually a good first candidate for retirement rather than migration. Migrating everything at once, without that triage step, is how a well-scoped migration turns into an open-ended one.

Triage decisions are easiest to make using the same inventory built during the framework step above. Confidence level, business value, and query frequency together indicate whether a workload should migrate as-is, get redesigned, or retire.

Both patterns tend to show up together in one telltale sign: a migration plan with a single cutover date and no separate validation milestone. If validation and triage are not separate line items in the project plan, they are probably not happening as separate steps in practice either.

Most timeline slips trace back to one of these two patterns, sometimes both at once. Neither is really a technology problem. Both are project management decisions made too late, or not explicitly made at all.

We covered both patterns in more depth, including how to structure the triage step, in our companion piece on the KMS modernization approach. Worth reading before you finalize your migration plan.

How KMS Reduces the Risk

The pattern above, VELOX for undocumented logic and triage before migrating everything, is also the shape of how KMS approaches a Databricks migration in practice.

VELOX handles the reconstruction problem for Hadoop and other undocumented legacy code, as covered above. The Databricks Migration Accelerator structures the migration itself, dependency mapping, phased cutover, and parity validation, into a repeatable engagement rather than a one-off project. Together, the two are how KMS’s Databricks migration services close the gap between knowing what to migrate and migrating it correctly.

In practice, a KMS-led migration usually runs through five phases:

• Discovery and workload inventory.
• Reconstruction of undocumented logic, where VELOX applies.
• A pilot migration on a contained, low-risk scope.
• Phased migration and parity validation for the remaining workloads.
• Final cutover with a defined rollback plan.

The Hadoop-specific reconstruction phase is the one most generic migration guides skip entirely. Most engagements begin with a short discovery phase that scopes which workloads need VELOX’s reconstruction versus which need only a standard code migration. Engagement length varies with scope, and Hadoop-heavy work with significant VELOX reconstruction tends to run longer than a narrower Snowflake or Teradata migration. The discovery phase is scoped and priced separately from the rest of the engagement, since its findings shape everything that follows.

The full mechanism, including the phase breakdown and the reusable patterns behind it, is covered in our KMS modernization piece instead of repeated here.

Getting Started: The First 30 Days

Whichever path applies to you, the first month of a Databricks migration looks similar regardless of source platform.

• Confirm which of the five migration paths applies, and read only that platform section in depth rather than the whole guide.
• Loop in whoever owns compliance and security sign-off before the technical work starts, not after.
• Decide whether the discovery and pilot work described above happens in-house or through Databricks migration services.
• Get executive sign-off on treating this as a phased project built around a pilot, rather than a single all-at-once cutover.

None of these steps require picking a final migration approach first. Sequencing decisions, lift-and-shift versus modernize, ETL-first versus BI-first, are easier to make once the discovery and pilot phases above are underway. They surface how much undocumented logic and dependency risk you are actually carrying.

Teams that want help with discovery, VELOX reconstruction, or the phased migration itself can start with an assessment rather than a full engagement. Either way, treat the first 30 days as a decision-making and de-risking phase, not a race to cutover.

If you are already on Databricks and the migration is behind you, the FinOps side of this is covered separately. DBU optimization, cluster sizing, and the full cost model live in our Databricks Cost Optimization Guide.