Specialized AI agents help generate and update tests. KMS engineers review and approve every new or changed test before it is used to gate a release, keeping those decisions under human control.
Keep Databricks data quality checks current as sources, schemas, and code change. Catch issues before they affect reporting and AI, while reducing the test maintenance burden on your engineers.
As sources, schemas, and business rules change, data quality checks need to change with them. Keeping those checks current takes ongoing engineering effort.
Databricks Data Quality Squad is a managed service for continuous data quality and pipeline testing. A Senior Data Quality Engineer and a DataOps Engineer oversee test maintenance and execution. They direct specialized AI agents, supported by Velox, KMS’s AI delivery accelerator.
The first 90 days cover assessment, test development, and the transition to ongoing service. Together, we prioritize critical pipelines and agree on responsibilities and success measures. Core coverage includes pipeline and code testing, data quality checks, and integration and end-to-end testing.
Specialized AI agents help generate and update tests. KMS engineers review and approve every new or changed test before it is used to gate a release, keeping those decisions under human control.
Test execution connects to your CI/CD, monitoring, and incident workflows. We agree who owns failure investigation and who coordinates corrective action, so unresolved issues have a defined owner.
Monthly reporting tracks agreed measures such as coverage, detection time, production defects, and testing effort. You can assess progress against your starting point and identify where further improvement is needed.
The squad updates tests as code, schemas, and business rules change. Clear ownership of regression coverage reduces the risk that new pipeline behavior goes unchecked while your team delivers new work.
During the first 90 days, your squad assesses priority pipelines, develops the initial tests, and transitions to ongoing service. The squad then maintains the tests, investigates failures, and reports progress each month.
Receive an assessment of your critical pipelines, tables, existing tests, and coverage gaps. The assessment establishes a quality baseline and identifies coverage priorities. We also agree on quality measures, service levels, and reporting responsibilities so you can assess progress throughout the engagement.
Receive unit tests, data quality rules, and integration and end-to-end tests for agreed pipelines written in PySpark, SQL, or Scala. Checks reflect the requirements of your bronze, silver, and gold layers, where applicable. Engineer-approved tests integrate with your CI/CD workflows, with agreed checks used as quality gates.
Test results and alerts feed into your agreed monitoring and incident workflows. For failures within the agreed scope, the squad triages the issue, investigates the cause, and documents findings and recommended corrective actions. Your engineers can use those findings to address the underlying problem.
Receive a monthly report and executive review covering agreed metrics, actions taken, unresolved issues, and recommended improvements. Each review sets priorities for the next cycle of test maintenance and coverage development.
A Senior Data Quality Engineer and a DataOps Engineer take responsibility for test design, execution, and failure investigation within the agreed scope. Your engineers spend less time coordinating test maintenance as pipelines evolve
Through Velox, KMS’s AI delivery accelerator, specialized agents help create unit tests, develop data quality rules, and support end-to-end pipeline testing. Engineers direct the work and validate outputs, retaining responsibility for testing decisions.
KMS brings rule libraries, configurable rule templates, PySpark unit test kits, CI/CD quality gate templates, and incident runbooks. The squad adapts these assets to your pipelines and business rules, then uses them to support ongoing test development and maintenance.
An optional migration module adds source-to-target reconciliation for migrations from platforms such as Teradata, Oracle, Hadoop, or Snowflake. Your team can identify differences between source and target datasets and investigate discrepancies before using the migrated data.
Optional modules add performance and DBU cost testing, alongside Unity Catalog access, sensitive-data, and lineage checks. You can extend the testing scope around specific platform requirements, with responsibilities and measures agreed before the additional work begins.
KMS combines a strategic Databricks partnership with dedicated data quality and DataOps expertise to maintain tests for your Databricks pipelines, integrate checks into release workflows, and investigate failures within the agreed scope.
Identify the pipelines your business depends on most and where existing tests fall short. Together, we’ll define the initial coverage, responsibilities, and measures for onboarding and the ongoing managed service.
Give your critical pipelines a dedicated owner for data quality testing. Start with a 90-day onboarding, then keep coverage current through ongoing test maintenance, failure investigation, and monthly reporting.
We value your privacy
We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies.
Customise Consent Preferences
Choose which cookies you allow. Necessary cookies are always on because the site needs them to work.
Required for basic site functions, such as remembering your cookie choice.
Enable features such as embedded videos, maps and podcasts.
Help us understand how visitors use the site, such as pages visited and time spent.
Used to show relevant ads and measure ad campaigns.
Contact Us