Why Your Data Platform Is Missing Its Most Important Layer

Your organization spent millions on Databricks, Snowflake, or a carefully assembled stack of Azure services. Leadership declared the data platform “modern.” And yet, every time a data team needs to build a new data product, they start from scratch — creating repositories, wiring up pipelines, configuring access, figuring out deployment. Somewhere between one and three weeks later, depending on how much manual provisioning and access management is involved, they write their first line of transformation logic.

The platform exists. The developer experience on top of it doesn’t. That gap is what this article is about.

The problem isn’t the platform — it’s the layer above it

In software engineering, the distinction between infrastructure and developer experience was settled years ago. Teams don’t just get access to Kubernetes — they get what’s called an Internal Developer Platform. An IDP is a layer that sits on top of infrastructure and gives teams self-service access to templates, environments, and guardrails. Tools like Backstage, Port, and Humanitec provide this layer — standardizing how teams create services, provision resources, and discover what already exists.

Data teams have nothing equivalent. They have a data platform — compute, storage, orchestration — but no standardized way to use it. No templates. No catalogue that tells you what data products exist, who owns them, and whether they’re healthy.

Even platforms designed to reduce friction don’t close this gap. Microsoft Fabric integrates compute, lakehouse, and BI into a single offering — but it doesn’t tell your teams how to structure projects, test data, or deploy consistently. Self-service tools like AWS data.all solve the provisioning and data sharing slice — but teams still start each project figuring out conventions, testing patterns, and deployment approaches on their own.

This isn’t a tooling gap. It’s a conceptual one. Most organizations conflate “data platform” with “data developer platform.” They’ve invested in the engine but haven’t built the cockpit.

I’ve come to believe this is the single biggest source of friction in data organizations today — and the one that gets the least attention.

Four patterns that signal the gap

If you lead a data organization of any size, you’ve seen at least three of these:

The reference project problem. Somewhere in your organization, there’s one team’s repository that has become the unofficial template. Everyone forks it. Nobody maintains it. It carries forward outdated patterns, hardcoded configurations, and assumptions that made sense eighteen months ago. New teams inherit technical debt on day one before they’ve written a single transformation.

The platform team as ticket queue. Your platform or infrastructure team receives requests for new environments, storage accounts, compute clusters, access grants. Each request takes days or weeks to fulfil. After the initial effort of building the platform, the platform team — which should be building capabilities — shifts its focus to processing tickets. They become a bottleneck, and the data teams they’re supposed to enable start routing around them, finding shortcuts to achieve what they need.

The governance spreadsheets. Compliance tracking lives in spreadsheets initially filled in by the teams when requesting resources. Someone from the governance or architecture team periodically reviews projects, flags issues, and updates a spreadsheet that’s perpetually out of date. What happens when an audit comes? It always starts with the same step: preparing an inventory from scratch.

The onboarding cliff. A new data engineer joins. There’s no standardized project structure to learn, no consistent patterns across teams, no documentation that reflects how things actually work. They can commit code within weeks, but making a productive contribution that accounts for how things connect, what exists where, and why decisions were made — that takes months. And it depends entirely on which team they land on and how good that team’s internal documentation happens to be.

The cost model directors need to see

The most visible cost is engineering time. A conservative model: on a typical data team, 1–2 engineers — usually the tech lead or a senior engineer — spend 2–3 weeks on project scaffolding and infrastructure troubleshooting for each new data product. If the team delivers 3–4 products per year, that’s 6–12 weeks of senior engineering time annually spent on solved problems — per team. Scale that across 10 data teams in a large organization and you’re looking at 60–120 weeks of engineering time per year burned on project setup. That’s 1–2 senior engineers’ worth of capacity, and it’s your most experienced people doing the least valuable work.

But the less visible costs are larger:

Quality variance. Without standardized patterns, every team makes independent decisions about data validation, testing, documentation, and deployment. Some teams test rigorously. Some don’t test at all. Some document their data products with lineage and SLAs. Some leave behind tribal knowledge that evaporates when people move on. The variance isn’t random — it’s structural. You can’t get consistent quality without consistent infrastructure.

Governance overhead. When every project is structured differently, enforcing standards requires manual review. Your governance function either becomes a bottleneck — reviewing everything before production — or becomes performative, setting standards that nobody follows because following them is harder than ignoring them. Processes like handing over products from development to maintenance take ages, and maintenance requires more skill and time when every product differs in structure and quality.

Compounding drift. Every month without standardization, the problem gets worse. More projects go live with different structures, different naming conventions, different deployment patterns. The cost of retroactively standardizing grows exponentially. Organizations that wait too long find themselves facing a multi-quarter remediation effort that could have been avoided.

From weeks of setup to writing logic on day one

Rather than listing components, let me walk through what the experience should be.

A data engineer on the commercial analytics team needs to build a new customer churn data product. Today, they open a blank repository and spend a day figuring out which conventions apply: asking on Slack, reading outdated wiki pages, digging through other teams’ repos to see what actually works. They copy a CI/CD pipeline YAML from a colleague’s project and spend another day debugging it — the service connection doesn’t exist yet, the runner doesn’t have the right permissions, the secret store isn’t accessible from the pipeline. They file tickets for storage and compute provisioning, wait for access grants, and eventually, maybe one to two weeks later, start writing business logic.

With a data IDP, they open the platform portal. They select “New Data Product”, choose a blueprint (say, dbt + Databricks), read a short document describing what they’ll get at the end, and fill in metadata: domain, owner, SLA tier, data classification level. They click create.

Within minutes, they have:

• A repository scaffolded with the organization’s dbt project structure, including shared macros, naming conventions, and quality check configurations
• CI/CD pipelines pre-configured to run data tests, linting, and quality gates on every pull request
• A catalogue entry registered with lineage placeholders, ownership metadata, and SLA tracking
• Provisioned storage and compute with access policies that match the data classification
• Monitoring dashboards connected and ready for custom alerting thresholds

Right away they can clone the git repository, connect to their new Databricks workspace, and verify that the pipelines in the repository actually work — a deployed “Hello world” version of the product. They’re writing transformation logic the same day, not the same month. The infrastructure decisions that used to consume weeks have been encoded into the blueprint — made once, maintained centrally, and applied consistently.

This is what the software world calls a golden path, a term coined at Spotify, where Backstage was originally built. It means a recommended way of doing things — not the only way, but the easy way. Teams can deviate if they have good reasons, but the default path should be so convenient that most teams won’t want to.

The scaffolding problem is worse for AI/ML — and most organizations aren’t ready

Every large organization is under pressure to put AI into production. But the scaffolding problem for ML projects is significantly worse than for analytics. Model training infrastructure, experiment tracking, feature engineering pipelines, model registry, deployment and serving, monitoring for drift and performance degradation — the setup burden is enormous and almost never standardized.

Without an IDP, each ML team reinvents this stack. One team uses MLflow, another considers Weights & Biases (because they don’t know about MLflow, or don’t know whether they’re allowed to use it), a third builds something custom. Deployment patterns are inconsistent. There’s no shared approach to model governance, versioning, or rollback. The result: organizations that claim to be “doing AI” but can’t reliably move models from experimentation to production.

A data IDP addresses this directly. A “New ML Project” blueprint could provision an experiment tracking workspace, connect to the organization’s feature store, register a model registry entry, configure training compute with appropriate cost policies, and wire up a deployment pipeline with automated monitoring. The difference between “we’re experimenting with AI” and “we’re shipping AI products at scale” is often exactly this kind of operational infrastructure.

Now, an obvious question: what if a team needs something fancier — say, GPU-heavy experiments with significant cost implications? An IDP doesn’t mean everyone can have everything. There’s still a place for approvals and permissions so that impactful actions are properly gated. But here’s the difference: when approval is required, the approver gets structured information about what’s being requested and why. And once that approval is given, everything is provisioned automatically — no email chain to the infrastructure team, no manual resource creation, no two-week wait.

Governance by design, not governance by review

This is the argument that I think resonates most with directors, architects, and compliance stakeholders — and it’s the one that most IDP discussions underserve.

The traditional model is governance by review: a central team examines projects before they reach production, checking for compliance with standards, data classification, access controls, documentation completeness. This model doesn’t scale. It creates bottlenecks, breeds resentment between delivery teams and governance functions, and still misses issues because reviewers can’t deeply assess every project.

The IDP model enables governance by design. When every project is created from a blueprint, you can:

Enforce data classification at creation time. The blueprint asks for classification level and applies corresponding access policies, encryption settings, and retention rules automatically. There’s no possibility of accidentally deploying a PII dataset with public access.

Inject quality gates into every pipeline by default. Every CI/CD pipeline created from a blueprint includes data validation, schema checks, and quality thresholds. Teams can add to these gates, but they can’t remove the baseline. Minimum quality is a property of the platform, not a choice left to individual teams.

Auto-generate documentation and lineage. The template structure produces documentation artefacts as a side effect of normal development. Catalogue entries, lineage graphs, and dependency maps are populated automatically. Documentation isn’t a separate task — it’s built into the workflow.

Create an auditable trail by default. Every project created through the IDP has a traceable history: when it was created, from which blueprint version, what metadata was provided, what approvals were granted. When audit season arrives, the evidence already exists.

This shifts the governance conversation from “how do we review everything” to “how do we encode our standards into the platform so that compliance is the default state.”

When this doesn’t apply

I should be honest about the limits. A data IDP isn’t the right investment for every organization.

If you have a small data team — say, under ten people — the overhead of building and maintaining an IDP probably isn’t justified. A well-maintained Cookiecutter template, a solid wiki page, and consistent code reviews can get you 70–80% of the way there. The IDP investment starts to pay off when you have multiple teams, multiple project types, and enough organizational surface area that consistency can’t be maintained through conversation alone.

Similarly, if your organization only ships a handful of data products per year, the cost model I described earlier doesn’t work in your favour. The setup time you’d save doesn’t offset the platform investment. This is a scale problem — and it requires a certain scale to justify.

The question I’d ask is: are your teams solving the same infrastructure problems independently, more than twice? If yes, you have a platform problem. If not, you probably just have a documentation problem.

Start with the pain, not the vision

Adopting a data IDP is as much an organizational challenge as a technical one. Here’s what the path typically looks like.

Don’t pitch the platform — pitch the problem it solves. “We’re building an Internal Developer Platform” means nothing to most stakeholders. “We’re going to cut project setup time from weeks to a single day” means everything. Solve the problem teams already complain about. The broader platform vision emerges as you accumulate capabilities.

Fund it as a product, not a project. An IDP isn’t a one-time build — it’s a product that requires ongoing investment. The platform team needs to treat internal data teams as customers, gather feedback, iterate on templates, and continuously improve. This means dedicated headcount, not part-time allocation from an already-stretched infrastructure team.

Build one or two blueprints first. Don’t try to cover every project type on day one. Identify the most common data product pattern in your organization — probably a dbt analytics project or a batch pipeline — and build a single, polished blueprint for it. Get three teams using it. Gather feedback. Improve. Then expand to ML projects, streaming pipelines, and other patterns.

Lead with developer experience, let governance follow. If you lead with “this will improve our governance posture,” data teams will see the IDP as another compliance burden. If you lead with “this will make your life easier,” they’ll adopt it willingly — and governance comes along for free because it’s embedded in the platform.

Address the ownership question early. Who funds this? Where does the platform team sit? Options include a dedicated Data Platform Engineering team, a Centre of Excellence model, or embedding platform engineers within a central data organization. The right answer depends on your structure, but the wrong answer is “nobody owns it.”

What to do Monday morning

If any of this resonated, here’s a concrete starting point. Pick the most common type of data product your teams build. Talk to three teams that built one recently and ask them how long the setup took, what they had to figure out on their own, and what they copied from somewhere else. Write down the steps they went through before they wrote their first line of business logic.

That list of steps is your first blueprint. Not all of it will be automatable on day one — but you’ll have a clear picture of the friction, and you’ll be able to show anyone who asks exactly where the time goes. In my experience, that’s usually enough to start the conversation that matters.