CI/CD pipelines fail for dozens of reasons unrelated to code quality—network timeouts, resource exhaustion, flaky tests. Yet analytics tools treat every red build as a defect. Discover how to filter infrastructure noise and measure true engineering performance.
TL;DR: DORA metrics like Change Failure Rate and Lead Time for Changes depend on CI/CD pipeline telemetry. But pipelines fail for infrastructure reasons (AWS outages, Redis timeouts, flaky tests) 41% of the time—not code defects. Traditional analytics tools can't distinguish signal from noise, inflating CFR by 267% and hiding real engineering problems. The Keypup MCP Server filters infrastructure pollution automatically, reconstructing accurate DORA metrics from corrupted pipeline data.
DevOps teams measure software delivery performance with DORA metrics:
These metrics are critical for understanding engineering effectiveness. But they all depend on CI/CD pipeline telemetry—build statuses, test results, deployment outcomes.
The assumption: Pipeline failures indicate code defects.
The reality: Pipelines fail for dozens of reasons unrelated to code quality.
"Our CFR jumped from 6% to 18% in three months. Management thought code quality collapsed. Turns out 70% of failures were AWS RDS connection timeouts during integration tests. Not a single actual production defect."
Modern pipelines integrate with:
Any of these can introduce non-deterministic failures that corrupt your metrics:
Each source adds noise to your pipeline telemetry. Analytics tools count them all as "failures," distorting DORA metrics.
When infrastructure pollution goes unmeasured, DORA metrics lose their predictive power.
"We hit our target of <10% CFR for six straight months. Got a $50K team bonus. Then production melted down with a 4-hour outage from a bad database migration. Our CFR was a lie—half the 'failures' were flaky Selenium tests, not real bugs."
Here's what happens when you measure DORA metrics on polluted pipeline data:
Show me our pipeline telemetry pollution metrics for the last 90 days
What this means:
When your Change Failure Rate is 3.6x higher than reality, you can't:
CFR is calculated as:
CFR = (Failed Deployments) / (Total Deployments)
But "failed deployment" is detected by pipeline status:
This works only if pipeline failures correlate with production defects. In practice:
"We have 143 integration tests. 22 of them are flaky—they fail randomly 10-30% of the time due to Stripe API test mode returning 500s, Redis cache misses in parallel test runs, and Selenium timing issues. Every flaky test failure blocks a deploy and inflates our CFR."
Compare our real Change Failure Rate vs reported CFR over the last 6 months
What's happening:
Traditional analytics platforms see the red line and conclude:
"Code quality is deteriorating. We need mandatory code reviews, stricter linting, more testing."
Reality:
"AWS had maintenance windows. Our CI runners were under-provisioned. Redis connection pooling is misconfigured."
The wrong diagnosis leads to the wrong solution. Teams add more tests (increasing flakiness), more review stages (slowing Lead Time), and more quality gates (reducing Deployment Frequency)—all while the actual problem (infrastructure instability) remains unfixed.
Lead Time for Changes measures:
Lead Time = (Deployment Timestamp) - (First Commit Timestamp)
But if pipelines fail due to infrastructure, developers rerun the same build multiple times until it passes.
"Our E2E tests timeout 40% of the time because Puppeteer can't reach our staging environment through the VPN. I click 're-run failed jobs' 3-5 times until it passes. My actual coding time: 2 hours. Pipeline Lead Time: 8 hours."
Not all repositories suffer equally. Flakiness concentrates in specific services:
Rank our repositories by pipeline pollution and CFR distortion levels
Patterns emerge:
The repositories with the most critical business logic (payments, auth) have the worst metric quality—not because the code is worse, but because they integrate with more external systems.
Traditional analytics platforms can't make this distinction. They report:
"payment-service has 4x higher CFR than frontend-app. Reassign senior engineers to fix code quality issues."
Reality:
"payment-service needs better test isolation, service mocking, and retry strategies for external API calls."
Here's the breakdown of what actually causes pipeline failures:
Analyze our infrastructure failure patterns and their impact on pipeline telemetry
The pollution sources:
max_connections reached during parallel test executionNone of these indicate code defects. All of them inflate CFR.
"We migrated to Kubernetes. Our test suite started timing out randomly because pods weren't getting scheduled fast enough. Analytics dashboard showed a 40% spike in 'failures.' Leadership questioned our release process. The code was identical—infrastructure just got more complex."
Infrastructure issues don't appear overnight. They accumulate as systems scale:
Show me our pipeline stability degradation trend over the last 12 weeks
Key inflection points:
This is the infrastructure debt tax: As systems grow, pipeline reliability declines even if code quality improves.
Traditional DORA metrics would show:
"Deployment Frequency down 35%, Lead Time up 2.3x, CFR up 267%. Engineering productivity has collapsed."
Reality:
"Infrastructure complexity increased. Telemetry pollution grew. Metrics became unreliable."
Most analytics platforms try to solve flaky test problems with:
"Identify tests that fail inconsistently and mark them."
Problem: Identifying flakes doesn't remove them from CFR calculations. You still have polluted metrics.
"Automatically rerun failed tests to filter out transient failures."
Problem: Retries hide symptoms but don't fix the underlying instability. Metrics still reflect the post-retry outcome, not the infrastructure quality.
"Disable flaky tests until they're fixed."
Problem: Now you're measuring a smaller test suite that doesn't represent production risk. CFR becomes artificially low.
"Let teams label failures as 'infrastructure' vs 'code.'"
Problem: Doesn't scale. Requires discipline that breaks down under pressure. Developers game the system by blaming infrastructure.
"Just use production incident rates instead of pipeline failures."
Problem: Production failures are lagging indicators. By the time you detect a problem, customers are impacted. DORA metrics exist to catch issues earlier.
None of these approaches automatically filter infrastructure noise at scale while preserving the predictive power of DORA metrics.
The Keypup MCP Server solves pipeline pollution by correlating multiple data sources to distinguish code defects from infrastructure failures.
Instead of trusting pipeline status alone, Keypup MCP synthesizes:
Pipeline Telemetry
Infrastructure Events
Test Result Metadata
Historical Patterns
Production Monitoring
Scenario: A deployment fails CI/CD pipeline with "Test suite timeout."
Traditional Tool: "CFR +1. Code quality issue."
Keypup MCP Analysis:
Result: "Infrastructure failure, not code defect. CFR unchanged."
This approach works automatically—no manual classification, no test quarantine, no gaming the system.
The Keypup MCP Server integrates with MCP-compatible AI assistants (Claude Desktop, Cursor, ChatGPT Desktop, VS Code) to give you infrastructure-filtered DORA metrics on demand.
{
"mcpServers": {
"keypup": {
"url": "https://hq.keypup.io/mcp",
"headers": {
"Authorization": "Bearer YOUR_TOKEN_HERE"
}
}
}
}
"What's our real CFR after filtering out infrastructure failures?" "Which repositories have the worst pipeline pollution?" "Show me the impact of last week's AWS outage on our metrics"
"What percentage of our pipeline failures are infrastructure vs code defects?"
Keypup MCP correlates error signatures, timing, and test history to classify each failure automatically.
"Compare our reported CFR vs infrastructure-filtered CFR over the last quarter"
Get the dual-line chart (like above) showing how much your metrics are polluted.
"Which repositories are most impacted by flaky tests and infrastructure timeouts?"
Identify where to focus infrastructure improvements for maximum metric quality gain.
"What are our top infrastructure failure patterns causing false pipeline failures?"
Get the breakdown (database timeouts, S3 rate limits, CI runner OOM) with frequency and impact.
"Show me our pipeline stability trend over the last 12 weeks"
Visualize how infrastructure debt accumulates and where major degradations occurred.
"How many of our 'high CFR' weeks had corresponding production incidents?"
Validate whether your CFR spikes actually correlate with customer-impacting defects.
With Keypup MCP, your DORA metrics become actionable again:
CFR = (Deployments with Production Impact) / (Total Deployments)
Excludes:
Result: CFR that actually measures code quality, not infrastructure stability.
Lead Time = (First Green Build) - (First Commit)
Ignores:
Result: Lead Time that reflects actual development work, not infrastructure waiting time.
Deployment Frequency = Successful Deploys / Time Period
Counts only:
Result: Deployment Frequency that measures delivery capability, not infrastructure luck.
41% of CI/CD pipeline failures are caused by infrastructure instability, not code defects—but traditional analytics tools treat them identically. This pollutes DORA metrics:
You can't fix this by:
The solution is multi-signal correlation: Use pipeline telemetry, infrastructure events, test history, error signatures, and production monitoring together to automatically distinguish code defects from infrastructure noise.
That's what the Keypup MCP Server does—automatically.
Want accurate DORA metrics without infrastructure pollution? Try the Keypup MCP Server—it reconstructs ground truth from noisy pipeline data.
Join teams already using AI to make data-driven decisions faster than ever.
Squash merges, interactive rebases, and monorepos systematically corrupt SDLC analytics. Discover how modern Git workflows distort metrics by up to 95%—and how to get accurate insights anyway.
SDLC analytics tools excel at showing problems but fail at solving them. Learn how conversational AI analytics transforms "red dashboards" into actionable workflows that actually improve engineering performance.
Lines of code don't measure architectural thinking, code review depth, or mentoring. Discover how Keypup's MCP Server reveals the invisible glue work that automated metrics miss—and why your most valuable engineers often appear least productive.