The Vendor Black Box: How to Measure Outsourced Engineering Teams with Git and Jira Data

The quarterly business review with the vendor goes well. The slides show headcount, utilization and story points delivered. Then someone from the platform team mentions that half of their week goes to reviewing the vendor's pull requests, and someone from security asks which ticket a large change to the payments service belonged to. Nobody in the room has the answer. The contract measures hours. Nobody measures the work.

The Friction: Paying for Capacity You Can't See

Outsourcing engineering is supposed to add capacity without adding management overhead. In large organizations, several forces quietly turn it into a black box:

• The contract measures inputs. Invoices count hours, headcount and seniority. They say nothing about what was merged, how big it was or how often it came back for changes.
• Status comes from the vendor. Progress reports are written by the vendor's delivery manager, from the vendor's own tracking. The client sees a summary, not the work.
• Review is the hidden subsidy. Vendor code still has to be reviewed, and often merged, by in-house engineers. That effort shows up nowhere: not on the invoice, not in the vendor's metrics, not in the in-house team's roadmap.
• Time zones turn every question into a day. A review requested at the end of the vendor's day is answered after the vendor has logged off. Each round trip costs a working day on both sides.

The result: an organization that can tell you exactly what each vendor costs, and almost nothing about what each vendor delivers.

Why This Matters: The Real Cost of the Vendor Black Box

The Hidden Review Load

Every vendor pull request consumes in-house review time. When vendor throughput grows, the in-house team's review load grows with it, and roadmap work slows down for reasons nobody can point to. The vendor looks productive. The in-house team looks slow. Both conclusions are wrong.

The Traceability Problem

Regulated organizations must show which change was made, by whom, for what reason and who approved it. Vendor pull requests that don't reference a ticket, or vendor tickets closed without any code, are exactly the items an auditor, a security team or a procurement team will ask about.

The Renewal Problem

Vendor contracts are renewed, expanded or cut every year. Without delivery data, the decision rests on the vendor's own reporting and on anecdotes. Organizations end up renewing the vendor that writes the best status reports, not the one that delivers the most.

The Enterprise Discussion: What Engineering Leaders Are Actually Saying

"The offshore team is a black box", "we spend more time reviewing their code than writing our own" and "we can't tell if we're getting what we pay for" come up again and again wherever engineering managers, directors and procurement-minded CTOs compare notes. The quotes below are representative of recurring threads on r/ExperiencedDevs, r/EngineeringManagers, r/cscareerquestions and r/sysadmin.

The pattern holds: organizations manage vendors on hours and slide decks, but rarely look at the pull requests, reviews and tickets that show what the vendor actually delivered and what it cost the in-house team.

How Keypup MCP Opens the Vendor Black Box

The Keypup Model Context Protocol (MCP) Server gives your AI assistant structured access to the data Keypup imports from Jira and from GitHub, GitLab, Bitbucket and Azure DevOps. For this analysis, what matters is that each pull request carries its author, repository, creation and merge dates, review request date, first review date, approval count, lines changed and the Jira issues it resolves. Each review carries its reviewer, state (approved, changes requested, commented) and submission date, and each Jira issue its assignees, closure date and resolution state: whether its linked pull requests were all merged, are still open, or don't exist.

Keypup doesn't know which accounts belong to a vendor: that's a condition of the whole analysis. You provide the lists, the Git usernames and Jira accounts of each vendor's engineers, typically exported from your identity provider or the vendor contract, and include them in the prompt. A pull request is a vendor pull request when its author is in a vendor list. An in-house review is a review submitted by an account that isn't. A pull request is linked when it references a Jira issue key with a closing keyword such as "fixes", "closes" or "resolves". All figures below cover the last 12 full months, November 2025 to October 2026, for two vendors: a nearshore partner (Vendor A) and an offshore partner (Vendor B).

1. Sizing the Vendor Share of Delivery

The first question: how much of what you ship comes from vendors, and how much of your own team's effort goes into it?

MCP Prompt:

Using these vendor account lists (Vendor A: …, Vendor B: …), for
pull requests merged in the last 12 full months, show the share
of merged pull requests and lines changed authored by each vendor
vs in-house, the share of each group linked to a Jira issue, and
the share of reviews submitted by in-house engineers that went to
vendor pull requests.

Output: How Much of Our Delivery Comes From Vendors?

Key Insight: Vendors author 41% of merged pull requests and 52% of lines changed, and they absorb 46% of the reviews your in-house engineers submit. That review effort is a real cost of the vendor contract, paid by your own team, and it appears on no invoice.

2. Comparing the Delivery Profile of a Pull Request

Averages per vendor hide how the work arrives. A dot plot puts each group side by side on the same measures.

MCP Prompt:

For pull requests merged in the last 12 full months, compare
in-house, Vendor A and Vendor B as a dot plot: median lines
changed, median business hours from review request to first
review (America/New_York, 9–17), changes-requested reviews per
pull request, median business days from creation to merge, and
the share merged with no approval.

Output: What Does a Vendor Pull Request Look Like?

Key Insight: Vendor B's pull requests are 2.7 times larger, wait 4.5 times longer for a first review and get changes requested almost three times as often. That isn't a verdict on Vendor B's engineers. It's the signature of large batches handed over across time zones, and the fix is usually smaller pull requests and an overlapping review window, not a new vendor. The 11% merged with no approval deserves a separate look: check whether those repositories enforce branch protection.

3. Measuring the Time-Zone Tax on Reviews

A median hides the shape of the wait. The distribution shows how often a vendor pull request loses a full working day.

MCP Prompt:

As a grouped column chart, show the distribution of business hours
between review request and first review (America/New_York, 9–17)
for pull requests merged in the last 12 full months, for in-house,
Vendor A and Vendor B: under 2, 2–8, 8–16, 16+. Add the share of
each group waiting 8 business hours or more.

Output: How Long Do Vendor Pull Requests Wait for a First Review?

Key Insight: 64% of Vendor B's pull requests wait a full working day or more for their first review, against 23% in-house. Vendor A, which overlaps the in-house team by several hours, sits much closer to in-house. The time-zone tax is real and measurable, and it's paid in calendar days on every round of review. Business hours are computed in one time zone, the in-house team's: change it in the prompt to see the wait from the vendor's side.

4. Tracking the Review Load Over Time

Vendor contracts usually grow. The question is whether the in-house review effort grows with them.

MCP Prompt:

By month of merge over the last 12 full months, show as bars the
number of vendor pull requests merged (both vendors), and as a
line the number of reviews submitted by in-house engineers per
vendor pull request.

Output: Is the Vendor Review Load Growing?

Key Insight: Vendor throughput rose 41% in a year, and each vendor pull request now takes 2.3 in-house reviews instead of 1.7. Together, in-house review effort on vendor code nearly doubled, from about 1,040 to 1,980 reviews a month. If the in-house roadmap slowed down over the same period, this is a large part of the reason, and it belongs in the next contract discussion.

5. Mapping Where Vendor Code Lands and Who Checks It

The organization-wide view hides the risky spots. A scorecard by repository shows where vendor code concentrates and where in-house oversight is thin.

MCP Prompt:

For the 8 repositories with the most vendor lines changed in the
last 12 full months, show as a heatmap the vendor share of lines
changed, and for vendor pull requests: the share linked to a Jira
issue, the share with at least one review by an in-house engineer,
changes-requested reviews per pull request and median business
days from creation to merge.

Output: Where Does Vendor Code Land, and Who Checks It?

Key Insight: partner-integrations is 82% vendor code, and only 39% of its vendor pull requests were reviewed by anyone in-house: the vendor effectively reviews and merges its own work there. payments-api has the most rework and the slowest merges. core-ledger and identity-svc are in-house territory, with tight review. The risk sits where vendor share is high and in-house review is low: that's where knowledge leaves with the vendor at the end of the contract.

6. Checking Traceability in Both Directions

Auditors ask two questions: which ticket justified this change, and which change delivered this ticket?

MCP Prompt:

For the last 12 full months, show two donuts: vendor pull requests
merged, split by whether they are linked to a Jira issue; and Jira
issues closed while assigned to a vendor Jira account, split by
resolution state: all linked pull requests merged, a linked pull
request still open, or no merged linked pull request.

Output: Can We Trace What the Vendor Delivered?

Key Insight: 3,640 vendor pull requests changed code that nobody can tie to a ticket, and 1,890 vendor tickets were closed without any merged pull request. Some of those are legitimate: configuration, investigation, documentation, or pull requests that reference the issue without a closing keyword. The 370 tickets closed while their code was still under review are not: the board says done, the code isn't merged. Ask the assistant for the list per repository and per Jira project, and agree with each vendor on a linking convention.

7. Listing Vendor Work Waiting on Your Team

The last step turns the analysis into an agenda. Keypup computes a recommended action for each open pull request (review, assign a reviewer, merge, fix the build, rebase) and the people expected to take it.

MCP Prompt:

List open vendor pull requests where the next action is on an
in-house engineer (review, assign a reviewer or merge). Show the
pull request, vendor, linked Jira issue, recommended action, who
it is waiting on, business days since the last review request
and the age of the pull request. Sort by business days since the
review request.

Output: Vendor Work Waiting on Us This Week

Key Insight: payments-api #4821 has waited 6.2 business days for two in-house reviewers. Every row is vendor capacity you're paying for while it waits on your own team. This is the agenda for the weekly vendor sync, and half of it is about you, not them.

The Technical Implementation: What Keypup MCP Actually Reads

Every number above comes from fields Keypup already imports from Jira and your Git platforms. It helps to be precise about what they are, and what they aren't.

Vendor Membership Comes From You

Keypup imports usernames, not employment contracts. The vendor split only works if you give the assistant the list of Git and Jira accounts for each vendor, and keep it current as people join and leave. If vendor engineers share accounts with in-house staff, or use the same email domain, the split can't be made from Keypup data alone. Many organizations already keep these lists for access reviews: reuse them.

Reviews Come From the Reviews Dataset

Each submitted review is a record with its reviewer, its state and its submission date. In-house review load counts reviews submitted by accounts outside the vendor lists on vendor pull requests; changes-requested counts reviews in that state. Comments left outside a formal review, pair programming and calls aren't counted, so the real review effort is higher than these numbers.

Links Come From Closing Keywords

Keypup links a pull request to a Jira issue when the pull request references the issue key with a closing keyword. A vendor that mentions the key without a keyword, or puts it only in the branch name, will look unlinked. Before reading the traceability donuts, agree on the convention and check it on a sample.

Resolution State Describes the Code, Not the Work

An issue's resolution state says whether its linked pull requests are all merged, still open, or absent. An issue closed with no merged pull request can be perfectly legitimate: investigation, configuration in a tool Keypup doesn't import, or work delivered under another ticket. Use the list to ask questions, not to assign blame.

Business Hours and Time Zones

Review wait times use Keypup's BUSINESS_HOURS and BUSINESS_DAYS functions, which accept a time zone and working hours. They apply one time zone per query, so pick the team whose day you want to measure. Keypup doesn't import calendars, so public holidays in either country aren't excluded.

Recommended Actions Are Computed From Pull Request State

Keypup's recommended action and recommended actors are derived from each open pull request's review state, approvals, build status and mergeability. They tell you who is expected to act next, not whether that person is available.

What Keypup Doesn't See

Keypup doesn't import invoices, hours billed, contract terms or deployment events. Cost per merged pull request, or vendor delivery against a contractual SLA, needs those figures from your own systems: the assistant can combine them with Keypup's output if you provide them, but they don't come from Keypup.

Real-World Impact: Enterprise Case Studies

Case Study 1: European Insurance Group

Before Keypup MCP:

• 900 engineers, of whom 320 from three outsourcing partners
• Vendor performance reviewed on utilization and story points reported by each vendor
• In-house teams consistently late on roadmap commitments, with no clear reason

After Keypup MCP:

• The review load trend showed in-house engineers spent close to half their reviews on vendor code, a share that had doubled in 18 months
• The time-to-first-review distribution showed the offshore partner lost a working day on most pull requests; a two-hour overlapping review window was added to the contract
• Vendor pull requests above an agreed size were split before review, and changes requested per pull request fell by a third over two quarters

Case Study 2: North American Healthcare Technology Company

Before Keypup MCP:

• 450 engineers, with a nearshore partner owning two integration repositories
• An audit finding on changes without tickets in one of those repositories
• No in-house engineer with recent knowledge of the integration code

After Keypup MCP:

• The repository scorecard showed one repository was over 80% vendor code with less than half reviewed in-house, and became the first knowledge-transfer target
• A linking convention was agreed with the vendor, and the share of vendor pull requests linked to a Jira issue rose from 52% to 91% in one quarter
• The waiting list became the agenda of the weekly vendor sync, replacing a status deck written by the vendor

Implementation: Getting Started with Keypup MCP

1. Connect Jira and Your Git Platforms

Connect Jira and GitHub, GitLab, Bitbucket or Azure DevOps to Keypup. Pull requests, reviews and issues are imported automatically, across all repositories, including the ones vendors work in.

2. Build the Vendor Account Lists

Export the Git usernames and Jira accounts of each vendor's engineers from your identity provider or vendor onboarding records. Keep one list per vendor and update it when people join or leave.

3. Agree on a Linking Convention

Ask every vendor to reference the Jira issue key with a closing keyword in each pull request description. Check the share of linked vendor pull requests per repository after a month.

4. Connect the MCP Server to Your AI Assistant

Follow the MCP documentation to connect Keypup to Claude, ChatGPT, Cursor or any MCP-compatible assistant.

5. Bring the Data to Vendor Reviews

Run the delivery profile, review load and repository scorecard before each quarterly business review, and the waiting list before each weekly vendor sync. Add your own cost data if you want to discuss cost per merged pull request.

Frequently Asked Questions

How do you measure the performance of an outsourced development team?

Measure the work, not the hours. Compare vendor and in-house pull requests on size, time to first review, changes requested and time to merge, and check how many vendor pull requests are linked to a Jira issue. Add the in-house review effort each vendor consumes: it's part of the real cost of the contract.

Why do offshore pull requests take longer to merge?

Mostly because of handoffs across time zones. A review requested at the end of the vendor's day is answered the next day, and each round of changes costs another day. Large pull requests make it worse, because they get more changes requested. Measuring time to first review in business hours, by group, shows how much of the delay is waiting rather than working.

How much in-house time do vendor teams consume?

Count the reviews submitted by in-house engineers on vendor pull requests, and compare them with all in-house reviews. In many organizations, vendor code absorbs a large share of in-house review effort, and that share grows as the vendor contract grows.

Can you compare vendors with each other?

Yes, as long as each vendor's accounts are listed separately, and you compare like with like: same repositories, same types of work. A vendor working on legacy integrations won't have the same profile as one building a new front end.

The Bottom Line

Your vendor contract tells you what you pay. Your Git and Jira data tells you what you get, and what it costs your own team. Keypup MCP brings the two together:

✅ Sizing the vendor share of delivery and of in-house review effort ✅ Comparing the delivery profile of vendor and in-house pull requests ✅ Measuring the time-zone tax on reviews ✅ Tracking the in-house review load as vendor contracts grow ✅ Mapping where vendor code lands and how closely it's reviewed ✅ Checking traceability from pull request to ticket and back ✅ Listing vendor work waiting on your team this week

Ready to open your vendor black box? Connect the Keypup MCP Server and ask your AI assistant.