Buying GuidesSoftware Buying

How to Build a Vendor Scorecard and Run Quarterly Business Reviews: Metrics, Weighting, QBR Agenda and Follow Up

Build a vendor scorecard and run quarterly business reviews with key vendors: KPIs to track, weighting, a QBR agenda, the metrics to bring and how to follow up.

Buying Guides: How to Build a Vendor Scorecard and Run Quarterly Business Reviews: Metrics, Weighting, QBR Agenda and Follow Up

Choosing a vendor gets a lot of attention. Managing the vendor afterwards usually gets almost none, until a renewal notice arrives or an outage stops the business. A vendor scorecard closes that gap. It is a one page, repeatable report that rates a supplier you already use against the measures you agreed, so you can see whether you are getting what you pay for and act before small problems become expensive ones.

This guide is for small and mid sized businesses managing a handful of important suppliers and software vendors, such as your accounting system, payroll provider, managed IT partner, logistics carrier or main product supplier. It covers which metrics to track, how to turn them into measurable KPIs, how to weight them, how to run quarterly reviews and how to use the results in improvement plans, renewals and exits. It includes a worked weighted score and a template you can copy.

What a vendor scorecard is, and what it is not

A vendor scorecard (also called a supplier scorecard) is a structured rating of an existing vendor's performance against a fixed set of measures, updated on a regular schedule. The professional body CIPS describes supplier performance as something you assess across characteristics such as product or service quality, delivery accuracy, lead times, customer service, account management and commercial costs, and stresses that the objectives and targets you monitor should be agreed in the contract.

It helps to separate three related tools that Businesslang covers in different guides:

Tool When you use it Question it answers Where to learn more
Evaluation matrix Before you buy, once Which candidate vendor should we choose? Vendor selection and RFP guide
Vendor risk assessment Before signing and at least yearly What could this vendor's failure do to us, and are we protected? Vendor risk management guide
Vendor scorecard After you buy, every month or quarter Is this vendor delivering what we agreed, and is it getting better or worse? This guide

The scorecard borrows categories from the selection matrix, but it scores actual results, such as outages, tickets, deliveries and invoices, rather than proposals and demos.

The six metric categories to track

Most vendor scorecards, whether for software or physical goods, draw on the same six categories. Pick the ones that matter for each vendor; not every vendor needs all six.

Category What it covers Example KPIs for a software or IT vendor Example KPIs for a goods or services supplier
Quality Whether the product or service works as specified Severity 1 and 2 incidents per quarter, bugs reopened, data errors Defect or reject rate, returns, warranty claims
Delivery and SLA Whether it arrives or stays available as promised Monthly uptime versus SLA, scheduled maintenance outside agreed windows On-time in-full (OTIF) rate, lead time versus quoted
Cost Whether you pay what you agreed and get value Invoice accuracy, price change versus contract cap, unused licenses Price variance versus quote, disputed invoices
Support and responsiveness How fast and how well problems are handled First response time, time to resolution by severity, escalations needed Response time to queries, issue closure time
Compliance and security Whether contractual, legal and security obligations are met Current SOC 2 report or equivalent on file, breach notices on time, subprocessor changes notified Insurance certificates current, certifications valid, required documents on time
Relationship and innovation How the vendor works with you over time QBR held, roadmap items delivered, named contact stable Improvement ideas offered, cost reduction proposals

CIPS's procurement KPI guidance defines on-time-in-full (OTIF) as deliveries received in line with the delivery date with no short shipments or quality defects, which makes it a useful single delivery KPI for goods suppliers. For software, the equivalent is uptime measured against the SLA in your contract.

Step 1: Choose which vendors get a scorecard

Do not score every supplier. A scorecard takes data gathering and a meeting, so reserve it for vendors where performance actually matters. A practical rule:

  • Critical vendors (failure would stop you selling, shipping, paying staff or getting paid, or the vendor holds sensitive data): full scorecard, reviewed quarterly.
  • Important vendors (failure hurts but you could work around it): a shorter scorecard, reviewed twice a year.
  • Low risk vendors: no scorecard; check at renewal.

If you have not tiered your vendors yet, the inventory and tiering steps in our vendor risk management guide take an afternoon and give you this list. For most small businesses it ends up at three to ten vendors.

Step 2: Pull targets from the contract and agree what success means

Your strongest KPIs come from what the vendor already committed to in writing: the SLA, the order of service, the statement of work, the price schedule and the data processing terms. CIPS makes the same point: the targets you monitor should be the ones agreed in the contract. If a measure you care about is not in the contract, you can still track it, but treat it as an expectation you raise in reviews and add at renewal.

Gather for each scored vendor:

  • The uptime or delivery commitment and how it is measured.
  • Support hours and response times by severity.
  • The price schedule, renewal date and any cap on increases.
  • Security and compliance obligations, such as breach notice timing and audit reports.
  • Any service credits or remedies for missed targets.

If the contract is silent on most of these, that is a finding in itself. Our SaaS contract checklist lists the clauses to add next time.

Step 3: Turn each category into measurable KPIs

A KPI belongs on the scorecard only if two people looking at the same data would give it the same score. CIPS recommends KPIs that are SMART: specific, measurable, achievable, relevant and time-bound. In practice that means each KPI needs five things written down:

Field Example: SaaS uptime Example: support responsiveness
Definition Percentage of minutes in the month the service was available, excluding agreed maintenance windows Time from ticket creation to first human response, severity 1 and 2 tickets only
Target Contract SLA, for example 99.9% Contract response time, for example 1 business hour for severity 1
Data source Vendor status page history plus your own incident log Your helpdesk or the vendor's support portal export
Period Calendar month, reported quarterly Calendar quarter
Owner IT or operations lead The person who manages the vendor day to day

Keep it short: one to three KPIs per category and no more than about 12 KPIs per vendor. CIPS notes that overcomplicated KPIs are difficult to manage.

A note on uptime percentages: small differences are bigger than they look. In a 30 day month (43,200 minutes), 99.9% availability allows about 43 minutes of downtime, while 99.5% allows 216 minutes, or 3.6 hours. That is simple arithmetic, but it is worth putting next to the KPI so everyone understands what the target means.

Step 4: Weight the categories

Weights tell the scorecard what matters most. Without them, a vendor with excellent account management can hide repeated outages in an average.

Weighting methods

Method How it works Best for
Equal weights Every category counts the same A first draft, or vendors where you genuinely have no priority
Direct weighting Stakeholders agree a percentage per category, totaling 100% Most small businesses; quick and transparent
Points allocation Each stakeholder splits 100 points across categories; you average the results Teams where users, finance and IT disagree on priorities
Pairwise comparison Compare categories two at a time and count how often each wins Larger teams wanting a more defensible result; usually overkill for an SMB

Direct weighting or points allocation works for most small businesses. Agree weights with the people who use, manage and pay for the vendor, and then leave them alone for at least a year so scores stay comparable from quarter to quarter.

Gates versus weights

Some things should not be averaged. A vendor that stops providing security evidence, misses a legal obligation or fails a sanctions or insurance check is not a "3 out of 5 on compliance"; it is a problem that needs action regardless of the total. Treat these as gates: pass or fail items shown at the top of the scorecard. Keep a weighted compliance category for the measurable parts, such as documents delivered on time.

Example weights by vendor type

These weightings are illustrative starting points, not a standard. Adjust them to your situation.

Category SaaS or IT vendor Goods supplier Professional services firm
Quality 20% 30% 30%
Delivery and SLA 25% 30% 15%
Cost 15% 20% 20%
Support and responsiveness 20% 10% 15%
Compliance and security 10% 5% 10%
Relationship and innovation 10% 5% 10%
Total 100% 100% 100%

Step 5: Define a rating scale with anchors

Score each KPI on a 1 to 5 scale, and write down what each score means before the first review. Anchors stop scores from drifting with mood or with how friendly the account manager is.

Score General meaning Uptime example (SLA 99.9%) Support example (target: first response within 1 business hour for severity 1)
5 Consistently beats target SLA met every month, no unplanned outage over 15 minutes 100% of severity 1 tickets answered within target
4 Meets target SLA met every month 95% or more within target
3 Minor misses SLA missed in one month, by a small margin 85% to 94% within target
2 Repeated misses SLA missed in two months, or one major outage 70% to 84% within target
1 Unacceptable SLA missed every month, or an outage that stopped the business Below 70% within target

The percentage bands above are examples to adapt, not industry thresholds. When a category has more than one KPI, average the KPI scores to get the category score, then apply the weight.

Step 6: Collect the data with as little effort as possible

A scorecard that needs a day of spreadsheet work each quarter will be abandoned. Use data you already have:

Category Where the data usually lives
Quality Your helpdesk tickets, incident log, returns or inspection records
Delivery and SLA Vendor status page history, monitoring tool, receiving records, purchase orders
Cost Accounts payable, invoices matched to purchase orders, license admin console
Support The vendor's support portal export, your ticket timestamps
Compliance and security Your vendor file: audit reports, insurance certificates, data processing agreement
Relationship Meeting notes, a short survey of internal users

Two tips. First, do not rely only on the vendor's own report; cross-check at least one KPI against your own records. Second, include a short internal user survey (three questions, scored 1 to 5) for the relationship and support categories. CIPS recommends combining objective data such as lead times and pricing compliance with the experiences of the people who deal with the supplier.

Step 7: Calculate the weighted score

Multiply each category score by its weight and add up the results. The worked example below is illustrative, with made up numbers, for a SaaS helpdesk vendor using the SaaS weights from Step 4.

Category Weight KPI results this quarter (illustrative) Score (1 to 5) Weighted score
Quality 20% 2 severity 1 incidents, both fixed same day 4 0.80
Delivery and SLA 25% SLA met in two of three months 3 0.75
Cost 15% Invoices accurate; 18% of seats unused 3 0.45
Support and responsiveness 20% 72% of severity 1 tickets answered within target 2 0.40
Compliance and security 10% Current audit report on file, subprocessor change notified on time 5 0.50
Relationship and innovation 10% QBR held, two promised features delivered 4 0.40
Total 100% 3.30 out of 5
Gates Security evidence current: pass. Data processing agreement signed: pass.

Method. Weighted score equals score multiplied by weight. The total is the sum of the weighted scores.

The total of 3.30 looks acceptable, but the detail tells the real story: support responsiveness is red, uptime is slipping and nearly a fifth of seats are unused. That is what you take into the review.

Turn the total into a status

Total score Status Typical action
4.0 to 5.0 Green Keep going; consider the vendor for more business
3.0 to 3.9 Amber Agree fixes for any category scoring 2 or below
Below 3.0 Red Formal improvement plan with dates; start exit planning

Also flag any single category scoring 1 or 2, whatever the total. In the example above, the vendor is amber overall but has a red category that needs a plan.

Step 8: Run quarterly business reviews with the vendor

A quarterly business review (QBR) is a scheduled meeting where you and the vendor go through the scorecard together and agree what happens next. For critical vendors, quarterly is a common rhythm; for important vendors, twice a year is usually enough. Keep operational KPIs, such as uptime, on a monthly check so you do not wait three months to spot a problem.

Many vendors offer their own QBR, built around their slides: adoption charts, new features and an upsell. Take the meeting, but run it on your agenda and your data. The purpose is to check performance against the contract, fix problems and plan the next quarter, not to watch a product demo.

Who should attend

Keep the group small enough to make decisions.

Your side Vendor side
The vendor owner who manages the relationship day to day (chairs the meeting) Account manager or customer success manager
A senior sponsor for critical vendors, at least twice a year Someone who can commit resources, such as a support or delivery lead
One or two heavy users who know where the problems are A technical contact when incidents or integrations are on the agenda
Finance or procurement when the renewal is within six months A commercial contact when pricing or renewal is on the agenda

Prepare: the metrics pack to bring

Send a short pack to the vendor at least three working days before the meeting, and ask the vendor to send its own data in advance too. Compare the two before you meet, so the meeting is spent on decisions rather than arguing about numbers.

Pack item What to include Where it comes from
Scorecard This quarter's scores, last four quarters' trend, any failed gates Your scorecard from Steps 5 to 7
SLA results Uptime or on-time delivery by month against the contract target, with service credits due Status page history, monitoring, receiving records
Incidents Severity 1 and 2 incidents, root cause, time to resolve, whether the fix is permanent Your incident log and the vendor's reports
Support Tickets opened, first response and resolution times, oldest open tickets Helpdesk or vendor portal export
Usage and licenses Seats paid for versus active, features used, growth expected Admin console, license reports
Spend Spend this quarter and year to date versus budget, invoice errors, price changes Accounts payable, purchase orders
Compliance Status of security evidence, insurance, data processing terms, subprocessor changes Your vendor file
Open actions Actions agreed at the last QBR with owner, due date and status Last QBR notes
Contract dates Renewal date, notice deadline, price cap Contract

Unused seats deserve a line of their own. They are often the fastest saving available, and our research on SaaS license waste shows why it is worth checking every quarter.

A QBR agenda that works

This 60 minute agenda fits most critical vendors. For important vendors reviewed twice a year, 45 minutes is usually enough: shorten the roadmap and commercial items.

Time Item Purpose Lead
5 minutes Goals and actions from last QBR Confirm what was promised and what got done You
15 minutes Scorecard and SLA results Agree the facts: what went well, what missed target You
15 minutes Red and amber areas Root cause, the vendor's fix with a date, what it needs from you Vendor
5 minutes Your side CIPS advises considering the supplier's experience of working with your organization too. Ask what you are doing that slows them down, such as late approvals or unclear requests. Vendor
10 minutes Business changes and roadmap Your growth, new locations or teams; their roadmap items that matter to you, with dates Both
5 minutes Commercial items Upcoming price changes, renewal timeline, license true up, service credits owed You
5 minutes Actions and next date Read back actions, owners and dates; confirm the next QBR You

Three rules keep the meeting useful:

  • Facts first. Agree the numbers before discussing causes. If the vendor's data disagrees with yours, note it as an action rather than spending the meeting on it.
  • Roadmap claims need dates. A feature "coming soon" is not a commitment. Record roadmap items only with a target date, and treat them as unconfirmed until they ship.
  • No surprise negotiations. Raise pricing and renewal topics in the commercial slot, and hold the actual negotiation separately, ideally 90 or more days before the notice deadline.

Follow up after the QBR

The value of a QBR is in what happens afterwards.

  1. Send written notes within two working days. Include the agreed facts, decisions and an action table with owner and due date for each item. Ask the vendor to confirm by reply.
  2. Log actions where you will see them. Add them to your vendor file or task tool, with reminders a week before each due date.
  3. Claim what you are owed. If the contract gives service credits for missed SLAs, request them in writing using the QBR data.
  4. Escalate when needed. If a red area has no credible fix, or actions from the last QBR were missed, escalate to the vendor's management and start an improvement plan (see Step 9).
  5. Check progress at the monthly operational review. Do not wait for the next QBR to find out an action slipped.
  6. Open the next QBR with the action table. It becomes the first item on the agenda and the first page of next quarter's scorecard.

A simple action table is enough:

Action Owner Due Status
Root cause report for the March outage Vendor support lead Within 2 weeks Open
Remove 12 unused seats at next billing date Your vendor owner Next invoice Open
Approve integration test window Your IT lead End of month Open
Confirm roadmap date for audit log export Vendor account manager Next QBR Open

The rows above are illustrative examples, not a standard list.

Step 9: Act on the results

The scorecard only creates value through the decisions it supports. Use the status and trend, not a single quarter, to decide what to do.

Situation What to do
Green and stable Renew on the best terms you can get; consider giving the vendor more scope
One red category, others fine Targeted improvement plan for that category, reviewed next quarter
Amber trending down for two quarters Formal improvement plan, escalate to the vendor's management, check contract remedies such as service credits
Red for two quarters, or a failed gate Start exit planning: shortlist alternatives, check data export and notice periods

Improvement plans

A good improvement plan names the KPI, the current result, the target, the actions the vendor will take, the date by which the target must be met and what happens if it is not. Keep it to one page and agree it in writing. If your contract includes service credits for missed SLAs, claim them; the scorecard data is your evidence.

Renewal negotiations

Bring the last four quarters of scorecards to every renewal. Documented missed SLAs, slow support and unused licenses are concrete leverage for a lower price, a renewal cap, stronger service credits or a smaller seat count. Our guide to negotiating SaaS pricing and renewals covers the commercial levers. Start at least 90 days before the notice deadline so you have time to act on what the scorecard shows.

Exit

When performance does not recover, the scorecard history is your record of why you are leaving, which matters if the contract allows termination for repeated service failures. This guide is not legal advice; check termination rights with counsel before giving notice. Plan the switch with the same discipline you used to buy: re-run a light vendor selection process, estimate the total cost of ownership including migration, and follow our software rollout guide for the move.

Vendor scorecard template

Copy this into a spreadsheet, one tab per vendor, and add a column per quarter to see trends.

Category Weight KPI Definition and data source Target This quarter result Score (1 to 5) Weighted score Trend vs last quarter Comment and action
Gates n/a Security evidence current Audit report or certificate on file, not expired Pass Pass or fail n/a
Gates n/a Required contract documents in place DPA, insurance certificate Pass Pass or fail n/a
Quality
Delivery and SLA
Cost
Support and responsiveness
Compliance and security
Relationship and innovation
Total 100%

Add a header with the vendor name, internal owner, contract value, renewal date, notice deadline and tier. If your vendor list grows beyond what a spreadsheet can handle, supplier management features in source to pay software can hold scorecards alongside contracts and purchase orders.

Common vendor scorecard mistakes

  • Too many KPIs. Twenty measures per vendor means none get attention. Start with six to ten.
  • Undefined scores. "Support: 3" means nothing without written anchors. Define 1, 3 and 5 before the first review.
  • Scoring on feelings. A friendly account manager is not a substitute for SLA data. Use relationship scores only for the relationship category.
  • Letting averages hide failures. Show gates and red categories separately from the total.
  • Changing weights every quarter. It makes trends meaningless. Review weights once a year.
  • Keeping the scorecard secret. Vendors cannot improve on measures they do not know about. Share definitions and results.
  • Scoring without acting. If a red score never leads to a plan, a credit claim or a renewal conversation, stop scoring and save the time.
  • Ignoring your own part. Late approvals, unclear requirements and missing internal owners can cause the problems you are scoring.

Bottom line

A vendor scorecard does not need special software or a procurement team. Choose your critical vendors, pull targets from the contracts, define a few measurable KPIs per category, weight them by business impact, score on written anchors and review the results with the vendor every quarter. Then use what you learn: fix problems with dated improvement plans, bring the history to renewals and leave vendors that will not improve. Start this week with your most critical software vendor and one quarter of data; the second scorecard is always easier than the first. For the bigger picture, see how scorecards fit into your small business purchasing process.

Frequently asked questions

What is a vendor scorecard?

A vendor scorecard is a structured, regularly updated rating of an existing supplier's performance against agreed measures such as quality, on-time delivery or uptime, cost, support responsiveness and compliance. Unlike a one-off selection score, it tracks the same measures over time so you can see trends and act on them.

What is the difference between a vendor scorecard and a vendor evaluation matrix?

An evaluation matrix compares several candidate vendors once, before you buy. A scorecard measures one vendor you already use, repeatedly, after you buy. Many categories overlap, but scorecard KPIs are based on actual results such as tickets, outages and invoices rather than proposals and demos.

Which KPIs should a software vendor scorecard include?

For a SaaS or IT vendor, common choices are monthly uptime against the SLA, number of severity 1 incidents, first response and resolution time for support tickets, invoice accuracy, price change versus the contract cap, status of security evidence such as a current SOC 2 report, and adoption or license usage.

How do you weight a vendor scorecard?

Give each category a percentage so the total is 100%, based on what would hurt most if it failed. Most small businesses use simple direct weighting agreed by the people who use, manage and pay for the vendor. Keep weights stable for at least a year so scores stay comparable.

How often should you review a vendor scorecard, and what goes in a vendor QBR?

Quarterly is a common rhythm for critical vendors, with a short monthly check on operational KPIs such as uptime or delivery. Important vendors can be reviewed twice a year and low risk vendors at renewal. A vendor quarterly business review (QBR) usually runs 45 to 60 minutes: actions from last time, the scorecard, red and amber areas, issues on your side, commercial items and the roadmap, ending with written actions, owners and dates.

Should you share the scorecard with the vendor?

Yes. A vendor cannot fix what it cannot see. Share the scores, the underlying data and the definitions, and invite the vendor to flag obstacles on your side, which CIPS also recommends considering in supplier evaluations.

Sources

  1. CIPS: How to manage suppliers performance (checked November 12, 2026)
  2. CIPS: Procurement KPIs, KPIs and performance measures used in procurement (checked November 12, 2026)

Related guidance