A Director of Learning Operations at a 15,000-person manufacturer checks three vendors' own reports against her data and brings procurement a recommendation for each, with any AI chat her company allows.
A procurement email and three vendor reports, each with a big number and a small footnote. Her own LMS team sent one line. The footnotes and that line change the picture.
Here the AI checks numbers written by the people selling to you. The prompt tells it to be skeptical.
Use the AI tool your company approves: ChatGPT, Claude, Gemini or Copilot.
Copy the prompt and the five fictional inputs from the appendix into one message.
Read each footnote, check every date, then make the calls the AI shouldn't make for you.
I'm Director of Learning Operations at an electrical equipment manufacturer with about 15,000 people. Three learning vendors are up for renewal, and procurement and our CHRO want a recommendation for each at the December 10 vendor review. Today is November 16. Below are procurement's email, each vendor's own quarterly report, and a note from our LMS team.
Please give me:
Be skeptical: the vendors wrote these reports. Check how each metric is defined, what each comparison is really comparing, and what we pay for against what we use. Check dates. Don't add facts that aren't in the inputs.
Exactly as it came back, not edited. Read it the way Priya did: check each point against the source it cites.
| Vendor | Vendor's claim | What it actually measures | What our own data says |
|---|---|---|---|
| A: Course library ($900,000, 15,000 licences) | "Engagement: 82%, 12,300 engaged learners" | Their footnote defines "engaged" as logged in at least once in the quarter. That counts logins, not learning. 12,300 / 15,000 = 82%. | Our LMS team says 4,100 people completed at least one course this quarter. That is 27% of licences (4,100 / 15,000). About 8,200 people logged in but did not complete a course. A: same people? |
| "Estimated ROI: $4.1M in productivity gains" | No method, inputs or assumptions are given, and the vendor calls it "estimated". | We have no data that supports it. Treat it as unverified. found | |
| Cost | $900,000 / 15,000 = $60 per licence | $900,000 / 4,100 ≈ $220 per person who completed a course | |
| B: VR safety simulations ($600,000, 6 sites) | "Module completion: 96% (410 operators assigned)" | The share of assigned operators who completed modules, about 394 of 410. It shows completion, not changed behaviour. | We have no data on this. Cost is about $1,463 per assigned operator ($600,000 / 410). |
| "Incident rate at participating sites: down 31%" | Their footnote says this compares participating sites this year with all company sites last year. The groups differ, and so do the periods. "This year" is not the same as "this quarter", and the year may not be complete. | We have no data on this. The 31% does not show the program caused anything. We need the same six sites, measured over the same periods, from our own safety records. found | |
| Contract | Auto-renews November 30 unless we give notice. That is 14 days from today and 10 days before the December 10 review. | found | |
| C: Language app ($400,000, 3,000 licences) | "Average learning time: 7.2 hours per active user" | The average covers only 340 active users (footnote). It leaves out the 2,660 licence holders who did not use the app. "Active" is not defined. | 340 / 3,000 = 11% of licences used. Total time is about 2,448 hours, which works out to roughly 0.8 hours per licence. found |
| Cost | $400,000 / 3,000 ≈ $133 per licence | $400,000 / 340 ≈ $1,176 per active user |
The three contracts add up to $1.9M ($900k + $600k + $400k), which matches procurement's email.
Unknowns that stay open: the basis of Vendor A's ROI figure; Vendor B's real effect on safety; how Vendor C defines "active"; and the exact reporting period for every report.
Claude found every planted problem, including the deadline. Priya's edits remove one number the inputs don't support and turn two "or" answers into decisions.
Use it whenever a vendor's own report is part of a renewal, a budget review or a business case.
I'm the [your role] at a [industry] company with about [number] people. [Number] vendors are up for renewal, and [who] want a recommendation for each at [the meeting] on [date]. Today is [date]. Below are [the emails, each vendor's report, and our own usage data].
Please give me: 1. A table: each vendor's main claims, what each claim actually measures, and what our own data says. 2. Renew, renegotiate or exit for each vendor, with a one-line reason. 3. Anything I must do before the review.
Be skeptical: the vendors wrote these reports. Check how each metric is defined, what each comparison is really comparing, and what we pay for against what we use. Check dates. Don't add facts that aren't in the inputs.
Copy the prompt, then all five inputs, into one message in a new chat. Everything here is fictional and safe to paste.
I'm Director of Learning Operations at an electrical equipment manufacturer with about 15,000 people. Three learning vendors are up for renewal, and procurement and our CHRO want a recommendation for each at the December 10 vendor review. Today is November 16. Below are procurement's email, each vendor's own quarterly report, and a note from our LMS team. Please give me: 1. A table: each vendor's main claims, what each claim actually measures, and what our own data says. 2. Renew, renegotiate or exit for each vendor, with a one-line reason. 3. Anything I must do before the review. Be skeptical: the vendors wrote these reports. Check how each metric is defined, what each comparison is really comparing, and what we pay for against what we use. Check dates. Don't add facts that aren't in the inputs. [1] Email from Mark Ellison, Procurement Subject: Learning vendor renewals, Dec 10 Priya, three learning contracts are up, $1.9M a year in total. We need renew, renegotiate or exit for each by the Dec 10 review. Their quarterly reports are below. Note: Vendor B auto-renews on November 30 unless we give notice. [2] Vendor A: online course library ($900,000 a year, 15,000 licences) Engagement: 82%! 12,300 engaged learners this quarter. Estimated ROI: $4.1M in productivity gains. Footnote: engaged = logged in at least once in the quarter. [3] Vendor B: virtual-reality safety simulations ($600,000 a year, 6 sites) Module completion: 96% (410 operators assigned). Incident rate at participating sites: down 31%. Footnote: comparison is participating sites this year against all company sites last year. [4] Vendor C: language-learning app ($400,000 a year, 3,000 licences) Average learning time: 7.2 hours per active user this quarter! Footnote: active users: 340. [5] Note from our own LMS team Vendor A: 4,100 of our people completed at least one course this quarter.I'm Director of Learning Operations at an electrical equipment manufacturer with about 15,000 people. Three learning vendors are up for renewal, and procurement and our CHRO want a recommendation for each at the December 10 vendor review. Today is November 16. Below are procurement's email, each vendor's own quarterly report, and a note from our LMS team.
Please give me:
Be skeptical: the vendors wrote these reports. Check how each metric is defined, what each comparison is really comparing, and what we pay for against what we use. Check dates. Don't add facts that aren't in the inputs.
Subject: Learning vendor renewals, Dec 10
Priya, three learning contracts are up, $1.9M a year in total. We need renew, renegotiate or exit for each by the Dec 10 review. Their quarterly reports are below. Note: Vendor B auto-renews on November 30 unless we give notice.
Engagement: 82%! 12,300 engaged learners this quarter. Estimated ROI: $4.1M in productivity gains. Footnote: engaged = logged in at least once in the quarter.
Module completion: 96% (410 operators assigned). Incident rate at participating sites: down 31%. Footnote: comparison is participating sites this year against all company sites last year.
Average learning time: 7.2 hours per active user this quarter! Footnote: active users: 340.
Vendor A: 4,100 of our people completed at least one course this quarter.