Holmes & Cook review

Excellent — 5 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 25% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Strong 4 · Adequate 12. The typical agency here scores Adequate, and none scores higher than this one.

What this dimension measures: The heaviest weight, because an unvalidated model is an opinion with decimal places. Look for a described methodology, stated assumptions, and above all how the model is VALIDATED: holdout periods, backtesting, or reconciliation against a real experiment. A described method with no validation story scores 3.

Scores high — A firm that publishes its validation approach and names its limits scores 4-5.

Scores low — 'Proprietary algorithm' with no methodology at all scores 1-2 — proprietary is not a method, and in this category it is the single least checkable claim a vendor can make.

What we found — The principal's methodology is published in full and hosted independently by the IPA, which credits Louise Cook as author of 'Econometrics Explained 2' (October 2017) and hosts the 2004 original by Cook and Mike Holmes, edited by Les Binet. Both guides set out the validation regime rather than asserting one: statistical validity conditions (errors independent, random, normally distributed; no drift in error variance; stable coefficients over time), omitted-variable bias detection, R-Squared and Estimated Standard Error as fit measures, confidence intervals on each effect, and an explicit out-of-sample test stated as question 05 of six - 'Can a model estimated using part of your data forecast the remainder? This is one of the best tests of any model.' The 2017 guide also names the method's limits plainly ('A bad model will mislead and can be worse than no model... It is the modeller's responsibility to demonstrate their work is fit for purpose') and tells buyers they are 'well within your rights to ask for a run down of the model's validation diagnostics.' The site's own FAQ repeats the same test in buyer language: judge a model on whether the findings make theoretical sense AND whether it passes the appropriate statistical tests. No 'proprietary algorithm' language appears anywhere. That is the high band above, which is why it scored Excellent.

On the record — “The 2004 guide 'Econometrics Explained' by Louise Cook and Mike Holmes, edited by Les Binet and published by the IPA in February 2004, states that clients are 'well within your rights to ask for a run down of the model's validation diagnostics' and explains R-Squared, estimated standard error, omitted-variable bias and confidence intervals in buyer-facing terms.” holmesandcook.com ↗

On the record — “'Econometrics Explained 2', hosted on ipa.co.uk, sets out a six-question model-quality test whose fifth item is an out-of-sample holdout: 'Can a model estimated using part of your data forecast the remainder? This is one of the best tests of any model.' It also specifies the statistical validity conditions (errors independent, random and normally distributed; no drift in error variance; coefficients stable over time) and states that 'It is the modeller's responsibility to demonstrate their work is fit for purpose.'” ipa.co.uk ↗

Evidence: verified — checked against a named source you can open; the links under Sources below are where to check it yourself.

Strong — 4 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 20% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Excellent 1 · Strong 6 · Adequate 5 · Weak 4. The typical agency here scores Adequate, and 1 of them score higher than this one.

What this dimension measures: Whether the firm can establish causality rather than only correlation: geo holdout tests, matched-market design, PSA/ghost-ad tests, switchback designs, or reconciliation of modelled results against live experiments.

Scores high — Named experiment designs with described execution score 4-5.

Scores low — Correlation-only modelling with no experimental capability scores 2-3 — legitimate and common, but the buyer should know they are buying a correlational estimate. A firm that presents modelled attribution as proven causality scores 1-2, and the overclaim should be named in the verdict.

What we found — One published case study describes a matched-market design executed on a real engagement: modelling at regional level, regions 'classified on a range of criteria in order to weight the media received', then 'matched and paired thus enabling us to filter out a variety of macro effects and identify any media impacts', with a later round measuring displacement effects between messages. Both published guides carry a Media tests section stating the firm quantifies test effects with confidence intervals and advises on test design up front, noting that 'many media tests fail to produce conclusive results because insufficient account is taken of measurement difficulties when the test is designed.' No geo-holdout, PSA/ghost-ad or switchback programme is offered as a named service line, and no reconciliation of modelled results against a live experiment is shown. Against that, the firm does not overclaim: the 2017 guide explicitly separates what MMM can establish from what digital attribution can, warns that attribution 'may overstate digital ROI', and describes ROI from econometrics that ignores other media as a misattribution. That is the high band above, which is why it scored Strong.

On the record — “A published case study describes a matched-market design executed on a real engagement: 'The modelling was at regional level. Regions were classified on a range of criteria in order to weight the media received. The regions were then matched and paired thus enabling us to filter out a variety of macro effects and identify any media impacts.' A follow-up round examined displacement effects between messages.” holmesandcook.com ↗

attribution — working out which marketing touch actually caused a sale. Good practice names its model and its blind spots; bad practice quotes each ad platform’s self-graded numbers, which overlap and overclaim.

Evidence: partly checkable — corroborated in part against the sources below; the remainder rests on the agency’s own account.

Strong — 4 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 15% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Excellent 1 · Strong 8 · Adequate 6 · Weak 1. The typical agency here scores Strong, and 1 of them score higher than this one.

What this dimension measures: Attributable client work at a stated scale and category. Treat any 'we found X% waste' claim as a vendor-stated number unless a client is named and corroborates it.

Scores high — Named clients with described engagements score 4-5.

Scores low — Anonymised case studies score 2-3 — common here for genuine confidentiality reasons, so do not penalise beyond the band, but do not credit unverifiable lift figures either.

What we found — One engagement is corroborated off-site: Thinkbox's Foster's case study credits 'Econometric modelling agency: Holmes & Cook' with three named individuals (Jon Fox, Louise Flin, Andreas Georgiou) on the paper that took the Grand Prix and a Gold at the 2014 IPA Effectiveness Awards. On-site, clients are named in attributed testimonials with role and organisation - HJ Heinz, The Home Office, Bauer Media, MEC, Radio Advertising Bureau, London Midway Entertainments, Muse Strategy, plus Karcher UK & Ireland (Simon Keeping, MD) and VCCP Media (Marie Oldham, Chief Strategy Officer) named with individuals on the homepage - and agency relationships are listed (AMV BBDO, Leo Burnett, McCann Erickson, VCCP, MPG, RKCR/Y&R, BDH TBWA, Brilliant Media). An awards board lists brands whose winning papers used their analysis (Fosters, PG Tips, Tesco, O2, Olivio, Optrex, Heinz, Lidl, Easyjet, Sainsburys, Karcher and others). Held back from Excellent because the four written case studies are anonymised and the client logo wall carries no alt text or captions, so most named clients cannot be tied to a described engagement. Notably, the case studies quote no lift or waste percentages at all - there are no unverifiable outcome numbers to discount. That is the high band above, which is why it scored Strong.

On the record — “Thinkbox's Foster's case study credits 'Econometric modelling agency: Holmes & Cook' and names Jon Fox, Louise Flin and Andreas Georgiou of Holmes & Cook among the paper's contributors; the paper won the Grand Prix and a Gold at the 2014 IPA Effectiveness Awards, with a stated return of 32 pounds of revenue per 1 pound of advertising spend.” thinkbox.tv ↗

Evidence: verified — checked against a named source you can open; the links under Sources below are where to check it yourself.

Strong — 4 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 15% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Strong 10 · Adequate 4 · Weak 2. The typical agency here scores Strong, and none scores higher than this one.

What this dimension measures: What the engagement needs from the buyer (spend, conversion, and channel data at what granularity) and — critically — whether the firm also buys the media it is measuring. A firm that measures media it does not sell is structurally more credible; where the same firm buys and grades its own work, that conflict must be disclosed and should be named in the verdict whether or not the firm names it.

Scores high — Clear data requirements plus independence from media buying scores 4-5.

Scores low — Undisclosed conflict scores 1-2.

What we found — Data requirements are stated with a number: 'A minimum of three year's monthly (or more frequently reported data) is recommended' where seasonality exists, longer where more sales influences are present, covering sales plus every variable likely to move sales; the FAQ tells the buyer to allow about a month to collect and hand over data and to appoint a 'data Tsar' with authority to chase colleagues. The published guides go further into granularity (retailer, region, store, key account, customer group). On independence, the firm sells modelling and consultancy only - Companies House classifies it under 73200 market research and public opinion polling, it employs 2-10 people, and media agencies (MEC, MPG, VCCP Media, AMV BBDO) appear as collaborators and clients rather than as a media-buying arm of its own. There is no media buying to create a conflict, and no engagement here where the firm would be grading media it sold. Short of Excellent only because the site never states this independence explicitly - a buyer has to infer it from the whole offering rather than read it. That is the high band above, which is why it scored Strong.

On the record — “The site's FAQ states data and timeline requirements with numbers: 'A minimum of three year's monthly (or more frequently reported data) is recommended' where seasonality exists, and 'Once you have collected and handed over all the data (probably allow yourself a month for that) the econometricians will need anything from 8-12 weeks to final debrief.'” holmesandcook.com ↗

On the record — “HOLMES & COOK LIMITED (company number 04969797) is Active at Companies House, incorporated 19 November 2003, registered at 81 Newport Road, Hanslope, Milton Keynes MK19 7ND, SIC 73200 market research and public opinion polling, with accounts made up to 30 November 2025 and a confirmation statement dated 19 November 2025.” find-and-update.company-information.service.gov.uk ↗

Evidence: vendor stated — the agency’s own claim, recorded as theirs rather than ours.

Adequate — 3 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 10% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Excellent 1 · Strong 6 · Adequate 9. The typical agency here scores Adequate, and 7 of them score higher than this one.

What this dimension measures: What arrives and how often: a one-off model, a refreshed quarterly model, a live dashboard, a decision workshop. A model delivered once and never refreshed is a snapshot of a market that has moved, and should be scored as the ceiling it is.

Scores high — Stated deliverables with a stated refresh cadence score 4-5.

Scores low — Undefined deliverables score 2.

What we found — What arrives is described: models mounted in an Excel-based planning tool the client and its media agency run what-if scenarios in, a debrief presentation, and after-sales support for follow-up questions. Timeline is anchored - 'the econometricians will need anything from 8-12 weeks to final debrief' after data handover, plus roughly a month of data collection. Cadence is where it thins. Two case studies describe models being refreshed and extended over time ('Models have been updated and new models have been added over time'; one client tracking actual sales against model expectation as an early warning), but no refresh interval, retainer or update package is stated anywhere on the site, so a buyer cannot tell whether they are commissioning a one-off model or an ongoing one without asking. The 2017 guide notes cost rises with 'update frequency', which implies refreshes are sold, but the site does not price or schedule them. That is between the two bands, which is why it scored Adequate.

On the record — “The site's FAQ states data and timeline requirements with numbers: 'A minimum of three year's monthly (or more frequently reported data) is recommended' where seasonality exists, and 'Once you have collected and handed over all the data (probably allow yourself a month for that) the econometricians will need anything from 8-12 weeks to final debrief.'” holmesandcook.com ↗

On the record — “The only published cost anchor is category-wide and appears in Cook's IPA guide, not on the firm's site: 'Modelling projects cost from tens to hundreds of thousands of pounds. Cost rises with the number of models required, the difficulty of the task and the update frequency.'” ipa.co.uk ↗

deliverability — deliverability: whether cold email actually lands in the inbox rather than spam. The tell is infrastructure talk — warmed sending domains kept separate from your main domain — because a burned domain outlasts the engagement.

retainer — a fixed monthly fee regardless of hours or output — predictable, but worth tying to a defined scope.

Evidence: vendor stated — the agency’s own claim, recorded as theirs rather than ours.

Weak — 2 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 10% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Strong 2 · Adequate 3 · Weak 11. The typical agency here scores Weak, and 5 of them score higher than this one.

What this dimension measures: Note any minimum spend and whether the model licence continues after the engagement ends.

Scores high — Published fees, ranges, or a stated engagement structure with numbers score 4-5.

Scores low — A described structure without numbers scores 2-3. Bespoke-only with no anchor scores 1-2.

What we found — The site publishes no fee, no range, no minimum and no engagement structure in commercial terms - every route ends at a phone number or a contact form. The only cost anchor a buyer can find anywhere is a category-wide one in the principal's IPA guide, not a Holmes & Cook rate card: 'Modelling projects cost from tens to hundreds of thousands of pounds. Cost rises with the number of models required, the difficulty of the task and the update frequency.' That is useful for calibration but it is the industry's number, not this firm's. Nothing states whether the Excel planning tool and the model itself remain the client's to use after the engagement ends, though the case studies describe clients and their media agencies continuing to run the tool, which points that way without saying so. That is what the low band describes, which is why it scored Weak.

On the record — “The only published cost anchor is category-wide and appears in Cook's IPA guide, not on the firm's site: 'Modelling projects cost from tens to hundreds of thousands of pounds. Cost rises with the number of models required, the difficulty of the task and the update frequency.'” ipa.co.uk ↗

Evidence: inferred — our reading of indirect evidence, not a documented fact.

Stronger here: Nepa scores Strong on the same dimension.

Weak — 2 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 5% of the total score.

Benchmark — across the 16 other agencies evaluated in this discipline, this dimension runs Adequate 6 · Weak 10. The typical agency here scores Weak, and 6 of them score higher than this one.

What this dimension measures: Independent, verified reviews or checkable references. Low weight deliberately: measurement work sells through procurement and referral, so a thin public review footprint is normal and the method evidence above matters far more.

Scores low — A substantial verified base scores 4-5; a handful scores 2-3; none located scores 2.

What we found — No independent client review base was located or read - no Clutch, G2 or Google review profile surfaced for the firm, which is normal for a two-to-ten-person consultancy selling through procurement and referral. Eleven testimonials on the site are attributed to a role and a named organisation (HJ Heinz, The Home Office, Bauer Media, MEC, VCCP, RAB, London Midway Entertainments, Muse Strategy) and one to a named individual, Paul Feldwick, but all are self-published and none link to a source. What is independently checkable is professional standing rather than client satisfaction: the IPA publishes Louise Cook's guides and calls her a 'respected econometrician', and Thinkbox credits the firm on an IPA Grand Prix paper. Scored on the rubric's own terms - no independent reviews located - not on that standing.

On the record — “The IPA's own econometrics page credits Louise Cook as author of 'Econometrics Explained 2' (October 2017), calling it a 'free follow-up guide by respected econometrician Louise Cook' that takes account of the growth of digital media, and lists the February 2004 original as 'The original influential IPA starter guide.'” ipa.co.uk ↗

Evidence: inferred — our reading of indirect evidence, not a documented fact.