Pilothouse review

Adequate — 3 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 20% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Strong 7 · Adequate 29 · Weak 6. The typical agency here scores Adequate, and 7 of them score higher than this one.

What this dimension measures: How many assets does this vendor actually ship per client per month, and how fast is a variant round? A stated, specific throughput (e.g. Where the vendor names a client, check the client's live ads in the Meta Ad Library — POSITIVE EVIDENCE ONLY, and scope it properly. Correct procedure: run the keyword search, harvest the advertiser's page_id from the rendered HTML, then re-query with view_all_page_id=<id>. Scoped counts are 100% brand-attributed; keyword counts are not scoped to the advertiser and overstated by 41% in one measured case (HexClad, 410 keyword vs 290 scoped), while a search for a generically-named client returned 1,400 results consisting of Viator, Klook and Tripadvisor. A high-volume, recent, scoped result confirms a real programme and may raise this score. A low or absent result is INCONCLUSIVE and must never lower it. Critically: NEVER record an absence from a page ID you did not harvest from a live ad — the userID shown on a logged-out Facebook page is not the Ad Library page_id, and querying it returns a clean, convincing zero. That was proven by control on 2026-08-11 against a brand known to be running 290 ads. Absence of ads is never scored against an agency. See library/_notes/meta_ad_library_method_2026-08-11.json.

Scores high — 'X assets/month', published turnaround of 24-72h per round) with corroboration scores 4-5.

Scores low — Vague 'unlimited requests' claims with a one-at-a-time queue score 2-3 — unlimited requests with a single concurrent slot is a throughput of one. No stated volume anywhere scores 2.

What we found — The Meta service page headlines '5,000+ Creative Assets Monthly' and 'Volume and variation pipelines designed for algorithmic learning'. The Studios page cites a benchmark that 'the most successful brands build, test, and iterate 30-50 original pieces of content a month' but does not commit to that as a deliverable. The about page states 175 team members managing over 125 active brands, which puts the 5,000 figure at roughly 40 assets per brand per month if it is distributed evenly, but no page states a per-client volume, a round turnaround, or a revision SLA. One client testimonial (Happy Caps) describes 'ideating Monday/launching Wednesday', which is a single anecdote, not a published turnaround. A Meta Ad Library scoped check on a named client could not be run - the Ad Library returned HTTP 403 to the request, a broken instrument, so no ad-count evidence was recorded in either direction. Capped at Adequate under the corroboration rule: the throughput figure is specific but entirely self-published and not scoped to any client. That is between the two bands, which is why it scored Adequate.

On the record — “The about page states the company scaled to 175 team members managing over 125 active brands, and lists a leadership team including a Head of Creative Operations; office listed at 1011 Government Street, Victoria, BC.” pilothouse.co ↗

On the record — “The Meta service page states '5,000+ Creative Assets Monthly' and 'Iteration tied to performance, not subjective creative preference', and lists creative-level performance analysis under reporting.” pilothouse.co ↗

pipeline — the dollar value of open, qualified deals sales is working. “Pipeline created” ties marketing to money, where lead counts don’t.

deliverability — deliverability: whether cold email actually lands in the inbox rather than spam. The tell is infrastructure talk — warmed sending domains kept separate from your main domain — because a burned domain outlasts the engagement.

SLA — service-level agreement: a contractual promise about speed or quality of delivery.

Evidence: vendor stated — the agency’s own claim, recorded as theirs rather than ours.

Weak — 2 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 20% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Strong 5 · Adequate 20 · Weak 17. The typical agency here scores Adequate, and 25 of them score higher than this one.

What this dimension measures: Does the vendor work from the client's brand system — logo lockups and clear space, exact colour and type, tone of voice — or from a mood board and taste? Look for an explicit intake of brand guidelines, a named check step before client review, and any statement about who catches an off-brand asset.

Scores high — A documented preflight or QC step scores 4-5.

Scores low — 'We'll learn your brand over the first month' scores 2-3. Rotating contractor pools with no guideline intake score 1-2. Silence on the question scores 2: at volume, brand drift is the default outcome and a vendor that has not thought about it has not solved it.

What we found — No page read describes a brand-guideline intake, a named preflight or QC step before client review, or who is accountable for catching an off-brand asset. The Studios page offers 'a single account manager for all creative services' and 'data-informed designs tested across 100+ accounts' - a coordination claim and a data claim, neither of which is a brand check. The about page lists a Head of Creative Operations, which implies an ops function but describes nothing about it. The only guideline language found is platform guidelines, not brand systems: the CorneaCare case study says Studios made 'Amazon listing images that adhered to guidelines and converted'. This matters more than usual here because the delivery model mixes in-house creators with an external CGC talent network, and the creator-facing page discloses no briefing standard at all. Rubric guidance is explicit that silence on this question scores 2. That is what the low band describes, which is why it scored Weak.

On the record — “Creative iteration off test results is described concretely in case studies: Cane Brew split tested offers and hooks; CorneaCare moved from founder-led content to a multi-creator UGC engine.” pilothouse.co ↗

On the record — “Creative is a named, productized service line (Pilothouse Studios) covering direct response ads, motion graphics, videography, explainer video, ecommerce and lifestyle photography, email and landing page design, with access to in-house creators and a CGC talent network.” pilothouse.co ↗

Evidence: inferred — our reading of indirect evidence, not a documented fact.

Stronger here: New Engen scores Strong on the same dimension.

Adequate — 3 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 15% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Strong 13 · Adequate 29. The typical agency here scores Adequate, and 13 of them score higher than this one.

What this dimension measures: Can one approved concept become 100+ assets across aspect ratios, placements, languages and markets? Resizing only, single language, scores 3. Distinguish genuine transcreation (copy reworked for a market) from machine translation dropped into a template; the latter is not localization capacity and should be scored as resizing.

Scores high — Named format/placement coverage plus real multi-market or multi-language delivery scores 4-5.

Scores low — 'We make ten great ads' scores 2 — excellent and out of category.

What we found — Format and placement coverage is named and broad: statics, motion graphics, illustration, videography, commercials, explainer video, ecommerce and lifestyle photography, email and landing page design, plus placement-specific work across 'feeds, Reels, and Stories', TikTok feed and Spark Ads, and Amazon listing imagery. The TikTok page describes testing 'first-three-second variants' and hook variants, and the Cane Brew case study shows genuine variant families rather than resizes - flavour-specific creative (strawberry, watermelon), seasonal cuts (July 4th, National Iced Tea Day), split-tested offers and hooks, and USP callout statics. That is more than resizing. What is absent is localization: nothing on any page read describes multi-language delivery, transcreation, or non-US market work, and the client roster read is US and Canada DTC. Rubric puts named format coverage without real multi-market or multi-language delivery at 3. That is between the two bands, which is why it scored Adequate.

On the record — “Creative is a named, productized service line (Pilothouse Studios) covering direct response ads, motion graphics, videography, explainer video, ecommerce and lifestyle photography, email and landing page design, with access to in-house creators and a CGC talent network.” pilothouse.co ↗

On the record — “Creative iteration off test results is described concretely in case studies: Cane Brew split tested offers and hooks; CorneaCare moved from founder-led content to a multi-creator UGC engine.” pilothouse.co ↗

DTC — direct-to-consumer: brands selling on their own site rather than through retailers.

Evidence: vendor stated — the agency’s own claim, recorded as theirs rather than ours.

Strong — 4 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 15% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Excellent 2 · Strong 32 · Adequate 7 · Weak 1. The typical agency here scores Strong, and 2 of them score higher than this one.

What this dimension measures: Does anyone on this vendor's team read the ad account? Look for a stated feedback loop: winners scaled, losers cut, next round briefed off results. Iteration on client-relayed feedback only scores 3.

Scores high — Access to platform performance data plus a named cadence scores 4-5.

Scores low — Ship-and-stop production, where the vendor delivers files and never learns what happened, scores 1-2 — this is the most common failure in the design-subscription tier and it is what separates production from performance creative.

What we found — This is the dimension the vendor is structurally set up to pass. Pilothouse buys the media it makes creative for - Meta, TikTok, Google, Amazon, YouTube are all published service lines - so the creative team is not working blind to results, and ship-and-stop delivery is not the model. The Meta page states 'Iteration tied to performance, not subjective creative preference' and lists 'Creative-level performance analysis' under reporting; the TikTok page describes a creator network with 'briefing methodology and performance feedback loops'. Two case studies show the loop in specifics rather than in slogan: Cane Brew ('By split testing offers and hooks, we found the most effective way to overcome the taste-test barrier'), and CorneaCare, where founder-led content was replaced by a 'multi-creator UGC engine' after testing. Held back from Excellent because no page names an iteration cadence - reporting is described only as 'Regular business reviews', with no weekly or per-round rhythm stated - and the corroboration cap applies to a claim visible only on the vendor's own site. That is the high band above, which is why it scored Strong.

On the record — “Creative iteration off test results is described concretely in case studies: Cane Brew split tested offers and hooks; CorneaCare moved from founder-led content to a multi-creator UGC engine.” pilothouse.co ↗

On the record — “The Meta service page states '5,000+ Creative Assets Monthly' and 'Iteration tied to performance, not subjective creative preference', and lists creative-level performance analysis under reporting.” pilothouse.co ↗

UGC — user-generated-content style ads: creative shot to look like a customer’s own phone video rather than a produced commercial.

Evidence: partly checkable — corroborated in part against the sources below; the remainder rests on the agency’s own account.

Adequate — 3 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 10% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Strong 11 · Adequate 19 · Weak 12. The typical agency here scores Adequate, and 11 of them score higher than this one.

What this dimension measures: Legal disclaimers, claim substantiation, platform specs and safe zones, accessibility contrast, rights and licensing on music/footage/creator content. Platform-spec competence only scores 3. Do not fail a small shop for lacking pharma review workflows it was never asked for.

Scores high — Explicit process, or regulated-category experience (health, finance, alcohol, children's), scores 4-5.

Scores low — No evidence either way scores 2 with LOW confidence rather than a penalty — most SMB-facing shops never discuss this publicly, and absence here is weaker evidence than absence on volume or ownership.

What we found — Platform-spec competence is evidenced: Amazon listing images built to Amazon's own guidelines (CorneaCare case study), placement-native builds for Reels, Stories and Spark Ads, and format work across marketplace and social surfaces. There is also incidental regulated-adjacent client experience visible on the case study and testimonial pages - CorneaCare (eye health), Four Sigmatic (functional mushroom supplements), KilgourMD (dermatology), Prai Beauty. What is not evidenced anywhere: a claim-substantiation step, legal or disclaimer review, accessibility contrast, safe-zone checks, or any statement about music, stock footage or creator-content licensing. Rubric puts platform-spec competence alone at 3 and explicitly warns against penalising a shop for review workflows it was never asked for, so this sits at Adequate with low confidence rather than lower. That is between the two bands, which is why it scored Adequate.

Evidence: vendor stated — the agency’s own claim, recorded as theirs rather than ours.

Adequate — 3 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 10% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Strong 3 · Adequate 23 · Weak 15 · Poor 1. The typical agency here scores Adequate, and 3 of them score higher than this one.

What this dimension measures: Who owns the output — final files, source/working files, raw footage, and the rights to creator content? Documented designer discontinuity in independent reviews is a verified Poor, not an inference.

Scores high — Explicit transfer of source files and full usage rights scores 4-5.

Scores low — Finals only, working files withheld, scores 2. Silence scores 2; at this price point silence usually resolves against the buyer. Separately, continuity: the same named designers month to month scores 4-5, a rotating pool with no continuity guarantee scores 1-2.

What we found — Split result. Continuity is the well-evidenced half: the testimonials page carries named contacts at named companies describing multi-year relationships (Hestan Culinary four years, Santevia '40% YoY growth two years', Four Sigmatic's Head of Growth calling retention 'ridiculously high' and the team 'like an internal team'), and the Studios page promises 'a single account manager for all creative services'. With 175 staff against 125 active brands, the staffing ratio is consistent with assigned rather than pooled teams, though no page states that. Ownership is the silent half, and it is silent completely: no page read - homepage, Studios, the service pages, get-in-touch, or the creator recruitment page - states who owns final files, whether source or working files transfer, what usage rights attach to CGC or creator footage, or whether licences expire. Ownership silence alone scores 2 under the rubric; the documented continuity is what lifts this to Adequate rather than Weak. That is between the two bands, which is why it scored Adequate.

On the record — “The testimonials page carries named contacts at named client companies describing multi-year relationships, including Hestan Culinary (fourth year) and Santevia (40% YoY growth over two years).” pilothouse.co ↗

On the record — “The about page states the company scaled to 175 team members managing over 125 active brands, and lists a leadership team including a Head of Creative Operations; office listed at 1011 Government Street, Victoria, BC.” pilothouse.co ↗

Evidence: partly checkable — corroborated in part against the sources below; the remainder rests on the agency’s own account.

Weak — 2 of 5 on this rubric’s scale, from Poor (1) to Excellent (5). This dimension carries 10% of the total score.

Benchmark — across the 42 other agencies evaluated in this discipline, this dimension runs Excellent 1 · Strong 9 · Adequate 7 · Weak 25. The typical agency here scores Weak, and 17 of them score higher than this one.

What this dimension measures: Published tiers with an unstated volume ceiling score 3. Note any minimum spend, and note whether pausing is permitted, since seasonal advertisers pause.

Scores high — Published per-asset, per-month or per-round pricing scores 4-5. Month-to-month or per-project scores 4-5.

Scores low — Bespoke-only quoting with no public anchor scores 1-2. Twelve-month lock-in on a production service scores 2 — production is inherently variable in volume and a long lock is a transfer of that risk to the buyer.

What we found — Bespoke-only quoting with no public anchor. The homepage names a commercial model - 'monthly retainers, performance-based incentives, and custom project scopes' - which is more than most, but attaches no number to any of it. No per-asset rate, no per-month tier, no per-round price, no published minimum spend, and no stated minimum retainer. The get-in-touch page carries no pricing, no qualification criteria and no commitment terms; the Studios, Strategy, Meta and TikTok service pages all route to a call or a partnership inquiry instead of a figure. Nothing read states contract length, notice period, or whether a seasonal advertiser can pause - a material gap for a DTC buyer whose volume swings around BFCM, which the vendor itself markets around. Absence measured across six pages, not one. That is what the low band describes, which is why it scored Weak.

On the record — “No pricing, minimum spend, minimum retainer, contract length or qualification criteria appears on the contact page or on the Studios, Strategy, Meta or TikTok service pages; the homepage names only 'monthly retainers, performance-based incentives, and custom project scopes' with no figures.” pilothouse.co ↗

On the record — “The Meta service page states '5,000+ Creative Assets Monthly' and 'Iteration tied to performance, not subjective creative preference', and lists creative-level performance analysis under reporting.” pilothouse.co ↗

retainer — a fixed monthly fee regardless of hours or output — predictable, but worth tying to a defined scope.

DTC — direct-to-consumer: brands selling on their own site rather than through retailers.

Evidence: inferred — our reading of indirect evidence, not a documented fact.

Stronger here: Shuttlerock scores Excellent on the same dimension.