Reference checks that actually tell you something

Most reference calls take four minutes and produce nothing. "Are you happy with them?" invites a yes, and you both go back to work.

The call is worth twenty minutes if you use it correctly. You are interviewing someone who has already lived the next twelve months of your life.

What are references actually for?

Learning the texture, not verifying the sales pitch.

An agency's references are their best relationships. That is fine and expected. Even a best relationship contains friction, waiting, surprises, and things that did not work, and a satisfied client will describe all of it plainly because they have no reason not to.

What you want is a picture of a normal Tuesday. How fast things move, who answers, what the client has to supply, and what happens when something breaks. None of that requires a critical reference to surface.

You are not asking whether the agency is good. You are asking what it is like to be their client.

How selected is the sample I am being handed?

Precisely selected, and you can put a number on it in one question.

Ask how many clients they have. Say the answer is forty and they have given you three references. Three of forty is 7.5%derived of the client base, chosen by the person selling to you. That is not a sample, it is a shortlist, and treating it as evidence about the other 92.5%derived is the mistake most buyers make.

Then ask how many clients left in the last twelve months. Say six of forty, which is 15%derived annual departure. Now you know something the reference list will never show you, because a departed client is by definition not on it. Whether 15% is high or low depends on the model, and there is no published benchmark for agency churn with a disclosed sample, so do not pretend the number has a grade. What it does is start a real conversation, and the reaction to the question is itself informative.

Then use the references to audit the document rather than the relationship. Take three promises from the proposal, the reporting frequency, the response time, and who does the work. Ask each reference what actually happened on all three. If the proposal says monthly reporting and two of three references describe every other month, that is 67%derived of your sample contradicting the document you were about to sign. That is a specific, checkable finding, and it is far more useful than an overall impression.

Which questions get past the script?

Eight, and they take about twenty minutes.

What did the first ninety days feel like? You will hear whether the setup was organized or chaotic, which is the phase you are about to enter.

What surprised you, good or bad? This is the single most productive question in the set.

Tell me about a time something went wrong. Every relationship of a year has one. How the agency handled it is the closest thing to a test you can run without hiring them.

How long does a normal request take, from email to done? Ask for an actual example rather than a policy.

Who does your work, and has that changed? Turnover on your account is the most common quiet degradation.

What do you end up doing yourself that you expected them to do? This uncovers scope gaps that no proposal reveals.

Do you hold your own logins? A client who does not know is telling you something about the agency's habits, and access and ownership are the terms that matter most when a relationship ends.

Would you hire them again for this, or for something different? A yes with a qualifier is the most useful answer you will get all week.

Are the reviews on their website a form of reference?

Not the way most agencies present them, and there is now a federal rule that says so in specific terms.

16 CFR Part 465, the Federal Trade Commission's rule on consumer reviews and testimonials, took effect on 21 October 2024. Three of its sections apply directly to what you are reading on an agency's website.

Section 465.6 prohibits misrepresenting that a website you control provides independent reviews. That is aimed squarely at the "best agencies in your city" pages, and the same logic applies to any directory or roundup that quietly features the publisher's own clients or paying members.

Section 465.5 covers insider reviews. An officer, manager or employee reviewing their own business has to disclose the relationship clearly and conspicuously, and the rule's definition of clear and conspicuous excludes anything a reader has to click or hover to see. The trap is that liability attaches for omission: a business that asks its own staff to leave a review and says nothing about disclosure is exposed even if nobody wrote a false word.

Section 465.7(b) prohibits representing that displayed reviews are most or all of those submitted when reviews have been suppressed by rating or sentiment. A testimonial widget on a homepage, labelled as "our reviews," showing only the top rated ones, is the pattern this section describes.

So the debunk is narrow and worth stating precisely: a curated testimonial page is not an independent review source, and calling it one is now regulated conduct rather than a matter of taste. None of this makes an agency's testimonial page evidence of dishonesty. It makes it evidence of selection, which is exactly what a reference list is, and the correct response to both is to go find the unselected version.

Where it breaks down: a testimonial page that is plainly labelled as selected client comments, with no claim of independence and no suppression claim, is just marketing and there is nothing wrong with it. The rule is about the representation, not the page.

How do I get a reference they did not choose?

Two ways, both ordinary.

Ask the given references for a name. "Do you know anyone else who uses them?" People say yes more often than you would think, and the second degree reference was not selected for you.

Or find clients yourself. Agency portfolios, case studies, and website footers name businesses. Pick one in a different city, call the owner, and ask for five minutes. Small business owners take these calls. You are not competing with them and they remember being where you are.

If you can find a former client, that call is worth more than the rest combined. Ask what changed and whether the parting was clean. You are listening for how the exit was handled, not for whether it happened.

What do the answers actually tell me?

Three things, and you can score them as you go.

Whether the agency does what it says. Compare the reference's description of the monthly rhythm to what the proposal promises you, using the arithmetic above. This kind of gap belongs alongside the other things in a proposal worth questioning before you sign.

Whether they tell clients uncomfortable things. Ask whether the agency ever told them not to do something, or warned them about a change in advance. The best available probe right now is what clients were told about AI answers appearing above the results, because the click impact is real and it is measurable.

Ahrefs compared 150,000 keywords showing an AI Overview against 150,000 matched informational keywords without one, using aggregated Search Console click through rates from March 2024 against March 2025, and measured a 34.5% decrease in clicks. A later cut of the same work revised the figure to 58%, and SparkToro's panel work with Similarweb puts the drop at around 60% when an AI Overview is present. Quote that as a range, never as a point, because the spread comes from different baselines, dates and query mixes. Separately, SparkToro measured 68.01% of US Google searches ending without a click in early 2026, and Google has publicly disputed the methodology behind panel based estimates of this kind, so use the direction rather than the decimal.

AI overviews changed what a ranking is worth, and clients who were warned early had a very different year from clients who were not. A reference who says their agency explained this in advance is describing a firm that reads primary sources.

Whether the work is templated. Ask what was built for them. If the answer is a large number of near identical city pages, ask how those performed, because publishing many thin local pages can hurt rather than help. A volume answer with no reasoning behind it usually means you are buying the same template.

Then check anything the agency claimed against what the reference says happened. Numbers cited in a pitch should match a client's memory, and claims that do not survive checking are more common than they should be.

What to do this week

Ask for three references in your industry or a similar one. Take whoever you get, and note whether all three are recent, because references from four years ago describe a different company.

Ask the two counting questions first: how many clients do you have, and how many left in the last year. Do the division before the calls, so you know what fraction of the picture you are about to see.

Schedule twenty minutes each rather than trading emails. Tone carries information that text does not.

Use the eight questions in order and resist the urge to fill silences. The pause after "what surprised you" is where the useful answer lives.

Then run the check that any agency with a testimonial page should be able to survive, and we are included. Ask where the complete, unselected set of their reviews lives, and go read it yourself. SEOD publishes that it holds 128 verified five star reviews. That figure describes the five star ones, on a public profile where we cannot choose which reviews appear, and the honest instruction is to open the profile and read all of them rather than the page we wrote. Ask every candidate the same question. An agency that can only point you at its own website has given you marketing, not references.

Write one line per call: what would be true for me if this experience repeated. Three lines is a better decision input than any proposal.

Questions about references you have been given?

Email me at eric@seod.com with the reference list an agency provided, first names and industries only if you prefer, and I will send back six questions tailored to that list, including which gaps in it are worth asking about. You make the calls. I have no interest in the account and there is no message after the reply.

There is more on hiring and working with agencies while you are deciding.