Reading a monthly marketing report critically

Reports are written by the people being judged by them. That is not a scandal, it is a structural fact, and it means the reader has to do some work. A good agency will welcome the questions. A weak one will explain why the questions are complicated.

You do not need to understand marketing to read one of these well. You need the same skepticism you would bring to a supplier invoice.

What should I check first in any report?

The comparison period, before anything else.

A report comparing this month to last month, this month to the same month last year, and this month to a rolling average will tell three different stories from the same data. All three can be honest. Only one was chosen.

Watch for the period that changes between reports. If last month's report compared year over year and this month's compares to the previous month, ask why. The answer is usually that one of them looked better.

Then find the base. Every percentage in the document should have an absolute number next to it, and if it does not, write the question in the margin. A forty percent increase in conversions means eight became eleven or two hundred became two hundred and eighty, and those are entirely different months.

Percentages on small bases are the most common way an unremarkable month is made to sound like a good one. This is not usually deliberate. Reporting tools default to percentage change because it looks clean.

What does restoring the base actually look like?

Take a real-shaped headline and rebuild it. This takes four minutes with a calculator.

The claim as printed. "Conversion rate improved from 2.1% to 2.9%, a 38% lift month over month."

The numbers underneath it, once you ask. March: 620 sessions, 13 conversions, which is 2.10%derived. April: 655 sessions, 19 conversions, which is 2.90%derived. The relative change is 38%derived and the absolute change is six conversions.

Now say it out loud. Six more inquiries in a month. That may well be a good month. It is not a 38% improvement in anything you can plan against, because six is inside the range a single week of weather produces.

The next question is how much data a real answer would take, and that has a published answer. Kohavi, Deng, Longbotham and Xu, in the KDD 2014 proceedings, state it directly: minimum sample size depends on "the metric's variance and sensitivity (the amount of change one wants to detect)," which means there is no universal traffic threshold. Their own worked table for Bing shows the requirement swinging by more than a factor of twenty depending on which metric you pick. A high-variance revenue metric needed 114k users per variant to detect a 4.4% change. Their least volatile metric, sessions per user, still needed 4.70k users per variant to detect 5.4%.

Apply the friendly end of that to the report above. At 655 sessions a month, accumulating 4.70k sessions for a single variant takes roughly seven months (derived), and a split test needing two of them takes roughly fourteen months (derived). That is for the lowest-variance metric at a search engine, on a page with far more stable traffic than yours. The honest reading of a one-month conversion movement at local volume is that it is a hint, not a result.

That does not mean the work is worthless. It means the report should say "six more inquiries" and not "38% lift," and it means the evidence you actually have is a long before-and-after against the same months last year, described as what it is.

How do I tell whether a number moved for a real reason?

Ask what the number would have done if nobody had done anything.

Seasonality, a competitor closing, a holiday landing in a different week, one large customer, a weather event. Any of these move a small business number further than a month of marketing work does.

The test is base size, and small samples fool people in a specific and predictable way that marketing reports are unusually good at hiding.

Ask for the raw counts in a table alongside the charts. Charts smooth. Tables do not. Anyone who reports monthly should be able to hand you the underlying numbers within a day.

And check the same numbers yourself. Search Console is free, takes two minutes to read, and is under your ownership, which makes it the simplest independent check available. Clicks and impressions there should be in the same neighbourhood as what the report claims for organic.

Which sections of a report deserve the most suspicion?

Four, and they show up in almost every template.

The activity summary. Posts published, keywords tracked, hours spent, links built. This is a work log. It tells you effort was expended. It says nothing about result, and when it appears before the outcome numbers, the order is doing persuasion.

Ranking screenshots. A cropped image of one improved position, with no volume and no location noted. Rankings vary by where the search happens and personalise to the searcher, so a single screenshot is a sample of one taken from a place that may not be your market.

Any AI visibility score, and especially any AI ranking. Google's Search Console generative-AI reports, announced in June 2026, expose impressions, pages, countries, devices and dates. Clicks, click-through rate, position and the user's prompt are explicitly not available, and AI Mode is not separated from AI Overviews. So a report showing your rank inside an AI answer is reading a source that does not exist. Google states the point plainly in its own documentation: "No third-party tool has access to our internal ranking or AI systems." Where a mention rate is reported, ask how many runs per prompt, over what window, and with what variance. Absent those, the number is an estimate wearing the costume of a fact.

Borrowed case-study statistics. This is the one nobody flags, and it is the easiest to check.

What does a borrowed statistic look like when you check it?

It looks like a number with no document behind it, repeated until it sounds like common knowledge.

The example most likely to be in your report is this one: "reducing form fields from eleven to four increases conversions 120%." It circulates in agency decks constantly, it is usually credited to the Baymard Institute, and it is not a Baymard finding and has no traceable primary study. What Baymard actually published, in June 2024, is that the average checkout flow runs 5.1 steps and 11.3 form fields, that 17% of users have abandoned a purchase because checkout was too complicated, and that field count matters far more than step count. That is a directional principle from e-commerce checkout research, and Baymard sells UX benchmarking, so label it as vendor research. It is not a promise about your quote form.

The KDD 2014 authors have a rule for exactly this, and they name names. On published A/B case studies: "The quality varies... Was it peer reviewed? Was it properly run? Were there outliers? Was the p-value low." They cite the widely circulated "red beats green" call-to-action result as their example of a finding that does not generalise, concluding "we believe this is not a general result that replicates well." A tactic that worked on somebody else's site is a hypothesis about yours, not evidence.

The practical move is short. Any number in your report that came from outside your business gets one question: what document is that from, and can I have the link. A good vendor sends the link. A weak one sends a blog post that cites another blog post.

What should be in the report that usually is not?

The three things that matter most to you and least to a vendor.

Money. Cost per inquiry and cost per customer, with the customer number coming from your own records rather than the platform's. Without it, the conversion math that decides whether ads make sense never gets done, and you can spend a year on a channel that was never going to work at your ticket size.

What did not work. Every month contains something that underperformed. A report where everything improved is a report that is filtering. The KDD 2014 paper is blunt about the base rate here: at Bing, most experiments fail, and the ones that succeed move key metrics by 0.1% to 1.0% once diluted to overall impact. A monthly report in which every line improved is describing a success rate no large technology company achieves.

What changed in the business. A price rise, a menu change, a staffing gap, a closed street. These move the numbers and nobody outside the building knows about them. Raising prices without losing covers is exactly the kind of decision that will show up in marketing data and get credited or blamed to the wrong cause.

Add a plain language paragraph at the top: what happened, what we think caused it, what we are doing next. If a vendor cannot write that in five sentences, they may not know.

What to do this week

Take your most recent report. Highlight every percentage. Write the missing absolute number next to each one, from your own data if the report does not supply it.

Circle every metric and label it counted or estimated. Two categories, no debate.

Highlight every statistic that came from outside your business and ask for the source document for each.

Compare the organic numbers against Search Console yourself.

Then send three questions back. What period did you compare and why. What are the base numbers behind the top three percentages. What underperformed this month. Those three questions will tell you more about the vendor than the report did.

If you find the report is answering questions you never asked, rebuild the ask from your side. Decide the handful of numbers you want to see every week and require the monthly report to speak to those.

Questions about a report you received?

Email me at eric@seod.com with the first page of your most recent monthly report, as a screenshot or a paste. I will send back the three specific questions I would ask about it, written so you can forward them without it turning into a fight.

I do this myself and I will not use it to bid against your current provider. Plenty of the reports I see are fine, and I say so when they are.

More on measurement sits in the analytics library.