The crawl chart drops. The team opens a content backlog and starts looking for pages to rewrite.
Pause at the chart. What question does it answer? What evidence connects its change to the proposed work?
Google positions Crawl Stats as a way to investigate serving problems. The report requires a root-level Domain or URL-prefix property. Google says sites with fewer than a thousand pages generally do not need this detail. See the Crawl Stats documentation.
A small B2B site does not need a crawl-volume target to justify its existence. It needs to serve the material buyers and crawlers request. Use this report when there is a bounded availability question you can hand to a responsible reviewer.
The investigation method below is proposed editorial guidance. The host events and figures are fictional. They demonstrate how to write an evidence request, rather than diagnose a real website.
Write the Question Before Opening the Breakdown
Start with one sentence that names a host, an interval, and an observed issue.
For example: determine whether the documentation host failed to respond during the release window. That question names something an infrastructure reviewer can inspect.
“Improve crawling” leaves the investigation open in every direction. “Make the line go up” substitutes a chart preference for an operating requirement. Neither defines what the reviewer should establish.
Write what triggered the review. It might be an availability warning, a report from a reader, or a maintainer's release note. Keep the trigger separate from the proposed cause.
Choose the evidence that would answer the question before collecting unrelated exports. For a suspected host failure, ask for the affected host's serving evidence. For a suspected wrong redirect, retain the requested address and intended destination.
Set a stopping condition. The first review can end with a supported finding, a request for missing evidence, or a documented unknown. It does not need to end with a website change.
Use the broader Search Console techniques article for query-performance work. Keep this investigation focused on request handling.
Read Each Measure at Its Own Grain
The report counts repeated URL requests and same-site resource requests. Redirect-chain requests count separately. Totals include some attempts that never reach the server, including DNS failures and fetches prevented by robots.txt availability failures. Total download size measures bytes. Average response time covers fetched resources. Response and file-type percentages describe requests, not distinct pages. Google's report reference defines these measures.
Put the metric name beside every number you copy. A request count belongs beside its interval and scope. A response percentage needs the population it describes. A download total needs its unit.
Do not rename the total “pages crawled” in a slide. That change makes a different assertion. If a decision requires distinct page URLs, request evidence that supports that calculation separately.
Use the file-type breakdown to decide which evidence to request next. Ask whether the observed change concerns HTML, images, scripts, styles, PDFs, or another group. Keep those groups distinct in the investigation record.
The point is a focused handoff. A maintainer asked about PDF delivery should not have to infer that scope from an unlabeled aggregate chart.
Preserve the comparison period. If you compare release day with the previous week, record that choice and its reason. Avoid replacing the period during the review because a different comparison looks more dramatic.
Investigate Host Status as a Serving Question
Host-status details cover robots.txt fetching, DNS resolution, and server connectivity. The Crawl Stats help page explains these availability categories.
Copy the reported category rather than translating every warning into “server down.” Ask the technical owner which evidence can establish the specific failure under review.
For a DNS question, request the relevant DNS and service evidence for the affected host. For a connectivity question, request the edge and origin records that can show where handling stopped. For a robots.txt question, preserve the requested path and observed response separately from the intended rules.
Do not let a current successful browser visit erase a historical warning. Record the current observation at its own time. Ask for evidence from the interval the report describes.
Also avoid turning the warning into a diagnosis of every service behind the same domain. Identify the component named in your question. Ask the owner to expand scope only when evidence requires it.
Leave a missing record visible. “Historical edge data unavailable” gives the next reviewer a concrete limit. “Probably a temporary issue” closes the question without showing what supports the explanation.
Keep Response Codes Attached to the Requested URL
Google documents that 5xx and 429 responses can temporarily slow its crawlers. A successful 2xx response does not guarantee indexing. See Google's HTTP-status guidance.
Record the code Google observed, the requested URL, and the intended behavior. A redirect can be the intended answer for an old document. A missing page can be intentional. A working service page returning an error needs a different review.
Ask the maintainer to compare the observed response with the publishing decision. Avoid assigning the same correction to every response outside 200.
For a migration-related change, attach the relevant site-migration checklist and the approved destination map. The investigation still needs evidence for the specific address at issue.
Keep the access decision separate. An authenticated document may have a deliberate access boundary. A crawl warning does not authorize opening that document to the public.
Write the proposed next check before proposing a configuration change. If the evidence cannot distinguish an edge-generated error from an origin-generated error, resolving that distinction is the next task.
Compare Two Fictional Events
Consider two invented hosts reviewed over the same invented seven-day interval. All counts, times, and conditions below are synthetic. They illustrate reasoning, not normal crawl levels.
| Observation | Host A: documentation | Host B: services |
|---|---|---|
| Request-count change | 120 requests on the prior day; 60 on the review day | 120 requests on the prior day; 60 on the review day |
| Availability evidence | Fictional recent connectivity warning | No warning captured in the fictional record |
| Response evidence | Fictional examples show 503 responses |
Fictional examples show 200 responses |
| Change record | A release occurred during the interval | No release information supplied |
| Starting question | Did serving fail during the release window? | What explains the request change, if the evidence can establish it? |
The identical request change does not give these investigations an identical answer.
For Host A, request time-aligned edge and origin records, the deployment timeline, and relevant monitoring evidence. Ask the owner to identify where the 503 responses originated. A nearby release is a candidate explanation until the evidence connects it to the failure.
For Host B, preserve the volume observation. Review the property scope, resource mix, and comparison interval before inventing a serving incident. The supplied record does not establish an outage or a cause for the decline.
Do not label Host B healthy beyond what was inspected. An absence of captured warning evidence is a narrower statement than proof that every request succeeded.
Both records can remain open for different reasons. Host A needs a supported incident explanation. Host B needs a clearer question before further investigation earns its place.
Give the Reviewer a Worksheet They Can Answer
Use one row per question. If the host, interval, or type of evidence changes, start another row. This keeps a broad chart review from becoming an unbounded infrastructure request.
The filled row continues fictional Host A. Its reviewer titles are placeholders for responsibilities, not assigned people.
| Field | Fictional entry | Your working entry |
|---|---|---|
| Investigation question | Did documentation serving fail during the release window? | [One bounded question] |
| Selected property | Root URL-prefix property https://docs.example.com |
[Exact property] |
| Host and protocol scope | docs.example.com; HTTP and HTTPS totals, HTTPS examples |
[Hosts and protocols included in each dataset] |
| Report interval | 2026.10.20 through 2026.10.26 | [Interval as displayed] |
| Observed issue | Connectivity warning and example 503 responses |
[Exact observation] |
| Report evidence | Retained breakdown and example-request details | [Controlled record location] |
| Server evidence needed | Edge and origin records for the release window | [Specific evidence request] |
| Unknowns | Error origin, exact failure interval, and release relationship | [Unresolved questions] |
| Responsible reviewer | Hosting maintainer | [Named accountable reviewer] |
| Acceptance condition | Reviewer connects the observed failure to supporting records, or states the gap | [Inspectable completion condition] |
| Follow-up | Next available maintainer review; no recovery forecast | [Agreed review point] |
Keep the evidence in the team's controlled workspace. The worksheet can link to records without duplicating sensitive log contents into a general marketing report.
Ask the reviewer to answer the stated question and identify the evidence used. If they need a different interval or host, record why the scope changed. Keep the first observation intact.
Reconcile Examples and Logs Without Forcing Equality
Google says example URLs are incomplete, some requests may be omitted, and some failed attempts never reach the server. Its data and example limits explain why report and log totals can differ.
Treat an example as a starting point for investigation. Preserve the details supplied, then ask the maintainer whether a corresponding controlled record exists. A URL missing from the example list does not close a page-level question.
Crawl Stats combines HTTP and HTTPS crawl data even for a URL-prefix property. Its example URLs follow the selected prefix's protocol. A parent domain can also include child hosts. Check Google's scope rules before comparing a report total with one host's HTTPS logs.
Before comparing totals, record the selected property, included hosts and protocols, interval, time basis, request population, and logging location. Ask each owner to confirm what their dataset includes.
Keep the original exports. If the comparison excludes a resource class or narrows a host, record the revised population alongside the original one. Do not silently adjust filters until two totals agree.
If the team cannot reconcile the difference, describe the remaining mismatch and its possible scope limits. Avoid changing analytics or logging configuration merely to make this review appear complete.
Close the Serving Finding Before Claiming a Search Outcome
Use three separate statements in the handoff: what Google reported, what the infrastructure evidence establishes, and what remains unknown.
For the fictional Host A investigation, a completed review might connect specific error responses to a confirmed serving failure. That is a useful finding. Its scope remains the identified host and interval.
Give any proposed repair its own owner and acceptance condition. The person who explains the incident may be different from the person authorized to change the service.
For a resolved repair, preserve implementation evidence and a separate follow-up observation. A completed deployment and a later report change are different events. Keep their times and supporting records separate.
Google describes crawling and indexing as separate stages in its Search explanation. A serving review cannot establish a ranking improvement or explain a sales result.
Place the supported serving finding into the technical SEO audit. Keep query analysis and business measurement in their respective records. If there is no supported defect, report that limited result instead of filling a repair backlog.
Close the question the evidence answers.
Sources
- Google Search Console Help: Crawl Stats report. Eligibility, measures, availability categories, response and resource breakdowns, and data limits. Checked 2026.10.28.
- Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers. Server errors, rate responses, and the limit of successful responses as indexing evidence. Checked 2026.10.28.
- Google Search Central: In-depth guide to how Google Search works. Distinction between crawling, indexing, and serving search results. Checked 2026.10.28.