# Verify Googlebot in Server Logs Before Acting on Crawl Traffic

<a id="main-content"></a>

SEO &amp; Search

October 28, 2026  ·  8 min read  ·  Victor Valentine Romo

On this page

1. [Keep Four Different Statements Separate](<#keep-four-different-statements-separate>)
2. [Choose the Crawler Category Before the Lookup](<#choose-the-crawler-category-before-the-lookup>)
3. [Complete the Reverse and Forward DNS Sequence](<#complete-the-reverse-and-forward-dns-sequence>)
4. [Use Published Ranges Without Losing the Reference](<#use-published-ranges-without-losing-the-reference>)
5. [Establish the Address Before Trusting a Proxy Field](<#establish-the-address-before-trusting-a-proxy-field>)
6. [Fill In a Synthetic Log Worksheet](<#fill-in-a-synthetic-log-worksheet>)
7. [Review the Record Before Counting the Population](<#review-the-record-before-counting-the-population>)
8. [Hand Off Identity and Search Questions Separately](<#hand-off-identity-and-search-questions-separately>)
9. [Sources](<#sources>)

A log export says Googlebot. Someone copies the total into an SEO report. The next conversation starts with what Google wants from the site.

There is a missing step between the label and that conclusion: establish who made the requests.

Google provides two verification routes: reverse DNS followed by forward DNS, or membership in published crawler IP ranges. Its [request-verification documentation](<https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests>) describes both. A user-agent name alone does not complete either check.

Start with one request you can trace. Keep the original record, identify the address you are testing, and record the evidence behind the result. Then decide which larger question the request can help answer.

The worksheet below is a proposed review method. Its records are synthetic. It does not authorize firewall changes, crawler exemptions, or production configuration changes.

<a id="keep-four-different-statements-separate"></a>

## Keep Four Different Statements Separate

Give the review four fields: claimed agent, observed address, verification result, and search interpretation.

The claimed agent is the text in the request's user-agent field. Preserve that text rather than replacing it with your preferred category name. Your later interpretation needs the original input.

The observed address needs a location. Is it the network peer recorded at the public edge? Is it a value recorded at the origin? Is it a forwarded field? Write where the value came from before checking it.

The verification result records what your chosen method establishes. Use a bounded status such as category match, no match, or unresolved. Attach the method and supporting evidence.

The search interpretation is a separate question. A request to a service page might matter to a [crawl](</articles/crawl-stats-serving-diagnosis-b2b/>) investigation. It does not tell you where that page appears for a buyer's query.

Keeping these fields separate makes a useful correction possible. You can replace an unsupported identity label without throwing away the underlying request record. You can also preserve a verified identity while rejecting an unsupported ranking explanation.

<a id="choose-the-crawler-category-before-the-lookup"></a>

## Choose the Crawler Category Before the Lookup

Google separates common crawlers, special-case crawlers, and user-triggered fetchers. Googlebot belongs with common crawlers. AdsBot is a special-case example. User-triggered requests have separate lists and hostname patterns. The [verification category table](<https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests>) is the reference for that distinction.

Write the intended category into the review. If the question concerns organic-search Googlebot [traffic](</articles/seo-roi-measurement-b2b/>), do not accept a broad statement that the address belongs somewhere in Google's infrastructure.

Record two things when the evidence supports them: the infrastructure category and the claimed product agent. Do not silently convert a common-crawler range match into independent proof of every word in the user-agent string.

When the label and category disagree, preserve both. Ask the reviewer to resolve the discrepancy rather than folding it into the total. A convenient [spreadsheet](</articles/spreadsheet-formula-audit-business/>) category should not erase a different request purpose.

Use the same category rule across the interval you report. If the team changes that rule, [retain the previous definition](</articles/late-arriving-data-report-revisions/>) and identify which totals need another review.

<a id="complete-the-reverse-and-forward-dns-sequence"></a>

## Complete the Reverse and Forward DNS Sequence

For the manual method, Google instructs you to reverse-resolve the accessing address, validate the returned Google domain, then forward-resolve that hostname. The original address must appear in the forward result. Common-crawler examples include names under `googlebot.com`, including `geo.googlebot.com`. See [Google's documented sequence and examples](<https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests#use-command-line-tools>).

Treat that sequence as a chain of evidence. Save the starting address, returned hostname, forward addresses, lookup time, and result. A reviewer should be able to see which comparison you made.

Check the hostname's actual domain boundary. A name that contains the word Google somewhere is insufficient. For this review, require the returned name to fit the documented pattern for the category under investigation.

[Keep an incomplete lookup incomplete](</articles/null-blank-zero-business-data/>). A timeout leaves an unanswered check. A reverse result without the forward comparison leaves an unanswered check. Neither earns the same status as a completed match.

Ask the technical reviewer to explain a mismatch in the record. Avoid replacing the first result repeatedly until a favorable answer appears. If another attempt resolves a temporary failure, retain both attempts and their times.

This article describes the checks. Have the responsible technical owner perform lookups within the team's approved environment. No production command or access-control change is required to understand the evidence.

<a id="use-published-ranges-without-losing-the-reference"></a>

## Use Published Ranges Without Losing the Reference

For larger reviews, compare addresses against the category's published ranges. Google's current [common-crawlers JSON](<https://developers.google.com/static/crawling/ipranges/common-crawlers.json>) includes IPv4 and IPv6 prefixes and a creation [timestamp](</articles/event-time-processing-time-business-reports/>). Its range fields use network prefixes, rather than a list of individual requests.

Have the technical reviewer use an address-aware membership comparison. Do not ask an analyst to match a few starting characters in a spreadsheet. Keep the input address intact through the comparison.

Save the reference URL, retrieval time, and the copy used for the review. Put those beside the result so another reviewer can identify the reference without guessing which download was current.

When reviewing an older interval, state when you obtained the range list. A reference retrieved today gives you a check against today's published evidence. Avoid describing it as a captured record from the request date.

[Record disagreements between methods](</articles/reconcile-conflicting-business-dashboards/>). If a DNS check and a range check differ, retain the source address and both references. Mark the result for technical review rather than choosing whichever answer supports the original report.

Keep the output narrow: address checked, category checked, method, reference, and outcome. Let another worksheet handle how much crawl traffic reached each page.

<a id="establish-the-address-before-trusting-a-proxy-field"></a>

## Establish the Address Before Trusting a Proxy Field

Draw the request path from the public edge to the log you hold. Include a content delivery network (CDN), any worker layer, load balancer, and origin where applicable. Ask the infrastructure owner which field represents the client at each point.

For one concrete configuration, Cloudflare documents `CF-Connecting-IP` on edge-to-origin traffic. Its [HTTP-header reference](<https://developers.cloudflare.com/fundamentals/reference/http-headers/#cf-connecting-ip>) also describes Worker subrequest differences and Pseudo IPv4 behavior. Those conditions matter when choosing which value to inspect.

Do not generalize that field name to every proxy installation. Require the owner to identify the active product, relevant settings, and the path that produced the record.

Use an explicit acceptance condition: the reviewer can explain who sets the selected field and which requests can reach the logging endpoint. If the chain is unknown, leave the crawler identity unresolved.

An arbitrary `X-Forwarded-For` value is insufficient evidence for this worksheet. Do not trust a value merely because it resembles a Google address. Cloudflare's [forwarded-header documentation](<https://developers.cloudflare.com/fundamentals/reference/http-headers/#x-forwarded-for>) describes a chain that can include pre-existing values.

Preserve the peer address and forwarded values separately. Ask for a controlled infrastructure explanation. Do not turn this diagnostic exercise into instructions to expose an origin or bypass a security rule.

<a id="fill-in-a-synthetic-log-worksheet"></a>

## Fill In a Synthetic Log Worksheet

The following addresses use the documentation ranges in [RFC 5737](<https://www.rfc-editor.org/rfc/rfc5737>) and [RFC 3849](<https://www.rfc-editor.org/rfc/rfc3849>). Every row is invented. No DNS lookup or Google range match is asserted for these records.

Use the table to practice preserving uncertainty. The worksheet can describe a useful investigation even when it cannot yet verify a crawler.

| Record | Address recorded | Request time, UTC | Proposed method | Current result | Unresolved proxy detail |
| --- | --- | --- | --- | --- | --- |
| A | `192.0.2.10` | 2026.10.26 09:00:00 | Reverse then forward DNS | Training record only; no verification | Whether this is the edge client or origin peer |
| B | `198.51.100.20` | 2026.10.26 09:02:00 | Common-crawler range membership | Training record only; no verification | Address field supplied without its configuration |
| C | `2001:db8::30` | 2026.10.26 09:04:00 | Range membership after field validation | Unresolved; selected field not accepted | IPv6 handling and proxy path unknown |

Add the requested host, path, response code, raw agent label, and log location to each working record. Keep those fields outside a public report when they contain material the audience should not receive.

Give the real worksheet an evidence column. For DNS, link the two lookup results. For ranges, identify the retained reference and comparison outcome. A status cell without that evidence is an assertion awaiting review.

Keep request time and verification time in separate fields. One describes the event being investigated. The other describes when the reviewer obtained supporting evidence. If the reviewer later repeats a check, append the new result rather than replacing the earlier evidence.

Make the status refer to the completed check. A reviewer might accept the address field while leaving DNS unresolved. Another might finish the range comparison while still questioning the product label. Record those partial results separately. That gives the next reviewer a precise starting point instead of [a single green cell](</articles/color-only-status-business-dashboards/>) that hides unfinished work.

The documentation-only addresses illustrate record structure. Replace them with authorized records in a controlled review. Do not copy the table into an allowlist or describe its rows as verified Google traffic.

<a id="review-the-record-before-counting-the-population"></a>

## Review the Record Before Counting the Population

Have a second reader inspect the first completed record before grouping the whole export. They should identify the request, address field, category, verification method, and remaining uncertainty without asking what each cell means.

If the reader cannot trace the address, stop at that gap. More rows would repeat an unresolved assumption.

Once the definition is usable, keep separate totals for matched, unmatched, and unresolved records. State the interval and the log sources included. Do not distribute unresolved requests across the other groups to make the percentages add up neatly.

Report request counts as request counts. If you also report distinct URLs, [label that calculation separately](</articles/business-metric-dictionary-template/>). Preserve the grouping rule so a later analyst can reproduce the distinction.

For a recurring export, define the evidence fields before connecting systems. The broader [API-first operations discussion](</articles/api-first-business-operations/>) provides integration context. This worksheet's narrower requirement is that the exported identity result remains [traceable to its input and method](</articles/data-lineage-small-business-reports/>).

<a id="hand-off-identity-and-search-questions-separately"></a>

## Hand Off Identity and Search Questions Separately

Google describes crawling, indexing, and serving search results as different stages. It does not guarantee that every crawled page reaches the index or appears in results. See [how Google Search works](<https://developers.google.com/search/docs/fundamentals/how-search-works>).

Write the identity finding without adding a business outcome: the selected records matched the documented verification method for the stated category. Include the unresolved population and the scope limits.

If the question concerns [page indexing](</articles/technical-seo-checklist-b2b-websites/>), open a separate page-level investigation. If it concerns query performance, use a separate [Search Console analysis](</articles/google-search-console-advanced-techniques/>). Neither question becomes answered because the crawler label is now better supported.

Place broader infrastructure findings into the [technical SEO audit](</articles/technical-seo-audit-checklist/>) with their evidence and responsible owner. Do not bury the identity gap in an audit score.

Close this review when another reader can trace the finding from request to address to verification evidence. Leave search outcomes attached to the evidence that can establish them.

Verify the request before counting it as Googlebot.

<a id="sources"></a>

## Sources

- [Google Crawling Infrastructure: Verify requests from Google crawlers and fetchers](<https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests>). Categories, DNS verification sequence, and IP-range verification. Checked 2026.10.28.
- [Google: Common crawler IP ranges](<https://developers.google.com/static/crawling/ipranges/common-crawlers.json>). Current prefix fields and reference timestamp. Checked 2026.10.28.
- [Cloudflare: HTTP headers](<https://developers.cloudflare.com/fundamentals/reference/http-headers/>). Client-address headers, forwarded chains, Workers, and Pseudo IPv4 configuration differences. Checked 2026.10.28.
- [Google Search Central: In-depth guide to how Google Search works](<https://developers.google.com/search/docs/fundamentals/how-search-works>). Separate crawl, index, and search-result stages. Checked 2026.10.28.
- [RFC Editor: RFC 5737](<https://www.rfc-editor.org/rfc/rfc5737>) and [RFC 3849](<https://www.rfc-editor.org/rfc/rfc3849>). Documentation address ranges used in the synthetic worksheet. Checked 2026.10.28.
