FindwiseBot and methodology

FindwiseBot is the crawler behind the Findwise store scan. It reads a small sample of a store's public pages. The short answers are here; the full policy follows.

The exact user agent

FindwiseBot/0.1 (+https://getfindwise.com/bot)

Findwise's team may also run the same crawler from a development machine. Those requests are never signed, keep every rule and limit on this page, and name themselves:

FindwiseDev/0.1 (+https://getfindwise.com/bot)

How to block or allow it

To block it, add this to your robots.txt. FindwiseBot answers to the FindwiseBot group, and applies the * group when there is no group of its own.

User-agent: FindwiseBot
Disallow: /

It stops on the next scan, and blocking needs no message to us. A scan reads only the pages robots.txt allows: if that leaves none, it makes no report, and if robots.txt kept it off some pages, the report says so. There is no mode that crawls around a robots rule.

The same rule stops the team's developer runs too. To stop only those, write FindwiseDev in place of FindwiseBot. A bot-protection or firewall rule written for FindwiseBot's name alone does not match them; to stop them that way, add FindwiseDev to it.

To let it in where your * group keeps other crawlers out, give it a group of its own:

User-agent: FindwiseBot
Allow: /

The limits it keeps

What it fetches
The store's public pages, its robots.txt and sitemap files, a few pages of the store's own help centre when the store links it, and, on stores built on a common platform, that platform's standard policy addresses even when no page links them, robots.txt permitting. No logins, no forms, no checkout.
Signed identity
When signing is configured, Ed25519 HTTP message signatures identify FindwiseBot as https://www.getfindwise.com. The public keys are then published at /.well-known/http-message-signatures-directory on that hostname.
Developer runs
Findwise's team may also run the same crawler from a development machine. Those requests name themselves FindwiseDev/0.1 (+https://getfindwise.com/bot), are never signed, and keep every rule and limit on this page. They obey the robots.txt rules for FindwiseBot and also the rules for FindwiseDev, or for * when there are none for FindwiseDev: a page is read only when both allow it, and the longer Crawl-delay is kept. A robots.txt rule that blocks FindwiseBot stops them too; to stop only them, write one for FindwiseDev instead. A bot-protection or firewall rule written for FindwiseBot's name alone does not match them; to stop them that way, add FindwiseDev to it.
Pages per scan
At most 30.
Request rate
One request per second per host, two at a time at most.
Page size
2.5 MB per page. Anything larger is truncated.
Time
On a slow site, or one that sets a Crawl-delay, a visit can take several minutes. Each page request times out after 10 seconds.
Crawl-delay
Honoured as written, however long. A long delay means fewer pages are read, never faster requests.
Rate limits
A 429 Too Many Requests response pauses the scan. Findwise waits at least as long as the Retry-After header requests, then continues automatically at a slower rate, unless the scan ends as described below. This applies to pages, sitemap files and robots.txt, including the store's own help centre. Without a usable Retry-After value, Findwise applies an increasing delay. While it waits, the scan remains in progress and keeps the same page limits. Robots.txt is read again before requests continue. If the store keeps answering with rate limits, or asks for a longer wait than a scan stays open, Findwise may end the scan instead: its result reports the rate limit and does not assess the store. A new scan can start once the limit is lifted and any wait the store asked for has passed. If only the help centre keeps limiting requests, Findwise stops reading the help centre and the rest of the scan continues.
Redirects
At most 5 hops. The address you submit may be followed once to another public https site. A further hop is not followed.
Scope
One site at a time: www and the bare domain count as the same site, and other subdomains and sites are not followed, with two exceptions. When the store's own pages link its FAQ or help centre on the store's own domain or another address under it, the scan may read a few pages there, counted inside the page limit, after reading that address's robots.txt. If the address you submit redirects once to another public https address, the scan follows that one hop, after reading that address's robots.txt; a second hop is not followed.

Findwise methodology

How does Findwise score a store?

Versioned automated checks read a sample of a store's public pages and look at six areas: crawlability, structured data, product page detail, comparison coverage, trust signals and answer snippets. Scores are estimates from what the scan could read, and each finding points to the evidence behind it. A report gives an overall score only when at least one product page was read in full; when none was reached, every product page was only partly read, or every product page read in full carried a markup block too large to read and showed no readable product markup, the report says so instead. A report made before this rule or any part of it was added keeps the scoring it was made with. When robots.txt prevents access to every page, the result only describes that access problem, with one access finding and its evidence and fix. When a scan ends because the store keeps limiting our requests, or asks for a longer wait than a scan stays open, the result is one rate-limit finding and a suggested fix. These results have no scores and are not for sale. An optional AI step tags what kind of site was scanned and what it sells; it does not change the scores or write any of the report.

What does Estimated coverage mean?

Estimated coverage compares the captured store evidence with a versioned set of buyer-question patterns. It describes how much supporting information the scan found. Findwise does not test AI assistants directly and does not predict traffic or sales lift.

Where does the example report come from?

The example uses a sample store built from test pages. It is a stored scan produced by the same scanner and read through the same report service as a visitor's scan, not a live customer scan or a report written for display.

Does a scan cover every page?

A scan reads a bounded sample of publicly reachable pages and honours robots.txt. Missing or blocked pages limit what the report can conclude. The crawler limits and full policy on this page explain that boundary.

Findwise Bot And Crawl Policy

Last updated November 11, 2026 · Version bot-2026-11-11-v0.17

This policy explains how FindwiseBot is designed to crawl public ecommerce websites for Findwise scans and reports.

Contact: legal@getfindwise.com

1. What FindwiseBot Does

FindwiseBot reads public pages, robots.txt files and sitemap files of the sites Findwise scans, within the limits in section 4. Findwise uses what it reads to estimate how well AI shopping systems could find, understand, trust, compare, and recommend a store's products.

A FindwiseBot visit does not mean the website owner requested, endorsed, approved, purchased, or is affiliated with Findwise.

FindwiseBot does not intentionally crawl login, account, cart, checkout or admin pages, private APIs, links that perform an action rather than show a page (such as adding to cart or logging out), or other non-public content. If a public page contains information that should not have been public, contact legal@getfindwise.com and secure the page at the source.

2. User Agent

Production crawler user agent:

FindwiseBot/0.1 (+https://getfindwise.com/bot)

The version number may change.

When signing is configured, FindwiseBot also signs its requests with Ed25519 HTTP message signatures (Signature-Input, Signature and Signature-Agent) that identify it as https://www.getfindwise.com. The public keys are then published at https://www.getfindwise.com/.well-known/http-message-signatures-directory.

Findwise's team may also run the same crawler from a development machine. Those requests name themselves:

FindwiseDev/0.1 (+https://getfindwise.com/bot)

They are never signed, and they keep every rule and limit in this policy. They obey the robots.txt rules for FindwiseBot (section 3) and also the rules for FindwiseDev, or for * when there are none for FindwiseDev: a page is read only when both allow it, and the longer Crawl-delay is kept.

3. Robots.txt

FindwiseBot follows robots.txt on every address it reads, including the store's help centre: the rules for FindwiseBot if there are any, otherwise the rules for *.

Findwise may also honor reasonable requests sent to legal@getfindwise.com. Findwise may verify domain ownership or authority before applying a manual block, correction, or removal.

If the store's own robots.txt returns a server error or does not answer, FindwiseBot reads nothing else from the store, as the robots.txt standard (RFC 9309) asks, and the scan ends without a report. A 429 Too Many Requests response pauses the scan, or ends it as section 4 describes; no pages are requested until robots.txt can be read again after the wait. A missing robots.txt means no rules. Another answer it cannot read as rules, such as a web page served in its place, is read as no rules, and the report says so. It never treats an unreadable file as permission to crawl harder.

Robots.txt is not a security boundary. Website owners should not place private or sensitive information on public URLs, even if those URLs are disallowed in robots.txt.

4. Crawl Limits

FindwiseBot is designed to crawl politely:

  • One site per scan: www and the bare domain count as one, and other subdomains and sites are not followed, with two exceptions.
  • The store's own FAQ or help centre, on the store's own domain or another address under it: when the store's pages link it, the scan may read a few pages there, counted inside the page limit.
  • The address submitted for the scan: if its first response redirects to another public https address, the scan follows that one hop, and no further hop to another site.
  • Public HTML pages only, besides robots.txt and sitemap files.
  • On stores built on a common platform it may also request that platform's standard policy addresses, even when no page links them, robots.txt permitting.
  • Static assets ignored.
  • Crawl-delay honored as written, however long. A long delay means fewer pages are read, never faster requests.
  • A 429 Too Many Requests response pauses the scan. Findwise waits at least as long as the Retry-After header requests, then continues automatically at a slower rate, unless the scan ends as described below. This applies to pages, sitemap files and robots.txt, including the store's own help centre. Without a usable Retry-After value, Findwise applies an increasing delay. While it waits, the scan remains in progress and keeps the same page limits. Robots.txt is read again before requests continue. If the store keeps answering with rate limits, or asks for a longer wait than a scan stays open, Findwise may end the scan instead: its result reports the rate limit and does not assess the store. A new scan can start once the limit is lifted and any wait the store asked for has passed. If only the help centre keeps limiting requests, Findwise stops reading the help centre and the rest of the scan continues.
  • Other interruptions to the scan for a paid report may be retried. Each new scan keeps every limit on this page and the store's robots.txt rules.
  • Quick scans: up to 8 HTML pages.
  • Deeper scans, including all Free and Premium membership assessments: up to 30 HTML pages.
  • Maximum page size: 2.5 MB of HTML per page. A larger page is truncated, and the report says so.
  • Per-page timeout: 10 seconds.
  • Scan length: on a slow site, or one that sets a Crawl-delay, a visit can take several minutes.
  • Host request rate: approximately 1 request per second.
  • Concurrency: no more than 2 requests per host.
  • Redirects: at most 5 hops.

These limits may be adjusted to protect site owners, reduce abuse, improve reliability, or support paid scans.

Site owners should rely on the current published policy and their own server logs rather than assuming any single scan reflects all future behavior.

Findwise may stop, slow, deny, or block a scan if we detect unusual load, abuse risk, legal risk, security risk, inaccurate configuration, domain-owner objection, or other operational concern.

5. What Findwise Does Not Do

FindwiseBot does not:

  • Bypass login, paywalls, robots.txt, CAPTCHAs, or technical access controls.
  • Submit forms, add items to cart, initiate checkout, or perform destructive actions.
  • Intentionally collect payment card numbers, passwords, API keys, protected health information, government identifiers, or other sensitive personal information.
  • Sell personal information collected through scans, or use scans as a people-search, credit, employment, insurance, housing, or other regulated decisioning service.

Findwise does not claim that a scan is complete, exhaustive, or equivalent to how any particular AI provider, search engine, or shopping agent will read the site, and does not guarantee that crawled content is accurate, lawful, endorsed, current, complete, or suitable for any customer's purpose.

6. Raw Crawl Data And Reports

Findwise may store raw public HTML, extracted evidence, scan metadata, findings, and reports to provide the Services, debug scan quality, support customers, and improve scoring.

Raw crawl data is not exposed in public free reports. Paid reports may include selected evidence snippets, page references, and recommendations. Findwise may remove, redact, or withhold evidence where privacy, security, legal, or third-party concerns apply.

7. How To Block Or Contact Findwise

To block FindwiseBot with robots.txt:

User-agent: FindwiseBot
Disallow: /

This also stops the team's developer runs (section 2). To stop only those, write FindwiseDev in place of FindwiseBot. A bot-protection or firewall rule written for FindwiseBot's name alone does not match those runs; to stop them that way, add FindwiseDev to it.

For questions, removal requests, crawl issues, or abuse reports, contact legal@getfindwise.com and include:

  • Your domain.
  • The issue or request.
  • Relevant log lines if available.
  • A contact email for follow-up.

Findwise may verify requests before making changes.