About CrawlinxBot

Crawlinx runs an on-demand technical-SEO audit crawler. When your site is submitted for a scan we fetch and analyze its pages to produce a report. This page helps you let our crawler through so the audit sees your real pages instead of a bot-challenge page — and so your report is complete and accurate.

What CrawlinxBot does

• Requests a bounded set of pages (default 50, up to 200) starting from the submitted URL, following on-site links.
• Fetches over HTTP and, where a page needs JavaScript to render, with a headless browser — the same way a search engine renders.
• Rate-limits itself per host and does not attempt password/login-protected areas or anything behind an access control.

Why a scan can come back incomplete

To reduce false 429s and blocks on aggressive hosts, our fetcher presents a normal browser fingerprint (a current Chrome/Firefox User-Agent). It does not send a distinctive CrawlinxBot User-Agent, so you cannot allow us by UA string and a User-agent: CrawlinxBot line in robots.txt has no effect on us. If your CDN or WAF challenges automated traffic, our fetch may receive a CAPTCHA / interstitial / 403 instead of your page, and the report will reflect that page rather than your content.

The reliable signal to allow is our source IP. Run a scan, watch your origin/edge access logs for the burst of requests, note the source IP, and allow that IP in the relevant system below. (Because scans are on-demand, we do not publish a single fixed IP.)

Letting the audit through, by protection system

Cloudflare

Most common. Any one of these lets the audit complete:

• Ease Bot Fight Mode. Security → Settings, filter by Bot traffic. For Bot Fight Mode turn it off; for Super Bot Fight Mode set the bot groupings (especially “Definitely automated” / “Likely automated”) to Allow for the duration of the scan.
• Add a Skip rule (surgical, preferred). Security → Security rules (older dashboards: Security → WAF → Custom rules) → Create rule. Expression: ip.src eq YOUR.CRAWL.IP.HERE, action Skip, then check the components to skip (Super Bot Fight Mode, and any managed/custom rules that were blocking the fetch).
• Or an IP Access rule. Security → WAF → Tools → IP Access Rules: add our IP with action Allow. Note the newer Skip action supersedes the legacy Allow.
• If you use a high Security Level, lowering it during the scan also reduces spurious challenges.

Sucuri Firewall (WAF)

• In the Sucuri dashboard open your site → Firewall → Access Control.
• Under Whitelist IP Addresses add our source IP and save; whitelisted IPs bypass the firewall's security checks.
• If you locked the origin to Sucuri's IPs via .htaccess, that only affects direct-to-origin traffic and does not need changing here.

AWS WAF

• Create an IP set (WAF → IP sets) containing our source IP.
• In your web ACL add a rule that references that IP set with action Allow, placed above your blocking rules so it evaluates first.
• For a rate-based rule (which cannot use Allow), add a scope-down statement that excludes the IP set — a NOT wrapping an IP set reference statement — so our IP is not counted or throttled.

Akamai Bot Manager

• Create a custom bot category (you cannot add bots to Akamai's built-in categories), then define a custom-defined bot in it that matches our source IP.
• In your Bot Manager policy set that category's action to Allow.
• Allow-listing the built-in SEO/Analytics category will not include us, since we do not self-identify as a known SEO bot — use the custom category above.

Imperva / Incapsula

• Add a security exception (Website → Security settings) scoped by source IP so trusted traffic bypasses the WAF and bot rules.
• Exceptions can also key on IP range or client, but IP is the reliable match for our crawler.

DataDome

• In the DataDome dashboard add a custom rule with an allow-list matching our source IP (DataDome supports allow-listing good bots by IP / IP range / reverse DNS).

Server-side rate limiting

• nginx (limit_req): use a geo block to set a flag for our IP, then map that flag to an empty $limit_key so the rate-limit zone skips it — e.g. geo $rl { default 1; YOUR.CRAWL.IP 0; } then map $rl $key { 0 ""; 1 $binary_remote_addr; } and key the zone on $key.
• Apache mod_evasive: add our IP to DOSWhitelist in the module config.
• fail2ban: add our IP to ignoreip in the [DEFAULT] section of jail.local, then fail2ban-client reload. If we were already banned: fail2ban-client unban YOUR.CRAWL.IP.

Our limits and manners

We crawl a bounded page count, rate-limit per host, request only static assets we need, and never touch authenticated or access-controlled areas. If a scan is misbehaving on your infrastructure, tell us and we will stop.

robots.txt

Our crawler obeys the * user-agent group in your robots.txt (standard Allow/Disallow matching). It does not honor named user-agent groups, so a User-agent: CrawlinxBot block will not stop it — to keep us out entirely, use the opt-out below.

Remove my site / opt out

Don't want to be scanned? Use the Remove my site form to remove your domain from our index and opt out of autonomous discovery. This is honored.

Contact

Questions or abuse reports: abuse@crawlinx.com