Use robots.txt first if you want a polite request; use Cloudflare WAF if you need AhrefsBot traffic to actually stop reaching your origin. AhrefsBot usually respects crawl rules, so blocking it in robots.txt is clean and low-risk. But if bandwidth, server load, or log noise is already a problem, an edge-level block is more reliable.
TLDR: robots.txt tells AhrefsBot not to crawl, while Cloudflare WAF can block requests before they hit your server. For example, a site seeing 120,000 monthly AhrefsBot requests might cut origin traffic by over 90% with a WAF rule, while still keeping normal search bots untouched. A practical setup is: disallow AhrefsBot in robots.txt, monitor logs for 7 to 14 days, then add Cloudflare rules if requests continue or server costs stay high.
Table of Contents
Why site owners block AhrefsBot
AhrefsBot is the crawler used by Ahrefs, a major SEO data provider. It scans the web to build backlink, keyword, and content databases. That can be useful if you rely on third-party SEO tools. It can also be irritating when the bot creates heavy crawl volume on a small server.
The main reasons to block it are simple:
- Server load: frequent crawling can increase CPU, RAM, and database usage.
- Bandwidth cost: image-heavy sites can pay for bot traffic that brings no users.
- Log pollution: bot requests can bury real user behavior in analytics and server logs.
- Content control: some publishers do not want SEO tools indexing their pages.
- Security hygiene: fake bots may pretend to be AhrefsBot using the same user agent.
The key question is not whether blocking is possible. It is where to block: at the crawler instruction layer, the CDN edge, the application, or the server.
Option 1: Block AhrefsBot with robots.txt
The simplest method is to add a rule to your robots.txt file. This file sits at the root of your site, such as https://example.com/robots.txt. Crawlers check it before indexing pages.
A basic AhrefsBot block looks like this:
User-agent: AhrefsBot
Disallow: /This tells AhrefsBot not to crawl any part of the site. If you only want to block certain sections, you can be more selective:
User-agent: AhrefsBot
Disallow: /private/
Disallow: /search/
Disallow: /tag/The benefit is trust and simplicity. It is easy to implement, easy to reverse, and does not risk blocking real visitors. It also avoids messy firewall rules.
The downside is obvious. robots.txt is a request, not enforcement. Legitimate AhrefsBot traffic is expected to comply, but spoofed bots and bad crawlers can ignore it. If a bot lies about its identity, your server still receives the request.
The catch is that many teams think the job is done after editing one text file. Then they check access logs three days later and see thousands of bot-style hits still burning resources. That does not always mean Ahrefs is ignoring the rule. It may mean cached behavior, delayed recrawling, fake user agents, or other bots being grouped incorrectly.
Option 2: Block AhrefsBot with Cloudflare WAF
Cloudflare WAF blocks traffic at the edge before it reaches your origin server. That is the main reason to use it. If the problem is load, not just indexing, edge blocking is usually the stronger control.
A common Cloudflare custom rule checks the user agent:
http.user_agent contains "AhrefsBot"Then set the action to Block, Managed Challenge, or JS Challenge. For a crawler, Block is usually the cleanest choice.
You can also create a more cautious rule:
(http.user_agent contains "AhrefsBot") and not ip.src in {trusted_ip_list}This is useful if you maintain an allowlist. Still, user-agent blocking has a flaw: it trusts the request header. A bad bot can remove or fake that string in seconds. It drives me crazy that many dashboards make this look more precise than it really is.
Cloudflare WAF is best when you need measurable origin relief. You can check Firewall Events, compare origin logs, and watch bandwidth drop. If AhrefsBot made up 18% of requests last week, a successful WAF rule should show a visible fall in server hits within hours.
Robots.txt vs Cloudflare WAF
| Method | Best for | Main weakness |
|---|---|---|
| robots.txt | Polite crawler control | Does not stop requests at the server |
| Cloudflare WAF | Reducing origin traffic | User-agent rules can be spoofed |
| Bot management | Advanced detection and scoring | May require higher-tier plans or tuning |
If your goal is to tell Ahrefs not to crawl, use robots.txt. If your goal is to protect resources, use Cloudflare WAF. If your goal is to separate real users, search engines, SEO crawlers, scrapers, and fake bots at scale, consider bot-management tools.
Cloudflare bot-management alternatives
Cloudflare is not the only option. Depending on your stack, you may have better control elsewhere.
- Cloudflare Bot Management: available on higher plans. It scores traffic based on behavior, browser signals, and request patterns. It is stronger than simple user-agent rules.
- Cloudflare Super Bot Fight Mode: easier to enable, but less flexible. It can be useful for smaller sites, though false positives need monitoring.
- Server-level rules: Nginx, Apache, LiteSpeed, and Caddy can block user agents before application code runs.
- Application firewall plugins: WordPress security plugins can block bots, but they often act after PHP has loaded, so they may not save much CPU.
- Rate limiting: useful when you do not want a full block. For example, allow 60 requests per minute, then challenge or deny.
- Reverse proxy tools: Fastly, Akamai, Imperva, and AWS WAF can enforce bot controls similar to Cloudflare.
- Log-based blocking: tools such as Fail2ban can watch logs and block abusive patterns at the firewall level.
For most small and mid-size sites, the best path is layered. Start with robots.txt. Add Cloudflare WAF if crawl volume remains high. Add bot scoring or rate limits only when simple rules no longer work.
Recommended setup
- Add the robots.txt rule: block AhrefsBot cleanly and wait for the crawler to adjust.
- Verify the file: visit
/robots.txtin a browser and confirm it is public. - Monitor logs: compare requests before and after the change. Use a 7-day window at minimum.
- Add a Cloudflare WAF rule: block
AhrefsBotby user agent if traffic still reaches the origin. - Watch for fake bots: look for strange IP ranges, high request rates, and missing browser headers.
- Do not block Googlebot by accident: keep rules narrow and test with known good crawlers.
SEO and business trade-offs
Blocking AhrefsBot does not block Google from indexing your site. Googlebot is separate. Your rankings should not drop just because Ahrefs cannot crawl you.
Still, there is a business trade-off. If Ahrefs cannot crawl your site, its reports about your backlinks, pages, and keywords may become incomplete. Agencies, partners, or buyers using Ahrefs may see less data. That may matter for sites being audited, sold, or monitored by SEO teams.
If you run a private SaaS, paid content site, staging area, or cost-sensitive store, blocking can make sense. If your marketing team depends on Ahrefs data every week, use rate limits instead of a full block.
Practical rule of thumb
Use robots.txt for policy. Use Cloudflare WAF for enforcement. That pairing is simple, serious, and easy to explain during a technical review.
A good final configuration might look like this: robots.txt disallows AhrefsBot, Cloudflare blocks requests with the AhrefsBot user agent, and rate limiting catches unknown crawlers making hundreds of requests per minute. Review logs every month. Remove rules that no longer help. Keep the controls narrow, because blocking too broadly can hurt real users faster than any SEO crawler ever could.