AhrefsBot

Name: AhrefsBot
Author: Ahrefs

SEO Tools

Verify AhrefsBot IP Address

Verify if an IP address truly belongs to Ahrefs, using official verification methods. Enter both IP address and User-Agent from your logs for the most accurate bot verification.

AhrefsBot is the web crawler operated by Ahrefs that powers the link and content database used by both Ahrefs, a marketing intelligence platform, and Yep, a privacy-focused search engine. It crawls webpages to discover links, anchor text, content changes, and technical signals used for SEO analysis and search indexing. Crawl activity can be moderate to high depending on site size and update frequency. Its purpose is to build a comprehensive, up-to-date web graph for SEO insights and search relevance across Ahrefs’ products. RobotSense.io verifies AhrefsBot using Ahrefs official validation methods, ensuring only genuine AhrefsBot traffic is identified.

This bot officially honors Crawl-Delay rule.

User Agent Examples

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Robots.txt Configuration for AhrefsBot

Robots.txt User-Agent:AhrefsBot

Use this identifier in your robots.txt User-agent directive to target AhrefsBot.

Recommended Configuration

Our recommended robots.txt configuration for AhrefsBot:

User-agent: AhrefsBot
Allow: /

Completely Block AhrefsBot

Prevent this bot from crawling your entire site:

User-agent: AhrefsBot
Disallow: /

Completely Allow AhrefsBot

Allow this bot to crawl your entire site:

User-agent: AhrefsBot
Allow: /

Block Specific Paths

Block this bot from specific directories or pages:

User-agent: AhrefsBot
Disallow: /private/
Disallow: /admin/
Disallow: /api/

Allow Only Specific Paths

Block everything but allow specific directories:

User-agent: AhrefsBot
Disallow: /
Allow: /public/
Allow: /blog/

Set Crawl Delay

Limit how frequently AhrefsBot can request pages (in seconds):

User-agent: AhrefsBot
Allow: /
Crawl-delay: 10

Note: This bot officially honors the Crawl-delay directive.