Ways to Integrate

Get real-time insight into bots browsing your website. Track bot behavior with analytics or programmatically verify and enforce access rules.

Get API KeyView Code Examples

Overview

Get started by choosing from the integration options below based on how your website is built and the subscription plan you have. We're ready to answer any questions you have.

📊

Analytics.js

Free Plan

Track and analyze bot traffic with a simple script tag. See which bots visit your site, when they visit, and what they access.

🎯 When to use:

Perfect for blogs, static sites, SEOs, and content publishers who want to understand bot traffic without needing backend integration.

✅Bot identification and analytics
✅Traffic insights & bot referral tracking
❌No blocking or enforcement
See Analytics Integration →
🛡️

Verification + Enforcement

Get complete bot analytics plus real-time verification and enforcement. Track all bot behavior while programmatically controlling access. Recommended for most users.

🎯 When to use:

Ideal for web applications, SaaS products, and dynamic websites that need to verify bot identities and enforce access rules while tracking all analytics.

✅Real-time bot verification
✅Programmatic allow/block controls
✅Custom rate limiting rules
See Backend Integration →

Quick Start

  1. 1. Sign up at robotsense.io
  2. 2. Verify your domain via DNS TXT record
  3. 3. Choose your integration method below

Option 1: Analytics.js

Install Analytics.js to track bot traffic and gain insights into crawler behavior. Includes referral analytics to see where your traffic comes from. This provides complete visibility but does not block or enforce any rules.

Static Websites & Frontend Apps

Pure HTML, React, Vue, Next.js, or any frontend framework

Installation

Add a script tag just before the closing </head> tag. You can copy the exact tag for your verified domains from the domains page.

1<script 
2  src="https://cdn.robotsense.io/analytics.js" 
3  identifier="RS-XXXXXXXXXX"
4  async>
5</script>

What it Does

  • ✓Auto-tracks all requests
  • ✓Identifies bot vs human traffic
  • ✓Detects Googlebot, Bingbot, etc.
  • ✓GDPR compliant (no PII)

Configuration

identifier

Your domain identifier (starts with RS-)

Option 2: Verification + Enforcement

The complete solution: get all bot analytics plus real-time verification and enforcement. Track bot behavior while programmatically controlling access. Recommended for most users. Choose your integration based on your backend technology.

Connect From Your Backend

Choose your language/framework below. Simply forward all request headers - the API automatically extracts IP addresses, User-Agent, and other relevant information from standard headers like X-Forwarded-For, CF-Connecting-IP, and more. Remove any headers you don't want to pass before making the call.

🔐

Optional: Enable Enforcement Rules

Add evaluate_rules: true to automatically enforce rules you configure in your domain settings. The API will return enforcement actions (allow, block, throttle, delay) that you can implement in your application.

All code examples below show how to enable this feature - just uncomment the options section. See the Response Field Reference for enforcement fields.

Node.js / ExpressClick to expand
1// middleware/botVerification.js
2const axios = require('axios');
3
4async function verifyBot(req, res, next) {
5  try {
6    const response = await axios.post('https://api.robotsense.io/api/v1/check', {
7      request_headers: req.headers,  // Forward all headers
8      // Optional: Enable enforcement rules evaluation
9      // options: {
10      //   evaluate_rules: true
11      // }
12    }, {
13      headers: {
14        'X-API-Key': process.env.ROBOTSENSE_API_KEY,
15        'Content-Type': 'application/json'
16      },
17      timeout: 2000  // 2 second timeout
18    });
19
20    const result = response.data;
21
22    // Attach to request for downstream handlers
23    req.botVerification = result;
24
25    next();
26  } catch (error) {
27    console.error('Bot verification failed:', error);
28    // Continue on error (fail open)
29    next();
30  }
31}
32
33module.exports = verifyBot;
34
35// Usage in Express app:
36// const verifyBot = require('./middleware/botVerification');
37// app.use('/api', verifyBot);
Python / Django / FlaskClick to expand
1# middleware/bot_verification.py
2import requests
3import os
4
5class BotVerificationMiddleware:
6    def __init__(self, get_response):
7        self.get_response = get_response
8        self.api_key = os.getenv('ROBOTSENSE_API_KEY')
9        self.api_url = 'https://api.robotsense.io/api/v1/check'
10
11    def __call__(self, request):
12        # Verify bot
13        try:
14            # Convert Django META headers to standard format
15            headers = {
16                k.replace('HTTP_', '').replace('_', '-').title(): v
17                for k, v in request.META.items() 
18                if k.startswith('HTTP_')
19            }
20            
21            response = requests.post(
22                self.api_url,
23                json={
24                    'request_headers': headers,  # Forward all headers
25                    # Optional: Enable enforcement rules evaluation
26                    # 'options': {
27                    #     'evaluate_rules': True
28                    # }
29                },
30                headers={
31                    'X-API-Key': self.api_key,
32                    'Content-Type': 'application/json'
33                },
34                timeout=2
35            )
36            result = response.json()
37
38            # Attach to request
39            request.bot_verification = result
40
41        except Exception as e:
42            print(f'Bot verification failed: {e}')
43            # Continue on error (fail open)
44            request.bot_verification = None
45
46        return self.get_response(request)
47
48# Add to Django settings.py:
49# MIDDLEWARE = [
50#     'middleware.bot_verification.BotVerificationMiddleware',
51#     ...
52# ]
PHP / LaravelClick to expand
1<?php
2// app/Http/Middleware/BotVerification.php
3namespace App\Http\Middleware;
4
5use Closure;
6use Illuminate\Http\Request;
7use Illuminate\Support\Facades\Http;
8
9class BotVerification
10{
11    public function handle(Request $request, Closure $next)
12    {
13        try {
14            $response = Http::withHeaders([
15                'X-API-Key' => env('ROBOTSENSE_API_KEY'),
16                'Content-Type' => 'application/json',
17            ])->timeout(2)->post('https://api.robotsense.io/api/v1/check', [
18                'request_headers' => $request->headers->all(),  // Forward all headers
19                // Optional: Enable enforcement rules evaluation
20                // 'options' => [
21                //     'evaluate_rules' => true
22                // ]
23            ]);
24
25            $result = $response->json();
26
27            // Attach to request
28            $request->attributes->set('bot_verification', $result);
29
30        } catch (\Exception $e) {
31            \Log::warning('Bot verification failed: ' . $e->getMessage());
32            // Continue on error (fail open)
33        }
34
35        return $next($request);
36    }
37}
38
39// Register in app/Http/Kernel.php:
40// protected $middleware = [
41//     \App\Http\Middleware\BotVerification::class,
42// ];
GoClick to expand
1package middleware
2
3import (
4    "bytes"
5    "context"
6    "encoding/json"
7    "net/http"
8    "os"
9    "time"
10)
11
12type BotVerificationResult struct {
13    Classification         string  `json:"classification"`
14    Confidence             *int    `json:"confidence,omitempty"`
15    Provider               *string `json:"provider,omitempty"`
16    ProviderIdentification *string `json:"provider_identification,omitempty"`
17    BotName                *string `json:"bot_name,omitempty"`
18    BotIdentification      *string `json:"bot_identification,omitempty"`
19    Category               *string `json:"category,omitempty"`
20}
21
22type contextKey string
23
24const BotVerificationKey contextKey = "bot_verification"
25
26func BotVerification(next http.Handler) http.Handler {
27    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
28        // Call RobotSense API with all headers
29        result, err := verifyBot(r.Header)
30        if err != nil {
31            // Log error but continue (fail open)
32            next.ServeHTTP(w, r)
33            return
34        }
35
36        // Attach to context
37        ctx := context.WithValue(r.Context(), BotVerificationKey, result)
38
39        next.ServeHTTP(w, r.WithContext(ctx))
40    })
41}
42
43func verifyBot(headers http.Header) (*BotVerificationResult, error) {
44    client := &http.Client{Timeout: 2 * time.Second}
45
46    // Convert http.Header to map[string]string
47    headerMap := make(map[string]string)
48    for key, values := range headers {
49        if len(values) > 0 {
50            headerMap[key] = values[0]
51        }
52    }
53
54    payload := map[string]interface{}{
55        "request_headers": headerMap,  // Forward all headers
56        // Optional: Enable enforcement rules evaluation
57        // "options": map[string]interface{}{
58        //     "evaluate_rules": true,
59        // },
60    }
61
62    jsonData, _ := json.Marshal(payload)
63
64    req, _ := http.NewRequest("POST", 
65        "https://api.robotsense.io/api/v1/check", 
66        bytes.NewBuffer(jsonData))
67
68    req.Header.Set("X-API-Key", os.Getenv("ROBOTSENSE_API_KEY"))
69    req.Header.Set("Content-Type", "application/json")
70
71    resp, err := client.Do(req)
72    if err != nil {
73        return nil, err
74    }
75    defer resp.Body.Close()
76
77    var result BotVerificationResult
78    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
79        return nil, err
80    }
81
82    return &result, nil
83}
84
85// Usage:
86// r := chi.NewRouter()
87// r.Use(BotVerification)
88// r.Get("/api/data", handler)
Ruby / RailsClick to expand
1# app/middleware/bot_verification.rb
2require 'net/http'
3require 'json'
4
5class BotVerification
6  def initialize(app)
7    @app = app
8  end
9
10  def call(env)
11    request = Rack::Request.new(env)
12
13    begin
14      result = verify_bot(request.env)
15      
16      # Attach to env for downstream use
17      env['bot_verification'] = result
18
19    rescue => e
20      Rails.logger.warn "Bot verification failed: #{e.message}"
21      # Continue on error (fail open)
22    end
23
24    @app.call(env)
25  end
26
27  private
28
29  def verify_bot(env)
30    uri = URI('https://api.robotsense.io/api/v1/check')
31    
32    http = Net::HTTP.new(uri.host, uri.port)
33    http.use_ssl = true
34    http.read_timeout = 2
35
36    # Extract HTTP headers from Rack env
37    headers = env.select { |k, v| k.start_with?('HTTP_') }
38                 .transform_keys { |k| k.sub(/^HTTP_/, '').split('_').map(&:capitalize).join('-') }
39
40    request = Net::HTTP::Post.new(uri.path)
41    request['X-API-Key'] = ENV['ROBOTSENSE_API_KEY']
42    request['Content-Type'] = 'application/json'
43    request.body = {
44      request_headers: headers,  # Forward all headers
45      # Optional: Enable enforcement rules evaluation
46      # options: {
47      #   evaluate_rules: true
48      # }
49    }.to_json
50
51    response = http.request(request)
52    JSON.parse(response.body)
53  end
54end
55
56# Add to config/application.rb:
57# config.middleware.use BotVerification
Other Languages (REST API)Click to expand

Use direct HTTP requests from any language: Java, Rust, C#, Elixir, etc. Forward all request headers - the API intelligently extracts IP, User-Agent, and other needed information.

Endpoint
POST https://api.robotsense.io/api/v1/check
Request Format
1curl -X POST https://api.robotsense.io/api/v1/check \
2  --max-time 0.05 \
3  -H "X-API-Key: rs_your_api_key_here" \
4  -H "Content-Type: application/json" \
5  -d '{
6    "request_headers": {
7      "User-Agent": "...",
8      "X-Forwarded-For": "...",
9      "Accept": "...",
10      "Accept-Language": "..."
11    },
12    "options": {
13      "evaluate_rules": true
14    }
15  }'
16
17# Without enforcement (default):
18curl -X POST https://api.robotsense.io/api/v1/check \
19  -H "X-API-Key: rs_your_api_key_here" \
20  -H "Content-Type: application/json" \
21  -d '{
22    "request_headers": {
23      "User-Agent": "...",
24      "X-Forwarded-For": "..."
25    }
26  }'

Response Examples

Sample responses for different scenarios. See Response Field Reference below for complete field documentation.

👤 Human VisitorClick to expand

Regular human user (bot fields omitted)

1{
2  "classification": "human"
3}
✅ Verified BotClick to expand

Legitimate bot with verified identity (e.g., Googlebot, Bingbot)

1{
2  "classification": "bot",
3  "confidence": 100,
4  "provider": "Google",
5  "provider_identification": "verified",
6  "bot_name": "Googlebot",
7  "bot_identification": "matched",
8  "category": "Search"
9}
⚠️ Unverified BotClick to expand

Bot detected but identity cannot be verified (potential impersonation)

1{
2  "classification": "bot",
3  "confidence": 95,
4  "provider": "Google",
5  "provider_identification": "unverified",
6  "bot_name": "Googlebot",
7  "bot_identification": "matched",
8  "category": "Search"
9}
✅ With Enforcement: AllowClick to expand

When evaluate_rules: true - Bot allowed by rule

1{
2  "classification": "bot",
3  "confidence": 100,
4  "provider": "Google",
5  "provider_identification": "verified",
6  "bot_name": "Googlebot",
7  "category": "Search",
8  "enforcement": {
9    "evaluated": true,
10    "action": "allow",
11    "matched_rule": "Allow Verified Search Engines",
12    "status_code": 200,
13    "log_only_matches": 0
14  }
15}
🚫 With Enforcement: BlockClick to expand

When evaluate_rules: true - Bot blocked by rule

1{
2  "classification": "bot",
3  "confidence": 100,
4  "provider": "Google",
5  "provider_identification": "unverified",
6  "bot_name": "Googlebot",
7  "category": "Search",
8  "enforcement": {
9    "evaluated": true,
10    "action": "block",
11    "matched_rule": "Block Fake Googlebots",
12    "status_code": 403,
13    "log_only_matches": 0
14  }
15}
⏱️ With Enforcement: ThrottleClick to expand

When evaluate_rules: true - Bot rate limited

1{
2  "classification": "bot",
3  "confidence": 100,
4  "provider": "OpenAI",
5  "provider_identification": "verified",
6  "bot_name": "GPTBot",
7  "category": "AI",
8  "enforcement": {
9    "evaluated": true,
10    "action": "throttle",
11    "matched_rule": "Rate Limit AI Bots",
12    "status_code": 200,
13    "retry_after": 60,
14    "rate_limit_headers": {
15      "X-RateLimit-Limit": "30",
16      "X-RateLimit-Remaining": "15"
17    },
18    "log_only_matches": 0
19  }
20}
🐌 With Enforcement: DelayClick to expand

When evaluate_rules: true - Bot should crawl slower

1{
2  "classification": "bot",
3  "confidence": 100,
4  "provider": "SEMrush",
5  "provider_identification": "verified",
6  "bot_name": "SemrushBot",
7  "category": "SEO",
8  "enforcement": {
9    "evaluated": true,
10    "action": "delay",
11    "matched_rule": "Slow Down SEO Tools",
12    "status_code": 200,
13    "retry_after": 10,
14    "log_only_matches": 0
15  }
16}
🧪 With Enforcement: Log Only ModeClick to expand

When evaluate_rules: true - Rules matched but in testing mode (not enforced)

1{
2  "classification": "bot",
3  "confidence": 95,
4  "provider": "OpenAI",
5  "provider_identification": "unverified",
6  "bot_name": "GPTBot",
7  "category": "AI",
8  "enforcement": {
9    "evaluated": true,
10    "action": "allow",
11    "matched_rule": null,
12    "status_code": 200,
13    "log_only_matches": 2
14  }
15}

💡 log_only_matches: 2 indicates 2 rules matched in log-only/testing mode. Use this to test rules before enforcement.

Response Field Reference

Complete reference for all response fields from the verification API.

Bot Detection Fields

FieldDescription
classification"bot" or "human"
confidenceScore 0-100 for bots, omitted for humans
providerBot provider name (Google, Microsoft, Meta, etc.)
provider_identification"verified" or "unverified"
bot_nameSpecific bot identifier (Googlebot, Bingbot, etc.)
bot_identification"matched" (known pattern) or "unknown"
categoryBot category (Search, Social, SEO, Monitoring, etc.)

Enforcement Fields (Optional)

When evaluate_rules: true is enabled:

FieldDescription
enforcement.action"allow", "block", "throttle", or "delay"
enforcement.status_codeHTTP status to return (200, 403, 429)
enforcement.matched_ruleName of the rule that matched, or null
enforcement.retry_afterSeconds to wait (for throttle/delay actions)
enforcement.rate_limit_headersX-RateLimit-* headers for throttle action
enforcement.log_only_matchesCount of rules that matched in log-only mode (for testing)

🔐 Security Best Practices

  • ✅ Store API keys in environment variables
  • ✅ Use different keys for production and development
  • ✅ Rotate keys regularly (every 90 days)
  • ❌ Never commit keys to version control
  • ❌ Don't use production keys in client-side JavaScript

Enforcement Implementation (Optional)

If you enabled enforcement rules above, handle the enforcement actions in your application code:

1// Check enforcement and take action
2if (result.enforcement?.evaluated) {
3  const { action, status_code, retry_after, rate_limit_headers } = result.enforcement;
4
5  switch (action) {
6    case 'block':
7      return res.status(status_code).json({ error: 'Access denied' });
8
9    case 'throttle':
10      // Add rate limit headers
11      if (rate_limit_headers) {
12        Object.entries(rate_limit_headers).forEach(([key, val]) => {
13          res.setHeader(key, val);
14        });
15      }
16      if (retry_after) res.setHeader('Retry-After', retry_after);
17      break;
18
19    case 'delay':
20      if (retry_after) res.setHeader('Retry-After', retry_after);
21      break;
22
23    case 'allow':
24    default:
25      // Continue normally
26      break;
27  }
28}
29
30// Process request...

⚡ Performance Tips

  • ✅ Always use a reasonable timeout
  • ✅ Implement fail-open behavior (allow request if API call fails)
  • ✅ Cache enforcement results for repeat visitors
  • ✅ Test rules in log-only mode before enforcing
  • ❌ Don't block legitimate traffic due to API errors
🤖

robots.txt API

Programmatically access robots.txt files with version history, ETag caching, and content verification. Perfect for CDNs, proxies, monitoring systems, and automated compliance tools.

✅ETag caching & SHA-256 content hash
✅Historical version access
✅JSON and text format support
✅Fixed 10 requests/minute rate limit
View Full Documentation→

API Rate Limiting

Plan-Based Rate Limits

Rate limits vary by subscription plan. Check your current limits on the subscriptions page or view available plans on the pricing page.

All API responses include an X-RateLimit-Limit header showing your current rate limit.

Rate Limit Headers

API responses include the rate limit header:

1X-RateLimit-Limit: 120

Error Handling

Error Response Format

1{
2  "error": "error_code",
3  "message": "Human-readable error message"
4}
HTTP StatusError CodeDescription
400bad_requestInvalid request payload or missing required fields
400validation_errorRequest validation failed
401unauthorizedInvalid or missing API key
500server_errorInternal server error