Error Handling

WebPeek API uses conventional HTTP status codes and returns detailed error information in JSON format. This guide explains how to handle errors effectively in your application.

Error Response Format

All error responses follow a consistent JSON structure:

{
  "success": false,
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable error message",
    "details": {
      // Additional context about the error
    }
  }
}

HTTP Status Codes

WebPeek API uses standard HTTP status codes to indicate the success or failure of requests:

Status Code	Meaning	Description
200	OK	Request succeeded
400	Bad Request	Invalid request parameters
401	Unauthorized	Missing or invalid API key
403	Forbidden	API key doesn't have required permissions
404	Not Found	Endpoint or resource doesn't exist
422	Unprocessable Entity	Request cannot be processed (e.g., URL unreachable)
429	Too Many Requests	Rate limit exceeded
500	Internal Server Error	Server error on our side
503	Service Unavailable	Temporary service disruption

Common Error Codes

Here are the most common error codes you'll encounter:

INVALID_URL

400

The provided URL is malformed or invalid.

{
  "success": false,
  "error": {
    "code": "INVALID_URL",
    "message": "The provided URL is invalid",
    "details": {
      "url": "not-a-valid-url",
      "reason": "Missing protocol (http:// or https://)"
    }
  }
}

MISSING_PARAMETER

400

A required parameter is missing from the request.

{
  "success": false,
  "error": {
    "code": "MISSING_PARAMETER",
    "message": "Required parameter 'url' is missing",
    "details": {
      "parameter": "url",
      "location": "query"
    }
  }
}

INVALID_API_KEY

401

The API key is missing, invalid, or has been revoked.

{
  "success": false,
  "error": {
    "code": "INVALID_API_KEY",
    "message": "Invalid or missing API key",
    "details": {
      "hint": "Include your API key in the X-API-Key header"
    }
  }
}

RATE_LIMIT_EXCEEDED

429

You've exceeded your rate limit. Wait before making more requests.

{
  "success": false,
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Rate limit exceeded. Try again in 3600 seconds.",
    "details": {
      "limit": 100,
      "remaining": 0,
      "reset": 1699564800,
      "reset_after": 3600
    }
  }
}

URL_UNREACHABLE

422

The target URL could not be reached or returned an error.

{
  "success": false,
  "error": {
    "code": "URL_UNREACHABLE",
    "message": "Unable to reach the target URL",
    "details": {
      "url": "https://example-nonexistent-site.com",
      "reason": "DNS resolution failed",
      "status_code": null
    }
  }
}

TIMEOUT

422

Request timed out while trying to process the URL.

{
  "success": false,
  "error": {
    "code": "TIMEOUT",
    "message": "Request timed out after 30 seconds",
    "details": {
      "url": "https://very-slow-website.com",
      "timeout_ms": 30000
    }
  }
}

ROBOTS_TXT_BLOCKED

403

Access to the URL is blocked by the site's robots.txt file.

{
  "success": false,
  "error": {
    "code": "ROBOTS_TXT_BLOCKED",
    "message": "Access blocked by robots.txt",
    "details": {
      "url": "https://example.com/admin",
      "robots_txt_url": "https://example.com/robots.txt",
      "user_agent": "WebPeekBot"
    }
  }
}

INTERNAL_ERROR

500

An unexpected error occurred on our servers.

{
  "success": false,
  "error": {
    "code": "INTERNAL_ERROR",
    "message": "An internal error occurred while processing your request",
    "details": {
      "request_id": "req_abc123xyz",
      "timestamp": "2025-11-06T19:56:25.284Z"
    }
  }
}

Error Handling Best Practices

JavaScript / Node.js

error-handler.js

async function fetchWebPeek(url, options = {}) {
  try {
    const response = await fetch(`https://api.webpeek.dev/metadata?url=${encodeURIComponent(url)}`, {
      headers: {
        'X-API-Key': process.env.WEBPEEK_API_KEY,
        ...options.headers
      }
    });

    // Parse JSON response
    const data = await response.json();

    // Handle different status codes
    switch (response.status) {
      case 200:
        return data;

      case 400:
        throw new Error(`Invalid request: ${data.error.message}`);

      case 401:
        throw new Error('API key is invalid or missing');

      case 403:
        if (data.error.code === 'ROBOTS_TXT_BLOCKED') {
          throw new Error(`Access blocked by robots.txt: ${url}`);
        }
        throw new Error(`Access forbidden: ${data.error.message}`);

      case 404:
        throw new Error('Endpoint not found');

      case 422:
        if (data.error.code === 'URL_UNREACHABLE') {
          throw new Error(`URL unreachable: ${data.error.details.reason}`);
        }
        if (data.error.code === 'TIMEOUT') {
          throw new Error(`Request timed out after ${data.error.details.timeout_ms}ms`);
        }
        throw new Error(`Cannot process request: ${data.error.message}`);

      case 429:
        const retryAfter = data.error.details.reset_after || 60;
        throw new Error(`Rate limit exceeded. Retry after ${retryAfter} seconds`);

      case 500:
      case 503:
        throw new Error(`Server error: ${data.error.message}`);

      default:
        throw new Error(`Unexpected error: ${response.status}`);
    }
  } catch (error) {
    // Handle network errors
    if (error instanceof TypeError && error.message.includes('fetch')) {
      throw new Error('Network error: Unable to connect to WebPeek API');
    }
    throw error;
  }
}

// Usage with error handling
try {
  const metadata = await fetchWebPeek('https://github.com');
  console.log('Title:', metadata.data.title);
} catch (error) {
  console.error('Error:', error.message);
  // Handle error appropriately (log, retry, notify user, etc.)
}

Python

error_handler.py

import requests
import os
from typing import Dict, Any
from urllib.parse import urlencode

class WebPeekError(Exception):
    """Base exception for WebPeek API errors."""
    def __init__(self, message: str, code: str = None, details: Dict = None):
        super().__init__(message)
        self.code = code
        self.details = details or {}

class RateLimitError(WebPeekError):
    """Raised when rate limit is exceeded."""
    pass

class URLUnreachableError(WebPeekError):
    """Raised when target URL cannot be reached."""
    pass

def fetch_webpeek(url: str) -> Dict[str, Any]:
    """Fetch metadata with comprehensive error handling."""
    api_key = os.getenv('WEBPEEK_API_KEY')
    if not api_key:
        raise WebPeekError('API key not found in environment')

    try:
        response = requests.get(
            f'https://api.webpeek.dev/metadata',
            params={'url': url},
            headers={'X-API-Key': api_key},
            timeout=30
        )

        # Parse JSON response
        data = response.json()

        # Handle successful response
        if response.status_code == 200:
            return data

        # Handle error responses
        error_code = data.get('error', {}).get('code', 'UNKNOWN')
        error_message = data.get('error', {}).get('message', 'Unknown error')
        error_details = data.get('error', {}).get('details', {})

        if response.status_code == 400:
            raise WebPeekError(
                f'Invalid request: {error_message}',
                code=error_code,
                details=error_details
            )

        elif response.status_code == 401:
            raise WebPeekError('Invalid or missing API key', code=error_code)

        elif response.status_code == 403:
            if error_code == 'ROBOTS_TXT_BLOCKED':
                raise WebPeekError(
                    f'Access blocked by robots.txt',
                    code=error_code,
                    details=error_details
                )
            raise WebPeekError(
                f'Access forbidden: {error_message}',
                code=error_code
            )

        elif response.status_code == 404:
            raise WebPeekError('Endpoint not found', code=error_code)

        elif response.status_code == 422:
            if error_code == 'URL_UNREACHABLE':
                raise URLUnreachableError(
                    f"URL unreachable: {error_details.get('reason')}",
                    code=error_code,
                    details=error_details
                )
            if error_code == 'TIMEOUT':
                raise WebPeekError(
                    f"Request timed out after {error_details.get('timeout_ms')}ms",
                    code=error_code,
                    details=error_details
                )
            raise WebPeekError(
                f'Cannot process request: {error_message}',
                code=error_code
            )

        elif response.status_code == 429:
            retry_after = error_details.get('reset_after', 60)
            raise RateLimitError(
                f'Rate limit exceeded. Retry after {retry_after} seconds',
                code=error_code,
                details=error_details
            )

        elif response.status_code >= 500:
            raise WebPeekError(
                f'Server error: {error_message}',
                code=error_code
            )

        else:
            raise WebPeekError(
                f'Unexpected error (HTTP {response.status_code})',
                code=error_code
            )

    except requests.exceptions.Timeout:
        raise WebPeekError('Request timed out')

    except requests.exceptions.ConnectionError:
        raise WebPeekError('Unable to connect to WebPeek API')

    except requests.exceptions.RequestException as e:
        raise WebPeekError(f'Request failed: {str(e)}')

# Usage example
try:
    metadata = fetch_webpeek('https://github.com')
    print(f"Title: {metadata['data']['title']}")

except RateLimitError as e:
    print(f"Rate limit exceeded: {e}")
    print(f"Retry after: {e.details.get('reset_after')} seconds")

except URLUnreachableError as e:
    print(f"URL unreachable: {e}")
    print(f"Reason: {e.details.get('reason')}")

except WebPeekError as e:
    print(f"WebPeek error: {e}")
    if e.code:
        print(f"Error code: {e.code}")

except Exception as e:
    print(f"Unexpected error: {e}")

Error Handling with Retry Logic

retry-handler.js

async function fetchWithRetry(url, maxRetries = 3, retryDelay = 1000) {
  let lastError;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(
        `https://api.webpeek.dev/metadata?url=${encodeURIComponent(url)}`,
        {
          headers: {
            'X-API-Key': process.env.WEBPEEK_API_KEY
          }
        }
      );

      const data = await response.json();

      // Success
      if (response.ok) {
        return data;
      }

      // Don't retry client errors (400-499) except 429
      if (response.status >= 400 && response.status < 500 && response.status !== 429) {
        throw new Error(`Client error (${response.status}): ${data.error.message}`);
      }

      // Handle rate limiting with exponential backoff
      if (response.status === 429) {
        const retryAfter = (data.error.details?.reset_after || 60) * 1000;
        console.log(`Rate limited. Waiting ${retryAfter}ms before retry...`);
        await new Promise(resolve => setTimeout(resolve, retryAfter));
        continue;
      }

      // Server errors (500-599) - retry with exponential backoff
      if (response.status >= 500) {
        lastError = new Error(`Server error (${response.status}): ${data.error.message}`);
        const delay = retryDelay * Math.pow(2, attempt);
        console.log(`Retry attempt ${attempt + 1}/${maxRetries} after ${delay}ms`);
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }

      // Other errors
      throw new Error(`HTTP ${response.status}: ${data.error.message}`);

    } catch (error) {
      lastError = error;

      // Don't retry on specific errors
      if (error.message.includes('Client error')) {
        throw error;
      }

      // Network errors - retry
      if (attempt < maxRetries - 1) {
        const delay = retryDelay * Math.pow(2, attempt);
        console.log(`Network error. Retry attempt ${attempt + 1}/${maxRetries} after ${delay}ms`);
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }
    }
  }

  throw lastError || new Error('Max retries exceeded');
}

// Usage
try {
  const data = await fetchWithRetry('https://github.com');
  console.log('Success:', data);
} catch (error) {
  console.error('Failed after retries:', error.message);
}

Debugging Tips

Log Request IDs

Server errors (500) include a request_id in the error details. Save this ID when reporting issues to support for faster debugging.

if (error.details?.request_id) {
  console.error('Request ID:', error.details.request_id);
  // Send to error tracking service
}

Validate URLs Before Sending

Validate URLs client-side to avoid unnecessary API calls and reduce errors.

function isValidUrl(string) {
  try {
    const url = new URL(string);
    return url.protocol === 'http:' || url.protocol === 'https:';
  } catch (_) {
    return false;
  }
}

if (!isValidUrl(userInput)) {
  console.error('Invalid URL format');
  return;
}

Use Error Monitoring

Integrate with error tracking services like Sentry or Rollbar to monitor API errors in production.

import * as Sentry from '@sentry/node';

try {
  const data = await fetchWebPeek(url);
} catch (error) {
  Sentry.captureException(error, {
    tags: {
      api: 'webpeek',
      endpoint: 'metadata'
    },
    extra: {
      url: url,
      errorCode: error.code
    }
  });
  throw error;
}

Handle Timeouts Appropriately

Some endpoints like Performance and Snapshot can take 10-30 seconds. Set appropriate timeout values.

// For performance endpoint - use longer timeout
const response = await fetch(url, {
  signal: AbortSignal.timeout(45000) // 45 seconds
});

Graceful Degradation

When API calls fail, have fallback behavior to maintain user experience.

try {
  const metadata = await fetchWebPeek(url);
  return metadata.data;
} catch (error) {
  // Fallback to basic metadata
  return {
    title: extractTitleFromUrl(url),
    description: 'Metadata unavailable',
    image: '/placeholder.png'
  };
}

Common Issues & Solutions

Issue: Getting 401 errors despite having an API key

Solution: Ensure you're sending the API key in the correct header format:

# Correct
curl -H "X-API-Key: your_api_key_here" https://api.webpeek.dev/metadata?url=...

# Incorrect (missing X- prefix)
curl -H "API-Key: your_api_key_here" https://api.webpeek.dev/metadata?url=...

Issue: Frequent URL_UNREACHABLE errors

Solution: This often happens with:

URLs behind authentication or paywalls
Localhost or internal network URLs
Sites that block automated access
Typos in the URL

Issue: Timeout errors on specific sites

Solution: Some sites are genuinely slow. For the Snapshot and Performance endpoints, the API has built-in timeouts of 30 seconds. If a site consistently times out, it may be too slow to process reliably.

Issue: CORS errors in browser

Solution: Make API calls from your backend, not directly from the browser. Exposing API keys in client-side code is a security risk.

// Backend (Node.js/Express)
app.get('/api/metadata', async (req, res) => {
  const { url } = req.query;
  const data = await fetchWebPeek(url);
  res.json(data);
});

// Frontend
const metadata = await fetch('/api/metadata?url=' + encodeURIComponent(url));

Issue: Rate limits hit unexpectedly

Solution: Implement caching on your end to avoid redundant requests for the same URLs. WebPeek also caches responses, but implementing your own cache reduces API usage.

Getting Help

If you encounter errors that aren't covered in this documentation:

Check API Status

Visit our status page to see if there are any ongoing issues.

View Status →

Contact Support

Reach out to our support team with your request ID for assistance.

Get Support →