Docs

Documentation

Learn how to automate your web scraping workflows with ActiCrawl

Introduction Quick Start Playground Installation Architecture Web Crawling Data Extraction API Endpoints Authentication Examples

Installation

ActiCrawl provides official SDKs for popular programming languages, making it easy to integrate web scraping into your applications.

Official SDKs

Node.js / JavaScript

Install via npm:

bash 
npm install @acticrawl/sdk

Or using yarn:

bash 
yarn add @acticrawl/sdk

Basic usage:

javascript 
import { ActiCrawl } from '@acticrawl/sdk';

const client = new ActiCrawl({
  apiKey: 'YOUR_API_KEY'
});

const result = await client.scrape({
  url: 'https://example.com',
  format: 'markdown'
});

console.log(result.content);

Python

Install via pip:

bash 
pip install acticrawl

Basic usage:

python 
from acticrawl import ActiCrawl

client = ActiCrawl(api_key='YOUR_API_KEY')

result = client.scrape(
    url='https://example.com',
    format='markdown'
)

print(result['content'])

Ruby

Add to your Gemfile:

ruby 
gem 'acticrawl'

Then run:

bash 
bundle install

Basic usage:

ruby 
require 'acticrawl'

client = ActiCrawl::Client.new(api_key: 'YOUR_API_KEY')

result = client.scrape(
  url: 'https://example.com',
  format: 'markdown'
)

puts result['content']

PHP

Install via Composer:

bash 
composer require acticrawl/php-sdk

Basic usage:

php 
<?php
require 'vendor/autoload.php';

use ActiCrawl\Client;

$client = new Client('YOUR_API_KEY');

$result = $client->scrape([
    'url' => 'https://example.com',
    'format' => 'markdown'
]);

echo $result['content'];

Go

Install:

bash 
go get github.com/acticrawl/go-sdk

Basic usage:

go 
package main

import (
    "fmt"
    "github.com/acticrawl/go-sdk"
)

func main() {
    client := acticrawl.NewClient("YOUR_API_KEY")

    result, err := client.Scrape(acticrawl.ScrapeOptions{
        URL:    "https://example.com",
        Format: "markdown",
    })

    if err != nil {
        panic(err)
    }

    fmt.Println(result.Content)
}

REST API

If your language doesn't have an official SDK, you can use the REST API directly:

bash 
curl -X POST https://api.acticrawl.com/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "markdown"
  }'

Environment Variables

For security, we recommend storing your API key in environment variables:

Unix/Linux/macOS

bash 
export ACTICRAWL_API_KEY="your_api_key_here"

Windows

cmd 
set ACTICRAWL_API_KEY=your_api_key_here

.env file

Create a .env file in your project root:

env 
ACTICRAWL_API_KEY=your_api_key_here

Then use in your code:

javascript 
// Node.js with dotenv
require('dotenv').config();

const client = new ActiCrawl({
  apiKey: process.env.ACTICRAWL_API_KEY
});
python 
# Python with python-dotenv
from dotenv import load_dotenv
import os

load_dotenv()

client = ActiCrawl(api_key=os.getenv('ACTICRAWL_API_KEY'))

Framework Integration

Next.js

javascript 
// pages/api/scrape.js
import { ActiCrawl } from '@acticrawl/sdk';

const client = new ActiCrawl({
  apiKey: process.env.ACTICRAWL_API_KEY
});

export default async function handler(req, res) {
  const { url } = req.body;

  try {
    const result = await client.scrape({
      url,
      format: 'markdown'
    });

    res.status(200).json(result);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
}

Django

python 
# views.py
from django.http import JsonResponse
from acticrawl import ActiCrawl
import os

client = ActiCrawl(api_key=os.environ.get('ACTICRAWL_API_KEY'))

def scrape_view(request):
    url = request.POST.get('url')

    try:
        result = client.scrape(url=url, format='markdown')
        return JsonResponse(result)
    except Exception as e:
        return JsonResponse({'error': str(e)}, status=500)

Laravel

php 
// app/Http/Controllers/ScrapeController.php
namespace App\Http\Controllers;

use ActiCrawl\Client;
use Illuminate\Http\Request;

class ScrapeController extends Controller
{
    private $client;

    public function __construct()
    {
        $this->client = new Client(env('ACTICRAWL_API_KEY'));
    }

    public function scrape(Request $request)
    {
        $result = $this->client->scrape([
            'url' => $request->input('url'),
            'format' => 'markdown'
        ]);

        return response()->json($result);
    }
}

Docker

If you're using Docker, you can pass the API key as an environment variable:

dockerfile 
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

ENV ACTICRAWL_API_KEY=${ACTICRAWL_API_KEY}

CMD ["node", "index.js"]

Run with:

bash 
docker run -e ACTICRAWL_API_KEY=your_api_key_here your-image

Troubleshooting

Common Issues

  1. Authentication Error

    • Verify your API key is correct
    • Check if the key has proper permissions
    • Ensure you're using the Bearer token format

  2. Network Errors

    • Check your internet connection
    • Verify firewall settings
    • Ensure API endpoint is accessible

  3. Rate Limit Errors

    • Check your plan's rate limits
    • Implement exponential backoff
    • Consider upgrading your plan

Debug Mode

Enable debug mode to see detailed logs:

javascript 
const client = new ActiCrawl({
  apiKey: 'YOUR_API_KEY',
  debug: true
});

Next Steps