PHP Web Crawler

GitHub
EN PT-BR

Crawl. Analyze. Generate.

A lightweight PHP crawler built to traverse websites and generate production-ready XML sitemaps. Includes support for canonical URLs, host limits, and an extensible architecture.

View on GitHub Install via Composer

Full Control, No Complexity

This crawler follows links from elements, normalizes URLs, resolves relative paths, removes fragments, and can optionally restrict navigation by host and scheme.

It respects noindex, nofollow, and canonical URLs while remaining lightweight and easy to extend.

It does not interpret robots.txt. It focuses on traversing the HTML and generating clean, efficient sitemaps.

example.php 
<?php

use Tonsoo\PhpCrawler\Extensions\SitemapExtension;
use Tonsoo\PhpCrawler\Sitemap\SitemapGenerator;
use Tonsoo\PhpCrawler\Sitemap\Writers\RotatingSitemapWriter;

crawler()
    ->preserveHost()
    ->respectCanonical(false)
    ->maxPages(1000)
    ->extension(
        new SitemapExtension(
            generator: new SitemapGenerator(
                writer: new RotatingSitemapWriter(
                    directory: __DIR__ . '/sitemap'
                )
            )
        )
    )
    ->start('https://example.com');

Try the Crawler

Enter the website URL you want to crawl. A sitemap will be generated in seconds.