spatie/robots-txt

Determine if a page may be crawled from robots.txt and robots meta tags

Parse robots.txt, robots meta and headers

, _(*1)

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers., _(*2)

Support us

, _(*3)

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products., _(*4)

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall., _(*5)

Installation

You can install the package via composer:, _(*6)

composer require spatie/robots-txt

Usage

``` php $robots = Spatie\Robots\Robots::create();, <sub>(*7)</sub>

$robots->mayIndex('https://www.spatie.be/nl/admin');, <sub>(*8)</sub>

$robots->mayFollowOn('https://www.spatie.be/nl/admin');, <sub>(*9)</sub>

You can also specify a user agent:

``` php
$robots = Spatie\Robots\Robots::create('UserAgent007');

By default, Robots will look for a robots.txt file on https://host.com/robots.txt. Another location can be specified like so:, <sub>(*10)</sub>

``` php $robots = Spatie\Robots\Robots::create() ->withTxt('https://www.spatie.be/robots-custom.txt');, <sub>(*11)</sub>

$robots = Spatie\Robots\Robots::create() ->withTxt(DIR . '/public/robots.txt');, <sub>(*12)</sub>

### Testing

``` bash
composer test

Changelog

Please see CHANGELOG for more information what has changed recently., <sub>(*13)</sub>

Contributing

Please see CONTRIBUTING for details., <sub>(*14)</sub>

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities., <sub>(*15)</sub>

Postcardware

You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using., <sub>(*16)</sub>

Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium., <sub>(*17)</sub>

We publish all received postcards on our company website., <sub>(*18)</sub>

Credits

• Brent Roose
• All Contributors

License

The MIT License (MIT). Please see License File for more information., <sub>(*19)</sub>