michaeljjames/silverstripe-robots

Adds a Robots.txt file that is configurable from /admin/settings/.

Silverstripe Robots

This module will allow you to have a dynamic robots.txt, _(*1)

Installation

$ composer require michaeljjames/silverstripe-robots

You can also install manually by downloading the zip and extracting it into your site root., _(*2)

You'll also need to run dev/build., _(*3)

Configuration

It is completely configurable inside /admin/settings, _(*4)

To configure your robots.txt file paste in your robots.txt confiuration into the textarea inside the robots tab in admin/settings/ or use the example below., _(*5)

User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /

User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /

Sitemap: https://www.domain.com/sitemap.xml