Feature: Robots.txt Manager #

Dynamic robots.txt generation with multiple security profiles, a bot database, AI crawler policies and sitemap integration.

Via FTP: Upload the robotstxt/ directory into /plugins/.

Via package manager: Select robotstxt from the available plugins list.

Paths:

• Main plugin: /plugins/robotstxt/robotstxt.php
• Configuration: /plugins/robotstxt/conf/robotstxt.conf.inc.php
• Handlers: /plugins/robotstxt/handlers/
• Classes: /plugins/robotstxt/lib/RobotsTxtGenerator.php, BotDatabase.php
• CSS: /plugins/robotstxt/css/

Configuration parameters:

| Variable | Default | Description |

|---|---|---|

| $basedisplevel | BASE_LEVEL_HIGHUSER | Minimum required level |

| $ftype | 3 | Plugin type (system) |

Security profiles #

| Profile | Description |

|---|---|

| default | Standard configuration |

| private | Blocks most crawlers |

| open_research | Allows academic crawlers |

| corporate | Strict enterprise configuration |

| aegis_sovereign | Maximises control |

AI policies #

| Policy | Description |

|---|---|

| allow | Allows AI crawlers |

| block | Blocks known AI crawlers |

| honeypot | Traps undeclared crawlers |

| ignore | No specific AI rules |

Workflow #

1. Select a profile and an AI policy

2. Configure crawl delay and elements to include (sitemap, honeypot)

3. Preview the generated file

4. Deploy (writes robots.txt to the site root)

Bot database migration #

The run_migration tool updates the known bot database and categorises entries.

?obj=robotstxt.php                — Main interface

POST actions:

| Action | Description |

|---|---|

| generate_preview | Generate a preview |

| deploy | Write the robots.txt file |

| run_migration | Update the bot database |

• Beamreactor\Database\SQL — bot database
• Write permission on robots.txt at the site root