WordPress generates automatically its own robots.txt and the best way to add your own rules in it, is to use the robots_txt filter.

Instead of using plugins or manually creating your robots.txt file, you can use WordPress robots_txt filter to modify the auto-generated robots.txt and add your rules.

Edit your theme (or child theme) functions.php and add the following code:

function ab_add_robotstxt($output, $public) {
$ab_rules = "
# Added by robots_txt filter in functions
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/

Sitemap: <?php echo site_url(); ?>/sitemap.xml
";
return $output . $ab_rules;
}

add_filter('robots_txt', 'ab_add_robotstxt');

That’s it. That’s the preferred way to do it.

Last Update on:March 5th, 2023 at 5:19 pm

---