How to create dynamic robots txt file in WordPress without plugin

WordPress generates automatically its own robots.txt and the best way to add your own rules in it, is to use the robots_txt filter.

Instead of using plugins or manually creating your robots.txt file, you can use WordPress robots_txt filter to modify the auto-generated robots.txt and add your rules.

Edit your theme (or child theme) functions.php and add the following code:

function ab_add_robotstxt($output, $public) {
$ab_rules = "
# Added by robots_txt filter in functions
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/

Sitemap: <?php echo site_url(); ?>/sitemap.xml
";
return $output . $ab_rules;
}

add_filter('robots_txt', 'ab_add_robotstxt');

That’s it. That’s the preferred way to do it.

Leave a Reply

Your email address will not be published.