LinkedIn
Editing and implementing a robots.txt file is, in many cases, an invaluable skill to have for an SEO. If you have product filters, internal searches, page variants, and listing page ordering mechanisms on your eCommerce site, and if these are indexed, they can water down the SEO ranking of your crucial money-making pages. This is when editing the Robots.txt file can be very useful.
What is a robots.txt file
A robots.txt file is designed to signal to Google how to crawl your site; and where Google might override canonical tags, it generally respects robots.txt files. Robots.txt files are uploaded by the owner of the site if they’re not automatically generated to the root directory of a website. Every CMS, including WooCommerce, has a different way of editing the file.
What are the rules of a robots.txt file
There is a set limit of rules that can be added to your robots.txt. Essentially, robots.txt consists of a series of “block” and “allow” directives. While the “user-agent” syntax can be used to target a specific spider, “Allow” and “Disallow” directives are used to target a URL or a group of URLs.
Therefore, a robots.txt file can end up looking like this:
What makes a good robots.txt file
Good robots.txt files stop search engines from indexing and crawling pages you don’t want them to focus on, so it can focus more “crawl budget” on the significant, money-making pages on your eCommerce site.
The biggest impact this has is when your eCommerce store grows and the number of URLs grows, a robots.txt file enables better control of what Google should focus energy on. However, if implemented wrong or against best practice, it can cause harm. Small mistakes we have seen in robots.txt files have disallowed the whole site or key pages. So before uploading your robots.txt, we urge testing it. We recommend using the Google Search Console robots.txt tester. Here, you can do a test upload of your robots.txt file and run different URLs through it to see if they are successfully allowed or blocked.
How to edit & implement your robots.txt file to WooComm- erce
The preferred way to edit & upload your robots.txt file on Woocommerce/Wordpress is through the Yoast plugin. This plugin comes with a robots.txt file generator which can be used to edit your robots.txt within your WordPress backend.
Steps:
1.Go to the SEO section in the side menu. Click for a drop-down menu and select “tools.”
2. From here you can select “File Editor.” Word of warning, if your Woocomerce site does not give you this option it is likely that your site has disabled file editing, as this site has done below: 
To fix this you’ll have to enable file editing. An alternative option is to edit using FTP, which we cover below.
3. Once you are in the file (this will be your existing robots.txt file) you can edit it to your liking. Yoast will automatically add specific rules for your robots.txt file. It is crucial that you remove these following rules as it can stop Google from being able to crawl your entire site:
- User-agent: *Disallow: / 4. From here click save and you are done.
How to edit/upload your robots.txt file using FTP (file transfer protocol)
Another way of editing or creating your robots.txt file is to edit it using an FTP (file transfer protocol). This is mainly necessary when having troubles with your plugins or site and enables you to update and upload files without login into your CMS.
Steps:
- You will have to log into your WordPress hosting account, and you have to do this as an FTP client. Word of warning, it is a good idea to back up your site while making changes like this. If you don’t know your credentials to log in, you can request them from your hosting provider.
- From here, you will be able to access your robots.txt file. If you can not find your robots.txt file within your root folder it is likely that you don’t have a robots.txt file yet. You can download and edit the file before uploading it again.
- Finally, upload your robots.txt file using your FTP.
To read more about how best to run your WooComerce site read our SEO limitations of WooCommerce blog now.
Alternatively, read more about our eCommerce SEO services here, or reach out to us and let’s work together to grow your eCommerce SEO non-brand traffic & revenue sustainably here!
