Websites are made for getting indexed on search engines like Google, Bing, etc. However, if you have an incomplete website with not much content, you will not want to index it at first. Here we are with the solution to discourage the Search Engines from indexing your WordPress website.
There are a lot of other reasons as well to discourage the Search Engine from indexing. But, Incomplete project on a website remains the most reasonable one. There are different methods that you can use for discouraging the search engines.
Reasons to stop your website from indexing on Search Engines
There are many reasons as we mentioned earlier for discouraging search engines from indexing. Here are a few of them which you might consider:
Incomplete Project: You do not want to index a website that is incomplete. It might have some pages that you don’t want to show to the public. If your website is under construction, It is better to not Index as it might create difficulties in the future.
Test Websites: There are chances that you might not want to index your Test websites. Developers create a test website just for testing purposes. It is not meant to be shared with the public. If it is indexed on search engines, the audience can check your test runs as well.
Private/Organizational Websites: In some cases, organizations have a separate dashboard for the employees. This is a confidential content & you would not want to show this to the public as well. Likewise, you don’t want to show your private website publicly.
Unarranged Website: If you have not enough content on your website, you do not want to show it to the public. In some cases, people prefer arranging articles in a specific manner. It allows them to index every content according to their needs.
Methods to prevent search engines from indexing your WordPress website
There are various ways you can prevent search engines from indexing your WordPress website. The main goal of all of these methods is to not allow the search engines to show your website URLs. You can achieve the same by using any of the methods below.
We are using these methods in order to prevent Search engines from crawling the URLs. It will eventually help us to not show the website links in the search results. And this is what we wanted to do.
Method 1: Discouraging the Search Engines directly
We are going to discourage the search engines directly through this method. It is going to be easier to let the Search engines know that we are not interested in indexing. Sometimes, it might take a little time for the Search engines as they are going to crawl periodically.
Within this method, We are going to use 2 ways to achieve our goals. The main goal will remain, just the way to do it is going to change in both.
Using WordPress Dashboard itself (Nothing Extra)
You can use the WordPress dashboard itself to discourage Search Engines. As it has an in-built feature to let the users allow discouraging the Search Engines. Isn’t it a great feature that you don’t need to do anything additional? Here’s how to do it:
- Go to WordPress Dashboard of your website
- Now, Click on Settings & Select the Reading tab.
- On the window, you will see a checkbox besides Search Engine Visibility.
- Tick the checkbox which says “Discourage Search engines from indexing this site”.
- Once you are done with enabling the feature, Save Changes of the settings.
That is all you have to do, WordPress will automatically configure the robots.txt file itself. If you want you can manually edit the robots.txt file as mentioned in the next section.
By Manually editing the robots.txt file
If you want, you can also do the same process manually. It doesn’t require to depend on the WordPress dashboard. All you have to do is make changes in the robots.txt file. You can access the robots.txt file from the File Manager or FTP Clients as well.
It is always not necessary that you would have the access to the WordPress dashboard. If you want to manually complete the process, you can prefer editing robots.txt file. Here’s how to do it with the help of File Manager:
- First of all, access the files of your website. You can use cPanel’s File Manager or prefer using FTP clients for the same.
- Once you have access to your WordPress files, open the public_html folder.
- Search for robots.txt file within the folder. Most likely you will find the file in this folder. If not, create a new one.
- Right-Click on the robots.txt file & select the Edit option from the menu.
- Once you are in the editor, paste the following code in your robots.txt file.
- Save the changes that you have made by clicking the Save button.
That’s all about editing the robots.txt file manually. If you want, you can delete the syntax anytime you would want to index your site.
If you just want to disallow a specific page or file from indexing, you can do that as well. Just write the sub-location of the page in the above syntax. For example, Disallow: /2020-guides/best-gaming-mouse
Method 2: Enable Password Protection on the WordPress website
Enabling password protection on your website is also going to help you achieve your goal. As the crawlers from different search engines will not be able to see the website without a password. This way, the content is only visible to the user who has the login credentials.
There are different ways to enable password protection on your website. Here are a few of them to help you prevent the search engines from indexing:
You can use cPanel of your hosting to enable password protection on your website. As the hosting is going to protect the visitors from accessing the content without a password. Follow the steps to enable password login on your website:
- Visit the cPanel of your hosting service.
- Click on the Directory Privacy under the Files section.
- Click on the root directory, which is usually public_html for general WordPress websites.
- Now, tick the checkbox which says “Password Protect this directory”.
- Write the new name of the protected directory that you want to set.
- Click on the Save button at the bottom.
- Now, Create a user from their & set the password as you like. Remember these credentials, as the content is now Password Protected on your site.
This will protect the website from being accessed by the public as well. This means, your website is now visible only to the users having its login credentials. So most people do not prefer password protection, as the website is not available for the public too.
If you ever change your mind, You can simply un-check the same checkbox that we enabled. This will vanish the password protection & your website will be visible to all.
You can even use various plugins for enabling password protection on your website. It becomes an easy task to enable password protection using plugins. You don’t have to do any manual tasks using the existing plugins.
You can use any of the plugins which do the job, We are using ‘Password Protected’ here. In order to use the plugin for password protecting your website, follow these steps:
- Go to your WordPress dashboard.
- Click on Plugins & Select Add New Plugin
- Now Search for ‘Password Protected’
- Install and Activate the plugin
- Now Click on the Settings on your WordPress dashboard
- Select ‘Password Protected’ settings
- Configure the options available on the plugin page.
- Save the changes and your website will have the password protection.
You will have to keep the plugin in order to have the password protection enabled. Make sure, you use the updated version of the plugin always.
Method 3: Remove the (already) Indexed pages from Google
You can also ask Google to remove the already indexed pages from the search results. It might be possible that you were late in discouraging your incomplete website. Google might have indexed the pages that you didn’t want to index.
In such cases, you can ask Google to remove the indexed URLs. You will need to verify your ownership of the domain. This will eventually help you to deindex the URL from your website. In order to do so, follow the steps below:
- Firstly, Make sure you have Google Search Console account.
- Verify the ownership of your domain by creating a property for the same.
- Now, you need to click on Legacy tools and reports. Then, Select Removals option from the sidebar
- Then, click on the Temporarily hide button.
- In the text field, Enter the URL that you want to deindex from Google.
- Now, you will see the option to Clear the URL from cache & temporarily remove from search.
- Click on the Submit Request button.
That’s all you have to do for removing the URL. Make sure that crawlers cannot access the same URL otherwise, it will be tough to remove. Once the URLs are removed from the search results, make sure to discourage the Search Engines from indexing again.
That’s it. These were the easiest & simplest way to prevent the search engine from indexing your site. We hope that you were successful in discouraging the Search Engines from indexing your WordPress website.
Facing any problems while following these methods? Let us know about that through the comments. Having any suggestions for us? We would love to hear it from you. Follow us on social media platforms to get the latest updates & similar tutorials.
Have an Awesome Day😍!!