SERPSTATBOT FAQ

What is the purpose of serpstatbot?

Our bot constantly crawls the web to add new links and track changes in our link database. We provide our users with access to one of the largest backlink databases on the market for planning and monitoring marketing campaigns.

What happens to the crawled data?

Crawled data is added to the backlink index. You can get access to the data from the index in Serpstat.

How do you handle 404 or 301 pages?

We collect historical data to make sure that no temporary change has an impact on your site. If the links to pages with a 404 or 301 server response code still exist, serpstatbot will continue to find and follow them. You can check Google 404 policy to find more information.

Does the bot crawl links with the rel = nofollow attribute?

Yes, it scans.

How can I block serpstatbot?

Serpstatbot addresses to the robots.txt standard. If you don’t want the bot to crawl your website then add the following text to your robots.txt:

User-agent: serpstatbot

Disallow: /

Please always ensure the bot can get robots.txt file itself. If not, then it will crawl your site by default.

If you think that serpstatbot doesn’t follow your robots.txt commands, please contact us via email: serpstatbot@serpstatbot.com. Please attach the URL to your website and log entries that show bot trying to retrieve pages that it was not supposed to.

What commands in robots.txt does serpstatbot support?

Serpstatbot supports the following additions to robots.txt:

Why doesn’t my robots.txt block work on serpstatbot?

There are several reasons:

How can I slow down serpstatbot?

You can slow down bot by adding the following settings to your robots.txt file:

User-Agent: serpstatbot

Crawl-Delay: 5

Crawl-Delay must be an integer: it stands for the number of seconds to wait between requests. Serpstatbot accepts a delay of up to 20 seconds between requests to your site. High Crawl-Delay should minimize the impact on your site. This parameter will also be valid if it is set for User-Agent: *.

If our bot finds out that you have used Crawl-Delay for any other bot, then it will automatically slow down the crawling process.

Can I get full list of Serpstatbot ip's?

Here is a list of all ip's used by Serpstatbot.