What is the purpose of serpstatbot?

Our bot constantly crawls the web to add new links and track changes in our link database. We provide our users with access to one of the largest backlink databases on the market for planning and monitoring marketing campaigns.

What happens to the crawled data?

Crawled data is added to the backlink index. You can get access to the data from the index in Serpstat.

How do you handle 404 or 301 pages?

We collect historical data to make sure that no temporary change has an impact on your site. If the links to pages with a 404 or 301 server response code still exist, serpstatbot will continue to find and follow them. You can check Google 404 policy to find more information.

Does the bot crawl links with the rel = nofollow attribute?

Yes, it scans.

How can I block serpstatbot?

Serpstatbot addresses to the robots.txt standard. If you don’t want the bot to crawl your website then add the following text to your robots.txt:

User-agent: serpstatbot

Disallow: /

Please always ensure the bot can get robots.txt file itself. If not, then it will crawl your site by default.

If you think that serpstatbot doesn’t follow your robots.txt commands, please contact us via email: Please attach the URL to your website and log entries that show bot trying to retrieve pages that it was not supposed to.

What commands in robots.txt does serpstatbot support?

Serpstatbot supports the following additions to robots.txt:

Why doesn’t my robots.txt block work on serpstatbot?

There are several reasons:

How can I slow down serpstatbot?

You can slow down bot by adding the following settings to your robots.txt file:

User-Agent: serpstatbot

Crawl-Delay: 5

Crawl-Delay must be an integer: it stands for the number of seconds to wait between requests. Serpstatbot accepts a delay of up to 20 seconds between requests to your site. High Crawl-Delay should minimize the impact on your site. This parameter will also be valid if it is set for User-Agent: *.

If our bot finds out that you have used Crawl-Delay for any other bot, then it will automatically slow down the crawling process.

Send article

Save the link and read the article later.
Choose where to send the link to the article.

email_image Mail
pocket_image Pocket
flipboard Flipboard
facebook_image Messenger
telegram_image Telegram

Поделитесь статьей с вашими друзьями

Войти Регистрация

Вы исчерпали лимит запросов.

Or email
Забыли пароль?
Or email
Back To Login

Don’t worry! Just fill in your email and we’ll send over your password.

Are you sure?


To complete your registration you need to enter your phone number


We sent confirmation code to your phone number

Your phone Resend code queriesLeft


telegramVerification telegramBotLink
Choose the project to view data on it

Персональная презентация сервиса

Serpstat is all about saving time, and we want to save yours! One of our specialists will contact you and discuss options going forward.

These may include a personal demonstration, a trial period, comprehensive training articles & webinar recordings, and custom advice from a Serpstat specialist. It is our goal to make you feel comfortable while using Serpstat.

We are glad of your comment

Upgrade your plan

Export is not available for your account. Please upgrade to Plan A or higher to get access to the tool. Learn more.

Sign Up Free

Спасибо, мы с вами свяжемся в ближайшее время

Просмотр Редактирование


У вас закончились лимиты

Вы достигли лимита на количество созданных проектов и больше не можете создавать новые проекты. Увеличьте лимиты или удалите существующие проекты.

Я хочу больше лимитов

Christmas is a time for miracles.

You are almost on the finish line of our Christmas quest. The last brick of your lego-promocode is left on the way ap 55% discount.

Did not find previous lego-bricks? Fill the form anyway.


Missing translations for "English" English | Русский | Українська | German

  • Такого пользователя...add
  • All-in-Oneadd
  • Блогadd
  • Поддержкаadd
  • This site uses cookies and other tracking...add
  • Accept and continueadd
  • Руководствоadd
  • Попробовать бесплатноadd
  • Pricesadd
  • Pressadd
  • Clients Serpstatadd
  • Customer Successadd
  • Communityadd
  • Member Perksadd
  • Affiliate Programadd
  • Webinarsadd
  • PPCadd
  • Toolsadd
  • Site auditadd
  • SERP crawlingadd
  • Languagesadd
  • in 2013. From 2015 - fully independent.add
  • Save the link and read the article later.add
  • Email is Emptyadd
  • The article is sent to you by mailadd
  • Регистрацияadd
  • Wrong emailadd
  • Passwordadd
  • Забыли пароль?add
  • I agree to Serpstat'sadd
  • Don’t worry! Just fill in your email and...add
  • Confirmadd
  • To complete your registration you need to...add
  • Backadd
  • Resend codeadd
  • contactSupportadd
  • Code is invalidadd
  • Персональная...add
  • Nameadd
  • Optionaladd
  • See Serpstat in actionadd
  • Upgrade your planadd
  • Upgrade planadd
  • Просмотрadd
  • Messageadd
  • Addadd
  • Select Alladd
  • У вас закончились лимитыadd
  • Christmas is a time for miracles.add
  • Write your discount codeadd
  • Successadd
  • Erroradd
  • Loading... Wait for a while.add
  • Некоторые функции...add
  • SEO platformadd
  • Тарифыadd
  • Улучшить планadd
  • By using the website, you agree to ouradd
  • Входadd
  • Language:add
  • or see our plans & pricesadd
  • About usadd
  • Bannersadd
  • Help centeradd
  • Knowledge baseadd
  • Blogadd
  • Experts We Loveadd
  • Discountsadd
  • Rankadd
  • Legal infoadd
  • Rank trackeradd
  • Competitor analysisadd
  • Keywords clusteringadd
  • Download pluginadd
  • Mailadd
  • Choose where to send the link to the article.add
  • Not valid emailadd
  • Поделитесь статьей с...add
  • Вы исчерпали лимит...add
  • Email Addressadd
  • Sign Inadd
  • Wrong phoneadd
  • Sign Up Freeadd
  • Send Emailadd
  • Canceladd
  • Telephone is invalidadd
  • We sent confirmation code to your phone...add
  • queriesLeftadd
  • telegramVerificationadd
  • Confirmation codeadd
  • Serpstat is all about saving time, and we...add
  • Emailadd
  • Textadd
  • Choose planadd
  • Export is not available for your account....add
  • Спасибо, мы с вами...add
  • Редактированиеadd
  • Optionaladd
  • Select somethingadd
  • Clear Alladd
  • Вы достигли лимита на количество...add
  • You are almost on the finish line of our...add
  • Add discount codeadd
  • Warningadd
  • Competitors list {start_name}{end_name}...add
  • Yesadd
  • Зарегистрируйтесь...add
  • Возможностиadd
  • Примерыadd
  • Запросить демоadd
  • Privacy Policyadd
  • FAQadd
  • Growth hacking tool for SEO, PPC and...add
  • Featuresadd
  • Statisticsadd
  • Contact usadd
  • Serpstat Академияadd
  • Serpsat APIadd
  • Send a guest postadd
  • User Testimonialsadd
  • Affiliate Program For Study Centersadd
  • SEOadd
  • Affiliate Agreementadd
  • Backlinksadd
  • Keyword researchadd
  • Text Analyticsadd
  • Was born and developed in SEO and PPC agencyadd
  • Send articleadd
  • Backadd
  • Submitadd
  • Войтиadd
  • Or emailadd
  • Cannot be emptyadd
  • Запомнить меняadd
  • Phoneadd
  • Back To Loginadd
  • Are you sure?add
  • Awesomeadd
  • Enter Phoneadd
  • Your phoneadd
  • smthWentWrongadd
  • telegramBotLinkadd
  • Choose the project to view data on itadd
  • These may include a personal...add
  • We are glad of your commentadd
  • Confirm that you are not a robotadd
  • Contact adminadd
  • Learn more.add
  • Inviteadd
  • E-mailadd
  • Canceladd
  • Searchadd
  • No results matchedadd
  • Я хочу больше лимитовadd
  • Did not find previous lego-bricks? Fill...add
  • No keywordsadd
  • Infoadd
  • Competitors list updated...add
  • Noadd