Web Hosting Forum | Lunarpages

Author Topic: blocking semrush bot  (Read 3614 times)

Offline dongna

  • Pong! (the videogame) Master
  • *****
  • Posts: 23
blocking semrush bot
« on: April 24, 2017, 09:34:39 AM »

My account has been flagged for excessive mySQL resource usage. Lunarpages disabled a script that was causing the excessive usage. I think I have determined that the script in question is getting hammered by a bot called semrush. So I've attempted to block access to my site for this bot.

First, I tried editing htaccess (using various suggested techniques I found). None of those seemed to work. Then I noticed in my "Recent Visitors" log that semrush seemed to always be within a small IP address range. So I went into cPanel and used the IPDeny feature to block this range of IPs.

As of last last Friday afternoon (4/21) that appeared to have worked as semrush had disappeared from my "Recent Visitors" log. I thought I was out of the woods! I was not able to check over the weekend, but now this morning (4/24) I see that semrush has returned (using the same IP addresses that should now be blocked).

Is there some other "trick" to blocking bot's such as these?

Also, as an aside, I have not heard from LunarPages support on my ticket now since 4/21-- 3 days ago. Is this normal? I'm not especially happy since my script is still disabled and I can't seem to get a response from support on my ticket. Just wondering what the expectation is.


Offline MrPhil

  • Senior Moderator
  • Berserker Poster
  • *****
  • Posts: 6264
Re: blocking semrush bot
« Reply #1 on: April 24, 2017, 12:45:17 PM »
Besides blocking the IP address range, can you determine the bot's name as given in HTTP_USER_AGENT? You could also block that way.

It's odd that IP address blocking would work for a day or two, and then stop. Have you checked your .htaccess to make sure someone didn't remove your blocks?

Could semrush have changed its ways, to jump directly to the directory in question, avoiding the root .htaccess? That's what happened on my site... I couldn't figure out why sometimes my root .htaccess worked, and other times it didn't, until I figured out that if a real path was given in the URL, control jumped directly to that directory, bypassing /.htaccess. Now all URLs I produce have fake paths, to force control to go through /.htaccess. BTW, this is a misconfiguration of my shared server, that LP refuses to acknowledge. Perhaps replicating your blocks in the target directory's .htaccess could work, too.

Is semrush a malicious bot, or just an over-eager but legitimate service? If the latter, it might obey robots.txt, if you haven't tried that yet.
« Last Edit: April 24, 2017, 12:51:46 PM by MrPhil »
Visit My Site

E-mail Me
-= From the ashes shall rise a sooty tern =-

Offline Pete

  • Alien Anomaly
  • Senior Moderator
  • Professor in Nanotechnology
  • *****
  • Posts: 4258
    • X-Visions Website Design
Re: blocking semrush bot
« Reply #2 on: April 25, 2017, 11:09:34 AM »

As I'm always saying.. (But nobody listens)
"Take a step back.. Take a deep breath and see if there a simple solution there, thats hiding" lol  :DLunarpages Web Hosting   Lunarpages Forums  Lunarpages Affiliate Program


Share |