robots.txt

• Stopping bad robots with honeytraps by David Naylor a UK SEO » DaveN »

  Helpful article with samples that shows how to detect and ban rouge robots that do not follow the robots.txt exclusion standard. E.g. scrapers and others.

  seo php robots robots.txt spiders sourcecode

  by Cumbrowski, 1866 days ago, 1 User

  save

• Meta Robots Tag 101: Blocking Spiders, Cached Pages & More

  seo searchengine robots robots.txt meta

  by Cumbrowski, 1909 days ago, 1 User

  save