Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	TechLord	Date:	Jul 4, 2018 12:26pm
Forum:	web	Subject:	Re: Why does the wayback machine pay attention to robots.txt

Dear Fizscy, “Honestly, half of the internet is being missed because some honest do-gooder decided that the garbage that is robots.txt should be followed by this archival service. This needs to stop.” could be me saying it.

You phrased my deep opinion so well.

When you wrote this, ytimg actually abused robots.txt, but this stopped at 20120229.