Another stupid spider mistake

To follow up my earlier entry on this stuff, I just saw another stunned monkey moment :

  • you can't randomly add a trailing slash to URLs any more than you can randomly remove them.
  • especially when the URL includes a query parameter, because then you're changing the query. And that always works really well.

From the pattern of the stealth spider's requests, I think it is adding the trailing slash on any URL that doesn't end in a filename with an extension. This is stunningly braindead, as extensions are nothing more than a hack workaround so webservers don't need real metadata about what MIME type a file is.

(It also has other problems, like not properly resolving relative URLs that use ' .. '.)

This stunned monkey moment is brought to you by the idiotic stealth spider running from 204.11.99.2, which is claimed to belong to a 'Goo Khim Yeung' as 204.11.99.0/29. (Assuming that the WHOIS information is accurate, which it isn't always.)