Part of the fun of writing programs that deal with HTTP is decodingthings like RFC 2616 to answer somewhat obscure questions about how various thingsinteract. Today's case is the following question:

When can your web server generate a 304 'content not modified'response if it receives a request with both an If-None-Match andan If-Modified-Since header?

If-None-Match and If-Modified-Since are HTTP request headers usedto implement 'conditional GET'

Dating back to our first experiences with msnbot,the MSN Search web crawler, I've known that it was kind of crazy aboutrepeatedly fetching large binary files. Since then, we have pointedthis issue out to MSN Search people more than once and switched to using accurate Content-Types .Recently we had a week of MSNbot not refetching those large binaries,so it looked like MSNbot had finally been fixed.

So much for that. Since 7pm Wednesday night, MSNbot has fetched 3.1gigabytes of various

Recently, I stumbled over a small issue in Python's cgi module that is agood illustration of how unintended consequences in Python can wind upshooting you in the foot.

The cgi module 's main purpose is to create a dictionary-likeobject that contains all of the parameters passed to your CGI programin the GET or POST HTTP command. DWiki uses it roughly like this:

form = cgi.FieldStorage()for k in form.keys():  ... stuff ...

Then one

Once upon a time I wrote a program to scan incoming mail messages duringthe SMTP conversation for signs of spam. Because this was running aspart of our mailer, reliability was very important; unhandled errorscould cause us to lose mail.

As part of reliability, I decided I wanted to catch any unhandledexception (which would normally abort the program with a backtrace),log the backtrace, save a copy of the message that caused the bug, andso on. So I wrote code that went like this: