A really stupid web spider
Today WanderingThoughts had a visit from the worst stealth spider thatI've ever seen. Given the previous contestants this is a fairly tall order, but I'm confidantI have a winner. The spider:
- made two requests for directories without the trailing slash, earningit redirections to the proper URLs.
- followed the redirections, making two valid requests.
- promptly made 95 bad requests by failing to treat URLs with a leading slash properly.
I've seen spiders
Link: Search engine page size limits for indexing
Search Engine Indexing Limits: Where Do the Bots Stop? takesan experimental approach to seeing how big a page various search enginebots will fetch, and how much of large pages they index.I find this an interesting question because it affects how you organizeyour content and generate indexes to it, especially for dynamic websiteswith auto-generated aggregate pages.
One area not investigated in the article is how far down the pages thesearch engine bots will go looking for links to follow. I smell afollowup project for someone
SCGI versus FastCGI
SCGI and FastCGI are both CGI 'replacements', in that they areprotocols for forwarding HTTP requests to persistent daemons instead ofstarting a possibly big, heavyweight program for each request. Ideallyyour web server will have a built in gateway for them; less ideally youcan run a tiny, fast to start CGI program to talk the protocol toyour persistent daemon.There's discussions around the Internet about which one is better; youcan find people on both sides, and to some extent it depends on whatyour
Weekly spam summary on May 6th, 2006
This week, we:
- got 11,443 messages from 213 different IP addresses.
- handled 15,802 sessions from 820 different IP addresses.
- received 219,841 connections from at least 43,156 different IPaddresses.
- hit a highwater of 50 connections being checked at once, reachingit Monday.
Connection volume is up significantly from the extrapolated levelsof last week . All of this is despite usbeing down for about half of
Using a stock kernel.org kernel on Fedora Core 5
If you like initrds, the stock Linux kernels from kernel.org should just compile and work. Install the
kernel-develRPM, copy the.configfrom/usr/src/kernels/to your source tree to duplicate the normal Fedora Core kernelconfiguration, make oldconfig, and go.I don't like initrds, though. So one of the usual benefits of compilingmy own kernel is not needing one, so there's one fewer moving part inmy system environment. Unfortunately,
Link: Readable colour text combinations
Color Test Results is a summary of the results of surveying a bunch of Internet users aboutwhat colour text on what background they found the easiest to read. Thesummary quote:
As you can see, the most readable color combination is black text onwhite background; overall, there is a stronger preference for anycombination containing black.
Not coincidentally, most of my text is black on white (or black onslightly off-white). I was interested to see that white on blackactually rates fairly highly, because
Peeking under mount points with NFS
Normally, one of NFS's irritating features is that when you mount afilesystem from a server, you don't automatically get access to anysub-filesystems mounted on that filesystem; you have to know about themand mount them yourself.
(Yes, yes, some NFS servers offer features to do this for you; suchfeatures have their own problems.)
But there's an old sysadmin trick that turns this into a feature. If youNFS mount a filesystem, such as
/, you can
A subtle advantage of simple wikis
One of the interesting things about programming DWiki is how it'swound up making me notice and appreciate the subtle cleverness of WardCunningham's original simple wiki design, in particular the choice of aflat page namespace. Many of the advantages are relatively technicalbut there are more abstract ones, one of which I stumbled into today:
You do not have to taxonomize information before you put it into aflat wiki; you just have to give it a reasonably good name.
When you have a page hierarchy of some
Fedora Core 5's missing 32-bit shared library symlinks
For my sins, I decided that my new machine should run the 64-bit versionof Fedora Core 5. I knew this was going to be somewhat painful, becausevarious important things (such as Flash and Sun's Java) don't yet have64-bit versions; I was going to have to build 32-bit versions of thingslike Firefox in order to use them.
Little did I know that Fedora has made it all but impossible to compilea 32-bit version
CSS and syndication (another CSS limitation)
In an earlier entry on Solaris patchexit codes I mentioned that its formatting might not look very nice insyndication feeds. And I was right; it did not look very good.
It didn't look good because it was styled with CSS, and you can't reallyput CSS in syndication feeds. CSS goes into the HTML element,but that only appears in HTML pages; syndication feeds and entries aredivorced from that sort of context. This is more or less implicit inRSS ( like
Menu