Why web spiders should not crawl syndication feeds

On the surface, crawling syndication feeds looks like an attractive ideafor web spider operators (although I am not convinced that the metadatathey get is on the whole any better than the metadata on web pages).But as things are today, it is a terrible idea and is highly likely toprovoke bad reactions if attempted.

The big problem is that right now, if you turned a web spider loose onsyndication feeds it would pull far too many of them. This is becausepeople (and websites) have lots


What promiscuous mode does on modern networks

Recently, I have wound up wondering if using or not using tcpdump 's -p switch made any difference on modern hardware and modern switchednetworks (partly because not using it causes your kernel to generate amessage every time you start and stop tcpdump , and I can live withoutthat). The somewhat unfortunate answer is that yes, it still makes adifference.

First, promiscuous mode is still at least partly a low level hardwarething in your network card, not just a software switch inside thenetworking stack.


Shimming modules for testing (and fun)

Suppose that you have a chunk of Python code that wants to properlymap IP addresses to hostnames , and you wantto test this code to make sure that it actually works (especially withunit tests). In order to do this you need to contrive for various IPaddress and hostname lookups to fail in various ways, and to do thison command.

The easy way to do this is to exploit Python's freedom by shimming (well, replacing) the gethostbyname() and gethostbyaddr() functions in the


The risks of forcing frequent password changes

People can talk about the risks of not changing passwords at the dropof a hat, but they rarely talk about the risks of forcing people to gothrough frequent password changes. This matters a lot, because thereare significant risks.

(And when people do acknowledge the risks, often it is to decry or mockthe 'bad habits' that people adopt.)

The obvious problems are that people can't remember things at the dropof a hat just because you want them to, and people have trouble coming


Making a good Unix glue language

The problem with a search for alternatives to the Bourne shell is that the Bourne shell isstill one of the best Unix programming languages. Or perhaps it wouldbe clearer to call it a 'Unix glue language', a language for connectingexisting programs together.

You might sensibly ask what the problems are with things like Perl andPython as Unix glue languages, and my answer would be that they make ittoo hard to interact with external programs. This is important, becausethe way you write small programs fast on Unix


Dear applications: WEP keys are not passwords

Every so often, I have the dubious pleasure of entering WEP keys in orderto get access to some wireless network around here. Invariably, theapplications treat the WEP key like a password, and has me type it in atext box where all the feedback I get is a ' * ' for each characterthat I type.

(We use WEP keys mostly as a way to keep random passers-by out. Notso random passers-by can crack the WEP key, at which point otherprecautions keep them


Frequent password changes as security mythology

One of the things I see trotted out in response to potential securityissues is a suggestion that we make users change their passwordfrequently. In addition to all of the practical reasons not to do this,it's useful to ask one of the most important question in security: whatrisks does this protect against, and how big are they?

The concise answer is that it doesn't actually protect against anyfundamental risks. What it does is limit the damage done to your systemsby compromised passwords (if you


Combining dual identity routing and isolated interfaces

In theory, combining dual identity routing with isolated interfaces should be simple.However, it turns out that there is a small trick to it.

This came up today because I am in the process of migrating my officemachine from one network to another. This means that I currentlyhave three active interfaces on the machine (old IP address, new IPaddress, and an internal network), plus one end of a GRE tunnel , and as part of the migration I wanted to force therouting to work right,


Getting live network bandwidth numbers on Solaris

After I wrote netvolmon for Linux ,I started getting curious about how much bandwidth our current SolarisNFS servers were using. Unfortunately, Solaris's version of ifconfig does not report byte counts; fortunately, the kernel does keep thisinformation and you can dig it out with kstat (information courtesyof here , which has abunch of more sophisticated programs to report on this stuff).

The magic kstat incantation is ' kstat -p "*:*::*bytes64" ',which


Segregating your outgoing email to get blocked as little as possible

Like many places with a lively and long email history, we generate a number of different sorts ofoutgoing email. These sorts of email have different characteristics andthus different chances of being seen as spam email or bulk email by outside places .

So here's a rough list of the different sorts of email that our systemwinds up sending out:

  • direct email from users; the least likely to be considered spam,since the users are typing it themselves.

  • users forwarding email to outside places; oftenwinds up forwarding spam