Adventures in network design, illustrated by our new backbone connection

Our current connection to the campus backbone is a 100 megabitconnection. While we have a (somewhat) new gigabit backboneconnection, we are not using it yet because we need to revise ournetwork architecture.

One of the issues with our current network setup is that it wasdesigned before firewalls were common. As a result, our currentbackbone connection connects directly to one of our /24 subnets, where(of course) a number of our servers live. This forces us to use abridging firewall


On the performance of Python longs being used as bitmaps

A comment on the previous entry got meinterested in what exactly cost all the time when using Python longs asbitmaps.Some poking shows that the time & takes scales up based on the smallerlong involved, and the time | takes scales based on the larger longinvolved. While this is not particularly surprising, it means thatsetting and clearing any bit has a cost based on how big the bitmap is,ie, what the largest set bit is.

(Checking low bits in a large bitmap is relatively cheap,


Clever large integers

Where bitmap is a Python long, consider the innocent looking Pythonexpression:

bitmap = bitmap & ~(1L << bit)

This is the perfectly normal way of clearing a bit in a bitmap, soit looks like there's nothing peculiar going on here.

Except that Python longs are of indefinite (and theoretically infinite)length, and what makes this expression work in normal integers is thatthe two pieces have the same (finite) size. So Python is clearly notjust doing straight binary operations here


A gotcha with the format of dump archives

Most archive formats, such as tarballs, cpio archives, and zip archives,interleave the file names (and their directory structure) with the filecontents. The dump filesystem backup program such as Solaris ufsdump do not. The dump archive format starts with an index that has all of thefilenames and the directory structure; after this comes the actual filecontent, labeled by inode number.

This has an important consequence for detecting damaged archives. In aninterleaved format like a tarfile, getting a full file listing requiresreading the


Explaining some university staff peculiarities

I think that universities not having a return on investment as such may go at least part of the way to explain someweird university behaviors around staff. The core issue is that withoutan ROI it becomes difficult to see the cost of how staff spend theirtime.

(With a ROI for projects it is easy to see how having staff spend theirtime on a lower-ROI activity more or less directly costs you money.)

Not only does this effectively make staff into a constant sunk costregardless of what you have


A surprise with the Provides header in RPM

Normally, one of the things RPM bases dependencies on is package names.The problem with doing only this is packages that want to depend oncapabilities instead of specific packages; for example, that there issome mailer installed, not specifically that Sendmail is installed. Todeal with this, RPM introduced the Provides: directive, which lets anRPM package tell the overall RPM system that it is providing somethingthat is not obvious from its name, its files, and so on.

In implementing this, RPM has chosen to


Implementing a preforking network server in Python

I recently read a great explanation of thegeneral Apache 2 preforking server model ,and found it so lucid and simple that I was immediately inspired toimplement a generic preforking server myself. Their approach has a bitof overhead in parent/child communication, but it does mean that you donot have to pass file descriptors between processes; all you need issome way for the parent to talk with its children.

(Since Python's standard library doesn't have support for filedescriptor passing, the pragmatic advantages of


Thinking about more text formatting for DWiki

What DWiki is used to write about has drifted a fair bit from what Ioriginally wrote it for, and some of the choices I originally made nolonger seem so ideal. I've been thinking for a while that DWikiText , thewiki-text formatting language here, needs some more formatting features;this entry is at least in part me thinking out loud about them.

So, here are the new features I think I want:

  • 'literal words', words that appear exactly was written withoutany special characters

Weekly spam summary on August 4th, 2007

This week, we:

  • got 111,59 messages from 243 different IP addresses.
  • handled 18,480 sessions from 1,401 different IP addresses.
  • received 393,665 connections from at least 102,514 different IPaddresses.
  • hit a highwater of 34 connections being checked at once.

Volume is up quite a lot from last week ; alsoup is how many different IP addresses are trying to send us email. My


Using WSGI for performance tuning

Doing profiling and performance tuning on a web application is normallya big pain. It's hard to profile things when processes (or threads)are always coming and going, and the typical web environment has somany sources of random delays that it is difficult to do fine-grainedperformance comparisons between two bits of code.

With WSGI, you can get out of this. Your WSGI application doesn't dealwith the web directly, so you can write a WSGI frontend that is just atiming and profiling harness