An important addition to how ZFS deduplication works on the disk
My entry on how ZFS deduplication works on the disk turns out to have missed one important aspect of how deduplicationaffects the on-disk ZFS data. Armed with this information we canfinally answer some long-standing uncertainties about ZFSdeduplication .
As I mentioned in passing earlier , ZFS uses block pointers to describe where the actual data for blocks are.Block pointers have the data virtual addresses of up to threecopies of the block's data, the block's checksum, and a number ofother bits and pieces
Backup systems, actual hosts, and logical hosts
One of the little but potentially important differences betweenbackup systems is whether they can back up logical hosts or if, forone reason or another, they can only back up actual hosts. Sincethis sounds like a completely abstract situation, let's set up aconcrete one.
Let's suppose that you have three fileserver hosts, call them A,B, and C, and two logical fileservers, fs1 and fs2 (and some sortof movable or shared storage system behind A, B, and
Failover versus sparing in theory and in practice
Suppose that you have a fileserver infrastructure with some number of physical servers, abackend storage network, and some number of logical fileservers embodiedon top of all of this. Broadly speaking, there are two strategies youcan follow if one of those physical servers has problems. You canfail the logical fileserver the physical server hosts over to anothermachine, perhaps a hot spare server, or you can replace the physicalhost in place with some amount of spare hardware, for example by simplyremoving the system disks and putting them
Static sites are stable sites
The usually cited advantage of generating static HTML for your websiteis the performance you easily get. But there's another, slightly lessobvious advantage of a static site: it's easy to preserve and tomaintain in operation with basically no attention and effort.
Dynamic sites need code. That means you need a web host where you canrun your kind of code and your programs, and it also means that you needto do all of the ongoing work to keep your site's code running. If it
How dynamic language code gets optimized
Yesterday I mentionedin passing that code that used Python dictionaries in predictablestruct-like ways could probably be optimized just as similar usesof Python class instances is (in JIT environments like eg PyPy).This may raise some people's eyebrows; after all, aren't Pythondictionaries too complex and flexible to get optimized?
It's my understanding that this is based on a misunderstanding ofhow modern dynamic languages are optimized. To simplify, the coreof optimizing dynamic languages like Javascript, Python, and so
Thoughts about Python classes as structures and optimization
I recently watched yet another video of a talk on getting goodperformance out of Python. One of the things it talked about wasthe standard issue of 'dictionary abuse', in this case in the contextof creating structures. If you want a collection of data, theequivalent of a C struct , things that speed up Python will do muchbetter if you say what you mean by representing it as a class:
class AStruct(object): def __init__(self, a, b,
What I can see about how ZFS deduplication seems to work on disk
There is a fair amount of high level information about how ZFSdeduplication works. There is much less that I could find aboutsome low level details of how deduplicated blocks exist on disk andsome implications of how the on-disk data structures are stored.Since I was just looking this up in the current Illumos source code,I want to jot down some notes before it all falls out of my headagain.
The core dedup data structure is the DDT, which holds the coreinformation for each deduplicated block: the
A Unix semantics issue if your filesystem can snapshot arbitrary directories
A commentator on my entry on where I think btrfs went wrong mentioned that HammerFS allows snapshots of arbitrarydirectories instead of requiring you to plan ahead and createsubvolumes or sub-filesystems as btrfs and ZFS do. As it happens,allowing this raises a little question about Unix semantics. Let'sillustrate it with some hypothetical commands:
mkdir a btouch a/fredln a/fred b/snapshot --readonly a@testls -li a/fred .snap/a@test/fredrm -
How Yahoo's and AOL's DMARC 'reject' policies affect us
My whole interest in understanding DMARC startedwith the simple question of how Yahoo's and AOL's change to a DMARC 'reject' policy would affect us and our users,and how much of an effect it would have. The answer turns out tobe that it will have some effects but nothing major.
The most important thing is that this change doesn't significantlyaffect either our users forwarding their email to places that payattention to DMARC or our simple mailing lists because neither of them normallymodify email on
At least partially understanding DMARC
DMARC is suddenly on my mind because of thenews that AOL changed its DMARC policy to 'reject' ,following the lead of Yahoo which did this a couple of weeks ago .The short version is that a DMARC 'reject' policy is what Ioriginally thought DKIM was doing : it locksdown email with a From: header of your domain so that only youcan send it. More specifically, all such email must not merely havea valid DKIM signature but a signature that is for the same domainas the