Putting the 'User-Agent' in your web crawler's User-Agent

In the "that's not how you do it" category, here are two HTTPUser-Agent values that I saw on Wandering Thoughts recently:

User-Agent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:103

Understanding ZFS ARC hit (and miss) kstat statistics

The ZFS ARC exposes a number of kstat statistics about its hit and missperformance, which are obviously quite relevant for understandingif your ARC size and possibly its failure to grow are badly affectingyou, or if your ARC hit rate is fine even with a smaller thanexpected ARC size. Complicating the picture are things like 'MFUhits' and 'MFU ghost hits', where it may not be clear how theyrelate to plain 'ARC hits'.

There are a number of different things that live in the ZFS


Thinking about our passive exposure to IPv6 issues

Over on the Fediverse, I recently saw a thought-provoking remarkby Michael Gebis ( via ):

If I were an evil threat actor, I'd be learning as much about #ipv6 aspossible right now. I'm convinced that many companies that say they"aren't using" IPv6 are in reality just ignoring IPv6, and it wouldbe easy to set up a "shadow network" consisting of IPv6 traffic whereyou could get away with murder. Nobody at the company


Programming on Unix and automatic memory management

Due to the pervasive influence of C, Unix is commonly thought ofas a programming environment of manual memory management; you programUnix in C and C requires manual memory management. Except that Imaintain this is somewhat of an illusion. From the days of V7 Unixonward (if not earlier), a substantial amount of Unix programminghas always been done in languages with automatic memory management.It's just that we called this programming "shell scripts", "awk","make", and so


The two types of C programmers (a provocative thesis)

Here is a provocative thesis:

There are two types of C programmers: people who chose C because theyliked various of its properties, or people who used C because it wastheir best or only option at the time.

Back in the days , C was somewhere between your bestoption or your only real option for doing certain sorts of programming.If you were writing a Unix program, for example, for quite a whileC was your only real choice (then later you could consider C++).The


The traditional workaround for stuck NFS(v3) locks

Up through NFS v3, advisory file locks over NFS were done through a separateprotocol and set of systems, the "Network Lock Manager" (NLM) setof protocols (which I believe are best covered in File Lockingover XNFS and Network Lock Manager Protocol ). Filelocking is naturally a stateful system, where the server and theclients have to have the same state, but unfortunately the NFS v3NLM protocol doesn't provide for any way for servers or clients toexplicitly check that they agree on


Setting the ARC target size in ZFS on Linux (as of ZoL 2.1)

In the past I've grumbled about wanting a way to explicitly setthe (target) ARC size . After all of my recent investigation into how the ARC grows and shrinks , I now believe that this can besafely done, at least some of the time. However, growing (or ingeneral resizing) the ZFS ARC comes with a number of caveats, because it's only going to beeffective some of the time.

The simple and brute force way to grow the ARC target size to agiven


An interesting mistake I made with a (Go) SSH client API

We have a custom system for NFS mount authentication on our Linux fileservers that works, in part, by havinga SSH client connect to would-be NFS clients to verify their SSHhost key . In the process of writing thecode for this, I made an interesting mistake that is fundamentallyenabled by a long-standing OpenSSH naming confusion.

What you have in a SSH known hosts file is a list of (public) keys,each of which has a key type like'ssh-rsa', 'ssh


When and how ZFS on Linux changes the ARC target size (as of ZoL 2.1)

Previously I discussed the various sizes of the ARC , some important ARC memory stats , and ARC memory reclaim stats . Today I can finally talk abouthow the ZFS ARC target size shrinks, and a bit about how it grows, which is asubject of significant interest and some frustration . I will be citing ZoL function names becausetools like bpftrace mean you canhook into them to monitor ARC target size changes.

(Changes in the actual size of the ARC are less interesting than changesin the ARC target size. Generally


ARC memory reclaim statistics exposed by ZFS on Linux (as of ZoL 2.1)

Yesterday I talked about some important ARC memory stats , to go with stats on howbig the ARC is . The ARCdoesn't just get big and have views on memory; it also hasinformation about when it shrinks and somewhat about why.Most of these are exposed as event counters in/proc/spl/kstat/zfs/arcstats, with arc_need_free as anexception (it counts how many bytes ZFS thinks it currentlywants to shrink the ARC by).

The Linux kernel's