You can't assume that your performance problems will be obvious
One of the things underlying us not checking to make sure our harmlesschange actually was harmless is that we made an assumption; we assumed that if we had aperformance problem it would be obvious and therefor the lack ofany obvious performance problem meant that everything was fine. Infact we assumed this all through our disk performance issues . For example, because only the mailspool had any particularly obvious performance issues we assumedthat only the mail spool had issues ( and that we understood them ).
What this drives home is
A reason for detailed commit messages: as a guard against errors
There's lots of common reasons to write good descriptive commit messagesin your version control system. But I've recently come to realize asomewhat unusual reason why they matter; writing good commit messagesmakes it much more likely that you'll be able to recognize and recoverfrom inadvertent modifications. In theory inadvertent modificationsshould never make it into your commits. In practice this is always arisk and it happens periodically. Now suppose that you are coming backto the modification after the fact and you've tracked
What goes into the terminal's 'cbreak' and 'raw' modes
Recently, Eevee tweeted :
things i never thought i'd need to know: the difference between'raw' and 'cbreak' is not just some flag. it's like 20! [link]
This inspires me to talk about what 'cbreak' and 'raw' modes are, bothat a high level and then at the low level of exactly what terminalsettings go into each mode.
The traditional 'raw' mode is the easier one to explain; it does noin-kernel processing
All kernel messages should be usefully ratelimited. No exceptions.
I've written about this before , but here's today'sversion. On some of our new Dell servers, the Ubuntu 12.04 kernel willproduce messages like these two:
[Firmware Warn]: GHES: Failed to read error status block address for hardware error source: 49378.ghes_read_estatus: 2 callbacks suppressed
(If you're coming here from a web search on this message text and wantto know how to fix it, see the
My new view on why you need to profile code
The usual and traditional story of why you want to profile your codewhen you're optimizing it is that people are terrible at predictingwhat their code is actually doing. It's not just that people don'tpredict where the time is going; often they don't even predict thingslike what functions are called a lot, how many times loops run, and soon. This view offers what I'll call a fantasy of perfect knowledge,the idea that if we really understood what our code
One reason why having metrics is important
Put simply, one of the reasons that having metrics is important is thatmetrics give you a backup check on changes that you think are harmless .
In many environments you can't exhaustively check the effects of everychange that you do. Trying to do performance checks all of the timeis simply both too time consuming and too much like monkey-work (andpeople tune out of monkey work). Sooner or later you'll start decidingthat some changes are safe enough that you can skip some or all of
When you make a harmless change, check to make sure that it is
As I mentioned before , our recent diskperformance issue has been good for a number oflearning experiences. One of the most painful ones (for me) comes fromthe fact that this was actually a self-inflicted performance problem.You see, we did not start out using two different sorts of switches on our two iSCSI networks . Initially bothnetworks used the lower-end switches until at one point where for various reasons we decided to swap one outfor the higher end and highly trusted core switch. After the
You don't necessarily know what matters for performance
One of the obvious lessons I could learn from the switch issue at the heart of our recent disk performanceissue is that low level detailsmatter, sometimes a lot. While this is true, I think it's asuperficial thing to take away from this learning experience . The lesson I really want to takeaway is a more general one:
I don't know in advance what matters for performance .
Until I had this experience you probably could have sat me down withthe switch specifications for both switches, asked
More thoughts on why Python doesn't see much monkey-patching
In yesterday's entry I advanced the ideathat a significant part of why Python doesn't see much monkey patching(and Ruby does) is that you can't monkey patch a lot of fundamentalPython classes and types because they're implemented in C. In soberhindsight, I think that I'm understating two intangibles (one of which Itouched on in passing) in favour of a technical explanation.
First, I think that the effects of culture matter more than I initiallythought. Culture plays
The limits of monkey patching in Python
One of the things I find interesting is the question of whymonkey-patching is a common thing in the Ruby world but not in thePython world. Certainly a significant part of this is cultural (Pythonculture is very much against monkey patching, Ruby culture seems to becompletely accepting of it), but it's hard for me to believe that that'sthe whole story. People do things to solve problems and I don't believethat Python is magically without the problems that cause people tomonkey