Sep 202009

I’ll be presenting on Requeue PI at LinuxCon this week, so I was able to attend the speaker’s dinner at Departure this evening. It was a nice venue, rooftop porch with maybe 40-50 people mingling. I’m no mingler, but I met a couple new people and put faces to some names. I’m particularly interested to hear more from Jesper Brouer on 10Gb routing. Jon Corbet, of LWN, was ever-patient with me for being late (OK, late doesn’t quite cut it… grossly remiss) with a series of articles on real-time Linux for LWN. It was a good kick-off to the next two weeks of conferences. Open the flood gates and let the information overload begin!

Mar 042009

Many of you know I have been working extremely hard to make some quality contributions to the Linux kernel. This can be a tough environment to work it, but also very rewarding. For the last couple of months I’ve been working on a patch series to allow for the requeueing of Priority Inheritance aware futexes in order to make glibc pthread_condvars PI aware and avoid some of the problems with the current implementation. I recently sent out an early copy of the patch series to be reviewed by a few core people, not even labeling the mails with [PATCH] as is customary. Apparently that didn’t stop someone from finding my new tongue-in-cheek error code for dealing with misbehaving user applications. I was cited on LWN’s Quotes of the Week, along with Steven Rostedt, Linus Torvalds, and Andrew Morton. I was the only one, however, to earn a nickname for my efforts. Darren “graceful” Hart, thank you very much :-)

If you don’t have an LWN account, I’ll try and post a new link once it’s publicly visible, but for now, the interesting code segment mentioned in the article follows:

+	/*
+	 * The pifutex has an owner, make sure it's us, if not complain
+	 * to userspace.
+	 * FIXME_LATER: handle this gracefully
+	 */
+	pid = curval & FUTEX_TID_MASK;
+	if (pid && pid != task_pid_vnr(current))
+		return -EMORON;

Feb 172009

Jon Corbet has posted an article on LWN calling for ideas for topics for this year’s Linux Plumbers Conference. Last year was a great success, and I looking forward to attending this year’s conference. If you haven’t seen it, hop over to LWN and post your thoughts on topics you’d like to see at the conference this year. Current topic suggestions include things like improved user-space I/O APIs, configuration storage, and security integration. I’m debating if I should add “making wireless not suck on Linux”…. Add your own, help make LPC a success.

Feb 032009

Geeks worldwide have been using IMAP mail servers to, among other things, allow them to have a consistent view of their email and their folders from various clients. Unfortunately, many useful IMAP features have had poor and spotty client support. The most notable for me has been server side tagging. There is the “!Important” flag… but come on… how much information can you really store with one flag? Is it something I need to do? Something I need to follow-up on? Something I just want to be sure I don’t delete? Sure you can use Labels, but most clients store those in local databases so when you pull up your mail in the office (or at home) all of your hard work tagging those mails is now invisible to you! Enter Thunderbird 2.0 with support for IMAP Keywords in the form of tags. So long as you restrict yourself to the default 5 tags mentioned in the latest IMAP spec (or rfc, or whatever it is) these tags are stored on the server! No kidding! The trick is to rename the existing Tags from the default Thunderbird profile rather than delete those and create your own. If you do that, they become custom tags and don’t get propagated, at least they didn’t appear to for me. Once you rename your tags, your prefs.js file should look something like this:

user_pref("mailnews.tags.$label1.color", "#FF0000");
user_pref("mailnews.tags.$label1.tag", "Important");
user_pref("mailnews.tags.$label2.color", "#CC33CC");
user_pref("mailnews.tags.$label2.tag", "Follow-Up");
user_pref("mailnews.tags.$label3.color", "#CC9933");
user_pref("mailnews.tags.$label3.tag", "Someday");
user_pref("mailnews.tags.$label4.color", "#3333FF");
user_pref("mailnews.tags.$label4.tag", "To Do");
user_pref("mailnews.tags.$label5.color", "#6600CC");
user_pref("mailnews.tags.$label5.tag", "Waiting");

If instead you find entries that look like:

user_pref("mailnews.tags.someday.tag", "Someday");
user_pref("mailnews.tags.someday.color", "#3333FF");

Then you are using custom labels. I found myself in this boat, so I edited prefs.js and added all the $label[1-5] pairs and then did a manual search to retag all my existing mail with the new labels. At this point it’s a good idea to copy your prefs.js to your other thunderbird installations so the client knows to look for those tags and render them in the same colors – lest you confuse yourself ;-)

Now… if only tbird would inline plain text attachments in replies….



Dec 042008

Hey, So I’m John*. I don’t blog. This isn’t even
my blog. I’m just borrowing it.

So with all the discussion on the planet gnome about future UI changes,
along with the long time rumbling about improving file storage, I
figured I’d throw this small idea for file management out into the fray.

Here’s a quick summary of the idea:

A fair amount of file data is managed by users not as files in folders,
but as objects in applications. Ask most Mac users where their music
is, and they don’t say “In their user folder, under the iTunes library
folder”, they’ll just say “it’s in iTunes”. The same is likely true of
their photos and iPhoto.

So my idea is this: Don’t fight user perception. Leverage file
management applications like Rhythmbox and F-Spot as top level storage
objects. Let them act like magic directories. They already deal with
searching, tagging, and storing meta-data, so lets not re-implement that
on the filesystem level.

If a user downloads a music file, they can just save to Rhythmbox in the
save-as dialog. Rhythmbox or any other file management application
functionally becomes a place to store things.

Here is the slide deck with more details.

Of course, I’m not a gnome developer. I’m just a user. So really this is
a bit of a personal vanity slide deck pitching my little homer-mobile of
an idea. But maybe, just maybe, a gnome developer might see it and think
of a better way to achieve something similar.

And since this is a blog post, here is a video of a cat:

* I’m speaking for myself in this post, and not for my employer.

Nov 222008

After a few weeks of disillusionment with my progress on BrainDump, John S. pumped me up with noise about casting a broad net and all that. You know, the percentage of people interested in BrainDump may be small, tiny even, but if I notify the entire planet about it, I may just get a handful of interested and talented people to participate in its development. So this weekend I spent some time and finished up the web presence of BrainDump.

Now I think I’ll sit on it a bit and get some feedback from a couple friends, I hope to start making noise on 43 Folders, Life Hacker, Freshmeat, etc. sometime next week. Please have a look and let me know what you think of the site.

Sep 212008

I spent the latter half of last week at the Linux Plumbers Conference in downtown Portland. And despite being a new conference, I found it to be one of the best conferences I’ve attended. I think this might be in part to its small size, there was a much better average developer to community leader ratio than at other conferences, indeed some of the key leaders don’t even attend some of the larger conferences. The fact that LPC piggy-backed on Kernel Summit had a lot to do with the turnout. So what was exciting?

“If you want to do it right, do what IBM does, do what Intel does. — Greg KH”
Well, maybe that was only funny to me…

While there was a lot of great content, the things that stood out to me were:

From Naught to Sixty in 5 Seconds

Arjam van de Ven and Auke Kok shared their work on getting Fedora and Ubuntu to boot in under 5 seconds on a netbook. While the SSD drives were key here, they still could do 10 seconds with rotating platters. Key bits including Super Read Ahead, disabling everthing you don’t use, and picking on everyone that wasted boot time :-) Canonical took another hit here for GDM taking 3 seconds to start (60% of their total schedule!) due to storing massive pngs that it scaled down at each run to fit the current resolution. Definitely plan to look at my boot process and see if I can knock a few 10s of seconds off.

Git Tutorial

So was this cool only because it was Linus presenting? Yes and no. Linus is actually a great presenter. He is bright (duh) but also witty and engaging. He had no slides and simply walked the room through some git usage. While most everyone in the room had used git before, no-one seemed ruffled by it. It was nice to hear first-hand why git is the way it is. Did I come away from it loving git – uh… no. But it did clear up some of the issues I had with it – mostly by correcting my perspective on how it should be used.

Graphics Drivers in the Kernel: 20 Years Late

Dave Ailie and Jesse Barnes (among others) discussed the work being done to move video drivers into the kernel (from xorg userspace where they are now). Specifically memory management and mode setting. There was some good debate between the speakers and Linus, which ended with three of the speakers simultaneously stating “Well, you’re wrong!” to the chief penguin. There was a good chuckle from the crowd, things usually go the other way. The presenters knew their work, they knew the interactions of the various pieces, and they were able to defend their approach which Linus didn’t accept at first. His pragmatism was very apparent during the discussion and his drive toward simplicity made very clear. An excellent discussion. So what does this mean for users? hopefully those of us with the right chipsets will not have to watch our screens struggle through resolution changes 3 times during a single boot :-)

Hallway Dialog

While the formal talks and discussions were great, the best part of any conference is the one on one time with the people you work with over email and irc the rest of the year. For me this was a lunch with Thomas Gleixner, Steven Rostedt, and Gregory Haskins. We discussed the problems pthread condition variables have with priority inheritance, and formed a game plan to get the solution moving forward. It’s a tough problem, and largely logistical in nature rather than technical. Changing APIs (and ABIs) at the glibc level is a painful process (for good reason).

All in all, it was a great conference. A lot of good discussion between the developers of various systems was had (which was of course the intent). Thanks to IBM for letting me attend, and thanks to the organizers who donated so much of their time to the event.

Jul 052008

I picked up a 1 TB Western Digital My Book World from Fry’s today – that’s a 1 Terabyte Gigabit Network Attached Storage box – for $220. I’ve seen a few consumer technical appliances that run Linux under the covers, and haven’t been terribly impressed with many of them. So far however, this box is slick. While WD doesn’t support Linux officially (bad WD!) it is pretty trivial to get the box into a techie-friendly state.

With only an hour or so of tinkering, I was able to enable ssh, disable the java server, add my own users, and mount my partitions via sshfs. The info on doing this is already covered by “Paul” at his How to setup My Book World Edition II page so I won’t duplicate that here. Should also probably link to Martin Hinner’s My Book Howto directly as he appears to have done the hard work. I’ll spend the next few days figuring out how to best make use of this device; most likely looking into something like rdiff-backup.

Something they don’t mention are the specs of this little beauty, for those of you who might care, read on…

[root@MyBookWorld ~]# uname -a
Linux MyBookWorld #1 PREEMPT Fri Jan 18 10:40:25 GMT 2008 armv5tejl GNU/Linux

[dvhart@MyBookWorld ~]$ head -n2 /proc/cpuinfo 
Processor       : ARM926EJ-Sid(wb) rev 5 (v5l)
BogoMIPS        : 99.73

[dvhart@MyBookWorld ~]$ head -n2 /proc/meminfo 
MemTotal:        30032 kB
MemFree:          4496 kB

[dvhart@MyBookWorld ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                2.8G  179M  2.5G   7% /
/dev/root             2.8G  179M  2.5G   7% /
/dev/md3              950M  151M  751M  17% /var
/dev/md4              914G  200M  913G   1% /shares/internal
Jun 192008

Today at the Red Hat Summit in Boston, Red Hat announced the official release of Red Hat Enterprise MRG V1 (Messaging Realtime Grid) [1]. A couple snippets of note:

“The Realtime component of Red Hat Enterprise MRG comprises numerous kernel enhancements that provide deterministic performance for time-critical and latency-sensitive applications.”

“IBM has worked together closely with Red Hat on the development of the real time Linux kernel and has optimized both WebSphere Real Time and BladeCenter servers on Red Hat Enterprise MRG. We are delighted that IBM and Raytheon have been recognized by Red Hat for this innovation which has led to the largest deployment of real time technology in the next generation of US Navy Destroyers.” –Jeff Smith, vice president, Open Source and Linux Middleware at IBM

Red Hat is the second vendor to announce an “Enterprise” Real-Time Linux offering (one that runs real-time applications in addition to all the workloads traditionally associated with an “Enterprise” distribution). Novell/SUSE has recently released SUSE Linux Enterprise Real Time 10 Service Pack 2 [2]. The availability of two COTS (commodity, off-the-shelf) offerings marks the beginning of a new era, one of “Real-Time for the Masses”. More than two years ago IBM began working with Raytheon on the DDG 1000 project, mentioned by Jeff Smith above, to prepare Real-Time Linux and WebSphere Realtime for use on next generation of US Navy Destroyers. I have enjoyed working closely with the RT folks at Red Hat in preparation for the release of MRG, and am excited to see more and more Enterprise applications take advantage of what Real-Time Linux has to offer.

1. Red Hat MRG V1 Press Release
2. SUSE Linux Enterprise Real Time 10 Sevice Pack 2

Jun 062008

As enterprise distributions continue to roll out their respective real-time offerings, the benefits (and drawbacks) of real-time are becoming available to the masses. The expectations for these distros are often a mixture of general purpose computing performance and real-time determinism and low-latency. Unfortunately, meeting one of these often comes at the expense of the other. Deterministic behavior requires more overhead in the OS, including some lock serialization and additional context switching, which naturally degrades throughput. Two real-time Linux developers, Steven Rostedt of Red Hat and Gregory Haskins of Novell have been actively working on two distinct approaches to minimizing the effects of determinism on throughput.

Steven’s rwlock-multi patches [1] (2.6.24.X-rt8) attempt to reduce the bottleneck of the conversion of the rwlock to an rtmutex in the PREEMPT_RT [2] tree. In order to provide Priority Inheritance [3] support, the rwlock was converted to an rtmutex, limiting it to one owner. This bottleneck hit some subsystems harder than others, networking was particularly affected. The rwlock-multi patch relieves the serialization by allowing a fixed number (configurable) of readers to hold the lock at any one time, bounding the wait time to a higher priority writer. As would be expected, if a high priority writer is blocked on an rwlock, it will boost the priority of all the readers, and prevent any new lower priority readers from getting the lock (as soon as the existing readers release the lock, the writer gets it). This approach certainly increases the latency a high priority writer may see when acquiring a contended lock, but the increase is bounded, and therefor still deterministic. The benefit is seen by all subsystems that leveraged the rwlock for additional throughput, bringing the rt kernel closer in performance to the mainline kernel, while maintaining the necessary deterministic behavior.

Greg’s adaptive locks [4] patches aim to minimize the impact of converting spinlocks to rtmutexes in the PREEMPT_RT tree. This conversion was made to ensure the highest priority task would get the lock using a locking mechanism that supports priority inheritance and priorty-ordered wait queues. The side-effects included a lot more context switches as tasks blocked on the lock rather than actively spinning. However, by design, spinlocks were always intended to be held for very short periods of time, the adaptive locks patch modifies the rtmutex to busy-wait for a short configurable period of time. Since the rtmutex is still being used, the patch maintains priority inheritance and preemption of the blocked task. Improvements of up to 500% were seen on certain I/O intensive workloads, without negatively impacting the deterministic behavior of the PREEMPT_RT patch.

These sorts of patches go a long way towards bringing low-latency computing to the masses, as the costs for determinism lessen.

1. New read/write locks for PI and multiple readers
3. Priority Inversion Avoidance for Enterprise Real-Time Linux Systems
4. adaptive-locks v3