Jan 162013

Over the years I have tended to specialize in a particularly focused areas of development. Until recently, this has primarily been in locking and scheduling, particularly with respect to multiprocessing and real-time.

Since I have joined Intel and been working on the Yocto Project, I have had to branch out quite a bit. From the Linux kernel side, I’ve updated serial drivers, forayed into accelerometers and industrial I/O, debugged x86 early boot errors in the VM, contributed to the start of an upstream modular Linux kernel configuration system, mapped out minimal configurations and tooling for whittling things down, as well as keeping an eye on some of what I used to contribute to and fixing bugs as they arise.

Outside of the Linux kernel, I’ve worked to enable EFI support of some new platforms, re-factored facets of image building and early boot, performed a similar minimal configuration exploration for user-space, fleshed out support for image generation using the ext[234] file-systems, and generally made a nuisance of myself to those who actually know what they’re doing.

While I miss the ability to truly focus on a particular problem and dig deep into brain-bending execution graphs involving multiple threads, atomic variables, and memory barriers, I also appreciate the value of an increased awareness of how all these pieces fit together to form a greater whole. I’ll continue to try and squirrel away some time to work on things I’m most passionate about, but overall, I believe this time spent on the Yocto Project has made me a better developer.

Oct 282011

Today concludes my eleventh day in Prague attending three conferences and enjoying a few quick days of vacation with my wife.

Carsten Emde of OSADL was kind enough to invite me to speak on the Yocto Project at the Realtime Linux Workshop. I thoroughly enjoy hearing from members of industry, academia, and the Linux development community all at the same conference. I received a lot of positive feedback regarding the Yocto Project and collected my fair share of “todos”. Spending time with the leading Linux kernel developers is always inspirational for me, I invariably return with renewed commitment to improving my technical skills. After so many of these events, I consider these people my friends, and I so enjoy the trips that they feel more like a social event than a professional one. That, in my opinion, is how it should be.

Mary Lou joined me here in Prague while Grandma and Grandpa Mickelson braved our two children so she could get away for the needed break. She joined us on an OSADL sponsored walking tour of Prague covering the caste, lesser town, old town, and new town. We enjoyed the sights, the food was great, and some quiet time alone in a place where we had no responsibilities was wonderful. Her visit was long enough that she got to relax and enjoy her vacation, as well as look forward to getting home to our kids, our home, and our life.

Following our break, the Linux Foundation events began. The Embedded Linux Conference Europe (ELCE) and LinuxCon were held in parallel here at the Clarion Congress Hotel Prague. I spent a good deal of time at the Yocto Project booth, talking with attendees and catching up with team members. People were still trying to figure out how to spell Yocto a year ago when the project was announced. Now that the message has gotten out, there was a surge of interest – largely from people tired of maintaining their own frankenstein OS and looking for a solution. Others came by to be mesmerized by the Yocto Project 1.1 contributions visualization video rendered with the excellent gource project.

I attended excellent talks from Jonathan Corbet of LWN fame, Mathieu Desnoyers (EfficiOS), Grant Likely (Secret Lab), Stephen Rostedt (Red Hat), Frank Rowand (Sony), and Koen Kooi (beagleboard). The hallway track was also excellent. I met with developers and friends whom I only see online, including someone I had never met in person and hadn’t heard from in nearly 10 years when we hacked on some opensource projects together.

I wrapped it up with my presentation on tuning Yocto Project built images for tiny systems. Despite an imposing stack of 48 slides, I delivered the talk in the allotted time with room for questions without sending my audience into a speed-talk-stupor. I received some valuable feedback and made some useful contacts.

I am now exhausted mentally and physically. While I’m rather dreading the travel home, I’m very excited to be home with my family again.

May 042011

With the soon to be legendary “roadmap” Thomas Gleixner presented at the 2011 Embedded Linux Conference for the PREEMPT_RT patch series, people are likely wondering what that means for projects like The Yocto Project which make the PREEMPT_RT patch series available as part of a larger integration project.

It is our goal to maintain the linux-yocto recipes and git repositories as closely aligned with the mainline kernel as possible. This primary goal sets the schedule for which kernel version will be used as the basis for the next major release. 0.9 released with 2.6.34, 1.0 with 2.6.37, and 2.6.39 or 40 are likely to follow. These won’t necessarily align with the supported PREEMPT_RT kernel versions.

We have a few options. First, we could port PREEMPT_RT to whatever the next kernel version turns out to be. Second, we could release a separate kernel tree just for RT. Third, we could just not include PREEMPT_RT when the versions do not align. There is precedent for the first option, and indeed this exactly how we provide a 2.6.34 PREEMPT_RT tree with 0.9 and 1.0. We took the third option for 1.0 and the 2.6.37 tree, preferring to stick with the 2.6.34 version for RT kernels rather than do yet another forward port to 2.6.37, duplicating much of Thomas’ work toward a 2.6.38 PREEMPT_RT. This is time consuming, error prone, and doesn’t contribute to the overall quality of the PREEMPT_RT patch series as it is somewhat apart from that which Thomas releases. The second option, a separate kernel tree, is contrary to another key goal of The Yocto Project, which is to reduce duplication of effort. A separate kernel tree would require duplication of the meta branch, and all the bug fixes, security fixes, and feature sets would have to be applied to each tree.

Going forward, our approach to PREEMPT_RT will be as follows. We will strive to align with the official PREEMPT_RT releases. When that is not possible, we will favor skipping RT support in a kernel version or two just as Thomas does with PREEMPT_RT. In very rare occasions, such as with the 2.6.34 kernel, we may opt to forward port PREEMPT_RT to align with the current linux-yocto recipe’s base kernel version.

Feb 242011
From Rage

The present truck(s) were good to me today. I received the two Intel Xeon x5680 CPUs, the two Seagate Barracuda 1TB drives, the Intel 160GB G2 SSD, and the second heat sink. The SuperMicro hot-swap trays don’t allow for mounting 2.5″ drives, so I had to mount the SSD in a 3.5″ bracket in a 5.25″ bracket. Lame. As I mentioned in my last post, the first CPU cooler’s fan conflicts with the rear chassis fan. Since I had to choose between the two, I chose to keep the larger (quieter) chassis fan, but I connected it to CPU 1 FAN instead of the FAN 5 header. This is a guess on my part, but I figure the CPU is first thing to get hot, and the most valuable component in the system, so it makes sense to me to let its temperature determine the fan speed. This may cause problems however as the fan speed used by the CPU 1 FAN is probably not appropriate for the larger fan, and I don’t know how removing the FAN 5 connection will impact how the system decides to use the forward fan (which is smaller, and louder). Any insight readers may have here is very welcome.

Initial power-on is always exciting, this was no different, perhaps more so. After pressing the power button, Rage jumped to life like a wild beast startled from slumber. Her fans roared and her many bright beady eyes flickered their discontent. After familiarizing myself with her BIOS settings, I ran a quick Ubuntu 10 install off USB (it was absurdly fast). The BIOS RAID options were confusing at best, and I felt I just might get better results with software RAID via mdadm (at least more control). Rage is currently resyncing a RAID 1 array composed of the two 1TB SATA drives. I’m not sure quite how long this will take, and with her periodic snoring (loud fan bursts), I may just have to force her back into hibernation so my better half can sleep tonight.

Just as soon as I can I’ll kick off a complete Yocto build and share the results. Following that, I’ll run some burn in tests to ensure the memory, CPUs, and HDDs are all functioning properly. I haven’t tested IPMI 2.0 support yet (remote access, KVM, etc.) I’ll get to that soon as well.

Feb 182011
From Rage

The first round of components arrived for my Yocto Project and Linux Kernel development system. I haven’t built a system like this (piece by piece) since I started using laptops in 2002. I had to learn all the new terms for all the same architectural bits. Spec’ing out the system was an interesting experience, and I learned something about categories at Newegg. Finding quality components can be a real challenge as you first have to sort through all the neon-lights-and-acrylic-chassis-viewing-window-crowd junk. But, there is a short cut – the term is “server”. It’s great, select “Server” to narrow the search for memory, CPUs, and especially cases and CPU coolers and all the teenage-gamer-consumer crap goes away and you’re left with no-nonsense computing hardware. The heatsinks were under “server accessories” and not “cpu fans”.

So first, the specs:

The machine will be put to a variety of uses, but most of the time it will be used for two things. First, as a build system for the Yocto Project. We build for four architectures, a variety of machines, and several image types. A typical build takes two hours (we are working on reducing that) and as my primary area of focus is the kernel, I try to build as many architectures as possible as I change things. Once built, these images can be tested in qemu. Being able to build these quickly and keep all the build trees around to facilitate incremental builds is important to keeping productive.

Secondly, I’ll use this beast to continue to work on the futex subsystem, parallel locking constructs, and real-time. When it comes to catching locking bugs or identifying bottlenecks – there is simply no substitute for core count.

When it isn’t busy with either of the above, I hope to use this system to build and test the mainline and tip Linux kernel trees.

Back to the assembly. For this stage, I only have the chassis, motherboard, and memory. I’m having to wait a bit on CPUs and disks. The assembly was straight forward, but I obsessed about airflow and cable management. Supermicro matches their chassis to their motherboards, so the usual time spent mapping and aligning every LED and switch connector was replaced with single ribbon connector – very nice. I still read through the manuals to make sure I was getting everything right. Turns out the motherboard has a built-in speaker where the manual says the speaker header should be, fine. There is some ducting to keep air flowing from the front of the chassis, over the motherboard, and out the back. I made sure I routed the SATA cables clear of that. Finally, the 665W ultraquiet PSU is not modular, so I had to find a place for all the cables I didn’t use while minimizing obstructions to airflow for the chassis and the PSU itself. Some careful bundling and a couple wire ties seems to have wrapped that up nicely.

I also discovered that CPU1’s fan conflicts with the rear chassis fan. I have a choice: I can remove the rear chassis fan, or I can remove the fan from the CPU heatsink (which was made easy by Supermicro). I’m somewhat disappointed in Supermicro here. This is their motherboard, with their recommended CPU fans, in their recommended chassis. Fortunately, the rear fan is immediately behind CPU1, and likely moves as much, if not more, air with less noise. If I do remove the CPU fan, do I connect the chassis fan to the CPU1FAN header, or leave it connected to the generic FAN5 header? I was pleased that both chassis fans and the CPU fans are four-wire fans, meaning their speed (and therefore noise level) can be controlled by the BIOS depending on temperature.

This motherboard support IPMI 2.0, meaning it has a service processor and a full graphical KVM. I’ll be running this system headless connected via two gigabit links to my home network. I was very pleased overall with the quality of the Supermicro components, they are a significant step up from what I’m used to seeing in consumer computing and while not cheap, they were not particularly expensive either. Only time will tell, but I’m becoming a Supermicro fan… er…. enthusiast.

Next time: CPUs, HDDs, RAID setup and benchmarking!

Jan 142011

I holed up in my cave with a Beagleboard XM rev B and spent the day learning how an ARM board boots. After a few hours of aimless wondering, misdirection, ill-conceived notions, and deep tangents, I am happy to say I managed to boot a minimal Yocto image on the Beagleboard XM!

beagleboard login: root
root@beagleboard:~# uname -r

From dvhart's blog

The key difference (in terms of booting) between the Beagleboard and the Beagleboard XM is the lack of NAND. This changes the boot process slightly. Specifically, it requires an X-Loader and a u-boot binary on the MMC (as opposed to having them in NAND) – or something along those lines. I am thrilled that in a single day (not even a very long one) I was able to get Yocto booted on an architecture I have exactly zero experience with.

I will be spending next week tweaking poky recipes to generate the proper binaries and user boot scripts. The linux-yocto meta-data for the Beagleboard will also need to be updated for the new ethernet inteface on the XM. For those of you eager to boot Yocto on your Beagleboard XM, you won’t have to wait long!

And in case you are wondering how your Kindle can help you with Beagleboard development, here are two ways. First, it can read the Beagleboard manual if you email it to your kindle account. And second, its power supply with the proper mini-usb cable provides adequate voltage and more current than my laptop and is able to power the Beagleboard XM, while my laptop could not source quite enough current, resulting in a kernel panic, oddly enough.

Oct 272010

The first question I received (and indeed the first one I asked a few weeks ago) regarding the Yocto Project was, “Yet another Linux distribution?” We certainly have plenty of those, adding one more would really only serve to further fragment the embedded space.

So no, Yocto is not a distribution (like Fedora and Ubuntu) and it is not a platform (like MeeGo and Android). The Yocto Project is a workgroup (as described in the Linux Foundation Announcement) and the core bit of software behind it is the Poky build system which has its roots in OpenEmbedded. So, the Yocto Project is targeted at making it easier to create your own Linux distribution. It also serves as an umbrella project to collect things like BSPs all in one place.

The ultimate goal is to provide a common collaboration point from which we can reduce the chaos in this space and make it easier for people to bring Linux based devices to market.

If you’re still not convinced, please take a look at the Quick Start Guide and even the Reference Manual which will introduce you to the sorts of things you can do with the Yocto Project.

Oct 272010

After weeks of barely being able to contain myself while being probed about what I’m working on at Intel (my new job as of about 6 weeks ago), I can finally share all the details publicly. We’re proud to announce the initial public release of The Yocto Project!

Check out all the buzz about Yocto:

I am personally working on the Linux kernel itself. I’ve spent the last few weeks learning the system, fixing a few bugs wherever I ran into them, and preparing the live demo for the CELF conference.