Work and family life was busy, so it was a few days before I could put the QNAP TS-419P+ to the test with some representative use cases. But before I did, I spent some time educating myself with RAID levels and came to the conclusion that until I am in desperate need of more storage space, RAID 5 just doesn't make sense. Here's why. RAID 5 distributes parity across all the drives in the array, this parity calculation is both compute intensive and IO intensive. Every write requires the parity calculation, and data must be written to every drive. With the low power CPU already the bottleneck for throughput, adding an additional load didn't seem like a good idea. More importantly is data integrity. RAID 5 allows for a single drive to fail without any loss of data. However, while rebuilding the array, if one of the remaining three drives were to fail, all the data is lost. Rebuilding a RAID 5 array after a single disk failure is also very compute and IO intensive as every disk must be read in order to restore the blocks to the new drive.
A better option for a four drive array is RAID 10, a striped pair of mirrored disks. In this configuration writes only affect the two drives in the mirror, and the data integrity is much improved. After a single drive fails, it is restored by copying it's sibling in the mirrored pair. If one of the three remaining drives were to fail, there is only a 33% chance that data will be unrecoverable as either drive from the other mirrored pair could fail without a problem.
The cost for this is total volume size. RAID 5 provides SIZE*(N-1) while RAID 10 provides SIZE*(N/2). With 4 1.5TB drives, RAID 5 yields a 4.5TB volume, while RAID 10 yields only 3TB. When drives were expensive, this 50% gain was significant, but when 2TB drives can be had for under $100, and larger drives becoming available every year (3TB drives now ship with consumer level NAS products), the principle value of RAID 5 is not as convincing as it once was. With RAID 10 and current technology (3TB drives) I can still double my capacity, and that should only improve in the coming years.
OK, on to configuration. I took to configuring the NAS for use with my home Linux network. I should preface this by saying I am not a network file system expert, not even an experienced user. I have setup NFS enough times to know the homogeneous uid/gid thing is a pain and that there are plenty of failing corner cases with respect to dropped connections, file locking, etc. QNAP claims to support NFS, so I expected them to provide the necessary tooling in their oft-praised web interface. The sad truth is that NFS appears to be an afterthought, and the implementation barely merits an "[x] NFS" string on their marketing material. The UI allows you to add users, but not to specify (or modify) the uid or gid. This means that standard Unix file permissions simply do not work, and their solution appears to be to make the shares globally read-only or globally read-write. Two words: cop out. Fortunately QNAP does provide a root-only ssh shell, and I was able to log in and manually edit /etc/passwd and /etc/group to make my users match the rest of the network. Some careful recursive 'chmod g+s ug+rw o-rwx' commands provided me with the permissions I wanted - but avoiding that sort of work is precisely why I opted for QNAP instead of building my own. In this regard, they failed miserably.
The SMB story is better. The QNAP UI supports per volume user and group permissions. While the on disk representation is still globally read-write, it's not a problem as SMB performs it's own user authentication and only the admin user has ssh access anyway. I tinkered with this enough to get it working with the GNOME desktop file manager and with autofs. This might be the best way to access shares on the QNAP, even from a Linux system. Still, something about running SMB makes me feel like I need to shower.
For throughput tests I used the rsync daemon to copy my MythTV recordings to the QNAP. This consisted of 300GB of mostly mpeg2 files from 2 to 6 GB each. I used the UI as well as top to monitor the system status periodically during the transfer. The CPU was pegged at 100% for the duration of the transfer, and it averaged just over 20MB/s.QNAP claims 45MB/s writes over SMB and 42MB/s over FTP. Rsync should be faster if anything. Throughput was a disappointment. Following the transfer the QNAP remained under heavy load (7-8), and became fairly unresponsive. Watching the kernel logs I found a few Out of Memory errors, with apache and php being among the OOM Killer's victims. I raised the throughput and OOM issues with QNAP and my supplier. They weren't able to suggest any changes to improve throughput or identify why the OOM occurred. They did agree to allow me to return the TS-419P+ in exchange for a TS-459Pro+. The latter replaces the ARM CPU with a dual core Atom, doubles the RAM, and replaces the 16MB flash with a 512MB DOM. 20MB/s just wasn't cutting it, and a kernel OOM was just unacceptable. I shipped the QNAP TS-419P+ back and am impatiently awaiting a TS-459Pro+. Whether I keep the QNAP firmware or replace it with Ubuntu Server or perhaps a custom Yocto image is the subject for a future project.
As if the arrival of the final components for Rage wasn't enough tech debauchery, the present trucks also delivered a shiny new QNAP TS-419P+ and 4 Samsung Spinpoint F2 1.5TB drives. Devon helped me unpack everything and carefully mount the drives in their trays. He even helped me plug it in and start the initial setup process.
The QNAP packaging and physical documentation is simple, nay, spartan. Which I like. The device itself is smaller than I expected (always nice) but I was disappointed to find a separate power brick instead of a built-in power supply - this educed my excitement about the compact size of the unit a somewhat. The recommended "Linux Setup" was to connect a PC directly to the NAS and configure your networks to talk to eachother - this didn't appeal to me, so I just looked up the QNAP IP on my dd-wrt router and followed the directions for Windows and Mac - just without installing a qnap finder application.
The QNAP web interface is highly polished. Initial setup included setting the hostname (I selected Toph in keeping with my heroine theme for my personal machines), installing the latest firmware, setting an initial password, which network services to enable, and an initial RAID configuration. Perhaps this is obvious to everyone else, but be sure to unzip the firmware you download from the QNAP website, otherwise you'll just get an unhelpful error complaining the image is bad. I found the initial RAID selection to be odd as it is very limited. I chose RAID 5 as that is probably what I want to do, but the device offers a lot more options than a single RAID array using all the disks. Given the amount of time it takes to resync a 4.5 TB RAID 5 array - it seems like this step could be skipped and the user sent directly to the full-featured volume management admin screen at first login. Instead, after completing the initial setup, you are presented with this iTunes-wanna-be AJAX interface:
Here you can see the volume management screen - and an ascending time remaining field in the Status column. I really don't know how I'll partition things up, or if I even need to. The QNAP offers a _ton_ of flexibility in how you access your data. I'll need to spend a good deal of time considering them before I make a final decision. I'll reserve judgement on these features until then.
Out of the box, several network services are available for immediate configuration:
And finally, QNAP offers add-on packages in the form of QPKG, which oddly enough includes an IPKG application for even greater selection of packages. There are several media streaming servers available, including one that is pre-installed. The installation process appears a bit cumbersome, requiring the user to download the package to a PC and then upload it to the NAS for installation. I am looking forward to installing Python, possibly Twonky, and maybe MySQL and WordPress (I'm considering moving this blog away from Drupal and to something else).
So for now, my QNAP is resyncing its RAID 5 array. I hope to have the time to explore its many features soon, and I'll share my experience as I do. My initial impressions are good, and I'm optimistic that this will turn out to have been a good choice for our needs.
The present truck(s) were good to me today. I received the two Intel Xeon x5680 CPUs, the two Seagate Barracuda 1TB drives, the Intel 160GB G2 SSD, and the second heat sink. The SuperMicro hot-swap trays don't allow for mounting 2.5" drives, so I had to mount the SSD in a 3.5" bracket in a 5.25" bracket. Lame. As I mentioned in my last post, the first CPU cooler's fan conflicts with the rear chassis fan. Since I had to choose between the two, I chose to keep the larger (quieter) chassis fan, but I connected it to CPU 1 FAN instead of the FAN 5 header. This is a guess on my part, but I figure the CPU is first thing to get hot, and the most valuable component in the system, so it makes sense to me to let its temperature determine the fan speed. This may cause problems however as the fan speed used by the CPU 1 FAN is probably not appropriate for the larger fan, and I don't know how removing the FAN 5 connection will impact how the system decides to use the forward fan (which is smaller, and louder). Any insight readers may have here is very welcome.
Initial power-on is always exciting, this was no different, perhaps more so. After pressing the power button, Rage jumped to life like a wild beast startled from slumber. Her fans roared and her many bright beady eyes flickered their discontent. After familiarizing myself with her BIOS settings, I ran a quick Ubuntu 10 install off USB (it was absurdly fast). The BIOS RAID options were confusing at best, and I felt I just might get better results with software RAID via mdadm (at least more control). Rage is currently resyncing a RAID 1 array composed of the two 1TB SATA drives. I'm not sure quite how long this will take, and with her periodic snoring (loud fan bursts), I may just have to force her back into hibernation so my better half can sleep tonight.
Just as soon as I can I'll kick off a complete Yocto build and share the results. Following that, I'll run some burn in tests to ensure the memory, CPUs, and HDDs are all functioning properly. I haven't tested IPMI 2.0 support yet (remote access, KVM, etc.) I'll get to that soon as well.
The first round of components arrived for my Yocto Project and Linux Kernel development system. I haven't built a system like this (piece by piece) since I started using laptops in 2002. I had to learn all the new terms for all the same architectural bits. Spec'ing out the system was an interesting experience, and I learned something about categories at Newegg. Finding quality components can be a real challenge as you first have to sort through all the neon-lights-and-acrylic-chassis-viewing-window-crowd junk. But, there is a short cut - the term is "server". It's great, select "Server" to narrow the search for memory, CPUs, and especially cases and CPU coolers and all the teenage-gamer-consumer crap goes away and you're left with no-nonsense computing hardware. The heatsinks were under "server accessories" and not "cpu fans".
So first, the specs:
- Supermicro SC733TQ-665B Chassis
- Supermicro MBD-X8DTL-iF-O Motherboard
- Supermicro SNK-P0040AP4 CPU Heatsink and Cooling Fan
- 2 x Intel Xeon X5680 Westmere 3.33GHz 12MB L3 Cache LGA 1366 130W Six-Core Server Processor
- 2 x Patriot Signature 8GB 240-Pin ECC Registered DDR3 SDRAM 1333 (PC3 10600)
- 2 x Seagate Barracuda 1TB Sata II HDD
- Intel 160 GB G2 SSD
The machine will be put to a variety of uses, but most of the time it will be used for two things. First, as a build system for the Yocto Project. We build for four architectures, a variety of machines, and several image types. A typical build takes two hours (we are working on reducing that) and as my primary area of focus is the kernel, I try to build as many architectures as possible as I change things. Once built, these images can be tested in qemu. Being able to build these quickly and keep all the build trees around to facilitate incremental builds is important to keeping productive.
Secondly, I'll use this beast to continue to work on the futex subsystem, parallel locking constructs, and real-time. When it comes to catching locking bugs or identifying bottlenecks - there is simply no substitute for core count.
When it isn't busy with either of the above, I hope to use this system to build and test the mainline and tip Linux kernel trees.
Back to the assembly. For this stage, I only have the chassis, motherboard, and memory. I'm having to wait a bit on CPUs and disks. The assembly was straight forward, but I obsessed about airflow and cable management. Supermicro matches their chassis to their motherboards, so the usual time spent mapping and aligning every LED and switch connector was replaced with single ribbon connector - very nice. I still read through the manuals to make sure I was getting everything right. Turns out the motherboard has a built-in speaker where the manual says the speaker header should be, fine. There is some ducting to keep air flowing from the front of the chassis, over the motherboard, and out the back. I made sure I routed the SATA cables clear of that. Finally, the 665W ultraquiet PSU is not modular, so I had to find a place for all the cables I didn't use while minimizing obstructions to airflow for the chassis and the PSU itself. Some careful bundling and a couple wire ties seems to have wrapped that up nicely.
I also discovered that CPU1's fan conflicts with the rear chassis fan. I have a choice: I can remove the rear chassis fan, or I can remove the fan from the CPU heatsink (which was made easy by Supermicro). I'm somewhat disappointed in Supermicro here. This is their motherboard, with their recommended CPU fans, in their recommended chassis. Fortunately, the rear fan is immediately behind CPU1, and likely moves as much, if not more, air with less noise. If I do remove the CPU fan, do I connect the chassis fan to the CPU1FAN header, or leave it connected to the generic FAN5 header? I was pleased that both chassis fans and the CPU fans are four-wire fans, meaning their speed (and therefore noise level) can be controlled by the BIOS depending on temperature.
This motherboard support IPMI 2.0, meaning it has a service processor and a full graphical KVM. I'll be running this system headless connected via two gigabit links to my home network. I was very pleased overall with the quality of the Supermicro components, they are a significant step up from what I'm used to seeing in consumer computing and while not cheap, they were not particularly expensive either. Only time will tell, but I'm becoming a Supermicro fan... er.... enthusiast.
Next time: CPUs, HDDs, RAID setup and benchmarking!
I just got back from a trip to Best Buy to fondle a wide selection of laptops. Considering they don't carry the "business" line from HP or others, there is a giant gap between the consumer laptops and the Mac Book Pro. It would be like going to a car dealer that sold Kias and BMWs (no offense to all the Kia drivers who wait with baited breath for my next awe-inspiring blog post). I spent about an hour typing on some 20 keyboards and testing the flex of the chassis and keyboard. Most were Chiclet style keyboards, which turns out to be a sub-optimal design (in my opinion) if you use more than two fingers to type.
The Toshiba Portege R705 wins the worst-keyboard-ever-manufactured-in-the-history-of-bad-keyboards award. The keys are glossy plastic and flop left to right due to the incredible amount of slop in the design. They don't depress like... well, a keyboard... they sort of click with a very unsatisfying travel distance. In fact, my 4 year-old son's toy Cars laptop from Radio Shack has a much better tactile response! The chassis was thin plastic and the keyboard, the palmrest, and the screen all flexed a great deal under only minimal force. Considering it's $900 price tag (about $1,800 with a respectable amount of RAM and an SSD), I found the keyboard and chassis to be remarkably bad.
The 13" ThinkPad Edge's newly designed chiclet version of the ThinkPad keyboard is surprisingly good, and by far the best of the chiclets I've tried. The laptop is a very nice blend of consumer features and professional construction, but it's closer in design to a consumer laptop, and I'm concerned it wouldn't hold up to the daily abuse of heavy user. The glossy screen is also a drawback for extended use as glare can be a real problem in certain environments (Best Buy doesn't carry the ThinkPad line, I tested this one at Office Depot).
And the 13" Mac Book Pro gets an admirable-job-implementing-a-sub-optimal-type-of-keyboard aware. In all honesty, I'm pretty sure I could get used to the Mac's keyboard - despite the very shallow travel of the keyboard. The Mac shines in chassis design. The MacBook unibody is a truly incredible piece of engineering. It is very rigid, light, and incorporates only minimal machining to allow for the required ports, vents, etc. In terms of pure industrial design, nothing comes close to a Mac Book Pro.
The ThinkPad x201s sports a tried and true ThinkPad design and traditional keyboard. With a 12.1" screen, it is the smallest of this group (as the x301 is now outdated, gets terrible battery life, and is absurdly expensive), but shares the same 1280x800 screen resolution of the 13" Mac Book Pro. I haven't got my hands on one of these yet, but I've seen them in the wild and they meet all the expectations of a ThinkPad. Rugged, no-frills, business focused workhorse.
It's now abundantly clear to me, there are only two choices when it comes to quality professional grade notebooks: Lenovo ThinkPad and Mac Book Pro. My second discovery is that my belief that Macs were overpriced is completely false. You can't compare a Mac to a Dell or an Acer (not even an HP consumer grade machine), you have to compare it to the top of the line Lenovo ThinkPads (T510, x201s, x301, etc.) and then, feature for feature, the Mac is competitively priced. Damn.
I've been twice bitten by this subtlety of the python __cmp__() method, I thought I'd try to save you the same pain (and perhaps by writing it down I won't repeat it... again).
When dealing with objects, it's often useful to be able to compare these objects:
if mySwartz > yourSwartz: print "Suck it."
An obvious (and wrong) __cmp__() method would be:
def __cmp__(mine, yours): if mine.val < yours.val: return -1 if mine.val > yours.val: return 1 return 0
Looks good right? Well, what if some yahoo tries to compare two totally disimilar things, i.e:
if mySwartz > yourForce: print "Neener Neener."
Such a comparison clearly makes no sense, so let's update __cmp__() to check for similar data types:
def __cmp__(mine, yours): if not isinstance(yours, Swartz): return -1 if mine.val < yours.val: return -1 if mine.val > yours.val: return 1 return 0
Good right? WRONG. Consider the following ridiculous scenario:
mySwartz.val == yourSwartz.val
While I'm sure no such thing could happen, let's imagine for the sake of argument that two objects might be compared on an internal value that just might not be unique across all instances of the object. What would happen if you had to retrieve your object from a container?
storage =  storage.append(yourSwartz) storage.append(mySwartz) # time passes for sw in storage: if sw == mySwartz: print "I found my Swartz!" print "...wait... this is your Swartz... gah!"
Yeah yeah, you could rewrite this to yada yada, that's not the point. The point is while the values of the objects are the same, they are not the same objects and python uses the __cmp__() method for both less than, greater than, equality, AND identify comparison if you're used to thinking of your objects as pointers!
If you're faced with a situation where you want to be able to easily sort objects, but still need to be able to identify them uniquely when they share an internal value, augment your __cmp__() routine to check for id as well:
def __cmp__(mine, yours): if not isinstance(yours, Swartz): return -1 if mine.val < yours.val: return -1 if mine.val > yours.val: return 1 if id(mine) < id(yours): return -1 if id(mine) > id(yours): return 1 return 0
If you're a python guru and think I'm missing something, please share, otherwise:
if mySwartz > yourSwartz: print "I see my Swartz is bigger than yours..."
Apple's follow-on to the iPad, the iSmudge, will contain a moisture sensor so you can lick your thumb and "erase" stuff. Works on files, text, and those photos of your ex. The new device will ship with a monogrammed microfiber cloth to keep your screen "smudge" free.
Many of you have heard my reasoning for not having a fancy cell phone. That extra $30-$40 a month for a dataplan just seems outrageous to me. That got me to thinking about my current plan. My wife and I share 500 minutes and pay $0.20 per text for about $54 per month. We never go over our minutes and one rarely would we even break even if we upgraded to an unlimiting texting plan. My service provider would like me to think I pay $54/500 = 0.10 per minute during the Anytime minutes and that all my other calls are "Free". I don't think of my usage that way. I pay $54 dollars a month and a certain number of "communications" for that. Most of those are about 1 minute long, and to make things simple, I'll consider a "Text" to be a 1 minute call.
So what is this bizarre plot to the left here you ask. Some highly technical computer modeling of some fringe science? Perhaps one of those new-fangled computer thingies we know dvhart is always going on about. Maybe some nostalgic artifact from his days studying electrical engineering? Well... no. In fact it is simply the beam plots of an over the air antenna I am looking at installing in my attic. Yeah... just trying to watch TV. Turns out that there is a lot that goes into picking up a TV signal - an awful lot. I'm a week into my search for the right antenna (or antennas as the case may be - VHF-HI an UHF bands you know), and so far I've determined that despite living within 3 miles of the two transmission towers I care about, and even though those towers are within 10 degrees of each other, I appear to be dealing with skyline multipath and some other issues resulting from living in a valley and being surrounded by lots of very large trees. Oh, and the foil backed insulation in my house really doesn't like radio waves. It's been fun, but I don't know how anyone without 5 years of intense math and electrical engineering coursework behind them would dare attempt a less than trivial antenna installation. Sheesh. No wonder nearly everyone I know pays for cable! So in case you're interested in what it would take to setup OTA DTV (over the air digital television) at your house, check out the following links. Happy Hacking.
I'm a hopelessly slow reader, and it continually frustrates me, and later shames me in front of my friends. I just can't read the information fast enough to keep up with the world! I blame elementary school who had me read a grand total of I think 2 books by the time I was 12 (both involving dying dogs ... what the hell is wrong with you people?). I've been reading some things on how to improve my reading speed while maintaining comprehension, and also looking into how technology might be able to help. A colleague pointed me to SpreedNews which turns out to be awesome, and I top out at just under 500 words per minute before I simply don't remember anything about what I read. Now if the Amazon Kindel could incorporate this technology and allow me to read anything on the Kindel with this fancy rate adjustable phrase flashing awesomeness, I'd buy two... today. Unfortunately, one of the core technologies of the Kindel is their fancy pants epaper screen, which while it is great to look at and easy on the batteries, its refresh rate wouldn't allow for 20 words per minute, let alone 500. Ah well, maybe when Devon is my age he can get me a nice ereader that solves all my problems, without killing any dogs.
1. "Old Yeller" and "Where the Red Fern Grows" - both belong on the banned books shelf at my local library.