Hart Blogs

I've converted to a wordpress blog. Please update your bookmarks to http://www.dvhart.com and your RSS readers to http://www.dvhart.com/?feed=rss2.

I've reviewed Ubuntu's NetBook remix and then 10.10's Unity, and wasn't impressed with what appeared to be beta-ware at best. Now that 11.04 is out, I felt I should give it a spin before abandoning my preferred distribution of the last several years. I grabbed the amd64 11.04 live cd, used unetbootin to make a bootable USB Key, and booted my x201 to the Natty Narwhal.

First the good. The Unity interface has been polished up a bit, it's smoother and less clunky. Some small space savings features are effective: the maximized window title bar in the panel and the mouse-over file menu in the panel (ala Mac OS). The narrow scrollbar will a mouse over grab bar is actually particularly nice (or is this just GTK3?).

We are being harassed by collections agencies looking for individuals we do not know and do not live here. We recently changed our number for a similar problem with agencies looking for individuals with the same initials (m. hart) as our previous public number. We now have a private number, but that number has been associated with persons with unpaid debts in the past. I am extremely frustrated that the primary use for my phone service is for collections agencies to try and reach people I do not know. Phone service providers need to provide their customers with a means to prevent this. For example, a blocked number list that I can add numbers to from the web interface. The caller should just hear the phone ring and never be directed to voicemail or receive any kind of message. If something doesn't change soon, I am considering canceling my voip service since I strongly object to paying for a service which provides more rights to collections agencies than to me.

For now, I've filed complaints against GE Money with anyone who will listen and will be canceling every account I have with them. Good-bye GAP card. If you object to being called repeatedly at inconvenient times, having your child woken up early, and being lied to over the phone - consider doing the same.

I recently returned from the Intel Open Source Technology Summit (OSTS). As with any conference worth attending, I returned inspired, and humbled. The attendees were top caliber people from engineering and management, along with special invitees, including Linus Torvalds. While my senior management's keynote was inspiring (truly, very well done), Linus's address was insightful in that signature why-didn't-I-see-that kind of way, and my team's technical leadership proved itself to be exceptionally competent once again, there is one concept I came away with that will have a larger impact than any other, and the individual who planted the seed likely doesn't even remember the conversation. Hacking away late into the evening in the lodge's "library" on too little sleep over a crippled network, my colleague said to me, regarding a problem I was having difficulty making progress on, "I always assume that the code was written the way it is because that was the easiest way for the person who wrote it." - or something to that effect. That thought has been percolating in the back of my mind for several days, slowly flagging memories of my programming history, building insight, and forming a new resolve.

I've done much of my best programming when the result wasn't critical, without regard for how it might be criticized, without much thought for how others might do it. No concern for why others wrote what I'm debugging the way they did. This same individual also noted that there is an optimal level of beer consumption for programming. That level being where inhibitions are diminished, but cognitive ability is still mostly intact. While it is always good to carefully consider how a thing might be done, obsessing over the perfect solution can hinder progress, and fear of rejection and criticism can halt it.

A colleague and good friend from my days at IBM was fond of the saying, "The perfect is the enemy of the good". A working solution is better than a plan for a perfect solution. In open source, we have the advantage of being able to expose our work to the critical eyes of a vast community of brilliant individuals with a wide range of experiences. This group is well known for harsh criticism. Criticism of a working solution eventually leads to something approaching the perfect solution.

I intend to take this insight to heart, to have more confidence in myself, despite my naturally self-critical nature. To obsess less over the perfect solution, despite my predisposition to obsessive behavior, and progress toward a working solution using my individual experience as my guide. Finally, to best my aversion to rejection, and proudly present my working solutions and leverage criticism to approach a perfect solution.

Work and family life was busy, so it was a few days before I could put the QNAP TS-419P+ to the test with some representative use cases. But before I did, I spent some time educating myself with RAID levels and came to the conclusion that until I am in desperate need of more storage space, RAID 5 just doesn't make sense. Here's why. RAID 5 distributes parity across all the drives in the array, this parity calculation is both compute intensive and IO intensive. Every write requires the parity calculation, and data must be written to every drive. With the low power CPU already the bottleneck for throughput, adding an additional load didn't seem like a good idea. More importantly is data integrity. RAID 5 allows for a single drive to fail without any loss of data. However, while rebuilding the array, if one of the remaining three drives were to fail, all the data is lost. Rebuilding a RAID 5 array after a single disk failure is also very compute and IO intensive as every disk must be read in order to restore the blocks to the new drive.

A better option for a four drive array is RAID 10, a striped pair of mirrored disks. In this configuration writes only affect the two drives in the mirror, and the data integrity is much improved. After a single drive fails, it is restored by copying it's sibling in the mirrored pair. If one of the three remaining drives were to fail, there is only a 33% chance that data will be unrecoverable as either drive from the other mirrored pair could fail without a problem.

The cost for this is total volume size. RAID 5 provides SIZE*(N-1) while RAID 10 provides SIZE*(N/2). With 4 1.5TB drives, RAID 5 yields a 4.5TB volume, while RAID 10 yields only 3TB. When drives were expensive, this 50% gain was significant, but when 2TB drives can be had for under $100, and larger drives becoming available every year (3TB drives now ship with consumer level NAS products), the principle value of RAID 5 is not as convincing as it once was. With RAID 10 and current technology (3TB drives) I can still double my capacity, and that should only improve in the coming years.

OK, on to configuration. I took to configuring the NAS for use with my home Linux network. I should preface this by saying I am not a network file system expert, not even an experienced user. I have setup NFS enough times to know the homogeneous uid/gid thing is a pain and that there are plenty of failing corner cases with respect to dropped connections, file locking, etc. QNAP claims to support NFS, so I expected them to provide the necessary tooling in their oft-praised web interface. The sad truth is that NFS appears to be an afterthought, and the implementation barely merits an "[x] NFS" string on their marketing material. The UI allows you to add users, but not to specify (or modify) the uid or gid. This means that standard Unix file permissions simply do not work, and their solution appears to be to make the shares globally read-only or globally read-write. Two words: cop out. Fortunately QNAP does provide a root-only ssh shell, and I was able to log in and manually edit /etc/passwd and /etc/group to make my users match the rest of the network. Some careful recursive 'chmod g+s ug+rw o-rwx' commands provided me with the permissions I wanted - but avoiding that sort of work is precisely why I opted for QNAP instead of building my own. In this regard, they failed miserably.

The SMB story is better. The QNAP UI supports per volume user and group permissions. While the on disk representation is still globally read-write, it's not a problem as SMB performs it's own user authentication and only the admin user has ssh access anyway. I tinkered with this enough to get it working with the GNOME desktop file manager and with autofs. This might be the best way to access shares on the QNAP, even from a Linux system. Still, something about running SMB makes me feel like I need to shower.

For throughput tests I used the rsync daemon to copy my MythTV recordings to the QNAP. This consisted of 300GB of mostly mpeg2 files from 2 to 6 GB each. I used the UI as well as top to monitor the system status periodically during the transfer. The CPU was pegged at 100% for the duration of the transfer, and it averaged just over 20MB/s.QNAP claims 45MB/s writes over SMB and 42MB/s over FTP. Rsync should be faster if anything. Throughput was a disappointment. Following the transfer the QNAP remained under heavy load (7-8), and became fairly unresponsive. Watching the kernel logs I found a few Out of Memory errors, with apache and php being among the OOM Killer's victims. I raised the throughput and OOM issues with QNAP and my supplier. They weren't able to suggest any changes to improve throughput or identify why the OOM occurred. They did agree to allow me to return the TS-419P+ in exchange for a TS-459Pro+. The latter replaces the ARM CPU with a dual core Atom, doubles the RAM, and replaces the 16MB flash with a 512MB DOM. 20MB/s just wasn't cutting it, and a kernel OOM was just unacceptable. I shipped the QNAP TS-419P+ back and am impatiently awaiting a TS-459Pro+. Whether I keep the QNAP firmware or replace it with Ubuntu Server or perhaps a custom Yocto image is the subject for a future project.

Work and family life was busy, so it was a few days before I could put the QNAP TS-419P+ to the test with some representative use cases. But before I did, I spent some time educating myself with RAID levels and came to the conclusion that until I am in desperate need of more storage space, RAID 5 just doesn't make sense. Here's why. RAID 5 distributes parity across all the drives in the array, this parity calculation is both compute intensive and IO intensive. Every write requires the parity calculation, and data must be written to every drive. With the low power CPU already the bottleneck for throughput, adding an additional load didn't seem like a good idea. More importantly is data integrity. RAID 5 allows for a single drive to fail without any loss of data. However, while rebuilding the array, if one of the remaining three drives were to fail, all the data is lost. Rebuilding a RAID 5 array after a single disk failure is also very compute and IO intensive as every disk must be read in order to restore the blocks to the new drive.

A better option for a four drive array is RAID 10, a striped pair of mirrored disks. In this configuration writes only affect the two drives in the mirror, and the data integrity is much improved. After a single drive fails, it is restored by copying it's sibling in the mirrored pair. If one of the three remaining drives were to fail, there is only a 33% chance that data will be unrecoverable as either drive from the other mirrored pair could fail without a problem.

The cost for this is total volume size. RAID 5 provides SIZE*(N-1) while RAID 10 provides SIZE*(N/2). With 4 1.5TB drives, RAID 5 yields a 4.5TB volume, while RAID 10 yields only 3TB. When drives were expensive, this 50% gain was significant, but when 2TB drives can be had for under $100, and larger drives becoming available every year (3TB drives now ship with consumer level NAS products), the principle value of RAID 5 is not as convincing as it once was. With RAID 10 and current technology (3TB drives) I can still double my capacity, and that should only improve in the coming years.

OK, on to configuration. I took to configuring the NAS for use with my home Linux network. I should preface this by saying I am not a network file system expert, not even an experienced user. I have setup NFS enough times to know the homogeneous uid/gid thing is a pain and that there are plenty of failing corner cases with respect to dropped connections, file locking, etc. QNAP claims to support NFS, so I expected them to provide the necessary tooling in their oft-praised web interface. The sad truth is that NFS appears to be an afterthought, and the implementation barely merits an "[x] NFS" string on their marketing material. The UI allows you to add users, but not to specify (or modify) the uid or gid. This means that standard Unix file permissions simply do not work, and their solution appears to be to make the shares globally read-only or globally read-write. Two words: cop out. Fortunately QNAP does provide a root-only ssh shell, and I was able to log in and manually edit /etc/passwd and /etc/group to make my users match the rest of the network. Some careful recursive 'chmod g+s ug+rw o-rwx' commands provided me with the permissions I wanted - but avoiding that sort of work is precisely why I opted for QNAP instead of building my own. In this regard, they failed miserably.

The SMB story is better. The QNAP UI supports per volume user and group permissions. While the on disk representation is still globally read-write, it's not a problem as SMB performs it's own user authentication and only the admin user has ssh access anyway. I tinkered with this enough to get it working with the GNOME desktop file manager and with autofs. This might be the best way to access shares on the QNAP, even from a Linux system. Still, something about running SMB makes me feel like I need to shower.

For throughput tests I used the rsync daemon to copy my MythTV recordings to the QNAP. This consisted of 300GB of mostly mpeg2 files from 2 to 6 GB each. I used the UI as well as top to monitor the system status periodically during the transfer. The CPU was pegged at 100% for the duration of the transfer, and it averaged just over 20MB/s.QNAP claims 45MB/s writes over SMB and 42MB/s over FTP. Rsync should be faster if anything. Throughput was a disappointment. Following the transfer the QNAP remained under heavy load (7-8), and became fairly unresponsive. Watching the kernel logs I found a few Out of Memory errors, with apache and php being among the OOM Killer's victims. I raised the throughput and OOM issues with QNAP and my supplier. They weren't able to suggest any changes to improve throughput or identify why the OOM occurred. They did agree to allow me to return the TS-419P+ in exchange for a TS-459Pro+. The latter replaces the ARM CPU with a dual core Atom, doubles the RAM, and replaces the 16MB flash with a 512MB DOM. 20MB/s just wasn't cutting it, and a kernel OOM was just unacceptable. I shipped the QNAP TS-419P+ back and am impatiently awaiting a TS-459Pro+. Whether I keep the QNAP firmware or replace it with Ubuntu Server or perhaps a custom Yocto image is the subject for a future project.

As if the arrival of the final components for Rage wasn't enough tech debauchery, the present trucks also delivered a shiny new QNAP TS-419P+ and 4 Samsung Spinpoint F2 1.5TB drives. Devon helped me unpack everything and carefully mount the drives in their trays. He even helped me plug it in and start the initial setup process.

The QNAP packaging and physical documentation is simple, nay, spartan. Which I like. The device itself is smaller than I expected (always nice) but I was disappointed to find a separate power brick instead of a built-in power supply - this educed my excitement about the compact size of the unit a somewhat. The recommended "Linux Setup" was to connect a PC directly to the NAS and configure your networks to talk to eachother - this didn't appeal to me, so I just looked up the QNAP IP on my dd-wrt router and followed the directions for Windows and Mac - just without installing a qnap finder application.

The QNAP web interface is highly polished. Initial setup included setting the hostname (I selected Toph in keeping with my heroine theme for my personal machines), installing the latest firmware, setting an initial password, which network services to enable, and an initial RAID configuration. Perhaps this is obvious to everyone else, but be sure to unzip the firmware you download from the QNAP website, otherwise you'll just get an unhelpful error complaining the image is bad. I found the initial RAID selection to be odd as it is very limited. I chose RAID 5 as that is probably what I want to do, but the device offers a lot more options than a single RAID array using all the disks. Given the amount of time it takes to resync a 4.5 TB RAID 5 array - it seems like this step could be skipped and the user sent directly to the full-featured volume management admin screen at first login. Instead, after completing the initial setup, you are presented with this iTunes-wanna-be AJAX interface:

From qnap

Here you can see the volume management screen - and an ascending time remaining field in the Status column. I really don't know how I'll partition things up, or if I even need to. The QNAP offers a _ton_ of flexibility in how you access your data. I'll need to spend a good deal of time considering them before I make a final decision. I'll reserve judgement on these features until then.

From qnap

Out of the box, several network services are available for immediate configuration:

From qnap

And finally, QNAP offers add-on packages in the form of QPKG, which oddly enough includes an IPKG application for even greater selection of packages. There are several media streaming servers available, including one that is pre-installed. The installation process appears a bit cumbersome, requiring the user to download the package to a PC and then upload it to the NAS for installation. I am looking forward to installing Python, possibly Twonky, and maybe MySQL and WordPress (I'm considering moving this blog away from Drupal and to something else).

From qnap

So for now, my QNAP is resyncing its RAID 5 array. I hope to have the time to explore its many features soon, and I'll share my experience as I do. My initial impressions are good, and I'm optimistic that this will turn out to have been a good choice for our needs.

As if the arrival of the final components for Rage wasn't enough tech debauchery, the present trucks also delivered a shiny new QNAP TS-419P+ and 4 Samsung Spinpoint F2 1.5TB drives. Devon helped me unpack everything and carefully mount the drives in their trays. He even helped me plug it in and start the initial setup process.

The QNAP packaging and physical documentation is simple, nay, spartan. Which I like. The device itself is smaller than I expected (always nice) but I was disappointed to find a separate power brick instead of a built-in power supply - this educed my excitement about the compact size of the unit a somewhat. The recommended "Linux Setup" was to connect a PC directly to the NAS and configure your networks to talk to eachother - this didn't appeal to me, so I just looked up the QNAP IP on my dd-wrt router and followed the directions for Windows and Mac - just without installing a qnap finder application.

The QNAP web interface is highly polished. Initial setup included setting the hostname (I selected Toph in keeping with my heroine theme for my personal machines), installing the latest firmware, setting an initial password, which network services to enable, and an initial RAID configuration. Perhaps this is obvious to everyone else, but be sure to unzip the firmware you download from the QNAP website, otherwise you'll just get an unhelpful error complaining the image is bad. I found the initial RAID selection to be odd as it is very limited. I chose RAID 5 as that is probably what I want to do, but the device offers a lot more options than a single RAID array using all the disks. Given the amount of time it takes to resync a 4.5 TB RAID 5 array - it seems like this step could be skipped and the user sent directly to the full-featured volume management admin screen at first login. Instead, after completing the initial setup, you are presented with this iTunes-wanna-be AJAX interface:

From qnap

Here you can see the volume management screen - and an ascending time remaining field in the Status column. I really don't know how I'll partition things up, or if I even need to. The QNAP offers a _ton_ of flexibility in how you access your data. I'll need to spend a good deal of time considering them before I make a final decision. I'll reserve judgement on these features until then.

From qnap

Out of the box, several network services are available for immediate configuration:

From qnap

And finally, QNAP offers add-on packages in the form of QPKG, which oddly enough includes an IPKG application for even greater selection of packages. There are several media streaming servers available, including one that is pre-installed. The installation process appears a bit cumbersome, requiring the user to download the package to a PC and then upload it to the NAS for installation. I am looking forward to installing Python, possibly Twonky, and maybe MySQL and WordPress (I'm considering moving this blog away from Drupal and to something else).

From qnap

So for now, my QNAP is resyncing its RAID 5 array. I hope to have the time to explore its many features soon, and I'll share my experience as I do. My initial impressions are good, and I'm optimistic that this will turn out to have been a good choice for our needs.

From Rage

The present truck(s) were good to me today. I received the two Intel Xeon x5680 CPUs, the two Seagate Barracuda 1TB drives, the Intel 160GB G2 SSD, and the second heat sink. The SuperMicro hot-swap trays don't allow for mounting 2.5" drives, so I had to mount the SSD in a 3.5" bracket in a 5.25" bracket. Lame. As I mentioned in my last post, the first CPU cooler's fan conflicts with the rear chassis fan. Since I had to choose between the two, I chose to keep the larger (quieter) chassis fan, but I connected it to CPU 1 FAN instead of the FAN 5 header. This is a guess on my part, but I figure the CPU is first thing to get hot, and the most valuable component in the system, so it makes sense to me to let its temperature determine the fan speed. This may cause problems however as the fan speed used by the CPU 1 FAN is probably not appropriate for the larger fan, and I don't know how removing the FAN 5 connection will impact how the system decides to use the forward fan (which is smaller, and louder). Any insight readers may have here is very welcome.

Initial power-on is always exciting, this was no different, perhaps more so. After pressing the power button, Rage jumped to life like a wild beast startled from slumber. Her fans roared and her many bright beady eyes flickered their discontent. After familiarizing myself with her BIOS settings, I ran a quick Ubuntu 10 install off USB (it was absurdly fast). The BIOS RAID options were confusing at best, and I felt I just might get better results with software RAID via mdadm (at least more control). Rage is currently resyncing a RAID 1 array composed of the two 1TB SATA drives. I'm not sure quite how long this will take, and with her periodic snoring (loud fan bursts), I may just have to force her back into hibernation so my better half can sleep tonight.

Just as soon as I can I'll kick off a complete Yocto build and share the results. Following that, I'll run some burn in tests to ensure the memory, CPUs, and HDDs are all functioning properly. I haven't tested IPMI 2.0 support yet (remote access, KVM, etc.) I'll get to that soon as well.

From Rage

The present truck(s) were good to me today. I received the two Intel Xeon x5680 CPUs, the two Seagate Barracuda 1TB drives, the Intel 160GB G2 SSD, and the second heat sink. The SuperMicro hot-swap trays don't allow for mounting 2.5" drives, so I had to mount the SSD in a 3.5" bracket in a 5.25" bracket. Lame. As I mentioned in my last post, the first CPU cooler's fan conflicts with the rear chassis fan. Since I had to choose between the two, I chose to keep the larger (quieter) chassis fan, but I connected it to CPU 1 FAN instead of the FAN 5 header. This is a guess on my part, but I figure the CPU is first thing to get hot, and the most valuable component in the system, so it makes sense to me to let its temperature determine the fan speed. This may cause problems however as the fan speed used by the CPU 1 FAN is probably not appropriate for the larger fan, and I don't know how removing the FAN 5 connection will impact how the system decides to use the forward fan (which is smaller, and louder). Any insight readers may have here is very welcome.

Initial power-on is always exciting, this was no different, perhaps more so. After pressing the power button, Rage jumped to life like a wild beast startled from slumber. Her fans roared and her many bright beady eyes flickered their discontent. After familiarizing myself with her BIOS settings, I ran a quick Ubuntu 10 install off USB (it was absurdly fast). The BIOS RAID options were confusing at best, and I felt I just might get better results with software RAID via mdadm (at least more control). Rage is currently resyncing a RAID 1 array composed of the two 1TB SATA drives. I'm not sure quite how long this will take, and with her periodic snoring (loud fan bursts), I may just have to force her back into hibernation so my better half can sleep tonight.

Just as soon as I can I'll kick off a complete Yocto build and share the results. Following that, I'll run some burn in tests to ensure the memory, CPUs, and HDDs are all functioning properly. I haven't tested IPMI 2.0 support yet (remote access, KVM, etc.) I'll get to that soon as well.

Okay nothing to exciting I'm just trying out a new blog site (that requires less maintenance on Darren's part). So check out Hart on my Sleeve and let me know what you think.

For those who are wondering what I look like here is a picture. Feeling the baby move and go in on the 16th for our ultrasound. Devon is still hoping for a sister so we'll see.

Thanksgiving is here again. It was quiet one for us here, but we enjoyed our time together as a family. We even skipped out the big food preparation and instead went to Marie Callendar's for their Holiday feast. we ate to much, and Devon had way to much Chocolate Satin pie but it was nice to come home to a clean house and just relax. We hope everyone else had a great day and time to relax and enjoy friends & family, and all of their blessings.

Well I know I am in trouble with my sister for not posting more this last month. It has been a crazy month and while I have had good intentions to post I just didn't ever feel like spend my limited free time posting. So what has happened in the last month, well lets see...

Trip to Mesquite: At the end of October we made the journey to Mesquite (2 hour plane ride & then 1 hour car ride after getting baggage & car rental - dvh3 was an angel!) My oldest nephew Jordan was speaking in church as he prepared to leave on his mission to The Baltic's (Estonia). My 10 year class reunion also happened to be the same weekend so we had lots to keep us busy. Really glad to have gotten to see a few of my old classmates but I must say I enjoyed the family time most of all. dvh3 got to play with his cousins and was following his new best friend Dayton everywhere.

Well for those that don't already know, Devon is going to be a big brother in May. Baby #2 is officially on the way. Mom & Dad are very excited and Devon is also excited to be a big brother. So far he is requesting a little sister so hopefully he won't be disappointed if it ends up being a little brother.

At 12 weeks I am stilling dealing with morning/all day sickness. But it does seem to be getting slightly better. I also find that I am exhausted nearly constantly. Big thanks to Darren for picking up so much of the slack. If anyone is interested you can see the 1st ultrasounds here: # 2.

The Portland Children's Museum is currently running a "Bob the Builder" exhibit. Devon was very excited to go and luckily got there in time to meet Bob (although once he got up close he wasn't quite as excited.) He had a great time playing with all the different exhibits, especially the water tank - pump blue "water" balls through the tank. He also liked getting to try out all the machines and guess what saying they'd repeat. There was also a "Machine petting" out side where the kids could climb on really construction equipment. Definitely something we need to go back for before it is gone.

Okay the weather is starting to turn so I thought I better finally get the blog about of summer. We had a very busy summer this year & relatively short (because the rain didn't stop until late June) but we enjoyed it. Devon took a Sports class through the local Rec. center and loved getting to go play with balls. Soccer balls, t-ball, balloons, basketballs, and one of his favorites bowling. He also took a swimming class which he enjoyed especially the finally day when he got to go down the slide. We also spent a lot of time visiting new parks this summer. You can see many pics from those excursions here: Parks.

We also enjoyed a few days at the coast with friends. We visited Pacific City, OR with our friends the Mauery's and we had a great time. We have known Vernon and Lauren for basic 10 years and 3 kids later we are enjoying getting back to the point where we can have adult conversation. Thanks to Nicole their 5 year old who puts of with Devon (I believe she is his favorite friends) and helped keep the parents informed when the boys started getting tired of each other (Nathan is only 6 months older then Devon but some days that is a BIG age difference!) Even though the beach was a bit windy we had a great time. More pictures posted here: Pacific City, OR. Oh and if your interested in a funny picture of me being attacked by a blanket check out Lauren's blog from our trip - she has some great pictures of the kids too!

Free-IQTest.net - Online IQ Test

So our Friend's the Mauery's posted their test scores so I had to try it. Not to bad considering I guessed on a few, I made it into the "Gifted" category. Lots of advertisements trying to get you to sign up for stuff at the end before they give you your score though. You can wade past it all but it takes a while.

The front yard landscape project continues on helped along by great friends who willing gave up an entire SUNNY Portland Saturday to work in our front yard last weekend. We were finally ready for plants so I took a trip to a recommended nursery, Farmington Gardens, and had a wonderful time picking out plants. We ended up with over 30 plants that need to planted and as quickly as possible. When the Mauery's offered their help we gladly to them up on the offer. We got to work at 10 am and other then a lunch break pretty much worked through until 5 that evening, but we got all the plants in. The kids were great and played together or watched tv most of the day so we could work. It was amazing how much we got done. Even more amazing was the transformation to the house, plants just add so much.