Are you an introverted person? If so, then you will definitely find a lot of truth in Dr. Carmella’s Guide to Understanding the Introverted, which was originally posted by starbug.. If you are extroverted, on the other hand, this guide might come in handy to improve your…
;)
This is a helpful little script I found on the net and modified to work around an issue with fast CPUs and slow IRC servers. If you find yourself needing to send messages from a shell script to an IRC server but keep coming against the “451 PRIVMSG: You have not registered” error, inserting a sleep between the USER and PRIVMSG commands does the trick.
#!/bin/bash -x
(
echo NICK backoffice
echo USER backoffice 8 \* : Notifier
sleep 1
# echo 'JOIN #netops'
echo 'PRIVMSG #netops' $1
echo QUIT
) | nc 127.0.0.1 6667
If you’ve been following my series of blog posts on my NAS rebuild, you probably came to this post expecting the final part of the series, where you’d find instructions on how to setup the applications and services that I rely on for my media management, such as: SABnzbd+, Sick-Beard, CouchPotato, AFP/Netatalk, Samba and some extras.
I’m afraid to say, however, that I have some bad news.
You see, when I started the NAS rebuild, my intentions were to replace Linux (Ubuntu 10.04, precisely) with FreeBSD which, in my previous encounters, showed itself as quite a stable and nice OS to build a server on (my NAS runs 24x7x365). And for many people out there, it works great.
However, after playing around with it for a while on my own NAS, I came to the conclusion that FreeBSD is not a good fit for my system. For an unknown reason, the network driver that is built into the FreeBSD kernel for my NAS’ network card is buggy, and locks up every few minutes when there’s a lot of traffic going through it. This caused the infamous ‘stuttering issue’ that I spent countless hours diagnosing in my spare time.
In the end, FreeBSD was removed and a new copy of Ubuntu 12.04 was used to setup my NAS from scratch. Or almost. As it turns out, the implementation of ZFS-on-Linux has reached stable status, and the kernel modules are available to install without much hassle. With that I was able to remount my ZFS array and browse through all my files from within Ubuntu.
This means that my original commitmment of posting a tutorial here about setting up SAB, SickBeard and others on FreeBSD, I will instead link to an awesome series of posts on another site about setting these apps up on Ubuntu 12.04.
So, without delaying any further, here are the posts on setting up SABnzbd+, SickBeard, CouchPotato and, to boot, Headphones on Ubuntu 12.04. Enjoy!
If you’ve been following my tweets (and if you haven’t, shame on you!), you’ll by now know that I undertook a challenge in June to stay alcohol-free for a whole month. I called it the #1MonthChallenge, and despite several (friendly) attempts from friends and family to derail me from my path, I succeeded in staying alcohol-free for a month. Woohoo!
A lot of people kept asking me during (and after) the challenge: “Do you feel different?”
Short answer: yes.
Long answer: I think so, yes.
Even longer answer: Yes, but not physically. If there were physical benefits to being off alcohol for a month, I did not feel them. Possibly because I already developed the habit of going to the gym 3 times a week, which keeps me energized and feeling “healthy”.
What I noticed, however, were mental changes. During the challenge, I became much more introspective (to be expected), but also much more observant. Suddenly eveything became fascinating. Watching the world turn became such a blast. I listened more than I talked, which was a bit different than what people were used to, but also quite challenging in itself. Which had the unfortunate side effect of making people think I wasn’t entertained enough. If only they could see what I saw…
Would I do it again? Maybe, if necessary. Will I? Probably not.
Don’t take me wrong: I do not need alcohol to function, thankfully. But the world is a lot more fun with it.
In moderation. Mostly.
Before we dig into part II of the NAS rebuild, I’d like to entertain you with a tale of how I accidentally destroyed my newly-rebuilt NAS, and how I proceeded to recover it completely.
After the FreeBSD rebuild, I decided to expand the NAS by switching to a new case. I sourced a brand new Array R2 by Fractal Design from a local supplier, and salvaged two spare 1TB drives I had lying around in an unused 2-bay NAS.
The transplant of motherboard and disks went without issues, and the new case was heaps quieter. Unfortunately, as I found out, it is not possible to add more physical disks to an existing RAID-Z array. I tried a few methods and, during one of the attempts, I accidentally added a standalone disk to the ZFS pool. This had, I believe, the adverse effect that all performance gains I had by pooling 4 disks together got cancelled by the disk that was part of the pool, but not of the RAID-Z array. This is best explained by looking at the output below:
nas% zpool status
pool: pool0
state: ONLINE
scan: scrub repaired 0 in 43h35m with 0 errors on Tue Jul 17 12:13:06 2012
config:
NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
errors: No known data errors
As you can see, ada2 is part of pool0 but sits outside the raidz1-0 array. Although some data may live in the fast RAID-Z array, there’s a chance that reads or writes will be performed on the much slower single disk as well, thus slowing the whole pool down. Not to mention the single-point-of-failure that having just one disk presents. As you’ll see ahead.
It all started last weekend, when I was playing a movie (Total Recall, the 1987 one) through the network on my HTPC. For some reason, XBMC was stuttering quite a lot with buffering. Both my NAS and my HTPC (running XBMC) are connected to the same Gigabit-capable Netgear WNDR3700 (running DD-WRT), so I assumed that network wasn’t the bottleneck. Heck, I can play the same movie over WiFi to my iMac on the other room! At least I could.
After getting fed up with the stutters, I decided to focus my attention on fixing my earlier blunder with the array setup. The first thing I tried was to offline the disk (offlining means to unplug the disk virtually, before doing it physically). That, unfortunately, didn’t work, because ada2 is the only disk in its separate pool, as explained above. You cannot offline disks that have no replicas (copies). A good feature of ZFS, but an annoyance in this particular case.
Next I attempted to physically remove the disk (with the computer off, obviously). And that’s when things got… interesting.
For reasons that go beyond my understanding of ZFS, removing that drive had the adverse effect of making the ZFS pool unbootable. So even though the USB drive was still working, the root drive wasn’t being mounted, as the ZFS pool wasn’t available. Re-attaching the drive did nothing to improve the situation. I started to worry. Did I just accidentally wipe all my data? Could I have been that dumb?
Thankfully, I learned from my mistakes and knew better than to store my critical data on the NAS. With solutions like SkyDrive, Dropbox and others, keeping data on local disks is asking for trouble. In the current scenario, if I had to rebuild my array, only media files would have been lost. That is: video and audio files. A lesson to be learned here, folks!
After a couple days (and many expletives) of trial and error, I decided to re-format the USB stick and start fresh, already accepting the losses. So I followed part I of this guide again from the top and worked my way down, up to the point where I needed to create the array. With all the original drives attached, I tried my luck and attempted to do a zpool import pool0 instead of a zpool create…. And whaddya know?!? It worked!
The array came back up exactly as shown at the beginning of this post. I let ZFS scrub the drives (almost 2 days later), then proceeded to plug in an external 2TB drive and extract all the data from the array, before rebuilding it again. This time, however, I used RAID-Z2 (minimum of 6 disks). And that’s how it’s been running since:
nas% zpool status
pool: pool0
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 ONLINE 0 0 0
errors: No known data errors
Although the data was safe and sound (+ points to ZFS), the issue with the stuttering still persists. I am almost certain that ZFS is not the culprit, and have turned my focus instead to the network. Either the router or the FreeBSD network drivers for the network card in the NAS are to blame.
I was about to test my router theory by using a spare Gigabit switch I had, but the switch’s power supply decided to release its magic smoke right when I plugged it in (talk about luck!). So I’m back to the drawing board, testing network card and kernel settings in FreeBSD to find the bottleneck.
Thanks for your patience while waiting for this guide to be finished. I thought I’d share a cautionary tale with a (somewhat) happy ending of how ZFS, in the end, kept my data safe from the worst enemy possible: myself. :)
Living tip: if you have a smartphone, put one of these in the same pocket where you carry your phone. Keeps it smudge-free. Bonus points if you wear glasses, in which case you’ll have a cleaning cloth always at hand. ;) (Taken with Instagram at Brickhouse Espresso Bar)
If you follow this blog (it’s ok if you don’t), you may be aware that a couple of years back I decided to build my own NAS. I won’t go into the details of what drove me to build it myself (you can read more about it here), but I will tell you what motivated me to rebuild it recently.
After my NAS was built, I had roughly 1.5TB in storage space available using a mixture of 500GB and 1TB disks, arranged in a RAID10 array. That was fast and reliable, but costly to upgrade. A few months later, I changed my mind and reshaped the array as a RAID5, which gave me roughly 2TB of disk space. Over time, I upgraded all drives to 1TB, which, in a RAID5 configuration, left me with little over 2.3TB available.
Needless to say that, with internet data caps growing constantly, I found myself again constrained by disk space. But with HDD prices still higher than they were pre-Thailand floods, replacing disks was not an option.
Enter ZFS.
UPDATED 16/07/2012: After pretty much destroying my USB stick on the weekend, I’m rebuilding my NAS yet again, and will be seeing if I can recover my ZFS pool from the new system. I have opted to install the whole system to USB (yes, including logs) for now, and will move busy folders to the ZFS array once it’s all back to normal.
I’ll be deleting the existing RAID-Z array and replacing it with either: a) a RAID-Z2 config with all 6 drives; or b) a RAID-Z configuration of 3 x 2-disk mirrors.
UPDATED 13/07/2012: Added note about RAID-Z drive-adding limitations.
My original NAS build ran Ubuntu Server 10.04 LTS, which I use on a daily basis at work and am quite happy with. Being a popular Linux distribution, it has a great support community as well as plenty of software packages available. It was a no-brainer.
What it didn’t have, though, was support for ZFS. Since a full explanation of ZFS is beyond the scope of this post, I’ll give you the short version: ZFS is a combined filesystem and logical volume manager created by Sun Microsystems (heard of them?) for the Solaris OS. It eventually got ported to FreeBSD (and even Ubuntu, although not as stable). Great, right? But what does this mean, exactly?
In essence, you add a bunch of physical HDDs to your system, tell ZFS you want to combine them into one big container to store your files, and it will take care of managing the disks, integrity and replication of the data for you. You see, ZFS was designed from the ground up with transparent data integrity in mind. So it will always ensure your data is safe at all times. And if it can’t keep it safe, it will let you know.
But, as I mentioned before, my NAS was running Ubuntu, which doesn’t have a good enough (or up-to-date) implementation of ZFS. Seeing as Uubntu 12.04 was out, I thought it was about time I rebuilt the NAS with just the services I really need, and shave the unnecessary fat off the system. So why not take the time to install FreeBSD instead, which is known for its stability and set-it-and-forget-it approach?
Just so that you know what I have after following the instructions below, here’s the current status of my NAS:
Before we dig into details, here’s what you need to know:
So with the aid of a spare FreeNAS box I had lying around in the office, I transferred all my media out of the NAS and started the process of installing FreeBSD. The first executive decision made was to unplug the old 60GB laptop HDD that was used as a boot drive, and instead boot from a USB stick. Not only would I save up on some power consumption, it would be one less drive to generate noise and heat in the enclosure. Take that, Greenpeace!
Although there are plenty of tutorials on the web for putting the FreeBSD root on ZFS, I have not found a comprehensive one for having root-on-ZFS with the boot code on an external device. Having the bootcode on an external device guarantees that your device will be bootable even in the unlikely event that your ZFS pool is destroyed.
Ready for the terminal hackery?
For this process, I picked two empty USB sticks. One for the install media (my NAS does not have an optical drive) and one to write the boot code to. I downloaded the FreeBSD 9.0 memstick image from ftp.freebsd.org for a 64-bit processor, and wrote it to the install media USB stick with:
dd if=~/Downloads/FreeBSD-9.0-RELEASE-amd64-memstick.img of=/dev/disk2 bs=64k
Please note that /dev/disk2 was the path to the install media USB stick on my system, and it will most likely be different on yours. To find out, open a terminal window and type:
diskutil list
Which in turn would give you a listing similar to this:
/dev/disk0
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *500.1 GB disk0
1: EFI 209.7 MB disk0s1
2: Apple_CoreStorage 398.3 GB disk0s2
3: Apple_Boot Recovery HD 650.0 MB disk0s3
4: Microsoft Basic Data Untitled 100.9 GB disk0s4
/dev/disk1
#: TYPE NAME SIZE IDENTIFIER
0: Apple_HFS Macintosh HD *398.0 GB disk1
/dev/disk2
#: TYPE NAME SIZE IDENTIFIER
0: *4.0 GB disk2
That last one was my 4GB USB stick which I used to boot the FreeBSD installation from.
Once the image is written to disk, eject it (diskutil eject /dev/disk2), plug it into the NAS and boot off it (your NAS may need a key press or settings changed to boot from USB sticks). Select Shell at the dialog prompt after the initial boot. You’ll be dropped to a root shell.
At the root shell, it’s time to create your ZFS pool. First, let’s set up 4K block alignment. For a complete technical breakdown of what 4K sectors are, check this primer from Seagate. In short: it’s good.
The ada* devices you see here are my 1TB disks (ada0 to ada3). Depending on how many disks you have, you may have to go with a different RAID arrangement. RAID-Z requires at least 3 drives.
IMPORTANT: You cannot add drives to an existing RAID-Z. You can only replace existing with bigger ones. So when planning your array, ensure you fill all the available slots in your system with drives, even if they are not the ones you plan on using later on. This got me after completing this guide, since I switched to a bigger case and got 2 spare HDDs that I cannot add to pool0 without rebuilding the pool.
To see which devices you have available, type:
ls /dev/ada*
Let’s prepare ZFS for 4K sectors:
gnop create -S 4096 ada0
Create a folder to store the ZFS config during the install:
mkdir /boot/zfs
And create the actual ZFS pool:
zpool create -O mountpoint=none pool0 raidz ada0.nop ada1 ada2 ada3
Note how the first device has .nop appended to its name. That’s how we tell the ZFS pool creation command to optimize for 4K sectors. In my example, pool0 is the name of the ZFS pool. We’ll create partitions on it, so they’ll be named pool0/root and pool0/media, among others.
Now let’s create the root partition where all system files will reside:
zfs create -p pool0/root
Tell ZFS not to mount the partition automatically, or it will screw things up for the installer shell:
zfs set canmount=noauto pool0/root
zfs set mountpoint=/ pool0/root
And mount this partition on a temporary path so we can copy the system files to it:
mount -t zfs pool0/root /mnt
Time to decompress the files from the install media onto the root of our new system:
cd /mnt
tar Jvxpf /usr/freebsd-dist/base.txz
tar Jvxpf /usr/freebsd-dist/kernel.txz
tar Jvxpf /usr/freebsd-dist/ports.txz
tar Jvxpf /usr/freebsd-dist/doc.txz
The last two lines are optional. However, you’ll need to get the ports from somewhere, and it’s always good to have documentation available for the commands you don’t know by heart. :)
Create a new file under /mnt/boot called loader.conf, using the dreaded vi editor:
vi /mnt/boot/loader.conf
With the following lines, chaing the last one to match your root partition created on your ZFS pool:
padlock_load="YES"
zfs_load="YES"
vm.kmem_size_max="512M"
vm.kmem_size="512M"
vfs.root.mountfrom="zfs:pool0/root"
Press ESC, then :wq to save and quit the editor.
Create a blank fstab so that the bootloader doesn’t complain about it:
touch /mnt/etc/fstab
Copy the ZFS pool configuration to the new root:
zpool set bootfs=pool0/root pool0
cp /boot/zfs/zpool.cache /mnt/boot/zfs/
Put the following lines into /mnt/etc/rc.conf:
hostname="nas.lan"
ifconfig_DEFAULT="DHCP"
keymap="us.iso.kbd"
keyrate="fast"
sshd_enable="YES"
sendmail_enable="YES"
zfs_enable="YES"
Changing nas.lan to your NAS’ hostname. If you want to set a static IP address instead, replace the ifconfig_DEFAULT line with:
ipv4_addrs_vge0="192.168.0.4/24"
defaultrouter="192.168.0.1"
Make the appropriate changes to match your local network configuration, and change vge0 to the adapter that shows up when you run ifconfig.
Time to plug in that second USB stick into the NAS, so the bootcode can be written to it. When plugging it in, the console will display some diagnostic messages. Pay attention to the device name (da1 or similar), as that’s what we’ll use in the next few commands.
Let’s partition the disk and write the bootcode to it:
gpart create -s GPT da1
gpart add -t freebsd-boot -b 64k -s 64k da1
gpart add -t freebsd-ufs da1
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 da1
Create and mount the filesystem:
newfs -L usbboot /dev/da1p2
mkdir /mnt/usbboot
mount /dev/ufs/usbboot /mnt/usbboot
And finally copy the boot config over:
cp -Rpv /mnt/boot /mnt/usbboot
With the newly-formatted USB stick ready to go, it’s time to turn the NAS off, remove the install media and let the NAS boot off the new USB stick. With luck, you’ll get a nice login prompt after all is done.
By default, the root user won’t have a password set. Let’s fix that:
passwd
Set up your local timezone (yours may differ from mine):
cp /usr/share/zoneinfo/Pacific/Auckland /etc/localtime
Create sendmail aliases database:
newaliases
Create a non-root user:
adduser
And voilà! A new FreeBSD install ready to go.
In my scenario, ZFS was used to combine all four 1TB HDDs into one single volume, on top of which I store all the files on the NAS. By arranging the disks into a RAIDZ array, I not only added redundancy to the NAS but also increased capacity by 400GB compared to my old RAID5 arrangement. How’s that for a surprise?
Since a NAS without any sharing or downloading software is useless, we need to add apps to make it this a truly useful box. But to avoid making this long article TOO LONG, it will be split into two parts.
Stay tuned for part 2, where we’ll setup SABnzbd+, Sick-Beard, CouchPotato, AFP/Netatalk, Samba and some extras.
By this stage, I have spent half of a month without ingesting any alcohol. It certainly hasn’t been easy, and that’s not because I miss it.
The real annoyance has been the constant hassle from friends, trying to get me to make a “wee exception” to my no-alcohol rule. It usually comes accompanied by the “you’ve been so quiet today” statements, to which my reply (“I’m just paying more attention to you guys”) fails to satisfy. But, as the saying goes: “you can’t please them all”.
It’s not hard, however, to see that I am indeed a bit quieter at parties. Nothing to do with inhibition, I swear. It is, in fact, a social experiment I’m running on myself, in which I’m trying to pay more attention to the body language of the people I meet. It’s fascinating to see how much there is to see if you just look at someone.
So please don’t worry if I seem quieter than usual. I’m just paying more attention to you. :)
Totally made up statistics say that roughly 98% of people agree that going alcohol-free before a long weekend is insane. I’m part of that 98%.
But this didn’t stop me from attempting (and, so far, succeeding at) it. Not only that, but it was also the weekend where we:
Although, when left alone, I did not feel the urge to pour myself a scotch, the peer pressure during the parties sure takes a lot of will power to fight back. Thankfully, not having any alcohol also helps with the strenghtening of said will.
As to the noodle challenge (or, as I like to call it, the noodle side-quest), it hasn’t been started yet. And this is solely due to the fact that, after all these parties and BBQs, the fridge is stocked full with leftovers. Not a bad predicament, right?
All these choices, and I’m on the Coke #1MonthChallenge (Taken with Instagram at SPE Bar & Restaurant at The Langham)
As I write this, there are only a few hours left till one of the biggest personal challenges I have undertaken: to go alcohol-free for a whole month.
Not only that, but I’ll be upping the ante with another quest: eating just instant noodles for dinner, to completely break my “always-dine-out” habit currently in full effect.
This may prove extra difficult, in fact, because I’m opting to be even more social. That is, I’ll be going out more with friends, attending more functions and just generally stepping out of my apartment a bit more.
If I can keep up with the challenge, I’ll be celebrating with whoever makes it to Gather’s after-party, 30th of June as the day rolls over to July.
So keep an eye here, on my Twitter and on Facebook for what is sure to be an interesting month!
One of those perfect moments: this was the last one in the packet (Taken with Instagram at ActionStep)