bamed.org

Tomorrow is the day!  Everything seems to be working, so tomorrow I shut down all the server, move some hardware around and bring everything up inside VM’s using VMWare.  The final plan includes setting up the main server with 4 250GB SATA II hard drives setup with RAID5 and a hot spare, which gives us 500GB of storage space.  I’ve split up this storage into several VM’s.  There will be one VM acting as our PDC using Samba and LDAP, and another as a BDC.  There will be a VM running as a LAMP server (Linux Apache MySQL PHP) that will host a few open source programs we use.  Then there will be another VM running Nagios to monitor all the server, switches and printers.  Then there will be a second server with a removeable SATAII hard drive bay, and I will use two 500GB SATAII hard drives.  I will copy the RAID array from Server1 to the removeable hard drive in Server2 using dd and iSCSI over a direct Gigabit connection between the two servers.  The removable hard drive not in Server2 will be stored offsite and they will be switched daily.  With the RAID5 we could lose up to 2 hard drives in Server1 and still keep working without losing any data.  If Server1 has some other hardware failure, I can start the VM’s on Server2 from the removable hard drive and run off that until the hardware failure is taken care of.  If there is some sort of catastrophe that results in both servers becoming non-functional, I can stick the SATAII hard drive that was stored off-site in any computer I can come up with that has a SATA port and run off that.

The end result will be better disaster recovery.  I hope to automate as much of this as I can, setting up a heartbeat monitor between the two servers and making a script to run if one server fails.  In conjunction with Nagios I should stay informed of any problems that come up, and hopefully correct them before the users even notice.

Well, that’s the plan, I’ll let you know how it ends up.

I managed to get the RAID array working this week.  So we now have RAID5 with a hot spare working.  That with removable hard drives will make up our DR plan.  I’ve been getting all the VM’s setup this week and everything is working well.  Last night I tested the new PDC VM and it seemed to work fine.  I tried copying data from our old LAMP server to the LAMP VM and it all worked.  Today I’ll work on getting Nagios setup and monitoring everything.

This weekend we worked on some of the server upgrades I’ve been planning for so long. Unfortunately I’ve been having trouble with the hew RAID card I got for my SAN, so I wasn’t completely ready to do everything I had hoped. I ended up working more on the RAID card, and a volunteer helped me move two of our servers into new cases. Good News Productions donated a server rack to us last year, so this year I got some rack-mountable cases to put our servers in.

Here’s what inside the server rack looked like before:

And this is after:

I know, not a HUGE improvement just yet. The two rack-mounted cases don’t show up too well in this pic either. The two remaining towers in the rack should be gone soon. The one of the right will be converted to a VM, and the one on the left gone forever. The old 486 on the left is our voicemail server. If everything works out well, it should be replaced by a VOIP server very soon. Then I need to work on cable-management, and we’ll be set.

In case you were wondering, yes, our “server room” is my office.

I got most of the hardware I ordered and I started assembling everything.  I re-assembled my SAN in a new case and added all the new hard drives yesterday.  Then during the day this morning I started noticing a kind of burning smell.  I couldn’t track down the source of the smell, so I walked around the office and found a couple of volunteers to come to my office and track down the source of the smell.  Before I left my office to find them I was recompiling my kernel on the SAN to enable the drivers for my new SATA II PCI-X card.  When I got back to my office with my volunteer bloodhounds, the SAN was off and the power supply was burning hot.

No worries!  I already had a new power supply ordered because I wasn’t certain the old one would handle all the new hard drives.  It arrives tomorrow.  Let’s just hope it shut down before anything else burnt up.

Our fiscal year ends this week and a new one begins next week. So I’m looking over my budget and figuring out what I’m going to get when. I came up with some estimates a few months ago when we had to submit our budgets, but now that’s all done and it’s about time to start spending.

If you’ve been following my blog, which I would assume you have been if you’re reading this, then you should know that I have some plans to upgrade our servers a bit to help with our DR Plan. The other major item on my agenda is our phone system. We’ve been limping along on an old Samsung ProStar DCS for many years now. I couldn’t really tell you how long because it was here long before I was. We’ve taken a few lightening hits over the years, and we have several bad ports on the system. There’s lots of weird little quirks that are a result of our outgrowing the system as it is now, and the lightening I’m sure. So for a few years now I’ve been looking into upgrading. I’ve had several quotes that I’ve submitted over the past three years, but the money has never been there. The major budget issue is the number of phones. Any proprietary system I’ve had quoted requires special proprietary phones that usually cost $200-$300 and we need about 80 of them.

So now I’ve come up with a different solution. Asterisk is an open-source telephony solution that will run on any PC and is open-standards so I have a lot of different hardware vendors to choose from. What I have recently discovered is that I can use an Asterisk server in conjunction with our current proprietary system. Basically I would setup an Asterisk server and connect it to our PSTN lines, then connect then Asterisk Server to our proprietary system. Then we can keep all our proprietary phones in place for now, and add newer feature-rich VoIP phones as the need arises. We could also implement SoftPhones, etc. The configuration will be a bit tricky to get the VoIP phones on the Asterisk server to work seamlessly with the proprietary phones on the Samsung system, but it is doable. So I’m going to do it. The only up-front cost will be the cost of the server I install Asterisk on. It will need some special hardware that will drive the cost up, but it won’t be that bad. I will need 16FXO ports to connect out 16 analog phone lines coming into the building, then I can connect the Asterisk server to the Samsung system via a T1 interface. I’ve found some tutorials for setting up such a configuration with different proprietary PBX’s and I found a forum post by a guy who claims to have done this exact same configuration with a Samsung Prostar DCS, so I know it can be done.

Anyway, when I actually get the hardware and get it all setup I’ll be sure to include a detailed tutorial on my blog. Until then, join our game of Diplomacy at http://www.geeks-4-jesus.org/diplomacy. We only need ONE MORE PERSON to start a game!!! So PLEASE sign-up TODAY!!!

I’ve been working on our Disaster Recovery Plan (Backups) lately, and I’m trying to do what I can so that we don’t just have disaster recovery, but we also do what we can to guarantee business continuity (things keep working).  Of course the big problem is I don’t have any money to do this with (at least not much).  If you’ve been following my blog you know that we use virtual machines, and that I now have iSCSI capability.  So after a lot of thought, this is the Disaster Recovery and Business Continuity Plan I’ve come up with, or at least an overview of it.

First I do need to get a little more hardware ( < $1000 worth), but when it's all said and done I should have:

Server1:
2x Intel Xeon 2.4GHz
4GB DDR RAM
1 80GB HDD - for host OS and VM's to run on
2x 250GB HDD's, striped - to host all data
2 Gb NIC's

Server2:
AMD Athlon64 AM2 3800+
4GB DDR RAM
1 80GB HDD - for host OS and to backup VM's
2x 250GB HDD's, striped - to backup data
1 Removable HDD Bay with 2 tray's
2 500GB SATA drives for removable HDD trays - for offsite backup
2 Gb NIC's
Both servers will be running Ubuntu and VMWare.  Server2 will also be setup as an iSCSI-target while Server1 will have an iSCSI initiator.  I will then access the HDD's in Server2 through iSCSI from Server1 and setup software RAID1 between the drives of the same size.  So the 500GB striped RAID0 set from Server1 will be mirrored to the striped RAID1 set on Server2.  The 80GB HDD's on both servers will be partitioned so that the host OS is on it's own partition and the guest OS's on another.  The partition with the guest OS's will also be mirrored between the two servers using a combination of iSCSI and RAID1.  Then Server1 and Server2 will be directly connected to each other through the extra Gigabit NIC's on their own subnet and all iSCSI traffic will travel over this direct connection.  Then Server1 and Server2 will be connected to the network through separate switches.  Finally I'll setup a heartbeat monitor between the two servers and create a script so that if Server1 is unreachable for a set amount of time Server2 will automagically start up the VM's that were mirrored from Server1 and all the data also mirrored from Server1 will still be available.  I am a little concerned about the performance of RAID1 mirroring over iSCSI, so I'll have to do some testing to see how that will work.  As far as I can tell, this setup, if all done properly, should help us guarantee business continuity in the event of hardware failure.

The second part to this plan is disaster recovery.  That's where the removable SATA HDD's come into play.  I'm going to run a standard backup to these HDD's nightly.  I'll probably just tar and gzip all files from all the drives to the 500GB removable drive, then encrypt using openssl.  I've done this before way back here.  That script mounted a NAS drive using smbfs so there was a 2GB file size limit.  No such limit here.  I believe the script in my blog was for a full backup, but I have another one I made that handles differential backups as well.  Right now I’ll probably only be able to afford 2 500GB HDD’s for the offsite backup, but I plan to eventually get one for every day of the week.  I will also be getting a SATA controller card that supports hot-swapping so I can swap the removable drives while the computer is still on.

Well, that’s the basic overview of my DR & BC plan.  I still have to get the hardware and setup all the software.  I’ll probably write a tutorial when I do in case anyone is interested in doing something similar.  I still need to research the heartbeat monitor.  I also need to figure out where a safe and convenient offsite location might be.  I live within 1 mile of the church and (worst-case scenario) a big tornado could take out the church and my house.  I also need to figure out who our backup personnel will be.  Since I’m the only IT person on staff, (again worst-case scenario) a big tornado that took out the church and my house could take out me as well, so somebody else needs to know where the off-site backups will be and what to do with them.

There’s been so much talk about SAN’s lately, and I knew my budget would never allow me to purchase one, so this week I built my very own iSCSI SAN.

I put together a system with an AMD Athlon64 X2 AM2 3800+ with 2GB DDR2 800 RAM, a 20GB IDE HDD for the OS and 4 250GB HDD I had lying around. Three of the 250GB HDD’s are IDE and the fourth is SATA. When my budget allows I intent to replace the IDE’s for SATA’s. I installed Ubuntu Server 7.04 on the 20GB HDD. Then installed iscsi-target, configured my LUN’s and all was done. For the moment I’m just using one of the 250GB HDD’s as a backup for our server, but now that I know how easy it was to setup, I’m going to think of some more creative ways to use my homemade SAN.

For those of you interested, here’s how I setup the software.

First download and install Ubuntu. It was pretty straightforward so I won’t get into that part of it. Once Ubuntu is installed get all the updates, including any kernel updates:

# sudo apt-get update
# sudo apt-get dist-upgrade
# sudo apt-get upgrade
Once all the updates are installed reboot. Now you need to install a few more packages:

# sudo apt-get install make
# sudo apt-get install libssl-dev
# sudo apt-get install linux-headers-`uname -r`
# sudo apt-get install gcc
Now make a symlink to your kernel source:

# sudo ln -s /usr/src/linux-headers-`uname -r` /usr/src/linux

This way make can find your kernel source files when you compile scsi-target without any other configuration from you.

Now download iscsi-target from http://sourceforge.net/project/showfiles.php?group_id=108475. Unzip and untar the file, the cd into the directory you extracted it to. Now:

# make
# sudo make install

If you don’t get any errors iscsi-target is installed. I did find one compatibility problem with the startup script with Ubuntu. To fix it edit /etc/init.d/iscsi-target by changing the first line from

#!/bin/sh

to

#!/bin/bash

No copy the ietd.conf file from the etc/ directory under the directory where you compiled iscsi-target to the systems /etc directory. You should read the man files and all associated documentation, but the only things you need to worry about to get up and running are the ‘Target’, ‘Lun 0 Path=’, and ‘Alias’ options. Configure these according to your system, then start up iscsi-target:

# sudo /etc/init.d/iscsi-target start

Now you just need to configure your iscsi initiators on whatever your want to use the iscsi drives on. I tested it in Windows and in Linux and it works beautifully. So far I’ve only tested it by making one HDD = one LUN, but it should be able to support different types of configurations such as RAID or file LUN’s, and there’s failover support as well. I’ll look into these some more and get the details back to you.

I’m just excited to have a 1TB homemade SAN to play with now.

Last week I tried something I’ve been wanting to try for a long time. I ordered an old Ipaq 3650 on Ebay and installed Linux on it. In order to install Linux you need either a serial cable (usb won’t work) or a compact flash sleeve. I just happened to have a CF sleeve from an old Ipaq I used to have. So I went to http://www.handhelds.org, the home of the handheld Linux projects and found the instructions there that I needed. I should note that it looks like this project is dying. There haven’t been any updates for almost a year on Opie (one of the environments available), and the other options hadn’t seen any activity for several years.

So I copied the necessary files to my CF card, ran “Bootloader.exe” which replaced the Ipaq’s bootloader, then I was able to install the Opie image from my CF card. It rebooted and everything worked. I really liked the interface, but everything wasn’t as great as I’d hoped. First of all the 3650 comes with a 16M ROM plus 16MB of RAM. WinCE uses the RAM to store files and data on (which you lose on a reboot) making it appear you have 32MB of memory available. Using RAM as ROM has some obvious drawbacks, so Linux won’t do it, which means I only had 16MB of storage, and the Opie image was over 15MB. So I started out with less than 1MB of memory available. Luckily I have a 512MB CF card. So I started uninstalling software from the system ROM and moving it over to the CF card. The problem with that was the CF card was formatted using vFat. When software was installed to the CF, it would recreate the file structure of the root file system(/), then create symlinks. For example, when installing samba it created a directory /media/cf/etc/samba then create a symlink between that directory and /etc/samba. The problem with that is that the OS didn’t support creating symlinks on a vFat partition. So I created a loopback partition on the CF card. A loopback partition is a file on the CF card that I can mount as a hard drive. I then formatted it in ext3 and was able to install software to it w/o any problems. Until I rebooted. When installing software using the package manager, I found that I would often SEGFAULT if I tried to install more than 2 or 3 things at once(including dependencies). After a while it would SEGFAULT constantly and I would have to reboot. When I rebooted I would then have to manually mount the loopback partition, and run through a serious of commands for all the symlinks to be recreated. If the software was installed directly on the CF card, it tried to create the symlinks on its own, but it didn’t for the loopback partition.

Anyway, I created a bunch of scripts to make all this easier, and once I had all the software I wanted installed, I started looking into some way to synchronize my PIM data with my desktop. If my desktop was running Linux I believe it would have been simple. But as it is I didn’t find anything to work with my Windows desktop running Outlook. There is Qtopia Desktop, but I couldn’t get the latest version of it to run on Windows XP, and the earlier version that I could get to run I couldn’t get to sync with my PDA or with Outlook though it’s supposed to sync with both. So then I found the KDE PIM package. It worked! I was able to import my Outlook info(though not sync) and sync with my PDA. But it ran incredibly SLOOOOOWWWWW on the Ipaq, probably because it was running form the CF, but I couldn’t free up enough memory to install it to the ROM.

So here’s my conclusion. Linux on the PDA is definitely something I want to explore some more. The Sharp Zaurus looks great and comes with Linux already installed. I’ve read plenty about people running Linux on the Ipaq and having a good experience. But if you want to run Linux on an Ipaq, get one with a 32MB ROM or more. All the problems I had really came about from the simple reason that I didn’t have enough memory. The CF card is a great way to store data, but no suggested for running apps from.

So I bit the bullet and put WinCE back on my Ipaq for now. When I did I plugged it in and in 30 seconds had all my PIM data on it. I’m not giving up for good, but I’m too busy to put any more time into Linux on my Ipaq for now:(

I finally came to a conclusion why I could not get everything on my HackLab to work. It’s because I’m using spare parts out of my garage that don’t work. I’ve gone through two hard drives already. The PS/2 ports on my motherboard don’t work. I’m sure there’s got to be other things wrong with it all. I can’t get SVN and Metasploit to work. So I took the same ISO I’ve been using, and ran it in Qemu on one of my computers at home, and both SV and Metasploit work fine straight off the ISO.

Next week I’m retiring our Mail Server, so I’m holding off on this project until then. One the new mail service is up and running and I’m certain all e-mails are present and accounting for, I’ll start again. The good news is I can copy the VM’s I’ve made, so I could potentially get the HackLab up and running in a couple of hours. So keep checking back, and I’ll let you know when we’re ready to test this new service.

OK, so last Monday I sent my laptop to HP, and 2 days later I had it back… FIXED!!! I’m now in love with HP!! Thank you HP for fixing my laptop so quickly!

My wife collects animated gif’s she refers to as blinkies. So last night she was going over her blinky folder and telling me what she had and she ran across one that said, “I have CRS.” She was just looking through them in Windows Explorer so they weren’t animated, and she couldn’t remember what CRS stood for. She tried and tried but couldn’t remember. So finally she opend the blinkie to watch it animated to see what CRS stood for. CRS = Can’t Remember Stuff. I laughed at her for not remembering CRS= Can’t Remember Stuff, so she looked at me with a wild look in her eye and said, “My voices don’t like you!”