Network Woes!

If you’ve been following along you would know that I did some server upgrades this weekend. I didn’t do all the upgrades I planned on because of some driver issues. When everything was finished Friday I tested everything and it all worked fine, I also logged in remotely and tested everything again Saturday evening and all was well. Then Sunday morning, the whole network was down. It was all very strange. All the servers were working, but not aways accessible. Sometimes I could ping a server from my workstation, sometimes I couldn’t. After a few packets captures it appeared there was a lot of “junk” on the network. So we proceeded to unplug different switches, then plug them back in until the problem came back. Eventually we narrowed the problem down to a single switch. I rebooted it and everything worked. I plan on replacing that switch soon, so it’s not that big a deal. Again I logged in last night and everything worked fine.

Since I worked all day Saturday, and ended up putting in a full day yesterday I decided to take today off. I got up and started mowing this morning, then about 9:00 I got a call from work. Apparently the server was down. I went in thinking it was the same switch again. It turns out the VM running our LDAP server had locked up. I rebooted it and all was well.

…

It turns out that the problem with the VM was my backup script.  I had unmounted the drive I was backing up to this weekend and didn’t remount it.  So when I backed up to /mnt/Backup, it copied all our data to the partition / is mounted to.  Therefore there was no free space on / and it caused problem.   Oops.  But I took care of it now.