TangoRangers.com's Blog

A Little History

I’ve been working for this small company for just over a year and a half. This is my first true system administration job. I’ve had several in the past where I did admin work part time, or on the side of my normal duties, but finally, after years of trying, I managed to land a great position. Most of my work consists of Linux servers, running everything from Ubuntu, to Debian, CentOS, and even a couple Slackware servers. Some are in house, some are in the cloud. Feels great to finally be doing what I love. There was just one problem… Windows. You see, many years ago my company was part of a much larger organization with hundreds of employees. This smaller company spun off and took with them 2 Windows Server 2003 systems. These server once did a lot of things. Managed E-Mail (Exchange), printers, file shares, Active Directory, VPN, and internal DNS/DHCP. Since I started, I’ve been fighting the AD (short for Active Directory) servers trying to keep it running (I should note, that I’m a pure Linux guy, I don’t know Windows hardly at all, and definitely not Windows Server). We also dropped down the number of users who actually used the AD. Most people in this newer, smaller company, all ran their own versions of Linux.

Damn Windows

A short time ago I informed management that our Windows Servers would not get updates for much longer, and we should plan on a migration. After talking to Microsoft and getting clarification on pricing, I found just to upgrade would cost somewhere about $3,000. While this is not much in company money, it also involves a lot of future headache for me having to keep them running, plus I’ve heard horror stories when dealing with the actual upgrade. I took it upon myself to get rid of this crap and just be done with it.

Now don’t get me wrong here. Sure, I’m not proficient with Windows Server, but I wouldn’t get rid of something that worked just fine. I’ve spent a lot of time keeping these servers running. They reboot at random times, half the patches fail to install, and sometimes the services on them will just stop working. The problem here is that these servers do not work! When they do reboot, they take 12 minutes and 34 seconds to boot (quad-core Xeon 3.6Ghz, 8GB RAM, 15K HDs on RAID 5). During that time, DNS stops working and I get complaints that the internet isn’t working, or it is very slow. For those who know what happens when your primary DNS goes down, you know what I’m talking about. I even had times where after a reboot, the services fail to start. Lets just say, these servers are very broken, and I don’t believe that Microsoft is worth keeping around when so few computers even need to be on Active Directory in the first place.

Lets Do This

The first step would be to find another DNS/DHCP solution. Easy, I’ve done this many times before. So I have the new server ready to put in place. I should note that we have a weird network configuration, and as of the writing of this portion, I have no idea if my solution will work. We have several VLANs and DHCP is “helped” through the system to get to the correct server. I’m not going to go into the technical details of how this all works (partly because I don’t fully understand it myself), so you’ll just have to hope I figured it all out.

As far as VPN goes, pptp is bad, weak, and apparently not a good one to use. I opted for using OpenVPN. Turns out of Windows setup is a little more difficult for the non-tech savvy users, but we will manage.

What Am I Doing?

Now, the nightmare begins. Removing machines from the AD. I was given 2 computers when I started at this company. A Windows laptop, and a desktop where I could install any Linux distribution I want. I started the AD removal on my laptop and everything went perfectly. At this point, I figured, how hard could this be. So I removed it from a Windows Server than manages our phone system (no one touches that machine, we are all too afraid). This one wasn’t too bad, but it did kill the backup user. Once I added it back in, all is good. I’m still getting backups. Next came a very old XP machine that hosts QuickBooks. After removing it from the AD, I couldn’t login! OK, boot a recovery disk, wipe the SAM Tables (Microsoft’s password database), reboot, add password, done! Woohoo… well, no. Turns out it had a share for our CFO. Crap. It took me a while, but I finally got that share working and he was able to access QuickBooks. As before, this one became broken on the backup system, but it was an easy fix as all I had to do was change out the user our backup software uses to connect to the shared folder. All is good.

Before going too far, I want to let you all know that I’m writing this as things are happening. At this point I still have a machine in our conference room, 1 Windows 8, 4 more Windows 7 (one of which I’m worried about since it is our CFO’s machine), and a bunch of XP machines in our dev lab. Yep, XP. We have older equipment where the software will not run on anything above XP, so I have to keep them very hidden and spend a little extra time on them to ensure they are OK.

Anyway, I ended up having another issue when doing out conference room PC. It seems, this computer didn’t keep some of the group policy settings that others did. Granted, I was able to just change the backup user, but I actually had to create another share. What a pain right? To make matters worse, I also had to edit the firewall settings to allow ping (my backup software requires I can ping the machine) from any subnet, and allow SMB access from any subnet. You see, the backup software host and most other computers are on different subnets, so I had to adjust for that. Live and learn I guess.

Pressing On… And On… And On…

Here I am again, Thursday morning (almost a week after starting this whole process), wanting to remove more machines from the AD. Then I thought about it again. Due to the configuration of our switch, and the need to forward DHCP packets correctly to the Windows Servers, what are the chances that this DHCP helper option won’t work correctly in Linux? While the switch does have fantastic documentation, it doesn’t tell you squat about what to do on the DHCP server side. My heart is racing, fears are rising. What if I can’t get this to work? Am I doing to be stuck running Windows Server forever simply because I don’t understand this very complex switch? This thing’s config hasn’t been touched since 2008, and even contacting someone who worked with it back then (pretty cool of the guy to talk to me after not being with the company for over 5 years) proved no help as he worked with 3 other people on getting this thing set up, and he didn’t know the CLI password (which is where I have to change these settings). I can’t get a hold of anyone else from back then. Looks like I’m on my own… again.

Well, I can’t test this today, it would interrupt everyone. Looks like I’m going to have to work on something else for the next couple of days then come in over the weekend.

Fast forward a couple days, and here I am. Easter Sunday… at work. Oh well, it worked out pretty well this way. Sure, I would like to do spending time with family, but since everyone else is, I figure now is the perfect time to take down the network and move everything over.

So it begins, nice and early. First I disconnect the Windows servers from the network, then I change the IP address associated with the Linux box that will control this network. All services now up and running.

Time to Test

Well crap, it seems Windows computers had some issues getting DNS updates. Sometimes I can refer to the other computer by name, sometime by the full name (meaning adding the internal domain name to it), but only sometimes. After spending hours working on it, I still have no solution… I’m starting to think many of these had issues before, and it could have something to do with their own hostnames. After all, Linux likes it when you give it a domain name. Either way, it works as good as before… I think. So I continue on.

Had one Linux server get an IP address outside the DHCP range. I have no idea how this is possible. Screw it, you get a static IP and DNS name. Fixed.

After getting through a couple machines that just didn’t want to place nice. I got through the rest without hardly an issue. It actually went much more smoothly than I thought it would. After a few hours, the network was up and running again!

Now, unfortunately, I wasn’t done yet. There were a few more items that needed to be dealt with. First, the new OpenVPN. Done. Oh… that’s right, just needed to make a simple change, and everything works. I actually forgot to adjust the server IP in the configuration files to reflect the server’s new IP. Tested, and working great. Cool.

D’oh!

What about pptp you ask? Well, yes, I did want to get rid of it. The problem was many of my users were still setup to use it and hadn’t been given their new keys with OpenVPN. I’ll deal with that next week, but the problem remains of ensure they can still get in. So I fire up a machine and get to testing (using a 4g modem so I’m outside the corporate network). Connecting… connecting… verifying username and password (I didn’t know pptp was this slow, holy crap!)… damn. It just isn’t going to work for me. I’ve actually never used the old VPN, so I have no idea if it would have ever worked before or not. I hate to say it, but I think I’m going to have to wait until tomorrow morning and see who complains.

UPDATE: Upon further investigation, I’ve found that Windows 2003 will NOT let you use VPN unless it can run the DNS/DHCP. Shit… Just another reason to move away from Windows.

Up and Running

Now, where was I? Oh right. So, everything is up and running. I found a few machines that were given static IPs from the Windows servers, but were not listed, so once they got their new addresses I found their services not working. This is because of our firewall rules. So I adjusted the rules and set static IPs for those systems, so they should now always keep those addresses.

Most internal DNS is working without having to put in the domain name. I’ll work on the rest of those throughout the week. I’m not anticipating this will cause any issues with my local users (users from the other office still have to type the internal domain name. I’ll work on that later). So Maybe now I’ll finally call it a day. I’ve been here for roughly 12 hours now, and I would like to call it a day.

From this point, all I need to do is gets all the workstations off the AD that no longer exists. I worry there will be issues just leaving them alone since I have no idea how long it will take before these workstations say they can no longer login from the missing AD controller. So over the next week I’ll get this taken care of. Just need to get backups working and any mounted drives working for each user, then I’m done! Oh I can’t wait!

Getting There

Fast forward a bit here and another week has passed. During this time I’ve ended up with just 2 desktops that needed to be taken off the old AD. One is Windows 7, the other is 8.1. During this process of getting off the AD, I do have to reboot and perform some work on the firewall, so I like to set up times where I can do this with each user.

First came the Windows 8.1 machine. Oh man do I hate Windows, especially 8.1. This thing caused all sorts of problems. There is so much hidden shit all over Windows, that is just drives me insane. I couldn’t get his account to login because of group policy. That was the actual error. Access denied by group policy. So I checked the group policy on the computer… there wasn’t any! Eventually I had to just create him another account, under a different name, and copy his personal files over. What a joke. I even had something of the same type of issue with the Windows 7 machine. Which sucked because the guy handles all our finances and I really didn’t want to cause issues for him. His wasn’t nearly as bad, but it ended up as where I had to copy his files over to a new account anyway. Fortunately he was able to keep the same user name. Some things ended up in other folders, so after several hours, we located all his files and got him all setup and good to go. I was pretty happy about that. I always like when things work out pretty well when they seem to be going so badly.

Userland

Speaking on Windows and their userland. I hate how Windows handles this. It makes it very hard to move over to a new machine with all your same settings and everything exactly how you like it. I ended up screwing up one email account because he uses Outlook, and apparently you can’t just “import”. You have to export, then import, and you can’t just copy config files over. This is according to MS! This is why I run Linux. Last time I moved from one machine to another, I copied (scp) my userland over to the new computer, started X (I like KDE), and guess what? Everything was there exactly how I had it on the other computer! Amazing! Also, I know there are tools provided by Microsoft designed to help with this, unfortunately those don’t work in my situation.

DONE!

So here I am, almost 3 weeks after starting this project, and I’m finally done. Every computer is getting backups, they can access and be accessed from other VLANs (I know, I know, that’s not how you use VLANs, shut it, I like it). It has been a pain, and I wouldn’t recommend it to anyone. Especially if you are the ONLY one doing it, and you are not a Windows guy. So in the end, here is my advice. If you are running AD, just keep giving MS all your money and hope it keeps working. If you are not running AD. DON’T! Stay away! If you are going to have it, be sure to hire someone just to handle it, and make sure that is their only job.

Thank you for letting me tell my story, and if you made it this far, good on you!

UPDATE: I wish I saved the link, I found something that MS apparently does with the newer version of Windows Server. I already knew that I have to pay a lot just for the OS, but then I have to pay an additional price for each user, or CAL as they call it. Well, apparently CALs are not just for users of the Active Directory. You have to have one for each machine that uses the DHCP! I have a lot of Linux servers and even more as Virtual Machines. I refuse to keep giving money to MS for each machine just to use DHCP/DNS. That is a load of crap. Some people who commented on the article said you don’t really have to, but if MS decided to audit your network, you could end up having to pay a lot of money. I don’t know how true this is, but I wouldn’t be surprised. Glad I got away from that train wreck.

Like many of you, I’m concerned about security, especially when working remotely. Generally I would simply create a tunnel using SSH, but then I must set all my programs to use the socks5 tunnel. This isn’t always possible without first opening the program, which will generally try to form a connection. Perhaps, not the best way to keep safe on a network you don’t trust (like a coffee shop).

Unlike using SSH to create a secured tunnel, which requires setting proxy settings for all your programs, using something like OpenVPN you can redirect all your traffic through the encrypted tunnel without having to configure anything. All thanks to using iptables.

Here is my quick and dirty guide on getting your very own OpenVPN server setup on Linux, as well as setup for two types of clients. One being a Linux client, the other being Cyanogenmod’s Android.

With this guide, I’m going to assume you already have OpenVPN installed and ready to go. Also that the configuration files are in /etc/openvpn/

Server Setup

First off, we need to generate some keys. This will be used to secure the connection. OpenVPN comes with all the tools you need to generate keys and indexes. Look for the easy-rsa directory that comes with OpenVPN. In my case, it’s in /usr/doc/openvpn-2.2.2/easy-rsa/2.0/

In that directory you will see a lot of scripts. Before doing anything you need to edit the file vars. In this file are several settings. Most important is with dealing with the openssl key. Here is a quick example you can base your configuration off of with all the comments removed.

export EASY_RSA="`pwd`"
export OPENSSL="openssl"
export PKCS11TOOL="pkcs11-tool"
export GREP="grep"
export KEY_CONFIG=`$EASY_RSA/whichopensslcnf $EASY_RSA`
export KEY_DIR="/etc/openvpn/keys"
export PKCS11_MODULE_PATH="dummy"
export PKCS11_PIN="dummy"
export KEY_SIZE=1024
export CA_EXPIRE=3650
export KEY_EXPIRE=3650
export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="City"
export KEY_ORG="domain name"
export KEY_EMAIL="emailaddress@domain"

Note the export KEY_DIR. This is important. You will get warnings about running ./clean-all. This will delete ALL your keys.

After editing the vars file, we need to execute it to store the values in memory, then clean the keys directory. Do so by running:

. vars
./clean-all

Yes, you read that right, period, space, vars.

Now we are going to generate keys for the server and two clients.

For the server, we just need to run a couple of quick and easy commands.

./build-ca
./build-dh
./build-key-server server

The last command will build a server.key file. This is needed when running the server for key exchanges and such.

Now there are 3 different ways to build keys for clients.
./build-key client (no password protection, not recommended)
./build-key-pass client (with password protection, recommended)
./build-key-pkcs12 client (PKCS #12 format, good for Android)

For the client configuration. I’m not sure if you can use the PKCS #12 format. I haven’t tried, but if it works for you, please let me know.

Now we need to edit /etc/openvpn/openvpn.conf for our network setup. Most of the config files are self explanatory. Here is my example:

cd /etc/openvpn #yes, you do need this for some damn reason
local localIP
proto udp
port 1194
comp-lzo
verb 3
log-append /var/log/openvpn.log
dev tun0
persist-tun
persist-key
server 172.16.1.0 255.255.255.0
ifconfig-pool-persist /var/log/ipp.txt
client-to-client
push "route 10.0.0.0 255.255.255.0"
push "dhcp-option DNS 10.0.0.1"
push "dhcp-option DOMAIN domain.tld"
push "redirect-gateway def1"
keepalive 10 120
cipher BF-CBC
ca keys/ca.crt
dh keys/dh1024.pem
key keys/server.key
user nobody
group nobody
status /var/log/openvpn-status.log

Be sure to change localIP to the server’s IP address AND (if applicable) forward UDP port 1194 to the server.

NOTE: There is one issue I have run into. By using the option push “redirect-gateway def1” does seem to work fine and redirect all through the VPN, I have an issue getting the DNS and DOMAIN to work through both the OpenVPN software or my Android. This means that all DNS queries do not appear to be going through the VPN. This may not be the case. I have yet to setup a packet sniffer to check. So for the time being, I simply created a bash script that will edit my /etc/resolv.conf file when I start the VPN, and revert it back when done. If someone knows of a really easy way to check without having to use a sniffer, please let me know.

Now that all of the keys are built, and the openvpn.conf file is setup, we are ready to start the server. While I have run into some strange behavior in my configuration, you may have better luck in yours. In mine, I had to create the device tun edit ip_forward and manually configure the IP tables.

Here is my simple script I run on the server what I want to have the OpenVPN server up and running (yes, I do this at boot). Explanation of items below.

mkdir /dev/net
mknod /dev/net/tun c 10 200
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I FORWARD -i tun0 -o eth0 -s 172.16.1.0/24 -d 10.0.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i tun0 -o eth0 -s 172.16.1.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i eth0 -o eth0 -s 10.0.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o eth0 -s 172.16.1.0/24 -j MASQUERADE
iptables -t nat -I POSTROUTING -o eth0 -s 10.0.0.0/24 -j MASQUERADE
openvpn --config /etc/openvpn/openvpn.conf --cert /etc/openvpn/keys/server.crt &

Most places I have found this stuff are not very specific about IPs. So let me give you a quick rundown on each item.

First we create the device with some special settings. That is the mkdir /dev/net (if /dev/net already exists, it will do nothing), then mknod /dev/net/tun c 10 200. Then set ip_forward to true. The fun part is with the iptables.

So in my example, tun0 is the virtual device that is the VPN and eth0 is my ethernet. 172.16.1.0/24 is the IP range I’m giving to the VPN (tun0), and my physical network is 10.0.0.0/24. You can leave the VPN network on the 172.16.1.0/24 network, simply adjust the 10.0.0.0/24 to your networking configuration (ie 192.168.0.0/24). How all those iptables work… yea, I’m not going into it. They work, I’m fine with that.

After running those commands, your OpenVPN server should be up and running. The final process is background so you get your terminal back. Wait a few seconds and hit enter again. If you don’t see the process has ended, then you have done everything correctly. If it did error, check /var/log/openvpn.log for information on what is causing the problem.

Client Configuration

Now that the server is setup, lets get the client side going. This part will be for the OpenVPN software running on Linux. See the next section for CyanogenMod’s Android.

This part is much easier than the server setup, but you need to get your keys to the client. I highly recommend you do with via scp. You will need ca.crt, client.crt, and client.key. Assuming you called your keys “client”. Put these files in /etc/openvpn/keys. Then create the file /etc/openvpn/openvpn.conf and put this in it.

remote IP/DNS 1194
proto udp
dev tun  
cd /etc/openvpn/
ca keys/ca.crt
cert keys/client.crt
key keys/client.key
client
ns-cert-type server
keepalive 10 120
comp-lzo 
user nobody
group nobody
persist-key
persist-tun
status /var/log/openvpn-status.log

Change IP/DNS to the IP or DNS name your server is reachable at. You should now be able to connect to your OpenVPN server by typing:

openvpn --config /etc/openvpn/openvpn.conf

That’s pretty much it. Once you get a handle on the settings, it is actually pretty easy. However, as mentioned before. I have found a possible issue with DNS. I would highly recommend editing /etc/resolv.conf to point to your DNS server. In my example, the DNS server is also at the gateway (10.0.0.1). You can script this. In fact, use my script.

#!/bin/bash
pid=`pgrep openvpn`
if [ -z "$pid" ]; then
echo "Starting OpenVPN Client"
cp /etc/resolv.conf /etc/resolv.conf.backup
echo "nameserver 10.0.0.1" > /etc/resolv.conf
openvpn --config /etc/openvpn/openvpn.conf &
else
echo "Stopping OpenVPN Client"
mv /etc/resolv.conf.backup /etc/resolv.conf
kill $pid
fi

Pretty strait forward if I do say so myself. You may have an issue if you have a passphrase on your key! If you are having an issue, remove the ampersand (&) from the end of the openvpn –config line. This will not background the process, but you can do it manually by typing ctrl+z then bg which will background the process.

CyanogenMod’s Android Configuration

Because I don’t run the Android that came with my phone, I can use OpenVPN with ease. If you are not running a custom rom, you can still run OpenVPN by getting the client software from the Android Market (now called the Play Store). The following instructions are for CyanogenMod 7.2, but should work in newer versions just fine.

Remember when you made your client key? Well you need to make one that works great with Android. It’s the PKCS #12 format. This will give you a file that ends in a .p12 extension. Copy this file over to the root of your sdcard.

Install the certificate by going to Settings->Location & Security->Install from SD card (under Credential storage at the bottom on the menu). It should find the file and ask for the password to unlock it. Then it will ask for a new password (you can use the same one as before) and you can also give it a custom name.

Build the client by going to Settings->Wireless & Networks->VPN Settings->Add VPN. You just need to select the OpenVPN type. In the new menu there are several settings.

VPN name (this can be anything you want)
Set VPN server (the IP or domain name of the server)
User authentication (leave unchecked)
Set CA certificate (click this and select the key you just installed)
Set user certificate (same as above)
DNS search domains (these are optional, but you can set 10.0.0.1 like in the bash script above)

Hit the menu button then Advanced.

Server port (default is 1194)
Protocol to use (udp is default)
Device to use (tun, which is fine)
LZO compression (check it!)
Redirect gateway (check it!)
Remote Sets Addresses (Should also be checked)

Everything below that I left as default. You do NOT need to enable TLS-Auth. For this type of setup it is unnecessary.

Hit back, then save. From here you should be able to connect to your VPN. Note that in my tests, the VPN is much slower. I’m not sure if it is something I have done wrong in my setup, or if my provider throttles VPNs.

Conclusion

Everything should be up and running now. I hope you found this useful. Please feel free to leave a comment below. If you have any suggestions or questions you can drop those below as well. I’m not an expert on OpenVPN, I just like learning.

Sources:
http://openvpn.net/index.php/open-source/documentation/miscellaneous/77-rsa-key-management.html
http://openvpn.net/index.php/open-source/documentation/howto.html
http://blog.johnford.org/openvpn-tunnel-to-home-server/

Like many of you, I run my own SVN repository. I have several projects and several people that use them. So I wanted a quick and easy way to perform backups. Here are the two methods I use.

The first is an incremental backup where I use a post-commit hook. The other is a bash script that I setup in cron to run once per week (you can do daily, monthly, or when ever. It depends on how quickly you want the full backup script to run). The bash script is designed to create a full backup using svnadmin dump.

The idea of a full backup is nothing new, but the way my script works is a little different than others. Their script just creates a new backup when ever run. Mine creates a full backup of every revision number. Even if there have been several updates since the last backup.

Lets get started.

Incremental backup using a post-commit

Post-commit hooks are very powerful. You can do a lot with them. In this case we are going to do an incremental backup. I don’t do a full because my repository is very large and I don’t want to be slowed down every time I commit new files.

First you need to go to the directory where your repository is. Lets call it /srv/svn/myproject. In that directory there is a folder called hooks. Looks for post-commit (it maybe listed as post-commit.tmpl, just rename it without the .tmpl). Open the file in an editor (vi, nano, emacks) and add this line to the end.

svnadmin dump "$REPOS" --revision "$REV" --incremental >/srv/backups/myproject/incremental/commit-$REV 2>> /srv/backups/myproject/incremental/backup.log

NOTE: If mailer.py is not commented out, put a hash in front of it “#”. You don’t need it.

Take note of the directories I’m using. Just change them to where your backups will go. Also, you need to give those directories the same ownership as your svn archive! Otherwise when the post-commit runs, it will error out because it cannot write to the new location.

And that is it! Now every commit you make will create a new incremental backup.

There is one down side to this method. If you already have several commits, you will not get your entire history. I did this a lazy way since at the time I was only up to around 20 commits.

./post-commit /srv/svn/myproject 0

This will create the backup starting at revision 0. Just keep running it and moving the number up. If you have hundreds or even thousands of commits… you might want to write something to do all that hard work for you. You might even get some ideas from the next section.

Full backup

This is where I’m different from everyone else… at least that I could find. While this is not the most elegant way of doing things, it really doesn’t take very long to run (unless there are many revisions to create new archives for.

#!/bin/bash
 
svnLocale=/srv/svn/myproject/
backupLocale=/srv/backups/svn/myproject/full/
fileName=myproject.rev.
extension=.svndump.bz2
 
latestRevision=`svnlook youngest $svnLocale`
let stopCounting=latestRevision+1
COUNTER=0
while [ $COUNTER != $stopCounting ]; do
        rev=$COUNTER
        # check to see if file exists
        if [ ! -e $backupLocale$fileName$rev$extension ]; then
                svnadmin dump $svnLocale -r $rev -q > $backupLocale$fileName$rev.svndump
                bzip2 -z9q $backupLocale$fileName$rev.svndump
        fi
        let COUNTER=COUNTER+1
done

Seems a bit much, but all you would need to worry about is changing svnLocale, backupLocale, and fileName. You can change extension if you wish, but I would leave it unless you plan on changing the svnadmin or bzip2.

Please note that the first time you run this, nothing is dumped on the screen. I ran this on an archive with 1215 revisions as a test. I went to lunch. You can modify the script and remote the -q from both svnadmin and bzip2 to see your progress. I silence them while running as a cron job.

The benefit to this script is it does check to see if an archive has been created for each revision. If not, it gets created. This script will even get backups you may have deleted and recreate them, then continue on. It doesn’t matter. It’s so simple that it just works.

So that’s pretty much it. Questions? Comments? Let me know what you think.

Do you want to use SVN to update your website? This guide will help you and includes information not found anywhere else on how to get it working. Many websites give basic information and comments complain about it not working. If those other website didn’t work for you. This will.

After digging around for hours and seeing everyone complain about having the same problems after being told the same solutions, I decided to figure it out for myself. If you read this carefully you too will have a website setup that can be updated via subversion.

Getting Started

I’m going to assume that your SVN server and website are on the same machine and that you are using Apache to serve both. I’m also going to assume that you are somewhat familiar with SVN and Apache. I’m not going to go into much detail on how to get everything setup from scratch.

The tools you will need to have installed are as follows (may need more depending on your configuration). Apache HTTPd, Subversion, and gcc.

Many Linux users and distributions use sudo. I do not, and I will not provide information on how to use sudo. My recommendation, before doing anything, is to type sudo su which will make you root. Do everything from there so you don’t have to sudo anymore.

Setup a new SVN repository

Alright, now we are going to setup the new SVN repository. This can be done easily with:

svnadmin create <path to repository>

Place it in a good location that you will remember since you need to point Apache to it. In my examples we are going to use /home/www/domain/svn/website/ for our project location and /home/www/domain/htdocs/ for our website.

For user access controls I use a 2 part process. I have an authz and passwd file in the root of my SVN directory (/home/www/domain/svn/). The reason I do this is because I host several SVN projects and I don’t want everyone to have access to everything. This way I can also setup (this comes later) Apache to have read-only access to the repository.

First, we need to create our authz and passwd files. Lets call them svn.authz and svn.passwd so we know what they are for.

In the svn.authz file we need something like this:

[groups]
project = dkun
other1 = dkun, someguy
website = dkun, friend1
readonly = redmine,apache
 
[project:/]
@project = rw
@readonly = r
* =
 
[other1:/]
@other1 = rw
* =
 
[website:/]
@website = rw
@readonly = r
* =

Looks a bit overwhelming. Let me explain.

Starting from the top, we have [groups]. These are groups of users that we give project names to. Notice that 2 of the projects have more that one user. There is also a read-only group. You can name the groups what every you want. I do it by project name to make it easy to read.

After that you see [project:/]. This is the project group. It has one user and is set to read/write. It also has a readonly group. This way I can setup both Apache and Redmine with read only access.

Lastly you might have noticed *=. What is that? It turns off anonymous access. So unless your project has code going out to the web, make sure to put this in.

Now we need to setup the svn.passwd file. This is pretty easy. Just type:

htpasswd svn.passwd <username>

You will be prompted for the password. That’s it! Pretty easy right? Just don’t forget to setup the user apache (keep it lower case, trust me), and any others you might need.

Setup Apache

Now that we have SVN and out new project setup, we need to configure Apache to serve everything. Setup your VirtualHost to point to where you want your website to be hosted from. So for me it would be /home/www/domain/htdocs/. I’m going to assume you already know how to do this. All you need to add is a little directive.

<DirectoryMatch "^/.*/\.svn/">
     Order deny,allow
     Deny from all
</DirectoryMatch>

This will stop someone from browsing your .svn folder on the website.

Now setup an SVN VirtualHost. This gets a bit more complicated. I’m going to avoid going into detail here. I want to assume you have some background.

In your Virtual host we need to set a few options and point to the SVN repository root. Here is an example of the options you need to put inside the VirtualHost section for SVN to work. Don’t forget to set other items you need!

<Location /svn>
   DAV svn
   SVNParentPath /home/www/domain/svn
   AuthType Basic
   AuthName "My SVN Repository"
   AuthUserFile /home/www/domain/svn/svn.passwd
   AuthzSVNAccessFile /home/www/domain/svn/svn.authz
   Require valid-user
</Location>
<Directory /home/www/domain/svn>
   Options +Indexes FollowSymLinks +ExecCGI
   AllowOverride AuthConfig FileInfo
   Order allow,deny
   Allow from all
</Directory>

Note all the directories. Change them to match your server. Also, verify that your SVN modules are enabled in Apache. They should be in your apache.conf or httpd.conf (depending on your server). You will need to enable:

LoadModule dav_svn_module lib/httpd/modules/mod_dav_svn.so
LoadModule authz_svn_module lib/httpd/modules/mod_authz_svn.so

That’s it! Now type apachectl configtest and you will get messages back if there is a problem in your Apache config. If it is a warning about your website directory missing, don’t worry. Just create the directory and give full ownership to the apache user (We will get back to this in a moment).

Before continuing, restart Apache with apachectl restart and make sure your SVN works! If it does not then the next section will fail.

Do some weird stuff with Apache

Here is where things get a little weird. This is also where 99% of people fail because there is no good information out there. I will do my best to explain what to do and why you should do it.

The first step is you need to find out what user and group Apache runs under. For me, both are apache, but for you it could use www or www-user. You can find this by checking /etc/passwd and /etc/group, or by going through your apache.conf file and look for User and Group directives.

Now look in /etc/passwd for the apache user. The second to last option shows Apache’s home directory. In my case it is /srv/httpd. Go to that directory and type:

chown apache.apache ./

Set the user and group to which ever Apache runs under. This will allow Apache to create new directories there without changing ownership of anything else. Also, that is just one (1) period before the slash!

Open /etc/passwd and find the apache user again. At the end of the line you should see something like /bin/false. This keeps the Apache user from logging in. This is for security reasons. We are going to temporarily change it to /bin/bash.

Now that you are already root, type:

su – apache

Once again I assume your Apache’s user is apache. Adjust as needed.

Now you are logged in as the Apache user. You should see a new shell and be ready to go. Make sure you are in the /var/httpd directory (as stated previously as being Apache’s home directory) by typing pwd. If you are, then you have created a new session and are ready for the next step.

Navigate to the directory where the website will be served from. So if the website is in /home/www/domain/htdocs/ then go one level up; /home/www/domain/. Now we are going to perform an SVN checkout and have subversion remember our credentials.

svn co http://<domain>/svn/website

Adjust the http directive as needed. If you use SSL have subversion permanently accept the certificate. You should see a prompt for apache’s password. This is the password you created for it in svn.passwd. Type it in and when asked if you want to store the password, say yes!

If everything went as planned then you should see a successful checkout as revision 0. If not… well you did something wrong. Double check your paths, apache config files, and passwords. If needed, you may need to give ownership of the directory to Apache so that user can create a new directory and write to it.

Check to make sure a .subversion directory was created in Apache’s home directory and that there is a .svn folder in /home/www/domain/htdocs/. If now you need to give ownership of that directory to the Apache user (you will have to be root for this). If there is so .svn directory, then change ownership of the htdocs directory to Apache and run the checkout again.

If everything went smoothly then we are done as the Apache user. Type logout to return to being root.

Last thing. Open up /etc/passwd and change /bin/bash back to /bin/false. No need to have a security hole.

Create an executable

Now that we have SVN, Apache, and a SVN checkout setup, we can finally finish this. The first thing you need to do is find where svn is located:

whereis svn
svn: /usr/bin/svn /usr/man/man1/svn.1.gz /usr/share/man/man1/svn.1.gz

As you can see, I used whereis to find the svn executable. It’s the first one, /usr/bin/svn.

Navigate over to the htdocs directory. Open up your favorite text editor and dump this in it:

#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
int main(void)
{
   execl("/usr/bin/svn", "svn", "update", "/home/www/domain/htdocs/",
      (const char *) NULL);
   return(EXIT_FAILURE);
}

See where the directives of /usr/bin/svn and /home/www/domain/htdocs/ are? Just change those to match your server. Save the file as update.c and just back to the shell.
Now we need to compile the program:

cc -o update update.c

This will create an executable called update. You can call it what ever you want, just as long as before renaming it you type chown apache.apache update; chmod +s update. It should be owned by the Apache user.

Setup post-commit

Just over to your SVN repository, then into the hooks directory. So if you were following, cd /home/www/domain/svn/website/hooks. There you will see a bunch of .tmpl files. Rename of copy post-commit.tmpl to post-commit, then chmod 755 post-commit.

cp post-commit.tmpl post-commit
chmod 755 post-commit

Open post-commit in your editor. You may see a bunch of stuff in there. Just put a hash (#) sign in front of any other commands there. They are examples. Add:

/home/www/domain/htdocs/update

That’s it. Don’t add sh, or bash to the front, just put in the path with executable at the end. Save and exit.

Try it!

Now, as long as you followed along and I didn’t skip anything. This should work. Try it out. Try committing something to SVN and see what happens! If you can view the website right away, they you did it correctly. If now, you should check your apache log files. Specifically error_log. If SVN threw an error you will have to see what went wrong there. Sometimes just putting a commit message in will fix any error.

That’s it. I hope you found this informative. Please feel free to drop my a comment below if you have anything you wish to add or just want to say thanks for the info. I appreciate all feedback. Thanks for reading!

Recently my work switched to Redmine to keep track of our projects and such, I loved it so much, I wanted to use it at home for my personal projects. Seriously, if you have never used it, give it a try, you won’t be disappointed. However, there is a problem that I ran into when setting it up at home, my Ruby was too new! At the time of this writing Redmine version 1.2.2 was the current stable, and required Ruby 1.8.6 or 1.8.7. My server came with 1.9.1! I could either downgrade and risk possible incompatibility problems, or I could take another approach. After days of Goggling, I found a solution.

Check out https://rvm.beginrescueend.com/rvm/install/

I recommend the single user install. I did it with no problems. Things get a bit tricky when using RVM, so I decided to create this Quick and Dirty Guide.

How to Install and Setup Redmine with RVM

Be sure to pay attention to directories, I will make notes to help make things easy. Also, I recommend trying this out in a VM or a dev box. Please read the HowTos on redmine.org and do some research and testing before following this guide.

Note: This was done running Slackware 13.1. I do NOT use sudo on my machines. If you get a permissions error you can use sudo if your Linux distro is setup for it. The first part of this I did as the root user. If you use sudo, type ‘sudo su -‘ to become full root. If you see Redmine (with the capital R), I’m referring to the Redmine program or website. Lowercase, is the user redmine.

After downloading the source from redmine.org extract it and rename the folder to redmine, you don’t have to do this, it’s just how I did it.

Setup a Redmine user

This user doesn’t need remote access, or the ability to login directly. We are going to create a new user and group for Redmine to run under.

groupadd redmine
useradd redmine -g redmine -m

Now set permission to where you extracted Redmine to. We are going to give everything to the redmine user and group.

chown redmine:redmine redmine/ -R

Now you need to su – redmine. Make sure you are in the redmine user’s home directory. Usually /home/redmine/

Setup RVM

Now that you are the redmine user and in their home directory, it’s time to install RVM. Be sure to check out their website for full and up to date information. The information listed here maybe out of date!

Lets install RVM for single user use. You are welcome to use the multi-user version, but this way worked just fine for me. Once again, make sure you are in the redmine user’s home directory.

bash < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer )

This will get the installer going. Once it finishes, run this:

echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bash_profile

Now logout. Log back in as the redmine user. and type:

type rvm | head -1

You should get back rvm in a function. If not, something went wrong. Try the commands listed above again. If you get an error about a broken pipe, don’t worry about it. I got it once, but never again. I have no idea why.

If you do get back the good news type:

rvm install 1.8.7

This will install ruby 1.8.7! Once it is complete type:

rvm use 1.8.7

If you don’t get back any errors, type:

rvm use 1.8.7 --default

This will set that ruby version as your default (only for the redmine user!) RVM is now setup. Good job!

Install Some Ruby Gems

Now that RVM is setup and Ruby 1.8.7 is installed, we need to setup some gems for ruby. First step is getting the correct version of RubyGems installed. As per the Redmine documentation, we want version 1.3.7. I find the easiest way of doing this is to grab their .tgz file. Once you get it (you can also find a list of versions available for download here. Extract it, and run setup.rb.

wget http://rubyforge.org/frs/download.php/70696/rubygems-1.3.7.tgz
tar zxf rubygems-1.3.7.tgz
ruby rubygems-1.3.7/setup.rb

It should only take a second or two to install. Once it’s complete there are two more gems that need to be installed before continuing. It is easy and straight forward. Note: If your system appears to freeze when running the commands below, do not worry. It’s working in the background a may take a minute or two.

gem install rails -v=2.3.11
gem install -v=0.4.2 i18n

Setup MySQL

Assuming you are still the redmine user and MySQL is setup and ready to go, these next steps are very easy and were taken right from Redmine’s website. Follow the steps below.

mysql -u root -p
create database redmine character set utf8;
create user 'redmine'@'localhost' identified by 'redpass';
grant all privileges on redmine.* to 'redmine'@'localhost';
flush privileges;

Take note of redpass. Change it to the password you want to use!

Log out of MySQL and stay logged in as the redmine user. Change directory to where the redmine program is stored. For this example, and the rest of the article, I’m going to assume it’s in /srv/redmine

Copy config/database.yml.example to config/database.yml then edit config/database.yml to include your db setup (under production. You can remove the others)

Copy public/dispatch.cgi.example to public/dispatch.cgi then changes first line to the path of the redmine user’s ruby location (/home/redmine/.rvm/bin/ruby in my case)

public/dispatch.cgi needs to have execute permissions. If it does not, type: chmod 755 public/dispatch.cgi

Uncomment NV[‘RAILS_ENV’] ||= ‘production’ in config/environment.rb

Now we need to setup MySQL access. Below are four (4) commands that will do all this including building the MySQL gem. Please note the path to mysql_config. Make sure this matches your setup!

rake generate_session_store
gem install mysql -- --with-mysql-config=/usr/bin/mysql_config
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data

If any of these fail, something may have gone wrong with RVM or MySQL. Check you meet the minimum qualifications listed on Redmine’s website and your paths are correct.

Setup Passenger

Passenger is designed to run as an apache module to keep Ruby of Rails running. If Redmine isn’t used for a while, Ruby of Rails may stop running, it’s not a big deal, it just means it will take a second longer to access your first request. It’s easy to setup, and can be done as the redmine user. So make sure you are still logged in as your redmine user.

I’m running version 3.0.11 or Passenger, you are welcome to do the same. There are only three (3) simple steps involved. So I put them below. Just give it some time to build, it can take a few minutes.

wget http://rubyforge.org/frs/download.php/75548/passenger-3.0.11.tar.gz
tar zxf passenger-3.0.11.tar.gz
./passenger-3.0.11/bin/passenger-install-apache2-module

Once the build finishes there are some lines you need to add to your apache.conf file. They are listed above, they are easy to see. Now, depending on your Linux distro, there are several names and locations for this file. You will need root access to edit the Apache config.

At this point you can stop being logged in as the redmine user. Go back to being root (or your regular user with sudo access)

Setup Apache

This setup worked well for me, I would recommend reading up on Apache config options at http://www.redmine.org/projects/redmine/wiki/HowTo_configure_Apache_to_run_Redmine

The page is a little hard to understand when talking about mod_fcgi and mod_fastcgi. You do NOT need them. They maybe faster, but I find that mod_cgi works just fine.

I’m getting a bit lazy here, and I don’t feel much like going into a lot of explanation on all the options you can use in Apache. Below is a sample of what I used.

<VirtualHost *:80>
   ServerName redmine.<YOUR-DOMAIN>.com
   ServerAdmin webmaster@<YOUR-DOMAIN>.com
   DocumentRoot /srv/redmine/redmine/public/
   ErrorLog /srv/redmine/redmine/log/redmine_error_log
   <Directory "/srv/redmine/redmine/public/">
      Options Indexes ExecCGI FollowSymLinks
      Order allow,deny
      Allow from all
      AllowOverride all
   </Directory>
</VirtualHost>

Obviously there are changes you need to make, like ServerName and ServerAdmin. Note how all the directories start with /srv/redmine/, Update it to where you extracted the Redmine program.

All done!

At this point you should just have to restart apache and you are good to go! Assuming you actually read the Redmine documentation, you can now login as admin (password admin) and give it a go. The first thing I recommend doing is creating a new Admin Level User and removing the default admin. If you get an Internal Error 500 when trying to add a new user, then something is definitely wrong. Check the redmine_error_log and production.log file (both should be in the same directory). If you see something about an error with US_ASCII, then Ruby is not running out of the redmine user’s home directory. Go back through and double check your settings.

I have been told that creating the redmine user in the fashion shown above can create a security risk. As root open /etc/shadow in your favorite editor. At the bottom of the list will be the redmine user. You may see something like:

redmine:!:12585:0:99999:7:::

If you change the exclamation point to an asterisk, it disables the account. That way you can not log into it remotely. Also, if you really want to play it safe, you can edit /etc/passwd also. Just change the last part of the file from /bin/bash to /bin/false. However, by doing this you will not be able to login, or su to that user.

Well?

Did this guide work for you? Do you have any suggestions that may work better? I would love to hear! Feel free to post a comment and let me know how it worked for you, and let me know what Linux distro and version you are running. Thank you.