TangoRangers.com's Blog

As I talked about in a previous post about BackupPC, it is a very powerful tool when is comes to doing self-hosted backups. The downside is when you want to archive out a machine. For example, you have backups of a host, but the host is long gone, and you just want to archive the data. Well there are 2 ways to go about this. Either you can use the web interface to create a restore.tar/zip file to download (which doesn’t always work, especially if done over the internet), or you can create the tar backup on the server, compress, md5, and download it using sftp. I like the second option. Mostly because I’m going through it right now. I have a backup server out in the cloud that I need to archive some 50 hosts from, so here is how I did it.

Simply log into the server and su to the backuppc user and go to where ever you want an archive.

/usr/share/backuppc/bin/BackupPC_tarCreate -h nameOfHost -n -1 -s '/home' / > ./home.tar

In the example above, I’m getting an archive of the home directory for host “nameOfHost”. You can do this for any backed up folder. Once done, you can create an md5sum of the file to help verify you got it downloaded right. You can also bzip2 the file and hopefully make it smaller. Even md5sum that one as well.

Either way, if is a great way to get very large archives created so you don’t have to go through the browser for everything. Feel free to script it, that’s what I did. I was able to start the archive and let it run over the weekend before downloading once the work week started again.

Did this command work for you? Did it not? What did work for you? Please let me know in the comments.

BackupPC is a fantastic tool for backing up all your machines. I use it to back up both Windows and Linux machines. Linux is easy, all you need is SSH and rsync, but Windows is kind of a pain. You need to use Windows shares in most every case. In the future, I’ll talk about how to use Cygwin to use SSH and rsync to backup a Windows machine.

The problem that I have, is there is a bug in Samba versions 3.6 to 4.1 that will cause the tarbackup function to stop the backup before it finished, and BackupPC will report the backup was complete. I haven’t run into this with every Windows machine, but I have in most. Generally what causes this is using another user account to login and perform the backups, instead of using the normal user account. If you backup a Windows machine using the smb method and it appears not everything is being backed up, then this is the guide you want to follow.

To start, I’m currently running Debian 7 (Wheezy) with Samba version 3.6. I tried getting Samba 4.2 to build, but several of my libraries are out of date. If you are currently running 4.0 or 4.1, you might be able to build 4.2 on your server. Otherwise, go with 3.5.22 (being the latest 3.5 series at the time of this writing). (https://bugzilla.samba.org/show_bug.cgi?id=10605)

There are several packages that need to be installed for this to work. Every config is different, but all I have to install was autoconf, make, and gcc.

apt-get install autoconf make gcc

Now we need to download the Samba sources and build it, but not install.

cd /opt
wget https://download.samba.org/pub/samba/stable/samba-3.5.22.tar.gz
tar -zxf samba-3.5.22.tar.gz
cd samba-3.5.22/source3/
./autogen.sh
./configure
make

That was the hard part, if Samba didn’t build correctly, you might be missing other packages. You maybe told what they are, otherwise, Google.

Now set the path for $SmbClientPath to /opt/samba-3.5.22/source3/bin/smbclient. You can either change the $SmbClientPath in your backuppc config, or just change it for hosts that are having issues. If you are reading this, I’m going to assume you know how to do that.

Now test (this will do a FULL backup, so it can take some time):

/usr/share/backuppc/bin/BackupPC_dump -v -f <computer name>

You can watch as it goes along (note that you will NOT see it running in the GUI). This can take some time, but when complete you will have an idea of if everything worked or not.

Did this work for you? Did you build Samba 4.2 or newer? Let me know in the comments.

Recently I upgraded some on my servers to use PFS. For those not familiar with PFS please read https://en.wikipedia.org/wiki/Forward_secrecy. Now I must admit, I have a strange setup of servers. I have a set of servers for each client and each performs a different function. One is an app server that can be accessed by client.company.com. These servers run nginx. I Also have a Nagios and Backuppc server that are accessed by nagios.custom.company.com. I purchased a wildcard SSL certificate for *.company.com. Before you say anything, I realize that using multilevel subdomains is not complaint with wildcard certs (see RFC 2818 and RFC 2459). Nagios is ONLY for me, not the customer, so I don’t care about the SSL warnings. To help illustrate:

customer.company.com -> nginx
nagios.customer.company.com -> Apache

Unfortunately, I’m pretty locked down on what OS I can run, and currently it is Ubuntu 12.04 (Yes, 14.04 is out, but it has too many issues and our software don’t run stable under it). Which means I have nginx 1.6 and Apache 2.2. Nginx 1.6 supports PFS, which I implemented without an issue, while Apache 2.2 does… but doesn’t. It appears to be some hack job by Ubuntu (Version 2.2.22 doesn’t have PFS, but 2.2.27 appears to have it. As we know, Ubuntu will back-port patches, and it looks like a patch got in to give semi-support to PFS, I decided not to use it).

Here is the interesting thing. With this setup, whether I setup PFS on Apache or not, I got the same results. The instant I setup PFS on nginx, I could no longer use the Nagios server on the Apache machine. In Firefox I would get an SSL error that I couldn’t bypass. No matter what, it wouldn’t work, even with other browsers.

So, I did a test. What would happen if I followed the rules of using wildcard SSL certs? So I changed the Nagios server to work under nagios-custom.company.com, and guess what? It actually worked!

I have no idea how this is. It seems that the browser is remembering something about the certificate, because it is the same one used on all servers. I tested this by having a machine connect just to the nagios server (before the URL change), and it worked until I accessed the app server with the same browser.

I’m now following the rules on wildcard SSL certs, and my naming is as follows

customer.company.com -> nginx
nagios-customer.company.com -> Apache

Now everything works, even though I’m not using PFS on the Apache servers.

If you want to setup PFS, check out this page, https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy, as it has great information. Be sure to use https://www.ssllabs.com/ssltest/ to test your server.

Lastly, I put this up in the hopes that is anyone else runs into this issue it will provide some insight to resolving the issue. If you have any additional information, please leave a note in the comments below (You don’t even have to give a real email, if you do, I won’t spam you or anything).

Like many of you, I’m concerned about security, especially when working remotely. Generally I would simply create a tunnel using SSH, but then I must set all my programs to use the socks5 tunnel. This isn’t always possible without first opening the program, which will generally try to form a connection. Perhaps, not the best way to keep safe on a network you don’t trust (like a coffee shop).

Unlike using SSH to create a secured tunnel, which requires setting proxy settings for all your programs, using something like OpenVPN you can redirect all your traffic through the encrypted tunnel without having to configure anything. All thanks to using iptables.

Here is my quick and dirty guide on getting your very own OpenVPN server setup on Linux, as well as setup for two types of clients. One being a Linux client, the other being Cyanogenmod’s Android.

With this guide, I’m going to assume you already have OpenVPN installed and ready to go. Also that the configuration files are in /etc/openvpn/

Server Setup

First off, we need to generate some keys. This will be used to secure the connection. OpenVPN comes with all the tools you need to generate keys and indexes. Look for the easy-rsa directory that comes with OpenVPN. In my case, it’s in /usr/doc/openvpn-2.2.2/easy-rsa/2.0/

In that directory you will see a lot of scripts. Before doing anything you need to edit the file vars. In this file are several settings. Most important is with dealing with the openssl key. Here is a quick example you can base your configuration off of with all the comments removed.

export EASY_RSA="`pwd`"
export OPENSSL="openssl"
export PKCS11TOOL="pkcs11-tool"
export GREP="grep"
export KEY_CONFIG=`$EASY_RSA/whichopensslcnf $EASY_RSA`
export KEY_DIR="/etc/openvpn/keys"
export PKCS11_MODULE_PATH="dummy"
export PKCS11_PIN="dummy"
export KEY_SIZE=1024
export CA_EXPIRE=3650
export KEY_EXPIRE=3650
export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="City"
export KEY_ORG="domain name"
export KEY_EMAIL="emailaddress@domain"

Note the export KEY_DIR. This is important. You will get warnings about running ./clean-all. This will delete ALL your keys.

After editing the vars file, we need to execute it to store the values in memory, then clean the keys directory. Do so by running:

. vars
./clean-all

Yes, you read that right, period, space, vars.

Now we are going to generate keys for the server and two clients.

For the server, we just need to run a couple of quick and easy commands.

./build-ca
./build-dh
./build-key-server server

The last command will build a server.key file. This is needed when running the server for key exchanges and such.

Now there are 3 different ways to build keys for clients.
./build-key client (no password protection, not recommended)
./build-key-pass client (with password protection, recommended)
./build-key-pkcs12 client (PKCS #12 format, good for Android)

For the client configuration. I’m not sure if you can use the PKCS #12 format. I haven’t tried, but if it works for you, please let me know.

Now we need to edit /etc/openvpn/openvpn.conf for our network setup. Most of the config files are self explanatory. Here is my example:

cd /etc/openvpn #yes, you do need this for some damn reason
local localIP
proto udp
port 1194
comp-lzo
verb 3
log-append /var/log/openvpn.log
dev tun0
persist-tun
persist-key
server 172.16.1.0 255.255.255.0
ifconfig-pool-persist /var/log/ipp.txt
client-to-client
push "route 10.0.0.0 255.255.255.0"
push "dhcp-option DNS 10.0.0.1"
push "dhcp-option DOMAIN domain.tld"
push "redirect-gateway def1"
keepalive 10 120
cipher BF-CBC
ca keys/ca.crt
dh keys/dh1024.pem
key keys/server.key
user nobody
group nobody
status /var/log/openvpn-status.log

Be sure to change localIP to the server’s IP address AND (if applicable) forward UDP port 1194 to the server.

NOTE: There is one issue I have run into. By using the option push “redirect-gateway def1” does seem to work fine and redirect all through the VPN, I have an issue getting the DNS and DOMAIN to work through both the OpenVPN software or my Android. This means that all DNS queries do not appear to be going through the VPN. This may not be the case. I have yet to setup a packet sniffer to check. So for the time being, I simply created a bash script that will edit my /etc/resolv.conf file when I start the VPN, and revert it back when done. If someone knows of a really easy way to check without having to use a sniffer, please let me know.

Now that all of the keys are built, and the openvpn.conf file is setup, we are ready to start the server. While I have run into some strange behavior in my configuration, you may have better luck in yours. In mine, I had to create the device tun edit ip_forward and manually configure the IP tables.

Here is my simple script I run on the server what I want to have the OpenVPN server up and running (yes, I do this at boot). Explanation of items below.

mkdir /dev/net
mknod /dev/net/tun c 10 200
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I FORWARD -i tun0 -o eth0 -s 172.16.1.0/24 -d 10.0.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i tun0 -o eth0 -s 172.16.1.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i eth0 -o eth0 -s 10.0.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o eth0 -s 172.16.1.0/24 -j MASQUERADE
iptables -t nat -I POSTROUTING -o eth0 -s 10.0.0.0/24 -j MASQUERADE
openvpn --config /etc/openvpn/openvpn.conf --cert /etc/openvpn/keys/server.crt &

Most places I have found this stuff are not very specific about IPs. So let me give you a quick rundown on each item.

First we create the device with some special settings. That is the mkdir /dev/net (if /dev/net already exists, it will do nothing), then mknod /dev/net/tun c 10 200. Then set ip_forward to true. The fun part is with the iptables.

So in my example, tun0 is the virtual device that is the VPN and eth0 is my ethernet. 172.16.1.0/24 is the IP range I’m giving to the VPN (tun0), and my physical network is 10.0.0.0/24. You can leave the VPN network on the 172.16.1.0/24 network, simply adjust the 10.0.0.0/24 to your networking configuration (ie 192.168.0.0/24). How all those iptables work… yea, I’m not going into it. They work, I’m fine with that.

After running those commands, your OpenVPN server should be up and running. The final process is background so you get your terminal back. Wait a few seconds and hit enter again. If you don’t see the process has ended, then you have done everything correctly. If it did error, check /var/log/openvpn.log for information on what is causing the problem.

Client Configuration

Now that the server is setup, lets get the client side going. This part will be for the OpenVPN software running on Linux. See the next section for CyanogenMod’s Android.

This part is much easier than the server setup, but you need to get your keys to the client. I highly recommend you do with via scp. You will need ca.crt, client.crt, and client.key. Assuming you called your keys “client”. Put these files in /etc/openvpn/keys. Then create the file /etc/openvpn/openvpn.conf and put this in it.

remote IP/DNS 1194
proto udp
dev tun  
cd /etc/openvpn/
ca keys/ca.crt
cert keys/client.crt
key keys/client.key
client
ns-cert-type server
keepalive 10 120
comp-lzo 
user nobody
group nobody
persist-key
persist-tun
status /var/log/openvpn-status.log

Change IP/DNS to the IP or DNS name your server is reachable at. You should now be able to connect to your OpenVPN server by typing:

openvpn --config /etc/openvpn/openvpn.conf

That’s pretty much it. Once you get a handle on the settings, it is actually pretty easy. However, as mentioned before. I have found a possible issue with DNS. I would highly recommend editing /etc/resolv.conf to point to your DNS server. In my example, the DNS server is also at the gateway (10.0.0.1). You can script this. In fact, use my script.

#!/bin/bash
pid=`pgrep openvpn`
if [ -z "$pid" ]; then
echo "Starting OpenVPN Client"
cp /etc/resolv.conf /etc/resolv.conf.backup
echo "nameserver 10.0.0.1" > /etc/resolv.conf
openvpn --config /etc/openvpn/openvpn.conf &
else
echo "Stopping OpenVPN Client"
mv /etc/resolv.conf.backup /etc/resolv.conf
kill $pid
fi

Pretty strait forward if I do say so myself. You may have an issue if you have a passphrase on your key! If you are having an issue, remove the ampersand (&) from the end of the openvpn –config line. This will not background the process, but you can do it manually by typing ctrl+z then bg which will background the process.

CyanogenMod’s Android Configuration

Because I don’t run the Android that came with my phone, I can use OpenVPN with ease. If you are not running a custom rom, you can still run OpenVPN by getting the client software from the Android Market (now called the Play Store). The following instructions are for CyanogenMod 7.2, but should work in newer versions just fine.

Remember when you made your client key? Well you need to make one that works great with Android. It’s the PKCS #12 format. This will give you a file that ends in a .p12 extension. Copy this file over to the root of your sdcard.

Install the certificate by going to Settings->Location & Security->Install from SD card (under Credential storage at the bottom on the menu). It should find the file and ask for the password to unlock it. Then it will ask for a new password (you can use the same one as before) and you can also give it a custom name.

Build the client by going to Settings->Wireless & Networks->VPN Settings->Add VPN. You just need to select the OpenVPN type. In the new menu there are several settings.

VPN name (this can be anything you want)
Set VPN server (the IP or domain name of the server)
User authentication (leave unchecked)
Set CA certificate (click this and select the key you just installed)
Set user certificate (same as above)
DNS search domains (these are optional, but you can set 10.0.0.1 like in the bash script above)

Hit the menu button then Advanced.

Server port (default is 1194)
Protocol to use (udp is default)
Device to use (tun, which is fine)
LZO compression (check it!)
Redirect gateway (check it!)
Remote Sets Addresses (Should also be checked)

Everything below that I left as default. You do NOT need to enable TLS-Auth. For this type of setup it is unnecessary.

Hit back, then save. From here you should be able to connect to your VPN. Note that in my tests, the VPN is much slower. I’m not sure if it is something I have done wrong in my setup, or if my provider throttles VPNs.

Conclusion

Everything should be up and running now. I hope you found this useful. Please feel free to leave a comment below. If you have any suggestions or questions you can drop those below as well. I’m not an expert on OpenVPN, I just like learning.

Sources:
http://openvpn.net/index.php/open-source/documentation/miscellaneous/77-rsa-key-management.html
http://openvpn.net/index.php/open-source/documentation/howto.html
http://blog.johnford.org/openvpn-tunnel-to-home-server/

Like many of you, I run my own SVN repository. I have several projects and several people that use them. So I wanted a quick and easy way to perform backups. Here are the two methods I use.

The first is an incremental backup where I use a post-commit hook. The other is a bash script that I setup in cron to run once per week (you can do daily, monthly, or when ever. It depends on how quickly you want the full backup script to run). The bash script is designed to create a full backup using svnadmin dump.

The idea of a full backup is nothing new, but the way my script works is a little different than others. Their script just creates a new backup when ever run. Mine creates a full backup of every revision number. Even if there have been several updates since the last backup.

Lets get started.

Incremental backup using a post-commit

Post-commit hooks are very powerful. You can do a lot with them. In this case we are going to do an incremental backup. I don’t do a full because my repository is very large and I don’t want to be slowed down every time I commit new files.

First you need to go to the directory where your repository is. Lets call it /srv/svn/myproject. In that directory there is a folder called hooks. Looks for post-commit (it maybe listed as post-commit.tmpl, just rename it without the .tmpl). Open the file in an editor (vi, nano, emacks) and add this line to the end.

svnadmin dump "$REPOS" --revision "$REV" --incremental >/srv/backups/myproject/incremental/commit-$REV 2>> /srv/backups/myproject/incremental/backup.log

NOTE: If mailer.py is not commented out, put a hash in front of it “#”. You don’t need it.

Take note of the directories I’m using. Just change them to where your backups will go. Also, you need to give those directories the same ownership as your svn archive! Otherwise when the post-commit runs, it will error out because it cannot write to the new location.

And that is it! Now every commit you make will create a new incremental backup.

There is one down side to this method. If you already have several commits, you will not get your entire history. I did this a lazy way since at the time I was only up to around 20 commits.

./post-commit /srv/svn/myproject 0

This will create the backup starting at revision 0. Just keep running it and moving the number up. If you have hundreds or even thousands of commits… you might want to write something to do all that hard work for you. You might even get some ideas from the next section.

Full backup

This is where I’m different from everyone else… at least that I could find. While this is not the most elegant way of doing things, it really doesn’t take very long to run (unless there are many revisions to create new archives for.

#!/bin/bash
 
svnLocale=/srv/svn/myproject/
backupLocale=/srv/backups/svn/myproject/full/
fileName=myproject.rev.
extension=.svndump.bz2
 
latestRevision=`svnlook youngest $svnLocale`
let stopCounting=latestRevision+1
COUNTER=0
while [ $COUNTER != $stopCounting ]; do
        rev=$COUNTER
        # check to see if file exists
        if [ ! -e $backupLocale$fileName$rev$extension ]; then
                svnadmin dump $svnLocale -r $rev -q > $backupLocale$fileName$rev.svndump
                bzip2 -z9q $backupLocale$fileName$rev.svndump
        fi
        let COUNTER=COUNTER+1
done

Seems a bit much, but all you would need to worry about is changing svnLocale, backupLocale, and fileName. You can change extension if you wish, but I would leave it unless you plan on changing the svnadmin or bzip2.

Please note that the first time you run this, nothing is dumped on the screen. I ran this on an archive with 1215 revisions as a test. I went to lunch. You can modify the script and remote the -q from both svnadmin and bzip2 to see your progress. I silence them while running as a cron job.

The benefit to this script is it does check to see if an archive has been created for each revision. If not, it gets created. This script will even get backups you may have deleted and recreate them, then continue on. It doesn’t matter. It’s so simple that it just works.

So that’s pretty much it. Questions? Comments? Let me know what you think.