TangoRangers.com's Blog

Like many of you, I run my own SVN repository. I have several projects and several people that use them. So I wanted a quick and easy way to perform backups. Here are the two methods I use.

The first is an incremental backup where I use a post-commit hook. The other is a bash script that I setup in cron to run once per week (you can do daily, monthly, or when ever. It depends on how quickly you want the full backup script to run). The bash script is designed to create a full backup using svnadmin dump.

The idea of a full backup is nothing new, but the way my script works is a little different than others. Their script just creates a new backup when ever run. Mine creates a full backup of every revision number. Even if there have been several updates since the last backup.

Lets get started.

Incremental backup using a post-commit

Post-commit hooks are very powerful. You can do a lot with them. In this case we are going to do an incremental backup. I don’t do a full because my repository is very large and I don’t want to be slowed down every time I commit new files.

First you need to go to the directory where your repository is. Lets call it /srv/svn/myproject. In that directory there is a folder called hooks. Looks for post-commit (it maybe listed as post-commit.tmpl, just rename it without the .tmpl). Open the file in an editor (vi, nano, emacks) and add this line to the end.

svnadmin dump "$REPOS" --revision "$REV" --incremental >/srv/backups/myproject/incremental/commit-$REV 2>> /srv/backups/myproject/incremental/backup.log

NOTE: If mailer.py is not commented out, put a hash in front of it “#”. You don’t need it.

Take note of the directories I’m using. Just change them to where your backups will go. Also, you need to give those directories the same ownership as your svn archive! Otherwise when the post-commit runs, it will error out because it cannot write to the new location.

And that is it! Now every commit you make will create a new incremental backup.

There is one down side to this method. If you already have several commits, you will not get your entire history. I did this a lazy way since at the time I was only up to around 20 commits.

./post-commit /srv/svn/myproject 0

This will create the backup starting at revision 0. Just keep running it and moving the number up. If you have hundreds or even thousands of commits… you might want to write something to do all that hard work for you. You might even get some ideas from the next section.

Full backup

This is where I’m different from everyone else… at least that I could find. While this is not the most elegant way of doing things, it really doesn’t take very long to run (unless there are many revisions to create new archives for.

#!/bin/bash
 
svnLocale=/srv/svn/myproject/
backupLocale=/srv/backups/svn/myproject/full/
fileName=myproject.rev.
extension=.svndump.bz2
 
latestRevision=`svnlook youngest $svnLocale`
let stopCounting=latestRevision+1
COUNTER=0
while [ $COUNTER != $stopCounting ]; do
        rev=$COUNTER
        # check to see if file exists
        if [ ! -e $backupLocale$fileName$rev$extension ]; then
                svnadmin dump $svnLocale -r $rev -q > $backupLocale$fileName$rev.svndump
                bzip2 -z9q $backupLocale$fileName$rev.svndump
        fi
        let COUNTER=COUNTER+1
done

Seems a bit much, but all you would need to worry about is changing svnLocale, backupLocale, and fileName. You can change extension if you wish, but I would leave it unless you plan on changing the svnadmin or bzip2.

Please note that the first time you run this, nothing is dumped on the screen. I ran this on an archive with 1215 revisions as a test. I went to lunch. You can modify the script and remote the -q from both svnadmin and bzip2 to see your progress. I silence them while running as a cron job.

The benefit to this script is it does check to see if an archive has been created for each revision. If not, it gets created. This script will even get backups you may have deleted and recreate them, then continue on. It doesn’t matter. It’s so simple that it just works.

So that’s pretty much it. Questions? Comments? Let me know what you think.

Do you want to use SVN to update your website? This guide will help you and includes information not found anywhere else on how to get it working. Many websites give basic information and comments complain about it not working. If those other website didn’t work for you. This will.

After digging around for hours and seeing everyone complain about having the same problems after being told the same solutions, I decided to figure it out for myself. If you read this carefully you too will have a website setup that can be updated via subversion.

Getting Started

I’m going to assume that your SVN server and website are on the same machine and that you are using Apache to serve both. I’m also going to assume that you are somewhat familiar with SVN and Apache. I’m not going to go into much detail on how to get everything setup from scratch.

The tools you will need to have installed are as follows (may need more depending on your configuration). Apache HTTPd, Subversion, and gcc.

Many Linux users and distributions use sudo. I do not, and I will not provide information on how to use sudo. My recommendation, before doing anything, is to type sudo su which will make you root. Do everything from there so you don’t have to sudo anymore.

Setup a new SVN repository

Alright, now we are going to setup the new SVN repository. This can be done easily with:

svnadmin create <path to repository>

Place it in a good location that you will remember since you need to point Apache to it. In my examples we are going to use /home/www/domain/svn/website/ for our project location and /home/www/domain/htdocs/ for our website.

For user access controls I use a 2 part process. I have an authz and passwd file in the root of my SVN directory (/home/www/domain/svn/). The reason I do this is because I host several SVN projects and I don’t want everyone to have access to everything. This way I can also setup (this comes later) Apache to have read-only access to the repository.

First, we need to create our authz and passwd files. Lets call them svn.authz and svn.passwd so we know what they are for.

In the svn.authz file we need something like this:

[groups]
project = dkun
other1 = dkun, someguy
website = dkun, friend1
readonly = redmine,apache
 
[project:/]
@project = rw
@readonly = r
* =
 
[other1:/]
@other1 = rw
* =
 
[website:/]
@website = rw
@readonly = r
* =

Looks a bit overwhelming. Let me explain.

Starting from the top, we have [groups]. These are groups of users that we give project names to. Notice that 2 of the projects have more that one user. There is also a read-only group. You can name the groups what every you want. I do it by project name to make it easy to read.

After that you see [project:/]. This is the project group. It has one user and is set to read/write. It also has a readonly group. This way I can setup both Apache and Redmine with read only access.

Lastly you might have noticed *=. What is that? It turns off anonymous access. So unless your project has code going out to the web, make sure to put this in.

Now we need to setup the svn.passwd file. This is pretty easy. Just type:

htpasswd svn.passwd <username>

You will be prompted for the password. That’s it! Pretty easy right? Just don’t forget to setup the user apache (keep it lower case, trust me), and any others you might need.

Setup Apache

Now that we have SVN and out new project setup, we need to configure Apache to serve everything. Setup your VirtualHost to point to where you want your website to be hosted from. So for me it would be /home/www/domain/htdocs/. I’m going to assume you already know how to do this. All you need to add is a little directive.

<DirectoryMatch "^/.*/\.svn/">
     Order deny,allow
     Deny from all
</DirectoryMatch>

This will stop someone from browsing your .svn folder on the website.

Now setup an SVN VirtualHost. This gets a bit more complicated. I’m going to avoid going into detail here. I want to assume you have some background.

In your Virtual host we need to set a few options and point to the SVN repository root. Here is an example of the options you need to put inside the VirtualHost section for SVN to work. Don’t forget to set other items you need!

<Location /svn>
   DAV svn
   SVNParentPath /home/www/domain/svn
   AuthType Basic
   AuthName "My SVN Repository"
   AuthUserFile /home/www/domain/svn/svn.passwd
   AuthzSVNAccessFile /home/www/domain/svn/svn.authz
   Require valid-user
</Location>
<Directory /home/www/domain/svn>
   Options +Indexes FollowSymLinks +ExecCGI
   AllowOverride AuthConfig FileInfo
   Order allow,deny
   Allow from all
</Directory>

Note all the directories. Change them to match your server. Also, verify that your SVN modules are enabled in Apache. They should be in your apache.conf or httpd.conf (depending on your server). You will need to enable:

LoadModule dav_svn_module lib/httpd/modules/mod_dav_svn.so
LoadModule authz_svn_module lib/httpd/modules/mod_authz_svn.so

That’s it! Now type apachectl configtest and you will get messages back if there is a problem in your Apache config. If it is a warning about your website directory missing, don’t worry. Just create the directory and give full ownership to the apache user (We will get back to this in a moment).

Before continuing, restart Apache with apachectl restart and make sure your SVN works! If it does not then the next section will fail.

Do some weird stuff with Apache

Here is where things get a little weird. This is also where 99% of people fail because there is no good information out there. I will do my best to explain what to do and why you should do it.

The first step is you need to find out what user and group Apache runs under. For me, both are apache, but for you it could use www or www-user. You can find this by checking /etc/passwd and /etc/group, or by going through your apache.conf file and look for User and Group directives.

Now look in /etc/passwd for the apache user. The second to last option shows Apache’s home directory. In my case it is /srv/httpd. Go to that directory and type:

chown apache.apache ./

Set the user and group to which ever Apache runs under. This will allow Apache to create new directories there without changing ownership of anything else. Also, that is just one (1) period before the slash!

Open /etc/passwd and find the apache user again. At the end of the line you should see something like /bin/false. This keeps the Apache user from logging in. This is for security reasons. We are going to temporarily change it to /bin/bash.

Now that you are already root, type:

su – apache

Once again I assume your Apache’s user is apache. Adjust as needed.

Now you are logged in as the Apache user. You should see a new shell and be ready to go. Make sure you are in the /var/httpd directory (as stated previously as being Apache’s home directory) by typing pwd. If you are, then you have created a new session and are ready for the next step.

Navigate to the directory where the website will be served from. So if the website is in /home/www/domain/htdocs/ then go one level up; /home/www/domain/. Now we are going to perform an SVN checkout and have subversion remember our credentials.

svn co http://<domain>/svn/website

Adjust the http directive as needed. If you use SSL have subversion permanently accept the certificate. You should see a prompt for apache’s password. This is the password you created for it in svn.passwd. Type it in and when asked if you want to store the password, say yes!

If everything went as planned then you should see a successful checkout as revision 0. If not… well you did something wrong. Double check your paths, apache config files, and passwords. If needed, you may need to give ownership of the directory to Apache so that user can create a new directory and write to it.

Check to make sure a .subversion directory was created in Apache’s home directory and that there is a .svn folder in /home/www/domain/htdocs/. If now you need to give ownership of that directory to the Apache user (you will have to be root for this). If there is so .svn directory, then change ownership of the htdocs directory to Apache and run the checkout again.

If everything went smoothly then we are done as the Apache user. Type logout to return to being root.

Last thing. Open up /etc/passwd and change /bin/bash back to /bin/false. No need to have a security hole.

Create an executable

Now that we have SVN, Apache, and a SVN checkout setup, we can finally finish this. The first thing you need to do is find where svn is located:

whereis svn
svn: /usr/bin/svn /usr/man/man1/svn.1.gz /usr/share/man/man1/svn.1.gz

As you can see, I used whereis to find the svn executable. It’s the first one, /usr/bin/svn.

Navigate over to the htdocs directory. Open up your favorite text editor and dump this in it:

#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
int main(void)
{
   execl("/usr/bin/svn", "svn", "update", "/home/www/domain/htdocs/",
      (const char *) NULL);
   return(EXIT_FAILURE);
}

See where the directives of /usr/bin/svn and /home/www/domain/htdocs/ are? Just change those to match your server. Save the file as update.c and just back to the shell.
Now we need to compile the program:

cc -o update update.c

This will create an executable called update. You can call it what ever you want, just as long as before renaming it you type chown apache.apache update; chmod +s update. It should be owned by the Apache user.

Setup post-commit

Just over to your SVN repository, then into the hooks directory. So if you were following, cd /home/www/domain/svn/website/hooks. There you will see a bunch of .tmpl files. Rename of copy post-commit.tmpl to post-commit, then chmod 755 post-commit.

cp post-commit.tmpl post-commit
chmod 755 post-commit

Open post-commit in your editor. You may see a bunch of stuff in there. Just put a hash (#) sign in front of any other commands there. They are examples. Add:

/home/www/domain/htdocs/update

That’s it. Don’t add sh, or bash to the front, just put in the path with executable at the end. Save and exit.

Try it!

Now, as long as you followed along and I didn’t skip anything. This should work. Try it out. Try committing something to SVN and see what happens! If you can view the website right away, they you did it correctly. If now, you should check your apache log files. Specifically error_log. If SVN threw an error you will have to see what went wrong there. Sometimes just putting a commit message in will fix any error.

That’s it. I hope you found this informative. Please feel free to drop my a comment below if you have anything you wish to add or just want to say thanks for the info. I appreciate all feedback. Thanks for reading!