Things learned while using Perfect Forward Secrecy

Recently I upgraded some on my servers to use PFS. For those not familiar with PFS please read Now I must admit, I have a strange setup of servers. I have a set of servers for each client and each performs a different function. One is an app server that can be accessed by These servers run nginx. I Also have a Nagios and Backuppc server that are accessed by I purchased a wildcard SSL certificate for * Before you say anything, I realize that using multilevel subdomains is not complaint with wildcard certs (see RFC 2818 and RFC 2459). Nagios is ONLY for me, not the customer, so I don’t care about the SSL warnings. To help illustrate: -> nginx -> Apache

Unfortunately, I’m pretty locked down on what OS I can run, and currently it is Ubuntu 12.04 (Yes, 14.04 is out, but it has too many issues and our software don’t run stable under it). Which means I have nginx 1.6 and Apache 2.2. Nginx 1.6 supports PFS, which I implemented without an issue, while Apache 2.2 does… but doesn’t. It appears to be some hack job by Ubuntu (Version 2.2.22 doesn’t have PFS, but 2.2.27 appears to have it. As we know, Ubuntu will back-port patches, and it looks like a patch got in to give semi-support to PFS, I decided not to use it).

Here is the interesting thing. With this setup, whether I setup PFS on Apache or not, I got the same results. The instant I setup PFS on nginx, I could no longer use the Nagios server on the Apache machine. In Firefox I would get an SSL error that I couldn’t bypass. No matter what, it wouldn’t work, even with other browsers.

So, I did a test. What would happen if I followed the rules of using wildcard SSL certs? So I changed the Nagios server to work under, and guess what? It actually worked!

I have no idea how this is. It seems that the browser is remembering something about the certificate, because it is the same one used on all servers. I tested this by having a machine connect just to the nagios server (before the URL change), and it worked until I accessed the app server with the same browser.

I’m now following the rules on wildcard SSL certs, and my naming is as follows -> nginx -> Apache

Now everything works, even though I’m not using PFS on the Apache servers.

If you want to setup PFS, check out this page,, as it has great information. Be sure to use to test your server.

Lastly, I put this up in the hopes that is anyone else runs into this issue it will provide some insight to resolving the issue. If you have any additional information, please leave a note in the comments below (You don’t even have to give a real email, if you do, I won’t spam you or anything).

Another way to limit SSH key access to a specific IP

There are a couple ways to limit where users can and cannot use SSH keys for quick access to systems. The most common I have found is to put access requirements in the sshd_config file. While this is a great method, and has quite a few ways to customize it. What if you only want to limit access for one user? Or maybe you don’t have root/sudo access on the machine and you want to limit your own account. Well, these might be the best examples, but here is another nifty way to limit access with SSH keys. Please note that this will only block access with the key, but will still allow you to provide a password!

Lets get started. I’m going to assume you already have generated your key and used ssh-copy-id to get it to your remote host. SSH into your remote host and open your user’s authorized_keys file. This is usually in ~/.ssh/. You are going to see something like this.

ssh-rsa AAAAB3Nza...LiPk== user@host

Simply add from=”host” in front of it and you are set! Seriously, it is that easy.

from="" ssh-rsa AAAAB3Nza...LiPk== user@host

The best part here is you don’t need to do anything for the changes to take effect. No restarting sshd or even logging out of the machine! It just works right away.

You can add other options in authorized_keys and not just limit by IP. What if your IP changes? Maybe you are always in a particular domain range somewhere, perhaps you want to force a VPN connection before allowing SSH (I’m just throwing out ideas here). I use it for my backup system. I allow root access only from that one machine and otherwise you can’t login via root at all. I only allow the one key and no password. So right there is a good use.

To see more options and additional syntax, open a terminal and type “man authorized_keys” and scroll down to the section titled Authorized_keys File Format. There is a lot of good information there.

Quick and dirty guide to OpenVPN on Slackware Linux and Android

Like many of you, I’m concerned about security, especially when working remotely. Generally I would simply create a tunnel using SSH, but then I must set all my programs to use the socks5 tunnel. This isn’t always possible without first opening the program, which will generally try to form a connection. Perhaps, not the best way to keep safe on a network you don’t trust (like a coffee shop).

Unlike using SSH to create a secured tunnel, which requires setting proxy settings for all your programs, using something like OpenVPN you can redirect all your traffic through the encrypted tunnel without having to configure anything. All thanks to using iptables.

Here is my quick and dirty guide on getting your very own OpenVPN server setup on Linux, as well as setup for two types of clients. One being a Linux client, the other being Cyanogenmod’s Android.

With this guide, I’m going to assume you already have OpenVPN installed and ready to go. Also that the configuration files are in /etc/openvpn/

Server Setup

First off, we need to generate some keys. This will be used to secure the connection. OpenVPN comes with all the tools you need to generate keys and indexes. Look for the easy-rsa directory that comes with OpenVPN. In my case, it’s in /usr/doc/openvpn-2.2.2/easy-rsa/2.0/

In that directory you will see a lot of scripts. Before doing anything you need to edit the file vars. In this file are several settings. Most important is with dealing with the openssl key. Here is a quick example you can base your configuration off of with all the comments removed.

export EASY_RSA="`pwd`"
export OPENSSL="openssl"
export PKCS11TOOL="pkcs11-tool"
export GREP="grep"
export KEY_CONFIG=`$EASY_RSA/whichopensslcnf $EASY_RSA`
export KEY_DIR="/etc/openvpn/keys"
export PKCS11_MODULE_PATH="dummy"
export PKCS11_PIN="dummy"
export KEY_SIZE=1024
export CA_EXPIRE=3650
export KEY_EXPIRE=3650
export KEY_CITY="City"
export KEY_ORG="domain name"
export KEY_EMAIL="emailaddress@domain"

Note the export KEY_DIR. This is important. You will get warnings about running ./clean-all. This will delete ALL your keys.

After editing the vars file, we need to execute it to store the values in memory, then clean the keys directory. Do so by running:

. vars

Yes, you read that right, period, space, vars.

Now we are going to generate keys for the server and two clients.

For the server, we just need to run a couple of quick and easy commands.

./build-key-server server

The last command will build a server.key file. This is needed when running the server for key exchanges and such.

Now there are 3 different ways to build keys for clients.
./build-key client (no password protection, not recommended)
./build-key-pass client (with password protection, recommended)
./build-key-pkcs12 client (PKCS #12 format, good for Android)

For the client configuration. I’m not sure if you can use the PKCS #12 format. I haven’t tried, but if it works for you, please let me know.

Now we need to edit /etc/openvpn/openvpn.conf for our network setup. Most of the config files are self explanatory. Here is my example:

cd /etc/openvpn #yes, you do need this for some damn reason
local localIP
proto udp
port 1194
verb 3
log-append /var/log/openvpn.log
dev tun0
ifconfig-pool-persist /var/log/ipp.txt
push "route"
push "dhcp-option DNS"
push "dhcp-option DOMAIN domain.tld"
push "redirect-gateway def1"
keepalive 10 120
cipher BF-CBC
ca keys/ca.crt
dh keys/dh1024.pem
key keys/server.key
user nobody
group nobody
status /var/log/openvpn-status.log

Be sure to change localIP to the server’s IP address AND (if applicable) forward UDP port 1194 to the server.

NOTE: There is one issue I have run into. By using the option push “redirect-gateway def1” does seem to work fine and redirect all through the VPN, I have an issue getting the DNS and DOMAIN to work through both the OpenVPN software or my Android. This means that all DNS queries do not appear to be going through the VPN. This may not be the case. I have yet to setup a packet sniffer to check. So for the time being, I simply created a bash script that will edit my /etc/resolv.conf file when I start the VPN, and revert it back when done. If someone knows of a really easy way to check without having to use a sniffer, please let me know.

Now that all of the keys are built, and the openvpn.conf file is setup, we are ready to start the server. While I have run into some strange behavior in my configuration, you may have better luck in yours. In mine, I had to create the device tun edit ip_forward and manually configure the IP tables.

Here is my simple script I run on the server what I want to have the OpenVPN server up and running (yes, I do this at boot). Explanation of items below.

mkdir /dev/net
mknod /dev/net/tun c 10 200
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I FORWARD -i tun0 -o eth0 -s -d -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i tun0 -o eth0 -s -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -i eth0 -o eth0 -s -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o eth0 -s -j MASQUERADE
iptables -t nat -I POSTROUTING -o eth0 -s -j MASQUERADE
openvpn --config /etc/openvpn/openvpn.conf --cert /etc/openvpn/keys/server.crt &

Most places I have found this stuff are not very specific about IPs. So let me give you a quick rundown on each item.

First we create the device with some special settings. That is the mkdir /dev/net (if /dev/net already exists, it will do nothing), then mknod /dev/net/tun c 10 200. Then set ip_forward to true. The fun part is with the iptables.

So in my example, tun0 is the virtual device that is the VPN and eth0 is my ethernet. is the IP range I’m giving to the VPN (tun0), and my physical network is You can leave the VPN network on the network, simply adjust the to your networking configuration (ie How all those iptables work… yea, I’m not going into it. They work, I’m fine with that.

After running those commands, your OpenVPN server should be up and running. The final process is background so you get your terminal back. Wait a few seconds and hit enter again. If you don’t see the process has ended, then you have done everything correctly. If it did error, check /var/log/openvpn.log for information on what is causing the problem.

Client Configuration

Now that the server is setup, lets get the client side going. This part will be for the OpenVPN software running on Linux. See the next section for CyanogenMod’s Android.

This part is much easier than the server setup, but you need to get your keys to the client. I highly recommend you do with via scp. You will need ca.crt, client.crt, and client.key. Assuming you called your keys “client”. Put these files in /etc/openvpn/keys. Then create the file /etc/openvpn/openvpn.conf and put this in it.

remote IP/DNS 1194
proto udp
dev tun  
cd /etc/openvpn/
ca keys/ca.crt
cert keys/client.crt
key keys/client.key
ns-cert-type server
keepalive 10 120
user nobody
group nobody
status /var/log/openvpn-status.log

Change IP/DNS to the IP or DNS name your server is reachable at. You should now be able to connect to your OpenVPN server by typing:

openvpn --config /etc/openvpn/openvpn.conf

That’s pretty much it. Once you get a handle on the settings, it is actually pretty easy. However, as mentioned before. I have found a possible issue with DNS. I would highly recommend editing /etc/resolv.conf to point to your DNS server. In my example, the DNS server is also at the gateway ( You can script this. In fact, use my script.

pid=`pgrep openvpn`
if [ -z "$pid" ]; then
echo "Starting OpenVPN Client"
cp /etc/resolv.conf /etc/resolv.conf.backup
echo "nameserver" > /etc/resolv.conf
openvpn --config /etc/openvpn/openvpn.conf &
echo "Stopping OpenVPN Client"
mv /etc/resolv.conf.backup /etc/resolv.conf
kill $pid

Pretty strait forward if I do say so myself. You may have an issue if you have a passphrase on your key! If you are having an issue, remove the ampersand (&) from the end of the openvpn –config line. This will not background the process, but you can do it manually by typing ctrl+z then bg which will background the process.

CyanogenMod’s Android Configuration

Because I don’t run the Android that came with my phone, I can use OpenVPN with ease. If you are not running a custom rom, you can still run OpenVPN by getting the client software from the Android Market (now called the Play Store). The following instructions are for CyanogenMod 7.2, but should work in newer versions just fine.

Remember when you made your client key? Well you need to make one that works great with Android. It’s the PKCS #12 format. This will give you a file that ends in a .p12 extension. Copy this file over to the root of your sdcard.

Install the certificate by going to Settings->Location & Security->Install from SD card (under Credential storage at the bottom on the menu). It should find the file and ask for the password to unlock it. Then it will ask for a new password (you can use the same one as before) and you can also give it a custom name.

Build the client by going to Settings->Wireless & Networks->VPN Settings->Add VPN. You just need to select the OpenVPN type. In the new menu there are several settings.

VPN name (this can be anything you want)
Set VPN server (the IP or domain name of the server)
User authentication (leave unchecked)
Set CA certificate (click this and select the key you just installed)
Set user certificate (same as above)
DNS search domains (these are optional, but you can set like in the bash script above)

Hit the menu button then Advanced.

Server port (default is 1194)
Protocol to use (udp is default)
Device to use (tun, which is fine)
LZO compression (check it!)
Redirect gateway (check it!)
Remote Sets Addresses (Should also be checked)

Everything below that I left as default. You do NOT need to enable TLS-Auth. For this type of setup it is unnecessary.

Hit back, then save. From here you should be able to connect to your VPN. Note that in my tests, the VPN is much slower. I’m not sure if it is something I have done wrong in my setup, or if my provider throttles VPNs.


Everything should be up and running now. I hope you found this useful. Please feel free to leave a comment below. If you have any suggestions or questions you can drop those below as well. I’m not an expert on OpenVPN, I just like learning.


Create 2 types of SVN backups quickly and easily

Like many of you, I run my own SVN repository. I have several projects and several people that use them. So I wanted a quick and easy way to perform backups. Here are the two methods I use.

The first is an incremental backup where I use a post-commit hook. The other is a bash script that I setup in cron to run once per week (you can do daily, monthly, or when ever. It depends on how quickly you want the full backup script to run). The bash script is designed to create a full backup using svnadmin dump.

The idea of a full backup is nothing new, but the way my script works is a little different than others. Their script just creates a new backup when ever run. Mine creates a full backup of every revision number. Even if there have been several updates since the last backup.

Lets get started.

Incremental backup using a post-commit

Post-commit hooks are very powerful. You can do a lot with them. In this case we are going to do an incremental backup. I don’t do a full because my repository is very large and I don’t want to be slowed down every time I commit new files.

First you need to go to the directory where your repository is. Lets call it /srv/svn/myproject. In that directory there is a folder called hooks. Looks for post-commit (it maybe listed as post-commit.tmpl, just rename it without the .tmpl). Open the file in an editor (vi, nano, emacks) and add this line to the end.

svnadmin dump "$REPOS" --revision "$REV" --incremental >/srv/backups/myproject/incremental/commit-$REV 2>> /srv/backups/myproject/incremental/backup.log

NOTE: If is not commented out, put a hash in front of it “#”. You don’t need it.

Take note of the directories I’m using. Just change them to where your backups will go. Also, you need to give those directories the same ownership as your svn archive! Otherwise when the post-commit runs, it will error out because it cannot write to the new location.

And that is it! Now every commit you make will create a new incremental backup.

There is one down side to this method. If you already have several commits, you will not get your entire history. I did this a lazy way since at the time I was only up to around 20 commits.

./post-commit /srv/svn/myproject 0

This will create the backup starting at revision 0. Just keep running it and moving the number up. If you have hundreds or even thousands of commits… you might want to write something to do all that hard work for you. You might even get some ideas from the next section.

Full backup

This is where I’m different from everyone else… at least that I could find. While this is not the most elegant way of doing things, it really doesn’t take very long to run (unless there are many revisions to create new archives for.

latestRevision=`svnlook youngest $svnLocale`
let stopCounting=latestRevision+1
while [ $COUNTER != $stopCounting ]; do
        # check to see if file exists
        if [ ! -e $backupLocale$fileName$rev$extension ]; then
                svnadmin dump $svnLocale -r $rev -q > $backupLocale$fileName$rev.svndump
                bzip2 -z9q $backupLocale$fileName$rev.svndump
        let COUNTER=COUNTER+1

Seems a bit much, but all you would need to worry about is changing svnLocale, backupLocale, and fileName. You can change extension if you wish, but I would leave it unless you plan on changing the svnadmin or bzip2.

Please note that the first time you run this, nothing is dumped on the screen. I ran this on an archive with 1215 revisions as a test. I went to lunch. You can modify the script and remote the -q from both svnadmin and bzip2 to see your progress. I silence them while running as a cron job.

The benefit to this script is it does check to see if an archive has been created for each revision. If not, it gets created. This script will even get backups you may have deleted and recreate them, then continue on. It doesn’t matter. It’s so simple that it just works.

So that’s pretty much it. Questions? Comments? Let me know what you think.

How To Update a Live Website via SVN

Do you want to use SVN to update your website? This guide will help you and includes information not found anywhere else on how to get it working. Many websites give basic information and comments complain about it not working. If those other website didn’t work for you. This will.

After digging around for hours and seeing everyone complain about having the same problems after being told the same solutions, I decided to figure it out for myself. If you read this carefully you too will have a website setup that can be updated via subversion.

Getting Started

I’m going to assume that your SVN server and website are on the same machine and that you are using Apache to serve both. I’m also going to assume that you are somewhat familiar with SVN and Apache. I’m not going to go into much detail on how to get everything setup from scratch.

The tools you will need to have installed are as follows (may need more depending on your configuration). Apache HTTPd, Subversion, and gcc.

Many Linux users and distributions use sudo. I do not, and I will not provide information on how to use sudo. My recommendation, before doing anything, is to type sudo su which will make you root. Do everything from there so you don’t have to sudo anymore.

Setup a new SVN repository

Alright, now we are going to setup the new SVN repository. This can be done easily with:

svnadmin create <path to repository>

Place it in a good location that you will remember since you need to point Apache to it. In my examples we are going to use /home/www/domain/svn/website/ for our project location and /home/www/domain/htdocs/ for our website.

For user access controls I use a 2 part process. I have an authz and passwd file in the root of my SVN directory (/home/www/domain/svn/). The reason I do this is because I host several SVN projects and I don’t want everyone to have access to everything. This way I can also setup (this comes later) Apache to have read-only access to the repository.

First, we need to create our authz and passwd files. Lets call them svn.authz and svn.passwd so we know what they are for.

In the svn.authz file we need something like this:

project = dkun
other1 = dkun, someguy
website = dkun, friend1
readonly = redmine,apache
@project = rw
@readonly = r
* =
@other1 = rw
* =
@website = rw
@readonly = r
* =

Looks a bit overwhelming. Let me explain.

Starting from the top, we have [groups]. These are groups of users that we give project names to. Notice that 2 of the projects have more that one user. There is also a read-only group. You can name the groups what every you want. I do it by project name to make it easy to read.

After that you see [project:/]. This is the project group. It has one user and is set to read/write. It also has a readonly group. This way I can setup both Apache and Redmine with read only access.

Lastly you might have noticed *=. What is that? It turns off anonymous access. So unless your project has code going out to the web, make sure to put this in.

Now we need to setup the svn.passwd file. This is pretty easy. Just type:

htpasswd svn.passwd <username>

You will be prompted for the password. That’s it! Pretty easy right? Just don’t forget to setup the user apache (keep it lower case, trust me), and any others you might need.

Setup Apache

Now that we have SVN and out new project setup, we need to configure Apache to serve everything. Setup your VirtualHost to point to where you want your website to be hosted from. So for me it would be /home/www/domain/htdocs/. I’m going to assume you already know how to do this. All you need to add is a little directive.

<DirectoryMatch "^/.*/\.svn/">
     Order deny,allow
     Deny from all

This will stop someone from browsing your .svn folder on the website.

Now setup an SVN VirtualHost. This gets a bit more complicated. I’m going to avoid going into detail here. I want to assume you have some background.

In your Virtual host we need to set a few options and point to the SVN repository root. Here is an example of the options you need to put inside the VirtualHost section for SVN to work. Don’t forget to set other items you need!

<Location /svn>
   DAV svn
   SVNParentPath /home/www/domain/svn
   AuthType Basic
   AuthName "My SVN Repository"
   AuthUserFile /home/www/domain/svn/svn.passwd
   AuthzSVNAccessFile /home/www/domain/svn/svn.authz
   Require valid-user
<Directory /home/www/domain/svn>
   Options +Indexes FollowSymLinks +ExecCGI
   AllowOverride AuthConfig FileInfo
   Order allow,deny
   Allow from all

Note all the directories. Change them to match your server. Also, verify that your SVN modules are enabled in Apache. They should be in your apache.conf or httpd.conf (depending on your server). You will need to enable:

LoadModule dav_svn_module lib/httpd/modules/
LoadModule authz_svn_module lib/httpd/modules/

That’s it! Now type apachectl configtest and you will get messages back if there is a problem in your Apache config. If it is a warning about your website directory missing, don’t worry. Just create the directory and give full ownership to the apache user (We will get back to this in a moment).

Before continuing, restart Apache with apachectl restart and make sure your SVN works! If it does not then the next section will fail.

Do some weird stuff with Apache

Here is where things get a little weird. This is also where 99% of people fail because there is no good information out there. I will do my best to explain what to do and why you should do it.

The first step is you need to find out what user and group Apache runs under. For me, both are apache, but for you it could use www or www-user. You can find this by checking /etc/passwd and /etc/group, or by going through your apache.conf file and look for User and Group directives.

Now look in /etc/passwd for the apache user. The second to last option shows Apache’s home directory. In my case it is /srv/httpd. Go to that directory and type:

chown apache.apache ./

Set the user and group to which ever Apache runs under. This will allow Apache to create new directories there without changing ownership of anything else. Also, that is just one (1) period before the slash!

Open /etc/passwd and find the apache user again. At the end of the line you should see something like /bin/false. This keeps the Apache user from logging in. This is for security reasons. We are going to temporarily change it to /bin/bash.

Now that you are already root, type:

su – apache

Once again I assume your Apache’s user is apache. Adjust as needed.

Now you are logged in as the Apache user. You should see a new shell and be ready to go. Make sure you are in the /var/httpd directory (as stated previously as being Apache’s home directory) by typing pwd. If you are, then you have created a new session and are ready for the next step.

Navigate to the directory where the website will be served from. So if the website is in /home/www/domain/htdocs/ then go one level up; /home/www/domain/. Now we are going to perform an SVN checkout and have subversion remember our credentials.

svn co http://<domain>/svn/website

Adjust the http directive as needed. If you use SSL have subversion permanently accept the certificate. You should see a prompt for apache’s password. This is the password you created for it in svn.passwd. Type it in and when asked if you want to store the password, say yes!

If everything went as planned then you should see a successful checkout as revision 0. If not… well you did something wrong. Double check your paths, apache config files, and passwords. If needed, you may need to give ownership of the directory to Apache so that user can create a new directory and write to it.

Check to make sure a .subversion directory was created in Apache’s home directory and that there is a .svn folder in /home/www/domain/htdocs/. If now you need to give ownership of that directory to the Apache user (you will have to be root for this). If there is so .svn directory, then change ownership of the htdocs directory to Apache and run the checkout again.

If everything went smoothly then we are done as the Apache user. Type logout to return to being root.

Last thing. Open up /etc/passwd and change /bin/bash back to /bin/false. No need to have a security hole.

Create an executable

Now that we have SVN, Apache, and a SVN checkout setup, we can finally finish this. The first thing you need to do is find where svn is located:

whereis svn
svn: /usr/bin/svn /usr/man/man1/svn.1.gz /usr/share/man/man1/svn.1.gz

As you can see, I used whereis to find the svn executable. It’s the first one, /usr/bin/svn.

Navigate over to the htdocs directory. Open up your favorite text editor and dump this in it:

#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
int main(void)
   execl("/usr/bin/svn", "svn", "update", "/home/www/domain/htdocs/",
      (const char *) NULL);

See where the directives of /usr/bin/svn and /home/www/domain/htdocs/ are? Just change those to match your server. Save the file as update.c and just back to the shell.
Now we need to compile the program:

cc -o update update.c

This will create an executable called update. You can call it what ever you want, just as long as before renaming it you type chown apache.apache update; chmod +s update. It should be owned by the Apache user.

Setup post-commit

Just over to your SVN repository, then into the hooks directory. So if you were following, cd /home/www/domain/svn/website/hooks. There you will see a bunch of .tmpl files. Rename of copy post-commit.tmpl to post-commit, then chmod 755 post-commit.

cp post-commit.tmpl post-commit
chmod 755 post-commit

Open post-commit in your editor. You may see a bunch of stuff in there. Just put a hash (#) sign in front of any other commands there. They are examples. Add:


That’s it. Don’t add sh, or bash to the front, just put in the path with executable at the end. Save and exit.

Try it!

Now, as long as you followed along and I didn’t skip anything. This should work. Try it out. Try committing something to SVN and see what happens! If you can view the website right away, they you did it correctly. If now, you should check your apache log files. Specifically error_log. If SVN threw an error you will have to see what went wrong there. Sometimes just putting a commit message in will fix any error.

That’s it. I hope you found this informative. Please feel free to drop my a comment below if you have anything you wish to add or just want to say thanks for the info. I appreciate all feedback. Thanks for reading!