Posts in category english

Subversion and Trac setup with minimal privilege overlap

Over the years I developed (in discussion with a few colleagues and friends) a subversion and trac setup with a rather strong privilege separation. I've written it down for Debian wheezy, but it can be reproduced (with more or less effort) on other distributions as well. I once did it on SUSE, which was a nightmare, but this is more a judgment about the cleanness and flexibility of the distribution, not the setup described here. In addition some details will certainly need adjustments to new developments, but the basics are rather mature. Also, even when the focus of other setups vary, the whole concept is probably worth sharing. It also forced me to rewind my setup and write it down not in a bunch of snippets in a best practice folder on my machine (as I'm doing it all the time), but in a (still sparsely) documented blog post.

First of all, I will install trac from source as a regular user. As I frequently tweak details on the trac installation, I don't want to use the version that comes with the distribution. The important point here is, that trac will be installed neither as root nor as the user which runs the trac process in the end. On machines I'm managing myself (alone) I use my own user account for such installations, but you could use a specific software installation account. To make the installation process a little more universal, let us set an environment variable for this user (on a root shell we'll use in the course of this guide):

export SW=...

We install Apache, as it is still the solution of choice for serving subversion (webdav). We take subversion from the distribution as well as python and gunicorn (as a simple solution for serving trac):

apt-get install -y apache2 libapache2-svn python-virtualenv python-subversion python-dev gunicorn sudo

We don't want Apache to serve at Port 80, nor should it do anything else than subversion. As other use-cases of "specific" apache instances might come up, we disable the default configuration and setup an apache2-svn configuration. This apache instance will be run as a svn user to be created later on. The apache config is created by the following steps:

service apache2 stop
update-rc.d -f apache2 remove
echo -e '\necho "default apache disabled"\nexit' >> /etc/default/apache2
mkdir /etc/apache2-svn
cp /etc/apache2/{apache2.conf,magic} /etc/apache2-svn
sed -e 's/www-data/svn/' /etc/apache2/envvars > /etc/apache2-svn/envvars
echo "Listen" > /etc/apache2-svn/ports.conf
ln -s /etc/apache2/{conf.d,mods-available} /etc/apache2-svn
mkdir /etc/apache2-svn/{mods-enabled,sites-available,sites-enabled}
APACHE_CONFDIR=/etc/apache2-svn a2enmod authz_host mime auth_basic authn_file authz_groupfile dav_fs authz_svn alias ssl
touch /etc/default/apache2-svn
sed 's/#.*apache2/&-svn/' /etc/init.d/apache2 > /etc/init.d/apache2-svn
chmod +x /etc/init.d/apache2-svn
mkdir /var/log/apache2-svn

As subversion uses more of http than simple get and post requests, it is rather hard to switch the scheme on a reverse proyx in front of the subversion server. Thus we create a private key, a signing request and self-sign it with our own private key:

openssl genrsa -out /etc/apache2-svn/server.key 2048
openssl req -new -batch -subj "/C=US/CN=localhost" -key /etc/apache2-svn/server.key -out /etc/apache2-svn/server.csr
openssl x509 -req -days 3650 -in /etc/apache2-svn/server.csr -signkey /etc/apache2-svn/server.key -out /etc/apache2-svn/server.crt

Now comes the apache configuration:

echo -e "ServerName svn\n"\
"ServerAdmin webmaster@$(hostname)\n"\
"DocumentRoot /var/www/\n"\
"CustomLog /var/log/apache2-svn/access.log combined\n"\
"SSLEngine On\n"\
"SSLCertificateFile /etc/apache2-svn/server.crt\n"\
"SSLCertificateKeyFile /etc/apache2-svn/server.key\n"\
"SSLSessionCache shmcb:/var/run/apache2-svn/ssl_scache(512000)\n"\
"SSLMutex file:/var/run/apache2-svn/ssl_mutex\n"\
"<Location /repos/>\n"\
"    DAV svn\n"\
"    SVNParentPath /var/lib/svn/repos\n"\
"    SVNListParentPath On\n"\
"    AuthzSVNAccessFile /etc/apache2-svn/authz\n"\
"    AuthType Basic\n"\
'    AuthName "svn repos"\n'\
"    AuthUserFile /etc/auth/htpasswd\n"\
"    Require valid-user\n"\
'RedirectMatch /repos$ /repos/' > /etc/apache2-svn/sites-available/default
APACHE_CONFDIR=/etc/apache2-svn a2ensite default

In this setup a subversion and trac user will be authenticated via a passwd file and the subversion access is configured in an authz file:

htpasswd -c /etc/$newuser/htpasswd user
echo -e "[groups]\n"\
"@all=rw" > /etc/apache2-svn/authz

Now we're all done regarding subversion except for the svn user itself:

adduser --system --home /var/lib/svn --group svn
sudo -u svn mkdir /var/lib/svn/{repos,backups,skel}
echo -e '#!/bin/bash\n'\
'/usr/bin/sudo -u trac /home/'$SW'/trac/bin/trac-admin /var/lib/trac/projects/NAME changeset added "$1" "$2" &&\n'\
'/usr/bin/svnadmin dump "$1" --revision "$2" --incremental|/bin/gzip > "/var/lib/svn/backups/NAME/dump.$(/usr/bin/printf %05d $2).gz"' > /var/lib/svn/skel/post-commit
echo -e '#!/bin/bash\n'\
'/usr/bin/sudo -u trac /home/'$SW'/trac/bin/trac-admin /var/lib/trac/projects/NAME changeset modified "$1" "$2" &&\n'\
'/usr/bin/svnadmin dump "$1" --revision "$2" --incremental|/bin/gzip > "/var/lib/svn/backups/NAME/dump.$(/usr/bin/printf %05d $2).gz"' > /var/lib/svn/skel/post-revprop-change
chown svn.svn /var/lib/svn/skel/*
chmod u+x /var/lib/svn/skel/*

Here we've included skeleton files for some hooks, which will create svn dump backups for each revision at checkin and inform trac about the changes in the repository. As trac will run as a separate trac user user, we need to add a sudoers rule:

echo "svn ALL=(trac)NOPASSWD:/home/wobsta/trac/bin/trac-admin /var/lib/trac/projects/* changeset added *,/home/wobsta/trac/bin/trac-admin /var/lib/trac/projects/* changeset modified *" > /etc/sudoers.d/trac
chmod 440 /etc/sudoers.d/trac

Ok, now let's install trac:

cd /tmp
sudo -u $SW virtualenv /home/$SW/trac
sudo -u $SW /home/$SW/trac/bin/pip install trac
sudo -u $SW mkdir /home/$SW/trac/wsgi
echo -e "#!/home/$SW/trac/bin/python\n"\
"import site, sys\n"\
"from pkg_resources import working_set\n"\
"for path in sys.path:\n"\
"    working_set.add_entry(path)\n"\
"import os\n"\
"os.environ['PYTHON_EGG_CACHE'] = '/var/lib/trac/eggs'\n"\
"import trac.web.main\n"\
"def application(environ, start_response):\n"\
"    environ['SCRIPT_NAME'] = '/projects'\n"\
"    environ['REMOTE_USER'] = environ.get('HTTP_REMOTE_USER')\n"\
"    environ['trac.env_parent_dir'] = '/var/lib/trac/projects'\n"\
"    return trac.web.main.dispatch_request(environ, start_response)" | sudo -u $SW tee /home/$SW/trac/wsgi/ > /dev/null

In the last step we've created a wsgi adapter. Due to the component architecture of trac adding the site-package path of the virtual env is not enough in this script, but package discovery requires a working_set modification as shown. In addition, we fix the SCRIPT_NAME environment variable and copy the HTTP_REMOTE_USER to the REMOTE_USER environment variable as required by trac. This setup enables us to use nginx as the front-end webserver in the end.

To complete the trac installation, we have to add the trac user to the system:

adduser --system --home /var/lib/trac --group trac
sudo -u trac mkdir /var/lib/trac/{projects,eggs}

Now we're left with starting apache2-svn and adding trac to the gunicorn configuration:

service apache2-svn start
update-rc.d apache2-svn defaults
service gunicorn stop
echo -e "CONFIG = {\n"\
"    'working_dir': '/home/$SW/trac/wsgi',\n"\
"    'environment': {\n"\
"    },\n"\
"    'user': 'trac',\n"\
"    'group': 'trac',\n"\
"    'args': (\n"\
"        '--bind=',\n"\
"        '--workers=3',\n"\
"        '--timeout=30',\n"\
"        'projects:application',\n"\
"    ),\n"\
"}" > /etc/gunicorn.d/trac
service gunicorn start

To create a new subversion repository and corresponding trac instance with a given name, execute:

export NAME=test
sudo -u svn svnadmin create /var/lib/svn/repos/$NAME
sudo -u svn mkdir /var/lib/svn/backups/$NAME
rm -r /var/lib/svn/repos/$NAME/hooks
cp -ra /var/lib/svn/skel /var/lib/svn/repos/$NAME/hooks
sed -i s/NAME/$NAME/ /var/lib/svn/repos/$NAME/hooks/*
sudo -u trac /home/$SW/trac/bin/trac-admin /var/lib/trac/projects/$NAME initenv

The reverse proxy configuration is straight forward and shown for as an nginx config snipped here

location /repos { proxy_pass; }

location /projects {
  rewrite ^/projects/(.*)$ /$1 break;
location ~ ^/projects/[a-z]+/login {
  auth_basic "trac projects";
  auth_basic_user_file /etc/auth/htpasswd;
  proxy_set_header REMOTE_USER $remote_user;
  rewrite ^/projects/(.*)$ /$1 break;

Thank's all, folks.

Excel FollowHyperlink loads URL twice

I'm writing a web app which receives data from Excel 2010 on Windows 7 (64 bit). I'm using a small VBA macro for data upload and web page opening. However, when using ActiveWorkbook.FollowHyperlink in VBA the website is not opened by my default browser initially (Chrome in my example), but by some MS Office internal Internet Explorer instance. I'm redirecting from this first URL … and my real web browser is called on the redirected URL only. Got it? The header of the first access contains the following user agent:

User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; ms-office)

The server response with a redirect and the other url is called by the same user agent. At the end this second url is opened with my default browser, which happens to be:

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3

Unfortunately this is not useful, as I'm setting a cookie at the first request (where the redirect occurs). I only found an ugly solution using ShellExecute (so far):

Private Declare Function ShellExecute Lib "shell32.dll" Alias "ShellExecuteA" (ByVal hwnd As Long, ByVal lpOperation As String, ByVal lpFile As String, ByVal lpParameters As String, ByVal lpDirectory As String, ByVal nShowCmd As Long) As Long

and then opening the website by:

ShellExecute 0, vbNullString, "http://...", vbNullString, vbNullString, vbNormalFocus

Others seem to have the same problem ( But ShellExecute really should only be a last resort. Never mind, it's just Microsoft.

Py3k WSGI Test

I started a small project back in May where I was trying to use Python 3 for web development for the first time. While it somehow worked (with ugly hacks in the stdlib and being a memory hog), I finally decided it to be a failure. As I needed the project to get in production I switched back to Python 2 and everything worked out well. Still, I'm interested in web development on Python 3. Recently the author of the Python 3 enabled WSGI framework bottle started a new project multipart to fix the problems with cgi.FieldStorage. As the WSGI problems on Python 3 are under constant discussion I want to demonstrate the use of multipart in bottle even though multipart has not yet been integrated in bottle by the author. In my mind it can be used already (at least for development and test instances), it works well, and doesn't look ugly at all. So you are welcome to have a look at the source of my sample application and test the demo instance (edit: link to the demo instance removed, as it is not available anymore).

cgi multipart bugs in py3k

Recently I started to implement a small web app in py3k and wsgi. While none of the major python web frameworks are ready for py3k anytime soon and wsgi for py3k is still under major discussions (see for example this current post by Armin Ronacher), there are already options to actually start working on the subject.

I'm using bottle (dev tree) and the current releases of sqlalchemy, py-postgresql and jinja2 on Python 3.1.2. This actually works sort of. I will likely write some further posts about my progress. However, I quickly found a bug in the cgi multipart input handling, which actually turned out to arise from two different sources. I tried to solve it and just opened issue 8846 on the Python issue tracker showing my findings.