Posts in category english

Caution with string values in choice fields in Plone forms with z3c.form

I recently noticed a very strange behaviour while setting initial data for a choice field in a z3c.form. Suppose you have a simple, static vocabulary:

num_voc = SimpleVocabulary([SimpleTerm(value=1, token='1', title='one'),
                            SimpleTerm(value=2, token='2', title='two'),
                            SimpleTerm(value=3, token='3', title='three')])

Here you might skip token, as it is then created by str(<value>) automatically. We will use it in a form by the following schema

num = schema.Choice(title='number', vocabulary=num_voc)

The return value of the forms extractData method will return the value of the dictionary. For example, in a form buttons action method, the selected "two" in the form will result in the integer value 2:

data, errors = self.extractData()
assert data['num'] == 2    # correct

However, if you set the value in the updateWidgets method when the form is opened with a get request, you need to set the widget value to a list containing the token, which happens to be a string:

self.widgets['number'].value = ['2']

The reason is, that the selected attribute of the option in the widget is set by the expression term.token in self.value. While this is natural, as choice fields might support several selected items, it becomes a trap if you use string values. Consider the following example:

title_voc = SimpleVocabulary([SimpleTerm(value='Dr.', title='one'),
                              SimpleTerm(value='Prof.', title='two'),
                              SimpleTerm(value='Prof. Dr.', title='three')])

(Note that the tokens are equal to the values here. For non-ASCII values, value.encode('ascii', 'backslashreplace') is used as of zope.schema >= 4.6.0, i.e. Plone 5.2. But this is a different story and means to need to do the same when setting the value of the widget. Or better, just do not use string values at all.)

If you now miss to set the value as a list ['Prof. Dr.'] but use the string directly, i.e. self.widgets['number'].value = 'Prof. Dr.', you will find all the three options being selected without any error, as term.token in self.value is true for all three cases. Awkward!

Sorry Apple, too late

I would have brought the new Apple 16" machine for sure, if it would have been available last fall. But Apple just did not offer any reasonable laptop. For years. Until today. I was forced to replace my previous machine by the support end of El Capitan and no newer operating system being available on this old machine. (The situation might have been different, if I could have used Sierra or even High Sierra. Technically, this might have been possible as some hacks have shown, but Apple decided to not support this.)

I finally decided to end my 15 years of Mac user life and switched (back) to Linux for my laptop. Last September I brought a Lenovo X1 carbon 6th gen. right after they announced the support of the S3 sleep mode to improve Linux compatibility. (They added a Linux BIOS option for that, awesome.) I do not regret it. I am fully aware of my rather limited use-case. Linux is not at all a reasonable replacement for many situations. Still, it also has advantages. Furthermore I admit to having used Ubuntu 19.04 initially for better compatibility with the hardware compared to Debian 9. This, again, confirms the trouble to be taken into account. But since this summer I am happy with Debian 10. Apple has lost me, and it is not clear whether I will ever come back.

MacBook Pro SE

At present I still work on a MacBook Pro 17" (early 2009), which I ordered at the day of the presentation at the MacWorld in January 2009. This is just the second Mac I own. My first Mac was a 17" Powerbook I brought in 2003. It is time for another update. Actually a new machine is due for several years already, but I struggle with the options available.

A recent visit at an Apple Store in Munich was disillusioning once again. Apple does not provide me with any upgrade I can comfort myself with. They have progressed in a direction I refuse to join. While making the machines thinner and thinner, Apple seems to have completely lost the vision for users needing a workhorse. I'm just joining the chorus: The keyboard with the new butterfly mechanism is crap. I tried it, I cannot work with it. Also I cannot make myself comfortable to the touch bar missing a proper escape key any vi user just needs to have. I'm also curious about the removal of the inverse T for the cursor keys which most manufacturers have agreed to.

In my opinion it is unquestionable for a machine labeled "pro" to allow for the hard drive being exchangeable. Soldering it to the mainboard is just irrational. Currently I have a 500GB SSD and a 750 GB HD in my machine (having removed the CD drive years ago). I exchanged and upgraded my drives multiple times already, once due to a HD crash I had a few years ago. I survive it with ease as I just went to a store, brought a new drive and restored from a time machine backup. All done within a few hours. This was just perfect and as it should be.

One more thing while I'm grumbling anyway: I'm very angry about the missing matte screen option which got lost years ago already. Still, and given the fact, that most of the time I work on an external display, I could arrange myself following the spirit of the time in all that (even the hard drive). Except for the keyboard.

I'm already close to switch to a Thinkpad but I still hope for the better: A MacBook Pro SE. While the hardware of my present machine is already discontinued, I can only hope that it won't break in the near future, before the last macOS 10.11 supported on this machine becomes unmaintained as well. Maybe, hopefully, Apple will release a MacBook Pro SE along with the Mac Pro already announced for 2018. Hope dies last.

Subversion and Trac setup with minimal privilege overlap

Over the years I developed (in discussion with a few colleagues and friends) a subversion and trac setup with a rather strong privilege separation. I've written it down for Debian wheezy, but it can be reproduced (with more or less effort) on other distributions as well. I once did it on SUSE, which was a nightmare, but this is more a judgment about the cleanness and flexibility of the distribution, not the setup described here. In addition some details will certainly need adjustments to new developments, but the basics are rather mature. Also, even when the focus of other setups vary, the whole concept is probably worth sharing. It also forced me to rewind my setup and write it down not in a bunch of snippets in a best practice folder on my machine (as I'm doing it all the time), but in a (still sparsely) documented blog post.

First of all, I will install trac from source as a regular user. As I frequently tweak details on the trac installation, I don't want to use the version that comes with the distribution. The important point here is, that trac will be installed neither as root nor as the user which runs the trac process in the end. On machines I'm managing myself (alone) I use my own user account for such installations, but you could use a specific software installation account. To make the installation process a little more universal, let us set an environment variable for this user (on a root shell we'll use in the course of this guide):

export SW=...

We install Apache, as it is still the solution of choice for serving subversion (webdav). We take subversion from the distribution as well as python and gunicorn (as a simple solution for serving trac):

apt-get install -y apache2 libapache2-svn python-virtualenv python-subversion python-dev gunicorn sudo

We don't want Apache to serve at Port 80, nor should it do anything else than subversion. As other use-cases of "specific" apache instances might come up, we disable the default configuration and setup an apache2-svn configuration. This apache instance will be run as a svn user to be created later on. The apache config is created by the following steps:

service apache2 stop
update-rc.d -f apache2 remove
echo -e '\necho "default apache disabled"\nexit' >> /etc/default/apache2
mkdir /etc/apache2-svn
cp /etc/apache2/{apache2.conf,magic} /etc/apache2-svn
sed -e 's/www-data/svn/' /etc/apache2/envvars > /etc/apache2-svn/envvars
echo "Listen" > /etc/apache2-svn/ports.conf
ln -s /etc/apache2/{conf.d,mods-available} /etc/apache2-svn
mkdir /etc/apache2-svn/{mods-enabled,sites-available,sites-enabled}
APACHE_CONFDIR=/etc/apache2-svn a2enmod authz_host mime auth_basic authn_file authz_groupfile dav_fs authz_svn alias ssl
touch /etc/default/apache2-svn
sed 's/#.*apache2/&-svn/' /etc/init.d/apache2 > /etc/init.d/apache2-svn
chmod +x /etc/init.d/apache2-svn
mkdir /var/log/apache2-svn

As subversion uses more of http than simple get and post requests, it is rather hard to switch the scheme on a reverse proyx in front of the subversion server. Thus we create a private key, a signing request and self-sign it with our own private key:

openssl genrsa -out /etc/apache2-svn/server.key 2048
openssl req -new -batch -subj "/C=US/CN=localhost" -key /etc/apache2-svn/server.key -out /etc/apache2-svn/server.csr
openssl x509 -req -days 3650 -in /etc/apache2-svn/server.csr -signkey /etc/apache2-svn/server.key -out /etc/apache2-svn/server.crt

Now comes the apache configuration:

echo -e "ServerName svn\n"\
"ServerAdmin webmaster@$(hostname)\n"\
"DocumentRoot /var/www/\n"\
"CustomLog /var/log/apache2-svn/access.log combined\n"\
"SSLEngine On\n"\
"SSLCertificateFile /etc/apache2-svn/server.crt\n"\
"SSLCertificateKeyFile /etc/apache2-svn/server.key\n"\
"SSLSessionCache shmcb:/var/run/apache2-svn/ssl_scache(512000)\n"\
"SSLMutex file:/var/run/apache2-svn/ssl_mutex\n"\
"<Location /repos/>\n"\
"    DAV svn\n"\
"    SVNParentPath /var/lib/svn/repos\n"\
"    SVNListParentPath On\n"\
"    AuthzSVNAccessFile /etc/apache2-svn/authz\n"\
"    AuthType Basic\n"\
'    AuthName "svn repos"\n'\
"    AuthUserFile /etc/auth/htpasswd\n"\
"    Require valid-user\n"\
'RedirectMatch /repos$ /repos/' > /etc/apache2-svn/sites-available/default
APACHE_CONFDIR=/etc/apache2-svn a2ensite default

In this setup a subversion and trac user will be authenticated via a passwd file and the subversion access is configured in an authz file:

htpasswd -c /etc/$newuser/htpasswd user
echo -e "[groups]\n"\
"@all=rw" > /etc/apache2-svn/authz

Now we're all done regarding subversion except for the svn user itself:

adduser --system --home /var/lib/svn --group svn
sudo -u svn mkdir /var/lib/svn/{repos,backups,skel}
echo -e '#!/bin/bash\n'\
'/usr/bin/sudo -u trac /home/'$SW'/trac/bin/trac-admin /var/lib/trac/projects/NAME changeset added "$1" "$2" &&\n'\
'/usr/bin/svnadmin dump "$1" --revision "$2" --incremental|/bin/gzip > "/var/lib/svn/backups/NAME/dump.$(/usr/bin/printf %05d $2).gz"' > /var/lib/svn/skel/post-commit
echo -e '#!/bin/bash\n'\
'/usr/bin/sudo -u trac /home/'$SW'/trac/bin/trac-admin /var/lib/trac/projects/NAME changeset modified "$1" "$2" &&\n'\
'/usr/bin/svnadmin dump "$1" --revision "$2" --incremental|/bin/gzip > "/var/lib/svn/backups/NAME/dump.$(/usr/bin/printf %05d $2).gz"' > /var/lib/svn/skel/post-revprop-change
chown svn.svn /var/lib/svn/skel/*
chmod u+x /var/lib/svn/skel/*

Here we've included skeleton files for some hooks, which will create svn dump backups for each revision at checkin and inform trac about the changes in the repository. As trac will run as a separate trac user user, we need to add a sudoers rule:

echo "svn ALL=(trac)NOPASSWD:/home/wobsta/trac/bin/trac-admin /var/lib/trac/projects/* changeset added *,/home/wobsta/trac/bin/trac-admin /var/lib/trac/projects/* changeset modified *" > /etc/sudoers.d/trac
chmod 440 /etc/sudoers.d/trac

Ok, now let's install trac:

cd /tmp
sudo -u $SW virtualenv /home/$SW/trac
sudo -u $SW /home/$SW/trac/bin/pip install trac
sudo -u $SW mkdir /home/$SW/trac/wsgi
echo -e "#!/home/$SW/trac/bin/python\n"\
"import site, sys\n"\
"from pkg_resources import working_set\n"\
"for path in sys.path:\n"\
"    working_set.add_entry(path)\n"\
"import os\n"\
"os.environ['PYTHON_EGG_CACHE'] = '/var/lib/trac/eggs'\n"\
"import trac.web.main\n"\
"def application(environ, start_response):\n"\
"    environ['SCRIPT_NAME'] = '/projects'\n"\
"    environ['REMOTE_USER'] = environ.get('HTTP_REMOTE_USER')\n"\
"    environ['trac.env_parent_dir'] = '/var/lib/trac/projects'\n"\
"    return trac.web.main.dispatch_request(environ, start_response)" | sudo -u $SW tee /home/$SW/trac/wsgi/ > /dev/null

In the last step we've created a wsgi adapter. Due to the component architecture of trac adding the site-package path of the virtual env is not enough in this script, but package discovery requires a working_set modification as shown. In addition, we fix the SCRIPT_NAME environment variable and copy the HTTP_REMOTE_USER to the REMOTE_USER environment variable as required by trac. This setup enables us to use nginx as the front-end webserver in the end.

To complete the trac installation, we have to add the trac user to the system:

adduser --system --home /var/lib/trac --group trac
sudo -u trac mkdir /var/lib/trac/{projects,eggs}

Now we're left with starting apache2-svn and adding trac to the gunicorn configuration:

service apache2-svn start
update-rc.d apache2-svn defaults
service gunicorn stop
echo -e "CONFIG = {\n"\
"    'working_dir': '/home/$SW/trac/wsgi',\n"\
"    'environment': {\n"\
"    },\n"\
"    'user': 'trac',\n"\
"    'group': 'trac',\n"\
"    'args': (\n"\
"        '--bind=',\n"\
"        '--workers=3',\n"\
"        '--timeout=30',\n"\
"        'projects:application',\n"\
"    ),\n"\
"}" > /etc/gunicorn.d/trac
service gunicorn start

To create a new subversion repository and corresponding trac instance with a given name, execute:

export NAME=test
sudo -u svn svnadmin create /var/lib/svn/repos/$NAME
sudo -u svn mkdir /var/lib/svn/backups/$NAME
rm -r /var/lib/svn/repos/$NAME/hooks
cp -ra /var/lib/svn/skel /var/lib/svn/repos/$NAME/hooks
sed -i s/NAME/$NAME/ /var/lib/svn/repos/$NAME/hooks/*
sudo -u trac /home/$SW/trac/bin/trac-admin /var/lib/trac/projects/$NAME initenv

The reverse proxy configuration is straight forward and shown for as an nginx config snipped here

location /repos { proxy_pass; }

location /projects {
  rewrite ^/projects/(.*)$ /$1 break;
location ~ ^/projects/[a-z]+/login {
  auth_basic "trac projects";
  auth_basic_user_file /etc/auth/htpasswd;
  proxy_set_header REMOTE_USER $remote_user;
  rewrite ^/projects/(.*)$ /$1 break;

Thank's all, folks.

Excel FollowHyperlink loads URL twice

I'm writing a web app which receives data from Excel 2010 on Windows 7 (64 bit). I'm using a small VBA macro for data upload and web page opening. However, when using ActiveWorkbook.FollowHyperlink in VBA the website is not opened by my default browser initially (Chrome in my example), but by some MS Office internal Internet Explorer instance. I'm redirecting from this first URL … and my real web browser is called on the redirected URL only. Got it? The header of the first access contains the following user agent:

User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; ms-office)

The server response with a redirect and the other url is called by the same user agent. At the end this second url is opened with my default browser, which happens to be:

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3

Unfortunately this is not useful, as I'm setting a cookie at the first request (where the redirect occurs). I only found an ugly solution using ShellExecute (so far):

Private Declare Function ShellExecute Lib "shell32.dll" Alias "ShellExecuteA" (ByVal hwnd As Long, ByVal lpOperation As String, ByVal lpFile As String, ByVal lpParameters As String, ByVal lpDirectory As String, ByVal nShowCmd As Long) As Long

and then opening the website by:

ShellExecute 0, vbNullString, "http://...", vbNullString, vbNullString, vbNormalFocus

Others seem to have the same problem ( But ShellExecute really should only be a last resort. Never mind, it's just Microsoft.

Py3k WSGI Test

I started a small project back in May where I was trying to use Python 3 for web development for the first time. While it somehow worked (with ugly hacks in the stdlib and being a memory hog), I finally decided it to be a failure. As I needed the project to get in production I switched back to Python 2 and everything worked out well. Still, I'm interested in web development on Python 3. Recently the author of the Python 3 enabled WSGI framework bottle started a new project multipart to fix the problems with cgi.FieldStorage. As the WSGI problems on Python 3 are under constant discussion I want to demonstrate the use of multipart in bottle even though multipart has not yet been integrated in bottle by the author. In my mind it can be used already (at least for development and test instances), it works well, and doesn't look ugly at all. So you are welcome to have a look at the source of my sample application and test the demo instance (edit: link to the demo instance removed, as it is not available anymore).

cgi multipart bugs in py3k

Recently I started to implement a small web app in py3k and wsgi. While none of the major python web frameworks are ready for py3k anytime soon and wsgi for py3k is still under major discussions (see for example this current post by Armin Ronacher), there are already options to actually start working on the subject.

I'm using bottle (dev tree) and the current releases of sqlalchemy, py-postgresql and jinja2 on Python 3.1.2. This actually works sort of. I will likely write some further posts about my progress. However, I quickly found a bug in the cgi multipart input handling, which actually turned out to arise from two different sources. I tried to solve it and just opened issue 8846 on the Python issue tracker showing my findings.