Debsources got swag and continous integration
Posted: May 07, 2015
Categories: debian, debsources, python
Debsources (http://sources.debian.net) is still under active development. We recently had a Gnome Outreachy intern, Jingjie Jiang, and we're about to work with 2 GSoC students, Clément Schreiner and Orestis Ioannou.
I will present here the GitHub mirror we've set up, in order to allow external pull requests to be submitted, and to use the continous integration service provided by Travis-CI.
GitHub and Travis-CI
Debsources' source code is hosted on Debian's git servers, and from there is mirrored to GitHub. Every time a commit is pushed (to master or other branches) or a pull request is open, the test suite will be automatically run on Travis-CI, and the result (tests pass or don't) is displayed on GitHub. This allows us to quickly filter external contributions (when they are submitted on GitHub), and be sure everything works with our setup, before reviewing work.
Travis-CI runs the tests on OpenVZ containers. The complete infrastructure was a bit challenging to setup, but as we now have a Docker recipe to quicly begin to hack on Debsources, most of the work could be done using the Dockerfile instructions.
In average, a run on Travis-CI (which includes git cloning the code and test data, setup the server, and run the tests suite) takes 7 minutes, which is an ok amount of time to wait for before submitting a pull request, in my opinion.
Bugs discovered in the process
Setting up this continuous integration infrastructure made me discover a few bugs.
Python magic does black magic
Debsources runs fine on Debian (not surprisingly), but I got tricked by black magic when I tried to run it on Ubuntu (which is the OS run in Travis-CI's containers).
We use the magic library to guess the type of files we're dealing with, for instance when we need to decide between rendering a file (for text files) or downloading it (for binary files).
Here comes the tricky part: the Python bindings for libmagic are not the same in Debian and Pypi. Debsources uses Debian package python-magic, which is not in Ubuntu 12.04. Moreover, there's no Python egg for it on Pypi, which has however another package (called magic) which provides a different API.
I solved this with a dirty hack, using the fact python-magic lies in a single file:
mkdir /tmp/python-magic && wget https://raw.githubusercontent.com/file/file/master/python/magic.py -O /tmp/python-magic/magic.py && export PYTHONPATH=/tmp/python-magic/:$PYTHONPATH
It simply downloads the library, saves it in a temporary folder and includes it in the Python path. Let's see for how long it works before everything breaks!
Edit: Thanks to Clément Schreiner, python-magic is now installed by
cloning it's Github repository and
pip install'ing into it.
Size of a directory
One test in the suite was ensuring the information returned by
on a directory and stored in the DB was the right information. Inode
metadata was tested, such as name, permissions, type, or size.
Interestingly enough, the size of a directory was tested, and expected to be 4096 bytes. The size of a directory actually depends on the filesystem in use, and on the number of files this directory contains. We often see 4096 because it's the size of a not-too-big directory on ext4.
Travis-CI doesn't use ext4:
$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /vz/private/209140041 simfs 125829120 103460612 22368508 83% / none devtmpfs 1572864 8 1572856 1% /dev none tmpfs 314576 56 314520 1% /run none tmpfs 5120 4 5116 1% /run/lock none tmpfs 1572864 0 1572864 0% /run/shm /dev/null tmpfs 786432 171584 614848 22% /var/ramfs
Simfs is a container filesystem for OpenVZ, on which directories have different sizes than on ext4:
$ ls -al / total 0 drwxr-xr-x 23 root root 480 Feb 4 18:08 . drwxr-xr-x 23 root root 480 Feb 4 18:08 .. drwxr-xr-x 2 root root 2480 Feb 4 18:20 bin drwxr-xr-x 2 root root 40 Apr 19 2012 boot drwxr-xr-x 5 root root 660 Apr 30 13:56 dev drwxr-xr-x 99 root root 3560 Apr 30 13:56 etc -rw-r--r-- 1 root root 0 Feb 4 17:56 fastboot drwxr-xr-x 3 root root 80 Feb 4 17:57 home [...]
Directory sizes are not even powers of 2.
Hence I changed the test to not check directory sizes. Hopefully this will help to make Debsources work on more filesystems!
An empty file is hiding
Last but not least, because this
still open in the wild. A file, which appears to be empty, is not
taken into account by Debsources' updater. This file is
is present in the filesystem in the container, is not the only empty
file over there, but still doesn't appear in the database, and make
fail the test which counts files.
The test has been commented out (booooooh), so that we still can use Travis-CI's platform for our GSoC students, before it's fixed.
Making Debsources run automatically on a different platform as the one we usually use permitted us to spot bugs, write dirty hacks, and expand the filesystems it's supposed to run on.
Now, let's hope the continuous integration will help our GSoC students, and let's wish them good luck!