Python 3.4 and Django on Ubuntu 13.10

Why bother about Python versions? 

I recently started a new project creating a web application. As I have a lot of Python programming experience I chose Python with Django over Ruby on Rails. At the beginning of a new project I prefer using the latest versions of the frameworks the application will depend on. Starting now with Python 2.7 would mean that sooner or later there would be additional work porting the codebase to Python 3. Yesterday, Python 3.4 was released. One of the biggest improvements is that it has pip already included which makes handling virtual environments and installing the latest release of Django really easy.

Building Python from source

The downside is, that Linux distributions do not include the latest Python release yet. Most of them still ship with Python 2.7 as default version. The next Fedora and Ubuntu releases might change that, but for now you need to compile it from source. Luckily that is not a hard task. Go to the download page and grab the latest Python release (recommended if you read the post later and a newer version was released) or past the following command into a terminal.

First make sure you have everything installed to compile Python from source.

sudo apt-get install build-essential

Before downloading create a temporary directory to make the cleanup easier. At the end you can just delete “tmpPython”.

mkdir tmpPython
cd tmpPython
wget --no-check-certificate https://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar xvf Python-3.4.0.tgz

After the archive is extracted, cd into the source directory. Create a directory to install to, then run configure, build and install.

cd Python-3.4.0
sudo mkdir /opt/Python34
./configure --prefix=/opt/Python34 && make -j4
sudo make install

Now you have Python 3.4 installed on your system.
Add the the path containing the executable to your environment.

export PATH=/opt/Python34/bin:$PATH

Also make sure to add this line to your .bashrc file (or .zshrc if you’re using zsh).

echo "export PATH=/opt/Python3.4/bin:$PATH" >> $HOME/.bashrc

Creating a virtual environment

Go to the directory where you want to create the virtual environment. I recommend /opt if you collaborate with others within the environment (you have to create everything with sudo) or your home directory if you work alone. Then run pyvenv to create it.
pyvenv-3.4 djangoEnv
source djangoEnv/bin/activate

The bash prompt changes to

(djangoEnv) mpei@earth /opt

and that means that you are now within this virtual environment.
This command shows you what you have installed:

pip freeze

Installing Django

Just use pip to install the latest version of Django and its extension:
sudo pip install django django-extensions

And you’re done! You can check the installed versions by running “pip freeze” again. Maybe another blog post on Django and databases? Or the first steps in Django? We’ll see… bye bye!

Organizing C/C++ includes

After starting my new job programming in a big software project I spent some thought on organizing includes and give a recommendation. Here’s what I’ve come up with. As always, some things are obvious, some are not…

  1. You should only include what is necessary to compile the source code. Adding unnecessary
    includes means a longer compilation time, especially in large projects.
  2. Each header and corresponding source file should compile cleanly on its own. That
    means, if you have a source file that includes only the corresponding header, it
    should compile without errors. The header file should include not more
    than what is necessary for that.
  3. Try to use forward declarations as much as possible. If you’re using a
    class, but the header file only deals with pointers/references to
    objects of that class, then there’s no need to include the definition of
    the class. That is what forward declarations are designed for!

    // Forward declaration

    class MyClass;

  4. Note that some system headers might include others. But apart from a few
    exceptions, there is no requirement. So if you need both
    <iostream> and <string> include both, even if you can
    compile only with one of them.
  5. To prevent multiple-inclusions, with loops and all such attendant horrors is having an #ifndef guard.

    #ifndef _FOO_H
    #define _FOO_H
      …contents of header file…
    #endif

  6. The order in which the includes appear (system includes and user includes) is up to the coding standard you follow.
  7. If you have the choice of picking a coding standard regarding the order at the beginning of a new project, I recommend to go from local to global, each
    subsection in alphabetical order. That way you can avoid introducing
    hidden dependencies. If you reverse the order and i.e. myclass.cpp
    includes <string> then <myclass.h>, there is no way to catch
    at build time that myclass.h may itself depend on string. So if later someone includes myclass.h but does not need string, he’ll
    get an error that needs to be fixed either in the cpp or in the header
    itself. 
  8. So the recommended order would be:
    • header file corresponding to its .cpp file
    • headers from the same component
    • headers from other components
    • system headers

If you use the Eclipse IDE (which I highly recommend), you can use a very nice feature that helps you organizing includes (“Source -> Organize Includes”).

5 Steps: Build a custom kernel in Debian Wheezy (…including NVIDIA drivers)

There may be many reasons to build your custom kernel. Either you have bought a new piece of hardware that is not supported by your current distribution, enable or disable features or just because you can.

Here are five easy steps to build a custom kernel on a debian system including NVIDIA kernel module. There may be way better tutorials on building a custom kernel out there, I just want to give you the essential steps that I use…

As I am writing this the latest kernel from the mainline is version 3.5. The latest stable release, which is what I recommend to build, is 3.4.6. But the following steps are for building the latest release from the mainline. 
So become root by typing “su” and entering your password in the console.
Make sure you have the requirements for building a kernel and the nvidia modules installed. If not you can install them by running:
$ apt-get install -f module-assistant build-essential

1. Download the full source code from kernel.org and extract it to /usr/src:

$ cd /usr/src/

$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.5.tr.bz2
$ tar -jxf linux-3.5.tar.bz2

2. Take the configuration of your current kernel by running:

$ make oldconfig
The script asks you for settings of features that were not available in your current kernel and therefore not available in the current configuration. I recommend to select the default values (by simply pressing ENTER several times).

3. Edit the kernel configuration

make xconfig

This will bring up a GUI that makes it easy to edit the kernel configuration.

4. Build the kernel, install the modules and install the kernel

$ make
$ make modules_install
$ make install

Then reboot the system. Boot the new kernel you installed by selecting it from the GRUB boot menu. The system will not boot to graphical desktop unless you have configured X to use the nouveau driver. So the last step is installing the NVIDIA driver.

5. Install the NVIDIA driver

Log in as root and run the module assistant to compile and install the NVIDIA kernel module.

$ m-a auto-install nvidia-kernel
After the installation is finished reboot the system and you’re done 🙂


Update: How to patch the custom kernel 

Kernel 3.5.1 was released. There is no need to Download the full source. You can simply download the patch and apply it to your source. Become root by typing “su” and your password and change to /usr/src.
Download the patch:
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/patch-3.5.1.bz2
Extract it: 
$ bunzip2 patch-3.5.1.bz2
Change to source dir and apply the patch:
$ cd linux-3.5
$ patch -p1 < ../patch-3.5.1
Configure, build and install the updated kernel:
$ make oldconfig
$ make
$ make modules_install
$ make install
You also need to repeat the last step from above to rebuild the NVIDIA driver:
$ m-a auto-install nvidia-kernel
Reboot and enjoy the latest version of Linux 🙂 

TeX Live 2012

Last weekend, TeX Live 2012 has been released. Most Linux distributions are still stuck with TeX Live 2009. If you want to enjoy the latest versions of all TeX packages can download it from here.
Another huge advantage compared to using the package that comes with the Linux distribution is, that you can use tlmgr, the TeX Live package manager.
After unpacking the archive, move to the resulting
install-tl-* subdirectory and start the installation by running install-tl. Leave everything at default values and press “I” to start the installation.
This will take some time since the installer has to download all packages.
Then add the following to either your .bashrc or .zshrc depending if you’re using bash or zshell.
export PATH=”/usr/local/texlive/2012/bin/x86_64-linux:$PATH”
export MANPATH=”/usr/local/texlive/2012/texmf/doc/man:$MANPATH”
export INFOPATH=”/usr/local/texlive/2012/texmf/doc/info:$INFOPATH”
export TEXMFHOME=”/usr/local/texlive/2012/texmf”
export TEXMFCNF=”/usr/local/texlive/2012/texmf/web2c”
I suggest to add these lines also to the .bashrc or .zshrc of root.
If you want to update your Tex Live distribution later you now can simply type:
tlmgr update –all

Skype 4.0 for Linux

Since 14th of June the long awaited version 4.0 of Skype is available for download. The changes are listed by Marco in his blog.
The last update was 2.2 beta released in April 2011. After Microsoft bought Skype in October of 2011 I thought there would never be a new Linux version. But wonders will never cease!

Spotify on Debian Wheezy (testing)

As I have a premium account for Spotify I was disappointed when I discovered that the client does not install on a recent Debian testing installation.
It depends on libssl-0.9.8 but this is not available in Wheezy anymore. The problem is solved by adding a source for Debian stable additionally to the Spotify source to the file /etc/apt/sources.list:
    deb http://repository.spotify.com stable non-free
    deb http://ftp.debian.org/debian/ stable non-free

After fetching the new sources with

    apt-get update

run

    apt-get install spotify-client

to install Spotify 🙂

Ubuntu One on Debian Wheezy


PLEASE DO NOT USE THIS AT THE MOMENT. I’M WORKING ON A BETTER AND UPDATED VERSION.

I tried to update the electron repulsion integral handler libint in our
quantum chemical code AICCM. But the linking against gmp failed on my
Ubuntu 12.04 LTS. After discovering that it would build on a Debian Wheezy I
decided to install Debian on my computer. Since I am using the Ubuntu One cloud
service to sync all my data across various machines, including
smartphone and tablet, I was disappointed to find out that there are no
packages available for Debian. I did not want to abandon the service since I am a paid subscriber. It is only 29.99$ per year for 25GB.

After searching through forums and trying
several failed approaches to use the binary packages from Ubuntu 12.04 I
decided to build it from the sources.

In this blog entry I would like to give a detailed description how to build and install Ubuntu One on a Debian system. If you have any questions or comments please do not hesitate to contact me.

Create a subdirectory in your $HOME where you want to install Ubuntu One:

mkdir $HOME/UbuntuOne
cd $HOME/UbuntuOne

and install the dependencies:

 
apt-get install python-twisted pyqt4-dev-tools bzr python-lazr.restfulclient python-oauth python-pyinotify python-protobuf gnome-common gobject-introspection xutils-dev libnautilus-extension-dev libgconf2-dev libebook1.2-dev gnome-settings-daemon-dev python-twisted-names python-libproxy python-distutils-extra python-setuptools

There are two ways to obtain the source code (1b is recommended):

1a. From bazaar repository (latest development version)

bzr branch lp:configlue

bzr branch lp:dirspec
bzr branch lp:ubuntuone-client
bzr branch lp:ubuntuone-storage-protocol
bzr branch lp:ubuntu-sso-client

If you are using this option you need to neglect the version numbers in the following.

1b. Download the tarball from launchpad (latest stable version)

wget https://launchpad.net/dirspec/stable-3-0/3.0.0/+download/dirspec-3.0.0.tar.gz
wget https://launchpad.net/ubuntu-sso-client/trunk/1.3.3/+download/ubuntu-sso-client-1.3.3.tar.gz
wget https://launchpad.net/configglue/trunk/1.0.3/+download/configglue-1.0.3.tar.gz
wget https://launchpad.net/ubuntuone-storage-protocol/stable-3-0/3.0.0/+download/ubuntuone-storage-protocol-3.0.0.tar.gz
wget https://launchpad.net/ubuntuone-client/trunk/2.0.0/+download/ubuntuone-client-2.0.0.tar.gz

If you have downloaded the latest stable version you have to extract all downloaded archives

tar -zxvf configglue-1.0.3.tar.gz
tar -zxvf dirspec-3.0.0.tar.gz
tar -zxvf ubuntuone-client-3.0.1.tar.gz
tar -zxvf ubuntuone-storage-protocol-3.0.0.tar.gz
tar -zxvf ubuntu-sso-client-1.3.3.tar.gz

2. Set the $PYTHONPATH

As most parts of the Ubuntu One client are written in Python you need to add the folders to your  $PYTHONPATH either in the .zshrc if you are using the zshell or .bashrc if you are using bash shell. The files are in your $HOME directory. If you do not know what I am talking about you are using the bash shell. Then you need to replace .zshrc by .bashrc in the following.

So open the file in your favorite editor (I use vim):

vim ~/.zshrc

And add the following lines:
# Ubuntu One PYTHONPATH
export PYTHONPATH=”$HOME/UbuntuOne/configglue-1.0.3:$PYTHONPATH”
export PYTHONPATH=”$HOME/UbuntuOne/dirspec-3.0.0:$PYTHONPATH”
export PYTHONPATH=”$HOME/UbuntuOne/ubuntuone-client-3.0.1:$PYTHONPATH”
export PYTHONPATH=”$HOME/UbuntuOne/ubuntuone-storage-protocol-3.0.0:$PYTHONPATH”
export PYTHONPATH=”$HOME/UbuntuOne/ubuntu-sso-client-1.3.3:$PYTHONPATH”

The $PATH and $LD_LIBRARY_PATH variable needs modification, too. So also add this:

# Ubuntu One PATH

export PATH=”$HOME/UbuntuOne/ubuntu-sso-client-1.3.3/bin:$PATH”
export PATH=”$HOME/UbuntuOne/ubuntuone-client-3.0.1/bin:$PATH”
export LD_LIBRARY_PATH=”$HOME/UbuntuOne/ubuntuone-client-3.0.1/libsyncdaemon:$LD_LIBRARY_PATH”

Then reload the settings:

source ~/.zshrc

3. Build the client:

The next step is to build the client. Note that you do not need to run make install and pollute your $HOME 🙂

cd $HOME/UbuntuOne/configglue-1.0.3
python setup.py config build
cd $HOME/UbuntuOne/dirspec-3.0.0
python setup.py config build
cd $HOME/UbuntuOne/ubuntuone-storage-protocol-3.0.0
python setup.py config build
cd $HOME/UbuntuOne/ubuntu-sso-client-1.3.3
python setup.py config build

cd $HOME/UbuntuOne/ubuntuone-client-3.0.1
./configure && make
rehash

3. Get an auth token

Download this script from an Ubuntu One developer (Roman Yepishev) and run it to generate an auth key:

wget http://people.canonical.com/~roman.yepishev/us/ubuntuone-sso-login.py 
python ubuntuone-sso-login.py 
Ubuntu SSO Login: **your Ubuntu SSO Login**
Password: **your Ubuntu SSO Password**
oauth=hPQWPsH:rhOokmNiRuuoiHe... 

4. Copy the configuration file and add an auth token 

Create the directory and copy the config file of the server:

mkdir ~/.config/ubuntuone
cp $HOME/UbuntuOne/ubuntuone-client-2.0.0/data/syncdaemon.conf $HOME/.config/ubuntuone/syncdaemon.conf
Open the file and add the auth token:
[__main__]
oauth=hPQWPsH:rhOokmNiRuuoiHe... 

This part (3. & 4.) is best described here.

5. Wrapper files

To be able to run the client on a headless server or via ssh you need to create 3 files and put them to $HOME/bin and add it to your $PATH variable.

The files look like this:

u1sdtool-wrapper
#!/bin/bash
# u1sdtool wrapper for headless Ubuntu One
if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
    ENVVAR=”DBUS_SESSION_BUS_ADDRESS”
    eval $(ps xe | grep “[u]buntuone-syncdaemon.*$ENVVAR” |
           sed -E “s/.*($ENVVAR=[^ ]+).*/1/g” )
    if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
        # Ubuntu One is not running and we don’t have a dbus daemon
        eval `dbus-launch –sh-syntax`
    fi
    export DBUS_SESSION_BUS_ADDRESS
fi
exec u1sdtool “$@”
ubuntuone-syncdaemon-wrapper
#!/bin/bash
# wrapper for headless Ubuntu One
if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
    ENVVAR=”DBUS_SESSION_BUS_ADDRESS”
    eval $(ps xe | grep “[u]buntuone-syncdaemon.*$ENVVAR” |
           sed -E “s/.*($ENVVAR=[^ ]+).*/1/g” )
    if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
        # Ubuntu One is not running and we don’t have a dbus daemon
        eval `dbus-launch –sh-syntax`
    fi
    export DBUS_SESSION_BUS_ADDRESS
fi
exec ubuntuone-syncdaemon $HOME/.config/ubuntuone/syncdaemon.conf &
ubuntu-sso-login-wrapper
#!/bin/bash
# ubuntu-sso-login wrapper for headless Ubuntu One
if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
    ENVVAR=”DBUS_SESSION_BUS_ADDRESS”
    eval $(ps xe | grep “[u]buntuone-syncdaemon.*$ENVVAR” |
           sed -E “s/.*($ENVVAR=[^ ]+).*/1/g” )
    if [ -z “$DBUS_SESSION_BUS_ADDRESS” ]; then
        # Ubuntu One is not running and we don’t have a dbus daemon
        eval `dbus-launch –sh-syntax`
    fi
    export DBUS_SESSION_BUS_ADDRESS
fi
exec ubuntu-sso-login “$@”

6. Start the syncdeamon

ubuntuone-syncdaemon-wrapper

7. Use the u1sdtool-wrapper

This shows you the available commands:

u1sdtool-wrapper –help

These might be tho ones you need the most:

u1sdtool-wrapper –status
u1sdtool-wrapper –current-transfers

u1sdtool-wrapper –list-folders

Examples:

1. Add a folder to be synched (that is already in the cloud)

Type:

u1sdtool-wrapper –list-folders

Get:
Folder list:
  id=2ce31368-0a79-411e-XXX subscribed=False path=/home/mpei/Xoom
Type:
u1sdtool-wrapper –subscribe-folder=2ce31368-0a79-411e-XXX

Known issues:

1. network-manager / connection: With User Not Network

I have configured my network manually via config files. I got the connection status from u1sdtool-wrapper –status that I was not connected with the network. This was due to the fact that the network manager was telling everybody via dbus that the computer does not have an internet connection.
Uninstalling network-manager via 
apt-get purge network-manager

fixed the problem for me 🙂

Hochzeitstermin (Tag 92)

Da wir bis heute noch keine Nachricht vom Standesamt bekommen haben stellen wir uns unseren Wecker auf 4 Uhr um dort anrufen zu können. Als wir uns soweit aufgerappelt haben, auf die Dachterrasse gehen und gerade den Standesbeamten in Österreich anrufen wollen erhalten wir von eben diesem eine E-Mail. Der 18.08.2012 wird uns bestĂ€tigt. :-)) Jetzt mĂŒssen wir nur noch die Location reservieren bzw. bestĂ€tigen, den Rest organisieren wir, wenn wir wieder zu Hause sind.

Wir heiraten am 25.08.2012 um 12:00h im Standesamt in Bruck an der Mur/Österreich. Die Feier findet abends im Hotel Landskron statt. Einladungen folgen 🙂

Cinvestav, Pizza und HĂ€hnchen (Tag 91)

Um halb 9 wollen wir los zum Cinvestav. Als wir JosĂ© in der KĂŒche antreffen setzen wir uns noch ein paar Minuten dazu und frĂŒhstĂŒcken. Wir erzĂ€hlen von unseren zwei Tagen allein in Puerto Escondido und der RĂŒckfahrt. FĂŒr heute Abend um 9 Uhr verabreden wir uns zu einem HĂ€hnchen auf der Dachterrasse.

Jetzt hat uns der Alltag wieder. Wir fahren mit einem gewissen Automatismus via Metro bis nach Lindavista, holen uns einen Orangensaft an dem Stand, Schokoriegel in dem Oxxo an der Ecke und fahren mit dem Bus zur UniversitÀt.

Um 11:00 Uhr gibt es einen Kakao und Espresso in der Caferteria und um 15:00 Uhr bestellt Michael ein zweites Mal via Telefon Pizza bei DominoÂŽs. Als wir uns zum Pizzaessen in die NĂ€he eines Brunnens setzen sollte Gerald doch Recht behalten: Auch im Cinvestav gibt es Kolibris. :-)) Nur fĂŒr kurze Zeit flattert das tĂŒrkisfarbene Vögelchen vor unserer Nase herum und verschwindet dann wieder.

Was mich angeht ist der Tag heute wenig produktiv. Ich tĂŒftle ein bisschen an meinem Lebenslauf und Anschreiben fĂŒr meine Bewerbung in Deutschland und google hier und dort ein bisschen nach möglichen Stellen. Ab morgen sind dann wieder produktivere Zeiten angesagt… 😉

Um kurz nach 8 machen wir uns auf den Weg nach Hause. Dort angekommen holen wir das HÀhnchen am Stand, Baguette im Globo und Wasser im Supermarkt. Txema kommt kurze Zeit spÀter im Hostal an und wir essen zusammen.

Ankunft in Mexiko-Stadt (Tag 90)

Nachts bekomme ich mit, dass wir mal im Stau stehen, schlafe jedoch dann wieder ein. Morgens um halb 7 werden wir ĂŒber Lautsprecher von dem Busfahrer geweckt und wir halten 30 Minuten bei einer AutobahnraststĂ€tte. Zeit um mal aufzustehen und ZĂ€hne putzen zu gehen. Der Muskelkater vom Reiten erreicht jetzt seinen Höhepunkt und wir watscheln schwerlich aus dem Bus. Wir haben nur noch 20 Pesos und kaufen eine Fanta.
Weiter gehtÂŽs. Über die Bildschirme werden furchtbar langweilige Filme gezeigt. Laut Michael gehört der Regisseur gesteinigt. 🙂 Also hören wir wieder dem Hörbuch zu und schlafen ein. Als wir wieder aufwachen steigen zwei Polizisten (agente federal) in den Bus und fordern etwas forsch jeden auslĂ€ndisch aussehenden Fahrgast auf, die PĂ€sse und Aufenthaltsgenehmigungen vorzuzeigen. Unsere Ausweise und Kopien vom Reisepass haben wir dabei. Etwas angespannt erklĂ€re ich dem Mann, dass wir die Genehmigungen, die wichtigsten Formulare ohne die die Ausreise nicht gestattet wĂŒrde, im Hotel gelassen haben. Alles aber halb so wild. Das nĂ€chste Mal sollen wir zumindest Kopien mitfĂŒhren. Nochmal gut gegangen. Die Busfahrt geht weiter.
Gegen Mittag kommen wir an dem Busbahnhof an und gehen direkt zum Metrogleis weiter. 20 Minuten spĂ€ter schleppen wir unsere Taschen die Wendeltreppe hoch in unser Zimmer. Endlich und leider wieder “zu Hause”. Ab heute zĂ€hle ich zugegebenermaßen rĂŒckwĂ€rts. Es sind noch 19 Tage…
Wir gehen duschen und anschließend bei “Frutos Prohibidos” essen. Ich bin gar nicht hungrig und will eigentlich nur noch schlafen. Auf dem RĂŒckweg gehen wir noch bei El Globo vorbei, Michael möchte was SĂŒĂŸes kaufen.

ZurĂŒck im Hostal schlafe ich direkt ein, gebe zwischendurch wirres Zeug von mir und spĂ€ter bloggen wir. Um halb 10 holen wir uns noch Tacos al Pastor. Der Gestank der Stadt liegt uns schwer in der Nase. Willkommen zurĂŒck.