This blog post is about releasing your Python packages (eggs) in reliable, repeatable and automatic manner. The tools and systems presented in the post are Python specific, but general principles apply for other programming languages too. If you are looking forward to share your applications and libraries with fellow peer developers you may find the information here useful.

• Make sure you are building your packages on robust scaffold.
• Communicate the intent of your package to your fellow developers. Package repositories should not be dumps for random code. It is very annoying to skim through packages with bad descriptions.
• Automatized release process is less prone to human errors. The time to spend to automatize the process will be paid back generously in the future.

1. About PyPi package repository

When you have finished a new version of your library, app or thingy it is time to wrap it into a package and publish it to the wild. In Python world this happens by pushing packages to pypi.python.org also known as Python Package Index (PyPi was formerly known as Cheeseshop).

pypi.python.org

• Contains centralized listing of available Python packages, also known as eggs
• Has short package introduction page written in reStructuredText mark-up
• Can host downloads, both binary and source eggs
• Also can host documentation and other HTML content (though for the documentation hosting I recommend readthedocs.org which comes with continuous VCS integration)

In the past PyPi had some reputation problems of downtime and service interrupts. Nowadays it is well mirrored and usually if you have package installation problems with pip or easy_install it is because people are not hosting packages on PyPi, but on their own server (or on sourceforge.net which has some issues hosting packages for automated downloads).

For team internal releases you can use any HTTP capable server or just a shared network folder as long as you tell easy_install or pip to install from this location.

2. What makes a high quality package release

A high quality package release makes it easy for the package users to understand what the package does, how to install and use it.

Each published package should minimally contain

• Short description (one liner for the package listing)
• Long description (restructured text) which explains the context of the package – in what kind of environment and situations you are interested in using this package.
• Documentation link for in-depth manual
• Version control system link
• Issue tracker link
• Author information (who, where, motivation)
• The suggested method how to contact authors
• Developer information: followed coding conventions, recommendations for submitting patches
• Changelog: how the package has evolved in the past versions. This is very important for the fellow peer developers, but proper changelog management is something only rare packages manage do well. Luckily there is medicine for this down in the blog post.
• Host downloadable Python egg(s) on pypi.python.org
• Package releases properly tagged in the version control system (Highly annoying stuff to do manually)

Python packages use Trove classifiers for package classification. You define the classifiers for your package in setup.py.

For long descriptions, please use  in-page table of contents for jump links (.. contents :: :local: restructured text directive).

Provide as many as possible methods contacting the author. One email address is not often enough, especially if it belongs to a company. You, as the owner of the PyPi page, are the dictator of the releases. If you abandon the package in some point people need to be able to take over. I have had cases where I need to hunt down the authors of abandoned packages just to make a bug fix release after four years. Hunting people is not  fun.

An example of  a high quality package release: z3c.form (detailed change log, documentation link, etc.)

An example of a high profile, but low quality, package release: PIL (lacks all usable metadata, not propeply packaged, not hosted on PyPi)  Django, as the community, has also made a habit of doing useless low quality releases (probably because Django itself does not fully utilize or encourage the use of Python package mechanisms)

3.  What is a package release process

To make high quality releases you need to have a well-defined process for doing it. The process is where the repeatability, the necessity of quality, is born.

First your package must be based on a proper Python package files layout making things easier to manage. In Python world, it is recommend that you base your Python packages to Paste code skeleton templates a.k.a. scaffolds. Usually these are specific to the framework you are using.

• Generic Python packages can be based on templer.core basic_namespace template
• Django: not sure what they are doing nowadays
• Pyramid has its own Paste templates
• Plone has its own templates
• etc.

The scaffold generates boilerplate which is needed in order to package your egg properly. The scaffold provides setup.py defining your package metadata, README.txt (README.rst recommend nowadays), CHANGES.txt changelog file, MANIFEST.in to define included files besides normal .py files and so on.

Also if you are having a fresh project start on Github consider using the great feature of pythonpackages.com which generates the code skeleton through Github API. It creates a Github repo with the necessary boilerplate in place for you.

After the skeleton files are in place you can start thinking the actual release process. Below is the outlline of tasks you need to do every time you release a package. Like a lot of good things in Python world, some of the best practices were pioneered by Zope / Plone folks.

3. Prerelease

• Give a new release version number by fair dice roll
• Update setup.py with the new version number
• Update CHANGES.txt with the version number and the release date
• Commit changes to trunk

3. Release

• Tag the release in the version control system
• Checkout the tag into a clean folder
• Run necessary scripts to build binary files for the distribution (like .mo files – more about this later)
• Create a package (python setup.py sdist)
• Upload the package to PyPi (python setup.py upload)
• Tag and upload new documentation

3. Postrelease

• Update setup.py with the next version development version number and development tag in the version (-dev, -trunk)
• Create a fresh change log entry in CHANGES.txt where you start collecting changes for the next release
• Commit to the trunk

Together these three steps are called a full release. It’s a lot of work to do it by hand every time you need to make a release…

4. zest.releaser, the super star of Python packaging world

Now… what would you give if I told you you can automatize all the above tasks with a single press of a button (… or more technically a single CLI command). zest.releaser is a Python tool specific tool to automate the package release process.

5. Installing zest.releaser

Here is how to install zest.releaser with various version control support (important!) in a virtualenv. (Please note that zest.releaser internally uses setuptools which is not very user friendly and the complex install process and many other issues stem from this)

# Don't rely on system virtualenv as it has issues on older
# systems like Ubuntu 10.04
curl -L -o virtualenv.py https://raw.github.com/pypa/virtualenv/master/virtualenv.py
python virtualenv.py releaser-venv
source releaser-venv/bin/activate
easy_install zest.releaser zest.pocompile
easy_install setuptools-git setuptools_subversion
easy_install setuptools_hg
# easy_install setuptools_bzr Broken for me...

Alternatively you can use buildout based installation.

6. Making a release with zest.releaser

zest.releaser automates the process above. It does not just do what you ask, but also has helpful suggestions which could break otherwise fragile and human-error prone release process. Below is a captured terminal session how to a do release.

[~/code/gomobile/src/mfabrik.webandmobile]% ../../bin/fullrelease           
INFO: Starting prerelease.
Enter version [1.0.13]:
INFO: Set setup.py's version to '1.0.13'
INFO: Changed version from '1.0.13.dev0' to '1.0.13'
INFO: History file ./docs/HISTORY.txt updated.
INFO: The 'svn diff':

Index: docs/HISTORY.txt
===================================================================
--- docs/HISTORY.txt    (revision 1105)
+++ docs/HISTORY.txt    (working copy)
@@ -1,7 +1,7 @@
 Changelog
 =========

-1.0.14 (unreleased)
+1.0.13 (2012-08-14)
 -------------------

 - Handle external image resize in more fashioned manner [miohtama]
@@ -12,9 +12,6 @@

 - Added Plone 4.2 example buildout.cfg [miohtama]

-1.0.13 (unreleased)
--------------------
-
 - Fixed broken released eggs [miohtama], see https://github.com/zestsoftware/zest.releaser/issues/10

 1.0.12 (2012-06-20)
Index: setup.py
===================================================================
--- setup.py    (revision 1105)
+++ setup.py    (working copy)
@@ -1,7 +1,7 @@
 from setuptools import setup, find_packages
 import os

-version = '1.0.13.dev0'
+version = '1.0.13'

 setup(name='mfabrik.webandmobile',
       version=version,

OK to commit this (Y/n)? y
INFO: Sending        docs/HISTORY.txt
Sending        setup.py
Transmitting file data ..
Committed revision 1106.

INFO: Starting release.
Tag needed to proceed, you can use the following command:
svn cp https://plonegomobile.googlecode.com/svn/mfabrik.webandmobile/trunk https://plonegomobile.googlecode.com/svn/mfabrik.webandmobile/tags/1.0.13 -m "Tagging 1.0.13"
Run this command (Y/n)? y

Committed revision 1107.

Check out the tag (for tweaks or pypi/distutils server upload) (Y/n)? y
INFO: Doing a checkout...
A    /var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd/setup.py
...
A    /var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd/setup.cfg
Checked out revision 1107.

INFO: Tag checkout placed in /private/var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd
INFO: Making an egg of a fresh tag checkout.
running sdist
running egg_info
creating mfabrik.webandmobile.egg-info
...
copying setup.cfg -> mfabrik.webandmobile-1.0.13
Writing mfabrik.webandmobile-1.0.13/setup.cfg
creating dist
creating 'dist/mfabrik.webandmobile-1.0.13.zip' and adding 'mfabrik.webandmobile-1.0.13' to it
adding 'mfabrik.webandmobile-1.0.13/MANIFEST.in'
...
removing 'mfabrik.webandmobile-1.0.13' (and everything under it)
unrecognized .svn/entries format in
warning: no files found matching '*' under directory 'main_directory'
warning: no previously-included files matching '*.pyc' found anywhere in distribution

WARNING: This package is NOT registered on PyPI.
Register and upload to pypi (y/N)? y
Please explicitly answer yes/no in full (or accept the default)
Register and upload to pypi (y/N)? yes
INFO: Running: /Users/mikko/code/plone-venv/bin/python setup.py register -r pypi sdist --formats=zip upload -r pypi
Showing first few lines...
running register
running egg_info
writing paster_plugins to mfabrik.webandmobile.egg-info/paster_plugins.txt
writing requirements to mfabrik.webandmobile.egg-info/requires.txt
writing mfabrik.webandmobile.egg-info/PKG-INFO
...
Showing last few lines...
unrecognized .svn/entries format in
# HAHAHA %!"(%!" PYPI FAILED JUST TODAY
# WHEN I HAD TO WRITE THE BLOG POST
Upload failed (503): Server Maintenance

Register and upload to plone.org (Y/n)? yes
INFO: Running: /Users/mikko/code/plone-venv/bin/python setup.py register -r plone.org sdist --formats=zip upload -r plone.org
Showing first few lines...
running register
running egg_info
writing paster_plugins to mfabrik.webandmobile.egg-info/paster_plugins.txt
writing requirements to mfabrik.webandmobile.egg-info/requires.txt
writing mfabrik.webandmobile.egg-info/PKG-INFO
...
Showing last few lines...
Server response (200): OK
unrecognized .svn/entries format in
warning: no files found matching '*' under directory 'main_directory'
warning: no previously-included files matching '*.pyc' found anywhere in distribution

INFO: Starting postrelease.
Current version is '1.0.13'
Enter new development version ('.dev0' will be appended) [1.0.14]:
INFO: New version string is '1.0.14.dev0'
INFO: Set setup.py's version to '1.0.14.dev0'
INFO: Injected new section into the history: '1.0.14 (unreleased)'
INFO: The 'svn diff':

...

+1.0.14 (unreleased)
+-------------------
+
+- Nothing changed yet.
+
+
 1.0.13 (2012-08-14)
 -------------------

OK to commit this (Y/n)? y
INFO: Sending        docs/HISTORY.txt
Sending        setup.py
Transmitting file data ..
Committed revision 1108.

INFO: Finished full release.
INFO: Reminder: tag checkout is in /private/var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd

With simple yes/no questions zest.releaser does everything for you and is very informative about the progress.

7. Handling ignored and otherwise special files (i18n)

Normally zest.releaser (or more accurately the underling setuptools) does not include any files in the package which are in the version control ignore (e.g. .pyc, .svn, so on…).

However some machine generated files which should not go into version control should still packaged into the egg. The most notable example is gettext .mo files. Since they are binary files generated from .po, they are not subject to version control. But since the applications and libraries need those files and they are not automatically generated on start-up, they should be distributed in the egg.

zest.releaser has an add-on called zest.pocompile which will compile .po -> .mo in release step. This is 100% automatic and if zest.pocompile is installed it should just work. Just make sure you include everything in your MANIFEST.in to cover .mo files too – not just .py files. An example:

recursive-include yourpackagename *
recursive-include docs *
include *
global-exclude *.pyc

More about MANIFEST.in.

Note: setuptools does not natively support version control systems and may have issues with e.g. too new Subversion repository versions (1.7 issues recently) so MANIFEST file is suggested even if you hope to rely on version controlled file pick up for packaging.

8. Don’t abuse PyPi

• Never release beta or alpha packages on PyPi – this will most likely break your fellow peer developer applications, make them angry and alienate your user base. See notes below if you need a special egg index server for beta releases.
• Never ever take down old PyPi releases – there are 5-10 years old system which still may rely on these packages being on PyPI….
• Don’t release random crap on PyPi. Just keep it on the Github etc. if you are not committed to maintain high quality release process in the future. pip and setuptools can directly install from Gtihub. Random crap pollutes PyPi making it harder to find the actually useful stuff.

I will educate you about the meaning of Finnish proverb “taking one behind the sauna” if you, after reading this, do any of the things above.

9. pythonpackages.com, though-the-web releasing and test index server

pythonpackages.com is a recent venture by Alex Clark for more user friendly Python packaging and releasing.

pythonpackages.com has an automated release process through a web interface and Github API. However as far as I know it does not keep CHANGES.txt file in order with pre- and postrelease steps so it does not provide yet such automation degree as zest.releaser does.

However, one obvious benefit of pythonpackages.com is that it you can first upload your egg to a test index server, so you can more easily release non-QA versions like alpha and beta releases.

Feel free to share best practices of release processes of other programming ecosystems in the comments 🙂

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+