This blog post is about releasing your Python packages (eggs) in reliable, repeatable and automatic manner. The tools and systems presented in the post are Python specific, but general principles apply for other programming languages too. If you are looking forward to share your applications and libraries with fellow peer developers you may find the information here useful.
- Make sure you are building your packages on robust scaffold.
- Communicate the intent of your package to your fellow developers. Package repositories should not be dumps for random code. It is very annoying to skim through packages with bad descriptions.
- Automatized release process is less prone to human errors. The time to spend to automatize the process will be paid back generously in the future.
Table Of Content
1. About PyPi package repository
When you have finished a new version of your library, app or thingy it is time to wrap it into a package and publish it to the wild. In Python world this happens by pushing packages to pypi.python.org also known as Python Package Index (PyPi was formerly known as Cheeseshop).
- Contains centralized listing of available Python packages, also known as eggs
- Has short package introduction page written in reStructuredText mark-up
- Can host downloads, both binary and source eggs
- Also can host documentation and other HTML content (though for the documentation hosting I recommend readthedocs.org which comes with continuous VCS integration)
In the past PyPi had some reputation problems of downtime and service interrupts. Nowadays it is well mirrored and usually if you have package installation problems with pip or easy_install it is because people are not hosting packages on PyPi, but on their own server (or on sourceforge.net which has some issues hosting packages for automated downloads).
For team internal releases you can use any HTTP capable server or just a shared network folder as long as you tell easy_install or pip to install from this location.
2. What makes a high quality package release
A high quality package release makes it easy for the package users to understand what the package does, how to install and use it.
Each published package should minimally contain
- Short description (one liner for the package listing)
- Long description (restructured text) which explains the context of the package – in what kind of environment and situations you are interested in using this package.
- Documentation link for in-depth manual
- Version control system link
- Issue tracker link
- Author information (who, where, motivation)
- The suggested method how to contact authors
- Developer information: followed coding conventions, recommendations for submitting patches
- Changelog: how the package has evolved in the past versions. This is very important for the fellow peer developers, but proper changelog management is something only rare packages manage do well. Luckily there is medicine for this down in the blog post.
- Host downloadable Python egg(s) on pypi.python.org
- Package releases properly tagged in the version control system (Highly annoying stuff to do manually)
Python packages use Trove classifiers for package classification. You define the classifiers for your package in setup.py.
For long descriptions, please use in-page table of contents for jump links (.. contents :: :local: restructured text directive).
Provide as many as possible methods contacting the author. One email address is not often enough, especially if it belongs to a company. You, as the owner of the PyPi page, are the dictator of the releases. If you abandon the package in some point people need to be able to take over. I have had cases where I need to hunt down the authors of abandoned packages just to make a bug fix release after four years. Hunting people is not fun.
An example of a high quality package release: z3c.form (detailed change log, documentation link, etc.)
An example of a high profile, but low quality, package release: PIL (lacks all usable metadata, not propeply packaged, not hosted on PyPi) Django, as the community, has also made a habit of doing useless low quality releases (probably because Django itself does not fully utilize or encourage the use of Python package mechanisms)
3. What is a package release process
To make high quality releases you need to have a well-defined process for doing it. The process is where the repeatability, the necessity of quality, is born.
First your package must be based on a proper Python package files layout making things easier to manage. In Python world, it is recommend that you base your Python packages to Paste code skeleton templates a.k.a. scaffolds. Usually these are specific to the framework you are using.
- Generic Python packages can be based on templer.core basic_namespace template
- Django: not sure what they are doing nowadays
- Pyramid has its own Paste templates
- Plone has its own templates
The scaffold generates boilerplate which is needed in order to package your egg properly. The scaffold provides setup.py defining your package metadata, README.txt (README.rst recommend nowadays), CHANGES.txt changelog file, MANIFEST.in to define included files besides normal .py files and so on.
Also if you are having a fresh project start on Github consider using the great feature of pythonpackages.com which generates the code skeleton through Github API. It creates a Github repo with the necessary boilerplate in place for you.
After the skeleton files are in place you can start thinking the actual release process. Below is the outlline of tasks you need to do every time you release a package. Like a lot of good things in Python world, some of the best practices were pioneered by Zope / Plone folks.
- Give a new release version number by fair dice roll
- Update setup.py with the new version number
- Update CHANGES.txt with the version number and the release date
- Commit changes to trunk
- Tag the release in the version control system
- Checkout the tag into a clean folder
- Run necessary scripts to build binary files for the distribution (like .mo files – more about this later)
- Create a package (python setup.py sdist)
- Upload the package to PyPi (python setup.py upload)
- Tag and upload new documentation
- Update setup.py with the next version development version number and development tag in the version (-dev, -trunk)
- Create a fresh change log entry in CHANGES.txt where you start collecting changes for the next release
- Commit to the trunk
Together these three steps are called a full release. It’s a lot of work to do it by hand every time you need to make a release…
4. zest.releaser, the super star of Python packaging world
Now… what would you give if I told you you can automatize all the above tasks with a single press of a button (… or more technically a single CLI command). zest.releaser is a Python tool specific tool to automate the package release process.
5. Installing zest.releaser
Here is how to install zest.releaser with various version control support (important!) in a virtualenv. (Please note that zest.releaser internally uses setuptools which is not very user friendly and the complex install process and many other issues stem from this)
# Don't rely on system virtualenv as it has issues on older # systems like Ubuntu 10.04 curl -L -o virtualenv.py https://raw.github.com/pypa/virtualenv/master/virtualenv.py python virtualenv.py releaser-venv source releaser-venv/bin/activate easy_install zest.releaser zest.pocompile easy_install setuptools-git setuptools_subversion easy_install setuptools_hg # easy_install setuptools_bzr Broken for me...
Alternatively you can use buildout based installation.
6. Making a release with zest.releaser
zest.releaser automates the process above. It does not just do what you ask, but also has helpful suggestions which could break otherwise fragile and human-error prone release process. Below is a captured terminal session how to a do release.
[~/code/gomobile/src/mfabrik.webandmobile]% ../../bin/fullrelease INFO: Starting prerelease. Enter version [1.0.13]: INFO: Set setup.py's version to '1.0.13' INFO: Changed version from '1.0.13.dev0' to '1.0.13' INFO: History file ./docs/HISTORY.txt updated. INFO: The 'svn diff': Index: docs/HISTORY.txt =================================================================== --- docs/HISTORY.txt (revision 1105) +++ docs/HISTORY.txt (working copy) @@ -1,7 +1,7 @@ Changelog ========= -1.0.14 (unreleased) +1.0.13 (2012-08-14) ------------------- - Handle external image resize in more fashioned manner [miohtama] @@ -12,9 +12,6 @@ - Added Plone 4.2 example buildout.cfg [miohtama] -1.0.13 (unreleased) -------------------- - - Fixed broken released eggs [miohtama], see https://github.com/zestsoftware/zest.releaser/issues/10 1.0.12 (2012-06-20) Index: setup.py =================================================================== --- setup.py (revision 1105) +++ setup.py (working copy) @@ -1,7 +1,7 @@ from setuptools import setup, find_packages import os -version = '1.0.13.dev0' +version = '1.0.13' setup(name='mfabrik.webandmobile', version=version, OK to commit this (Y/n)? y INFO: Sending docs/HISTORY.txt Sending setup.py Transmitting file data .. Committed revision 1106. INFO: Starting release. Tag needed to proceed, you can use the following command: svn cp https://plonegomobile.googlecode.com/svn/mfabrik.webandmobile/trunk https://plonegomobile.googlecode.com/svn/mfabrik.webandmobile/tags/1.0.13 -m "Tagging 1.0.13" Run this command (Y/n)? y Committed revision 1107. Check out the tag (for tweaks or pypi/distutils server upload) (Y/n)? y INFO: Doing a checkout... A /var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd/setup.py ... A /var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd/setup.cfg Checked out revision 1107. INFO: Tag checkout placed in /private/var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd INFO: Making an egg of a fresh tag checkout. running sdist running egg_info creating mfabrik.webandmobile.egg-info ... copying setup.cfg -> mfabrik.webandmobile-1.0.13 Writing mfabrik.webandmobile-1.0.13/setup.cfg creating dist creating 'dist/mfabrik.webandmobile-1.0.13.zip' and adding 'mfabrik.webandmobile-1.0.13' to it adding 'mfabrik.webandmobile-1.0.13/MANIFEST.in' ... removing 'mfabrik.webandmobile-1.0.13' (and everything under it) unrecognized .svn/entries format in warning: no files found matching '*' under directory 'main_directory' warning: no previously-included files matching '*.pyc' found anywhere in distribution WARNING: This package is NOT registered on PyPI. Register and upload to pypi (y/N)? y Please explicitly answer yes/no in full (or accept the default) Register and upload to pypi (y/N)? yes INFO: Running: /Users/mikko/code/plone-venv/bin/python setup.py register -r pypi sdist --formats=zip upload -r pypi Showing first few lines... running register running egg_info writing paster_plugins to mfabrik.webandmobile.egg-info/paster_plugins.txt writing requirements to mfabrik.webandmobile.egg-info/requires.txt writing mfabrik.webandmobile.egg-info/PKG-INFO ... Showing last few lines... unrecognized .svn/entries format in # HAHAHA %!"(%!" PYPI FAILED JUST TODAY # WHEN I HAD TO WRITE THE BLOG POST Upload failed (503): Server Maintenance Register and upload to plone.org (Y/n)? yes INFO: Running: /Users/mikko/code/plone-venv/bin/python setup.py register -r plone.org sdist --formats=zip upload -r plone.org Showing first few lines... running register running egg_info writing paster_plugins to mfabrik.webandmobile.egg-info/paster_plugins.txt writing requirements to mfabrik.webandmobile.egg-info/requires.txt writing mfabrik.webandmobile.egg-info/PKG-INFO ... Showing last few lines... Server response (200): OK unrecognized .svn/entries format in warning: no files found matching '*' under directory 'main_directory' warning: no previously-included files matching '*.pyc' found anywhere in distribution INFO: Starting postrelease. Current version is '1.0.13' Enter new development version ('.dev0' will be appended) [1.0.14]: INFO: New version string is '1.0.14.dev0' INFO: Set setup.py's version to '1.0.14.dev0' INFO: Injected new section into the history: '1.0.14 (unreleased)' INFO: The 'svn diff': ... +1.0.14 (unreleased) +------------------- + +- Nothing changed yet. + + 1.0.13 (2012-08-14) ------------------- OK to commit this (Y/n)? y INFO: Sending docs/HISTORY.txt Sending setup.py Transmitting file data .. Committed revision 1108. INFO: Finished full release. INFO: Reminder: tag checkout is in /private/var/folders/6k/b150546j3yj_1_8823kgsfqc0000gn/T/mfabrik.webandmobile-1.0.13-jjNYOd
With simple yes/no questions zest.releaser does everything for you and is very informative about the progress.
7. Handling ignored and otherwise special files (i18n)
Normally zest.releaser (or more accurately the underling setuptools) does not include any files in the package which are in the version control ignore (e.g. .pyc, .svn, so on…).
However some machine generated files which should not go into version control should still packaged into the egg. The most notable example is gettext .mo files. Since they are binary files generated from .po, they are not subject to version control. But since the applications and libraries need those files and they are not automatically generated on start-up, they should be distributed in the egg.
zest.releaser has an add-on called zest.pocompile which will compile .po -> .mo in release step. This is 100% automatic and if zest.pocompile is installed it should just work. Just make sure you include everything in your MANIFEST.in to cover .mo files too – not just .py files. An example:
recursive-include yourpackagename * recursive-include docs * include * global-exclude *.pyc
Note: setuptools does not natively support version control systems and may have issues with e.g. too new Subversion repository versions (1.7 issues recently) so MANIFEST file is suggested even if you hope to rely on version controlled file pick up for packaging.
8. Don’t abuse PyPi
- Never release beta or alpha packages on PyPi – this will most likely break your fellow peer developer applications, make them angry and alienate your user base. See notes below if you need a special egg index server for beta releases.
- Never ever take down old PyPi releases – there are 5-10 years old system which still may rely on these packages being on PyPI….
- Don’t release random crap on PyPi. Just keep it on the Github etc. if you are not committed to maintain high quality release process in the future. pip and setuptools can directly install from Gtihub. Random crap pollutes PyPi making it harder to find the actually useful stuff.
I will educate you about the meaning of Finnish proverb “taking one behind the sauna” if you, after reading this, do any of the things above.
9. pythonpackages.com, though-the-web releasing and test index server
pythonpackages.com is a recent venture by Alex Clark for more user friendly Python packaging and releasing.
pythonpackages.com has an automated release process through a web interface and Github API. However as far as I know it does not keep CHANGES.txt file in order with pre- and postrelease steps so it does not provide yet such automation degree as zest.releaser does.
However, one obvious benefit of pythonpackages.com is that it you can first upload your egg to a test index server, so you can more easily release non-QA versions like alpha and beta releases.
Feel free to share best practices of release processes of other programming ecosystems in the comments