Using paster local commands with buildout and avoiding the infamous dependency issue

1) Introduction

Paste script is a part of Python Paste web development utilities.

Its paster create command is used by various Python frameworks to generate skeleton code for a development project.

2) Paster and local commands

Besides generic project templates, paster provides local commands which are project aware commands to add more modules into an existing project. Local commands are made available by paster extensions. For example, ZopeSkel product has various local commands to generate skeletons into Plone/Zope projects automatically

  • Views
  • Content types
  • Forms
  • Portlets

… and so on.

For example, you could generate a project template for Plone add-on module and then create content types there using a local paster command. The local commands become available when you execute paster command in the folder of your project.

Example:

paster create -t archetype myorg.customploneaddon
cd src/myorg.customploneaddon

# Now new paster commands are available
paster

Usage: ../../bin/paster COMMAND
usage: paster [paster_options] COMMAND [command_options]

...

Commands:
  ...

ZopeSkel local commands:
  addcontent   Adds plone content types to your project

Above, ZopeSkel paster template adds its addcontent templates. Now you can use addcontent local command to contribute to the existing project

paster addcontent -t contenttype MyShinyWebPage
2.1. ZopeSkel

For more information how to use paster to create add-ons and add-on submodules for Plone, see here.

To see list of available paster local commands, run paster command

../../bin/paster addcontent --list

… in your development project. For ZopeSkel specific projects the output should be something like this:

Available templates:
    atschema:     A handy AT schema builder
    contenttype:  A content type skeleton
    form:         A form skeleton

3) How paster local commands work

paster reads (evaluates) setup.py file which declares a Python egg. If it founds paster_plugins section there, it will look for local commands there. For example, Plone project templates declare the following paste_plugins in setup.py:

paster_plugins = ["ZopeSkel"]

4) setup.py install_requires

Python modules can dependencies to other modules using setup.py and install_requires section. For example, a Plone add-on might read:

install_requires=['setuptools',
                  # -*- Extra requirements: -*-
                  "five.grok",
                  "plone.directives.form"
                  ],

This means that when you use setuptools/buildout/pip/whatever Python package installation tool to install a package from Python Package Index (PyPi) it will also automatically install dependency packages declared in install_requires.

4.1. paster and install_requires

This is where things usually go haywire.

Let’s assume you are using paster in a project which contains N python packages. You probably use an external configuration system to manage your installed applications and their versions to make repeatable deployments possible (hint: buildout is gaining traction in Python community lately).

Paster is not aware of this external Python package configuration set (paster cannot see them in its PYTHONPATH). So what happens when you try to execute paster create which reads setup.py containing install_requires and encounters dependencies?

Paster will try automatically download and install them locally in that folder.

Plone and Zope ecosystem contains over hundreds of reusable components, in nice dependency hierarchy. paster create would try to pull all them in to your source tree as *.egg folders. See discussion here.

Warning

Do not never use system paster command.

Do not ever run sudo easy_install ZopeSkel. Do not ever run paster local commands using a paster command from your system-wide Python installation.

Warning

The internet is full of tutorial saying easy_install ZopeSkel. If you ever encounter this kind of tutorial, it’s wrong.

5) Paste and buildout

If you are using buildout to manage your Python application deployment, you can integrate paster nicely with it.

Add to your buildout.cfg:

parts =
    ...
    paster

[paster]
recipe = zc.recipe.egg
eggs =
        PasteScript
        ZopeSkel
        ${instance:eggs}

After rerunning buildout, buildout adds paster command to bin folder.

Then you can run paster from buildout folder:

bin/paster

… or in a buildout managed project under src folder…

../../bin/paster

This way paster is aware of your deployment configuration and local commands won’t explode on your face anymore.

Thanks Martin Aspeli to helping with how buildout + paster should be done.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

9 thoughts on “Using paster local commands with buildout and avoiding the infamous dependency issue

  1. If what I read is true, I think that the web is FULL of tutorial that say to install ZopeSkel system wide… 😐

    Only I don’t really understand what is the problem doing that (if the system python is the same python I use for a buildout, having a system ZopeSkel is bad?)

  2. Mikko,

    Thanks for the writeup here. I’ve been meaning to look up the cause of this problem for a while. It seems to me that this would be a useful addition to the standard buildout recipes created by ZopeSkel. Do you feel like adding it in?

    Thanks

    c

  3. system-wide Python and paster do not see eggs configured in buildout.cfg and thus redownloads them into your product source code folder, making it unusable. In some cases it downloads > 20 eggs.

  4. I use a system-wide ZopeSkel to create initial skeletons for new buildouts and new packages. This is very convenient.

    I don’t use ZopeSkel local commands, but if I did, I’d do what Mikko does here.

    I kind of wish the local command stuff was in an extra or another package, to be honest, to avoid this kind of confusion. 🙂

    Martin

  5. Usually, if a tutorial tells you to do “sudo easy_install”, don’t do it! Maybe one exception: virtualenv itself. For the rest, there is 3 main concerns:

    1) For linux folks (with a packaging system like .deb or .rpm) easy_install will compete with your distribution packaging system. This is bad! all distributions have special paths for custom software (/opt, /usr/local/ for example), /usr/lib/ is *not* one of them.

    2) In Ubuntu or Fedora for example there is a lot of python programs use by the system. If you make unattended update of system wide libraries, you may break them (and you may wrongly blame the distribution for not providing stable releases!).

    3) An egg may bring libraries updates as dependencies. I remember having zope2.7/plone installations broken because of such system-wide update (some egg wanted something like an updated zope.interface). And it’s not so easy to trace.

    For the case of ZopeSkel, I believe a virtualenv in your home folder is the right choice. It’s just installing a specific software for a specific task: all you want is having a command to run; you may adjust your PATH environment variable. I do this for tools such as collective.releaser and collective.dist.

    (BTW, I remember it was possible to do something like “sudo easy_install –prefix=/usr/local …”)

  6. After adding paster and zopeskel to my buildout, I try to create a new egg (plone3_theme) in my buildouts src folder.

    With ../bin/paster create -t plone3_theme

    But the newly created egg still gets the packages Paste, PasteDeploy, PasteScript locally in the eggs top folder. (After running the above command, didnt run buildout yet).

    Deleting them and running buildout: the eggs reappear even though they are in the paster scripts sys.path

  7. Make sure that you don’t have Paste, PasteDeploy and PasteScript in your system PYTHONPATH (/usr/lib/python2.4). Buildout sys.path cannot override system site-packages AFAIK.

  8. “The internet is full of tutorial saying easy_install ZopeSkel. If you ever encounter this kind of tutorial, it’s wrong.”

    Yes! It took me days to realize that this was causing me problems. Several books give these instructions as well. This is one of the problems I have with documentation on plone.org–it should be easier to remove/edit outdated/incorrect information, and there should be comments (like this) for each document, so users can tell which ones are helpful and which ones are problematic.

    I also ran into an issue where I had to run buildout, after creating the product, to enable the local commands. No tutorial describes that step, but for me it was necessary to make the local commands available.

Leave a Reply

Your email address will not be published. Required fields are marked *