This blog post is about sauna.reload which is a Python package adding an automatic reload feature to Plone CMS.

1. The short history of reloading

The web developer community takes reload-code-on-development as granted. You edit your source file, hit refresh and poof: your changed are there. It is a good way and the only real web development way and one of the sane things PHP community has taught to us. Things could be different: for example if you are developing an embedded code on mobile platforms you might need to build the ROM image for hours before you see your changed source code line in the action.

Side note: To check your code changes it is often faster the hit enter on the browsee address bar line as this does not reload CSS and JS files for the HTTP request

For PHP the reloading is easy as the software stack parses every PHP file again on every request. There is no reload – only load. In fact there is a whole “optimizer” open source business model spawned around PHP just to make it faster. PHP processes are stateless.

For Python the things are not so simple. Python web frameworks have separate process lifespan and HTTP request handling lifespans. Often a server process is started, it spawns N threads and each thread handles one HTTP request at a time. Python processes are stateful.

As being stateful, Python processes may start slowly. You need to parse all source code and create initial memory objects, etc. Frameworks do not optimize for fast start-up because it really doesn’t matter on the production service as you spawn Python processes only once, when the server is booted.

Plone CMS, coming with choking 250 MB of source code (Linux kernel has 400-500 MB) is the extreme of this slow initialization. Plone CMS is the biggest open source Python project out there (does anyone dare to challenge my clain?).  When Plone starts it loads most of source code to memory and parsing and initializing Python itself, plus various XML files, is quite an achievement.

What Tornado, Django, Paster etc. Python frameworks do that when they reload they simply zap the process dead and reinitialize 100% virgin process. This is ok as these low level frameworks have little overhead. Though it becomes painful slow too when your Django service grows for many applications and the code starts piling up…

For Plone 100% start up this is no go. Plone people care about the start up time, but there is not much they can do about it… even the smallest sites have start up times of dozens of seconds.

2. Python and module reload

Plone used to have (has) a helper package called plone.reload. It works by reloading Python modules. Python interpreter allows you to reload a module again in the memory. The code is actually based on xreload.py written by Guido van Rossum himself.

However, there is a catch. Naive code reload does not work. As xreload.py comments state there are issues. Global stateful objects, other initialization specific code paths, etc. are ignored. So with plone.reload you ended up with a broken Python process as many times you ended up with a successful reload. It’s so frustrating to use that you don’t actually want to use it.

3. From Finland with Love

sauna.reload was initially created in Sauna Sprint 2011 event, arranged by EESTEC and Plone Community. The event name comes from the abundance of sauna in Finland and the package name comes from the fact that a lot of sauna was put into the effort of creating it.

4.  Going trident

A fork. (The orignal image)

A spoon (The orignal image)

sauna.reload takes a different strategy of reloading

The idea behind sauna.reload goes back to the days as I was working with Python for Nokia Series 60 phones. Symbian seriously sucks, peace on its memory (though not sure if it’s dead yet or just a walking corpse). If you did any Symbian development you saw how it could never fly – such a monster it was.

One of the issues was start-up time of the apps, especially Python start up time. Symbian IO was slow and simple fact to running the application to the point where all imports had been run took several seconds. For all of this time you could just show some happy loading screen for the user. It was very difficult to deliver a Python application which would have acceptable user experience on Symbian.

Far back in the day I chatted with Jukka Laurila who gave me some tips. Emacs used to have unexec() command which simply dumped the application state to the disk and resumed later. This works nicely if your application is an isolated binary and does not have any references to DLLs, so it doesn’t work so nicely on any contemporary platform.

On the other hand, Maemo folks had something called PyLauncher. PyLauncher loaded a daemon to memory and this daemon had Python interpreter and pygtk library loaded. When Python script had to be run, the PyLauncher daemon simply fork()’ed a new child and this new child inherited already loaded Python interpreter and pygtk for free. Fork() is a copy-on-write operation on modern operating systems.

This gave me a cunning idea.

5. Taking a shortcut

When you develop a web application and you want to reload changes you usually don’t want to reload the whole stack. In fact, your custom code tends to sit up on the top of the framework stack and unless you are a framework core developers yourself, you never poke into those files.

What if we run Plone initialization process to the point it is just about to load your custom code, freeze the process and then just always go back to this point when we want to load our (changed) code?

Plone process is in fact a series of Python egg loads. Your custom eggs are loaded last.

6. With sofware only your imagination is the limit

So I presented the idea to Asko Soukka and Esa-Matti Suuronen from University of Jyväskylä (one of the largest Plone users in Finland) in Sauna Sprint 2011. If I recall correctly their first reaction was “wow dude that definitely is not going to work”.

But at the end of the tunnel was the freedom of Plone web developers – being released from the chains of Zope start-up times forever. Even if it had turned out sauna.reload was not possible we would have been stranded in the middle of the forest having nothing to do. So the guys decided to give it a crack. All the time they spent to develop sauna.reload would be paid back later with gracious multiplier.

sauna.reload is using awesome cross-platform Watchdog Python package to monitor the source code files. Plone is frozen in the start-up process before any custom source code eggs from src/ folder are being loaded. When there is a file-system change event sauna.reload kills the forked child process and reforks a new child from the frozen parent process, effectively continuing the loading from the clean table where none of your source code files were present.

7. Going uphill

San Francisco, where we are hosting Plone Conference 2011 and sprints, is a city of hills. They feel quite steep for a person who comes from a place where maps don’t have countours.

With Zope 2, the technology the ancients left to us, things were not that straightforward.

ZODB, the database used in Plone, happens in-process for the development instances (single process mode). When you go back to the orignal frozen copy of Plone process you’ll also travel back in append-only database time.

The sauna.reload devs managed to fix this. However this code is prone to ZODB internal changes as happened with Plone 4.1.

Another funky issues were how to monkey-patch Zope loading process early enough and reordering egg loading process so that we hold loading the custom code till very end.

8. For The Win

sauna.reload works on all Python packages which declare z3c.autoinclude loading in setup.py. Just go and use it. This, of course, is based on the assumption you use an operating system which supports fork() operation and unfortunately Windows is not privileged for this.

Note: OSX users might need to use trunk version of Watchdog until a bug fix is released

Your Plone restart time goes down to 1 second from 40 seconds. Also, the restart happens automatically every time you save your files. It works with Grok, Zope component architecture, everything… ZCML is reloaded too. You should feel an unsurmountable boost in your productivity.

We still have some quirks in sauna.reload. If you try hard enough you rarely might be able to corrupt database. We don’t know how, so keep trying and report your findings pack to GitHub issue tracker.  Also as sauna.reload is meant for the development only you shouldn’t really worry about breaking  things. We also have an old issue tracker which was not migrated to collective – how one can migrate a issue tracker on Github?

9. The Future

Who knows? Maybe someone gets inspiration of this work and ports it to other Python frameworks.

Our plan is to release a Plone developer distribution which comes with sauna.reload and other goodies by default. This way new Plone developers could instantly start hacking their code by just copy-pasting examples and keep hitting refresh.

 

 Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+