Monthly Archives: 2008-9

Python code management & deployment – a glance at zc.buildout and few others

Posted on by
4

We’ve been using zc.buildout for Plone deployment and it’s working out great. A few days ago implemented a buildout recipe for Django project deployment, automatic web configuration, symlinking, media-folder structuring etc. and while I got it working, I came up with twisted feelings.

Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure 🙂 but there’s of course some learning curve, so the real question is should you spend time to learn it or not?

I found out that zc.buildout has some nice features like:

  • Automatic requirements processing through setuptools
  • Automatic (yet simple) removal of directories during recipe uninstall
  • Clear structure (install(), update() & uninstall() methods)
  • INI-syntax, python does have a clear syntax but INI is always clearer for a newbie
  • Easy script creation (adjust python paths somewhat automatically)
  • Easily repeatable
  • Passing of arguments from one recipe to another
  • etc.

The problems?

  • It takes a while to learn zc.buildout
  • It takes ‘another while’ to learn to write recipes
  • Too much hassle for little things
  • INI-syntax is very limited in features
  • Buildout easily updates all your packages (that means also the ones you didn’t want to!)
  • Lack of documentation (it has good docs to get you going.. but after a while it leaves you with open questions)
  • Unnecessary overhead (for each script you launch, you’ll need a launcher script created via buildout)

There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however.

I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up.

Introducing Python for Series 60 Community Edition

Posted on by
14

This blog post will introduce Python for Series 60 Community Edition.

Pythor for Series 60 Community Edition is a new open source effort to push Python for high quality mobile phone development. It aims to provide a maintained software stack for creating real mobile applications. The codebase is derived from the original Nokia’s Python for Series 60 codebase, but has been refactored for better integration with third party extensions and patches and commercial grade application deployment.

Motivation

Building and distribution

It is difficult to distribute Python for Series 60 applications to the end user with the current Nokia’s PyS60 distribution. You probably want to modify or extend PyS60 in some way. Since the build chain and deployment model is not designed for changes this would collide with the other PyS60 installations. Symbian Platform security prevents installing conflicting binaries. Thus, one can effectively have one Nokia PyS60 application in the phone once.

There are other problems: Nokia PyS60 distribution has UIDs in Nokia protected range. Embedded SIS file cause extra installation dialog and an application manager uninstall entry. Trimming down Nokia PyS60 distribution is a little bit difficult.

To overcome all these issues we created a build chain which spits out monolithic PyS60 distributions. We build only one DLL whose name and UID can be decided. Also the build chain is Scons for Symbian, scrapping the obscure, inflexible and difficult to understand Symbian ABLD once for all.

Evolution towards higher quality

Currently there is no centralized authority to co-ordinate PyS60 open source developers and maintain the repository of all the extension and patches. This effectively prevents the biggest benefit of open source: open innovation and gradual evolution of the product. It would be very nice having all those third party extensions, now scattered around the internet, under one maintained source – making PyS60 more functional out of the box.

The community maintained repositories do not have the same restrictions as ones managed by a big public corporations. It is not a probable target of a trigger happy lawyer action and ungentlemanly competition: the discussion and plans can be public and due dilugence check of the code more relax.

We started the project in the Launchpad. Launchpad provide a distributed version control system (Bazaar) which streamlines the process of integrating third party commits and patches. This should encourage contribution. The standard build system makes it easy to roll out applications and extensions from bare  C++ source up to the end user distributable SIS files.

It is yet to see what kind of co-operation possibilities between the community and Nokia exists. In the future, it should be possible to cherry pick patches from PyS60 community edition to Nokia’s own version.

Showing the commercial potential of PyS60 in the mobile application development

On Python you can write native Series 60 applications with very little effort compared to hardcore C++ banging, lowering the barries to enter the mobile application development.

We do not deny that we have an extrinct motivation called money. Of course we have also instrict motivations like thinking Python is the best programming language in the world and we all want to be most respectable gurus in it. Gurus need to eat still, though. We hope that our effort does not go unnoticed in the mobile application development world and good subcontract offers fill our inboxes.

Also, there is the John McClane effect. Unless we had done it, no one had. Somebody has to save the world, despite the hangover.

It runs on Linux

Since we are no longer dependend on .BAT/Perl/Windows hindered ABLD buildchain, we can (almost) crosscompile and build native Symbian binaries in Linux and  Python applications. All good hackers use Linux – but currently there are still kinks and you need to use WINE for some parts – all sane Symbian developers are tied to Windows based tools for now and so are these instructions.

Prerequisitements

You need all this stuff to get things running.

Install Bazaar

You need Bazaar distributed version control client. We are not planning to have fixed releases for Python for Series 60 community edition any time soon. This is because 1) the most magic happens at a compiler level and we are providing a buildchain 2) we hope this fosters incoming patches.

Why Bazaar?

  • Distributed (patching made easy)
  • Works in Windows
  • User friendly
  • Integrates with Launcpad.

Install ActiveState Perl

Series 60 SDK has ActiveState as a prerequisitement for running its installer.

http://www.activestate.com/Products/activeperl/index.mhtml

Install Series 60 SDK

Use only Series 60 3.0 maintenance release. Other releases have SDK bugs preventing correct Python compilation.

Get the Windows installer from http://forum.nokia.com. Forum Nokia Registration is required.

Please use the default installation location C:\Symbian\9.1\S603rd_MR.

Install Carbide.c++ express

Carbide.c++ comes with a Windows compiler to compile the emulator binaries. You need this only if you indend to develop and test your applications on Series 60 emulator.

http://www.forum.nokia.com/info/sw.nokia.com/id/dbb8841d-832c-43a6-be13-f78119a2b4cb.html

Forum Nokia Registration is required.

Use Software updater in Carbide.c++ to install PyDev, Python developer extensions for Eclipse.

http://pydev.sourceforge.net/

Install Python 2.5

Scons build chain and our utility scripts use Python.

http://www.python.org/download/releases/2.5.2/

Use the installer EXE and the default installation location C:\Python25. If you want to use advanced Bluetooth shell (PUTools) you also need wxPython and pyserial packages.

Install SCons

Python for Series 60 build script are based on SCons. It is a build system using Python as a recipe langauge.

http://sourceforge.net/project/showfiles.php?group_id=30337

Install Subversion

Install Subversion client for Windows. This is needed for checking out Scons for Series 60.

http://www.collab.net/downloads/subversion/

Registration to CollabNet is needed to download Windows binaries.

Scons for Series 60

SCons for Series 60 is available as a separate project. SCons for Symbian is a build toolchain intended as a replacement for Perl and MMP files used on regular Symbian projects. SCons for Symbian is not limited to build Python – You can use it to build any Series 60 C++ application.

http://code.google.com/p/scons-for-symbian/

This is later checked out during to the environment construction, so you do not need to install it now.

We have included a workaround for a problem with limited command line length on Windows.

Included tools

The following tools are included in the trunk tools folder:

These tools are not licensed under Apache license. Some of them are under GPL license. However, we believe that distribution them is ok, since this falls under GPL’s mere aggregation clause. However if you indent to distribute commercial applications built from PyS60 Community codebase, make sure that you understand the set of different licenses involved.

Set up build environment

Ensure that Bazaar is properly in your Windows path.

Create a workspace folder

First you need to subst (make a folder appear as a driver letter) in Windows. Open command line. Go to SDK folder.

C:\Symbian\9.1>subst t: S60_3rd_MR
T:
mkdir workspace

Now choose this folder as a workspace folder in Carbide C++ and create an Empty Symbian C++ project called ”pys60” there.

Checkout PyS60 community edition

The go to this folder

T:
cd workspace\pys60
bzr branch lp:pys60community
cd pys60community\src

Preparing the build

This needs to be done only once.

We need to pacth the existing Series 60 SDK headers which have some bugs.

T:
cd \epoc32\include
\workspace\pys60\pys60community\src\tools\patch.exe -p1 < \workspace\pys60\pys60community\src\pys60-fix-3rded-sdk.diff

EPOCROOT must be set for some Series 60 SDK tools to work. We point to T: drive root.

T:
cd workspace\pys60\src
set EPOCROOT=\

As we still have some dependencies to the legacy system, one needs to configure the build system using PyS60 setup. This will generate some files and defines for Series 60 versio 3.0. 

c:\Python25\python.exe setup.py configure 30

Do not run bldmake bldfiles.

You need to convert legacy MMP build files to SCons based. First we need to possibly fix up PATH, since Carbide C++ might break it.

set PATH=c:\program files\bazaar;c:\program files\CSL Arm Toolchain\arm-none-symbian elf\bin;c:\program files\CSL Arm Toolchain\libexec\gcc\arm-none-symbianelf\3.4.3;C:\program files\CSL Arm Toolchain\bin;t:\epoc32\gcc\bin;t:\epoc32\tools;t:\epo c32\tools;C:\program files\CSL Arm Toolchain\bin;C:\Program Files\Common Files\Symbian\Tools;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem

c:\Python25\python tools\mmp2scons.py
ext\\calendar\\calendar.mmp:60:7: warning: no newline at end of file
Creating recipe ext\miso\build.py
Creating recipe ext\socket\build.py
...
Done!

Checkout SCons for Symbian. We assume it lives in src tree.

"c:\Program Files\CollabNet Subversion"\svn checkout http://scons-for-symbian.googlecode.com/svn/trunk/ scons_symbian

1. Running the build

Now we can execute the actual Python build script. This will create one monolithic emulator DLL which has almost all the PyS60 extensions built in – some extensions need manual building, since they rely on headers not found from standard Series 60 SDK.  You might need to reset PATH to default to Carbide C++ after the previous mmp2cons step by reopening the console window.

c:\Python25\Scripts\scons

You should see the following output. As you can see, UIDs are being allocated dynamically as instructed in Scontruct UID_BASE argument.

scons: Reading SConscript files ...
EPOCROOT=\
Info: SIS creation disabled
Building winscw udeb
Defines []
Getting dependencies for e32socket.pyd
Getting dependencies for _topwindow.pyd
Getting dependencies for zlib.pyd
Getting dependencies for _locationacq.pyd
Getting dependencies for _location.pyd
Getting dependencies for _graphics.pyd
Getting dependencies for _sysinfo.pyd
Getting dependencies for Python222Config.lib
Getting dependencies for Python222.dll
Getting dependencies for Python_appui.dll
Allocated UID:0xE1000000
Getting dependencies for Python.exe
Allocated UID:0xE1000001
Getting dependencies for Python_launcher.exe
Allocated UID:0xE1000002
scons: done reading SConscript files.
scons: Building targets ...

...

scons: done building targets.

Now you should be able to start a custom built Python shell in the emulator. You should see the following applications in Installation menu: helloworld, btconsole and filebrowser. Try launch helloworld and if it opens a pop up dialog the build has been succesful.

Building a SIS file for mobile phones

To build a target distribution type:

scons release=urel compiler=gcce dosis=true

This should yield to the result:

scons: Building targets ...
ensymble(["MyPythonForSymbian.sis"], [])
scons: warning: no package version given, using 1.0.0
scons: warning: no certificate given, using insecure built-in one
scons: done building targets.

If you want to build a SIS file signed with your developer certificate:

c:\python25\scripts\scons compiler=gcce release=urel dosis=true simplesis={'--privkey':'C:\\Certificates\\PrivateKeyNoPassphrase.pem','--cert':'C:\\Certificates\\MyApp.cer','--passphrase':''}

For now, installing the SIS file works only for C drive  – we’ll fix this little issue soon.

If you do not have a symbiansigned.com developer certificate you can sign the SIS file online for one phone (one IMEI code).

  1. Go to symbiansigned.com
  2. Register
  3. Enter your IMEI and upload the SIS file to OpenSigned Beta

Building your own application

The purpose of this project is to make rolling out your PyS60 applications possible – so here we go. Currently we make a quite bad assumption that all the application live in the same source tree with PyS60 community edition due to problems with absolute file paths with Symbian build tools – we will figure out a long term solution for this later.

PyS60 applications are stub Symbian executables which boostrap Python virtual machine and start the Python code execution. Executables are linked against a custom PyS60 DLL and they are restricted by capabilities given to the EXE file.

PyS60 applications live in applications source tree. The source tree comes with Bluetooth console, Hello world and File browser sample applications.

Scons build scripts takes as applications parameter a comma separated list which applications are included in the build.

scons builtin=all applications=helloworld,filebrowser

Applications consist of

  • default.py boostrapper module
  • icon.svg SVGT icon
  • Any number of Python modules (.py) files

See applications\helloworld folder to examine what files are needed to build an application. All application Python modules go to the private application folder (/private/myapplicationuid). Default.py must boostrap PYTHONPATH (sys.path) for this folder – PYTHONPATH defines where Python interpreter looks for the code. Application UIDs can be chosen manually or they are picked automatically by Scons for the unprotected test range.

Note that Python Script Shell application is handled out of this flow due to its legacy heritage.

Adding in your own extension

If you have development an PyS60 extension you can drop in into the buildchain easily. Each extension is defined in ext subfolder. It consists of necessary CPP, H and Python files. The building structure is defined in build.py using SConstruct command PyS60Extension().

Build.py files can be automatically generated from legacy code using mmp2scons.py converter.

  • Drop files to ext/yourmodule folder.
  • Run MMP -> build.py. This needs GCC or valid C compiler for preprocssing MMP files.
c:\Python25\python.exe tools\mmp2scons.py

ext\calendar\calendar.mmp:60:7: warning: no newline at end of file
ext\progressnotes\progressnotes.mmp:38:7: warning: no newline at end of file
ext\uikludges\uikludges.mmp:37:7: warning: no newline at end of file
Creating recipe ext\\socket\build.py
Creating recipe ext\\glcanvas\build.py
Creating recipe ext\\graphics\build.py
...
...Done!
  • Recompile. New ext modules are automatically picked as built in modules.
c:\Python25\scripts\scons
  • Now test importing your module in the script shell in emulator
>> import applicationmanager

If your extension is using thread local storage (Dll::Tls()) you might need to figure out how to workaround with it. See socket  module for example. You may also need to play around with the init function of the Python extension – it must be init + module name.

2. Developing on target

If you want to develop your application on a mobile phone, you do not need to go through the full development cycle for every little change. It is possible to update Python files on a phone without SIS installation. You can either automatically synchronize changed files from your PC to Phone (the example below) or you can edit files in-place on the Phone either using PCSuite or Series 60 SMB server.

Here are short instructions how to update files using PUTools console (btconsole). PUTools is wxPython based remote Python shell which allows you to run Python console commands over a Bluetooth connection from your PC. PUTools also has a file syncrhonization feature – after editing source code on the PC changes are reflected automatically to the phone.

  1. Put application Python files to a shared folder on the phone e.g. the memory card root (E:\). Add startup.py to E:\ which will modify sys.path to include your files.
  2. Add incoming Bluetooth serial port on windows (Control panel -> Bluetooth -> COM ports)
  3. Start Bluetooth shell on the computer (tools\putools\pcfiles\console.bat)
    4. T:\workspace\pys60\pys60community\src\tools\putools\pcfiles>c:\python25\python.exe putools.py com5
  4. Start Bluetooth shell on the phone (btconsole icon)
  5. Edit PUTools sync.config file on PC and run sync command on Bluetooth terminal to update changes made on PC to phone
  6. Run application launcher in the console (depends on the application structure how it is best to bootstrap in the shell):
>> import filebrowser
>> filebrowser.FileBrowser.run()

3. Release notes

Here is the short summary of differences with the current PyS60 community edition and one available from Nokia. This information is also available in divergence.txt file in the source folder.

2008-08-29 Mikko Ohtamaa <mikko@redinnovation.com>

    * PyS60 general

        New build chain and static config generation

        Migration tool for MMP -> Scons based extensions

        Added several tools included in the core distribution: sisinfo, ensymble, cog, patch

        Patched py2sis tool

        Contains extension: applicationmanager

        Contains extension: uikludges

        Contains extension: progressnotes

        Contains extension: miso

        Contains application: Bluetooth shell

        Contains example applications: filebrowser, helloworld

        Changed Bluetooth console bootstrap to e:\startup.py         

2008-08-15 Antti Haapala <antti@redinnovation.com>

    * e32socketmodule.cpp:

        socket.access_points has more information, two new
        fields is given per access point: isptype and bearertype,
        whose values are integers corresponding to values returned
        by CApSelect::Type and CApSelect::BearerType respectively.
        No symbolic constants are yet exported.

    * appuifwmodule.cpp:

        multi_select_list has a new argument, selected, which defaults
        to None. Given a list of integers, the items with the given indices
        are initially selected.

Conclusion

We hope this helps you to get started with PyS60 community edition. It’s still a bit complicated, since setting up the build environment on Windows is a such a pain. In the future, when the Linux based build system is reading settings up the development environment should be easier – all those boring steps happen automatically.

This might be still too difficult for some of the readers, since a lot of prerequirement work must be done before anything useful can be done. Feel free to comment the article in this blog, but we hope that you use Answers section in Launchpad to ask help and technical questions related to PyS60 community edition.