Run your Python application using PyPy – it’s fast!

Try your Python application with PyPy, the new hot Python JIT compiler

  • Your program will run faster…
  • … faster…
  • and ..faster!

Here are quick instructions for using PyPy on OSX. You should be able to apply these instructions to other UNIX systems as well.

Install PyPy by getting binary tarball.

wget http://pypy.org/download/pypy-1.4.1-osx64.tar.bz2
tar -xjf pypy-1.4.1-osx64.tar.bz

Install Distribute / easy_install command (setuptools fork) in order to install third party libraries.

cd pypy-1.4.1-osx64
wget http://python-distribute.org/distribute_setup.py
bin/pypy distribute_setup.py

Install libraries – easy_install seems to run just fine on PyPy as long as you don’t try to install Python native extensions

bin/easy_install plac # command line parser
bin/easy_install iso8601 # date format parser

Then run your application

~/code/pypy-1.4.1-osx64/bin/pypy exceptionanalysis.py     -s=2010-01-01T00:00:00 /Users/moo/xxx/log/client1.log

In my case the application was a simple log analyzer based on Python generators (code will be posted later). It reads a text file and tries to interpret it. The script utilizes only one CPU core and thus is very CPU bound. The speed comparison results are

  • With Python 2.4: Completed in 111.08 seconds
  • With PyPy: Completed in 30.39 seconds

That’s almost 4x speed-up!

Things did not work with PyPy

  • import multiprocessing – was not hard dependency, needed to comment out in plac_ext

Here is the traceback if anyone can help:

Traceback (most recent call last):
 File "app_main.py", line 53, in run_toplevel
 File "exceptionanalysis.py", line 20, in <module>
 import plac
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac.py", line 35, in <module>
 from plac_ext import Interpreter, import_main, ReadlineInput, stdout, runp, Monitor
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac_ext.py", line 7, in <module>
 import itertools, traceback, multiprocessing, signal, threading
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/__init__.py", line 87, in <module>
 import _multiprocessing
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 7, in <module>
 __bootstrap__()
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 6, in __bootstrap__
 imp.load_dynamic(__name__,__file__)
AttributeError: 'module' object has no attribute 'load_dynamic'

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Fixing a misbehaving terminal after after software interruption (pdb) or messed up output

1. Preface

Your terminal on Linux or OSX may end up to a bad state.

  • You cat a binary file to terminal
  • You interrupt pdb or some other application reading lines in a bad way

2. Symptoms

  • New lines don’t work
  • You cannot see your own typing to terminal
  • Backspace stops working

3. Fix

Type command

stty sane

This may fix some problems without needing to restart the terminal. More info.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Session affinity (sticky) with Nginx load balancing

These are instructions for setting up session affinity with Nginx web server and Plone CMS. Nginx can be used as a load balancer at the front of several Plone front-end processes (ZEO processes). By default, Nginx does not do session affinity, a.k.a. sticky sessions.  If you have site functionality which stores user specific data on the front end session data, let’s say an eCommerce site shopping cart, you must always redirect the user to the same ZEO process or they will have have different random shopping cart for each front end client.

Instructions assume you are installing Nginx via buildout.

Nginx sticky session can be loaded from here

Manually extract nginx-sticky-module under src folder:

cd src
wget http://nginx-sticky-module.googlecode.com/files/nginx-sticky-module-1.0-rc2.tar.gz

Then add it do Nginx configure options in buildout part:

[nginx-build]
recipe = zc.recipe.cmmi
url = http://sysoev.ru/nginx/nginx-0.7.65.tar.gz
extra_options = --add-module=${buildout:directory}/src/nginx-sticky-module-1.0-rc2

Now test recompling Nginx via buildout:

mv parts/nginx-build/ parts/nginx-build-old # Make sure full rebuild is done
bin/buildout install nginx-build

See that it compiles without erros. Here is the line of compiling sticky module:

gcc -c -O -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Wunused-function -Wunused-variable -Wunused-value -Werror -g   -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules -I src/mail \
        -o objs/addon/nginx-sticky-module-1.0-rc2/ngx_http_sticky_module.o \

Now add sticky to the load balancer section of nginx config:

[balancer]
recipe = gocept.nginx
nginx = nginx-build
...
http {
        client_max_body_size 64M;
        upstream zope {
                sticky;
                server ${hosts:client1}:${ports:client1} max_fails=3 fail_timeout=30s;
                server ${hosts:client2}:${ports:client2} max_fails=3 fail_timeout=30s;
                server ${hosts:client3}:${ports:client3} max_fails=3 fail_timeout=30s;
        }

Reinstall nginx balaner configs and start-up scripts:

bin/buildout install balancer

See that generated configuration is ok:

bin/nginx-balancer configtest

Restart Nginx:

bin/nginx-balancer stop ;bin/nginx-balancer start

Check that some (non-anonymous) page has route cookie set by sticky module:

Huiske-iMac:tmp moo$ wget -S http://yoursite.com/sisalto/secret-travel-destination-infoa
--2011-03-21 21:31:40--  http://yoursite.com/sisalto/secret-travel-destination-infoa
Resolving yoursite.com (yoursite.com)... 12.12.12.12
Connecting to yoursite.com (yoursite.com)|12.12.12.12|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/0.7.65
  Content-Type: text/html;charset=utf-8
  Set-Cookie: route=7136de9c531fcda112f24c3f32c3f52f
  Content-Language: fi
  Expires: Sat, 1 Jan 2000 00:00:00 GMT
  Set-Cookie: I18N_LANGUAGE="fi"; Path=/
  Content-Length: 41471
  Date: Mon, 21 Mar 2011 19:31:40 GMT
  X-Varnish: 1979481774
  Age: 0
  Via: 1.1 varnish
  Connection: keep-alive

Now test it by doing manual session related activity and see that your shopping cart is not “lost”.

More info

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+