The source code for the edit pop-up is here. I hope you can see my cool CSS3 effects!

Also this blog post is a story how you gain 180k words worth of developer documentation almost from the scratch for your favorite Python project. A Story worth of telling, I hope.

History

To understand the context and the level of complexity.

Plone CMS, mature (as in your great-great-great-grand-aunt) Python based content management system, has not been very developer  friendly. The goal of “to make it easy to customize and extend” has never appeared in Plone roadmap or conference keynotes. Instead, the most of the developers have generally been busy of doing more interesting stuff and pushing the technology forward.

Plone is also geared toward more heterogeneous end of codebases out there, due to its long history (>10 years – a long time for a Python product) and adoption of many third party-ish packages in the codebase (248 Python packages) . All these packages, including critical packages you should understand for extending Plone for your own purpose, come with less or even less documentation. Sometimes packages are not Plone related and authors may not be interested how to apply the re-usable components in the context of Plone.

It’s all sucks if you are a developer and simply want to get your little customization done into Plone. You need to drill through a stack and try to guess which piece of history to apply in which point of source code.

From closed official documentation to open-ended unofficial documentation

There exists an official Plone developer documentation – official in the sense Plone Foundation has copyright over it. However this level of ownership, right to relicense, comes with a price: it’s “hard” to contribute for it – you need to sign a contribution agreement (by hand) and snailmail it somewhere. Due to lack of content and contributions, the official documentation does not carry you far.

Somewhere in the past “unofficial” Plone developer documentation was started. The goal of this project was to have more open-ended documentation process to get some developer documentation for Plone in the first place and the fix some of the problems mentioned before. The spirit was “wiki-style” though the tools were different. The community went through some mailing list wars about the future of the documentation. This was the future then (and now is the current past):

• First the unofficial documentation was bunch of Plone pages (WYSIWYG) managed in plone.org. We were living the time when Sphinx was still a dream in its authors minds.
• Then it was converted to reStructured Text pages, still managed in plone.org
• Then it was pulled out from plone.org, converted to Sphinx, put into Plone “collective” Subversion control (no contrib agreement needed). We were living the time when first Sphinx release was put out into PyPi.
• Sphinx becomes de facto method of creating Python developer documentation
• Then it was hosted as standalone HTML files, under collective.developermanual Python package, on random hosts
• Then it was hosted on plone.org again. This time Sphinx generated HTML was imported to plone.org using a tool called Funnelweb. Funnelweb was originally created for importing arbitary sites into Plone as Plone content (it has been known to work for Liferay, WordPress, Drupal, etc.) Howver, this was still bad idea as Funnelweb does not have really have a concept of updating docs – just one time imports and data dumps.
• And, da-da, it’s 2010s, readthedocs.org is launched. The documentation management is moved to GitHub as, with readthedocs.org, this solves the need for automatic documentation deployments after a commit.
• GitHub creates a button Fork and edit this file
• readthedocs.org gains edit back link when issue 152 is worked around
• The documentation title is renamed from Plone community maintained blaablaablaa to more simple Plone Developer Documentation (as opposite to official Plone Developer Manual)
• This blog post is written – hoping to encourage even more open-ended developer documentation contributions

Plone Developer Documentation now contains 183 792 words. It is comprehensive, but not coherent. It also crucially lacks polished, hand holding, Hello world tutorial needed to lure in new developers (someone please take a ball on this). Even with these little hindrances, based on the readers’ comments, it’s slowly becoming really really good.

The unofficial documentation is licensed under GPL license. The rationale for this is simply the fact that when collective.developermanual package was created this was the default license coming in Paster templates. Also, Plone Foundation’s official stance is that when you import code from Plone your product should be GPL. The documentation contains a lot of code starting with import plone… This little gem does not have any practical impact, but I wish to live to see some arguments about this.

Fork and edit this file

I love GitHub; I hate Git. Git is the anti-thesis of user interfaces – what a version control system would look like if it were developed by persons knowing only C and Perl and who’d love to scare away all other developers how are not equally smart. But luckily with GitHub, everyone, even without theoretical version management course studies, can work with Git.

One of GitHub’s recent innovations is Fork and edit this file button. It allows anyone to suggest changes to source file. No prior relationship between the author and the contributor is needed – the suggestion goes to the pull request queue and you’ll get a notification about the contribution.

This was a holy grail for me who cares less about politics and more about getting in more contributions.

• Any person can edit the file, as Git/SVN checkout is not needed
• You don’t need to maintain user list of “authored editors”
• The GitHub editor itself (ACE based?) is actually nice: selecting text and hitting Tab key actually indents a block of text. It’s way beyond crappy <TEXTAREA>.
• You get pull request notifications
• You can post-edit proposed changes easily and communicate with the contributor

 Linking it back

A perfect documentation update workflow would be that when you want to edit a piece of text you simply click it in the place, edit in place and press Save. This is what Wikipedia was before it grew too large.

With readthedocs.org, GitHub and Javascript  magic we cannot reach such fidelity yet. We

• Don’t know if user is logged into GitHub, or his/her GitHub username
• Cannot edit individual Sphinx paragraphs in ACE
• Don’t have GitHub Javascript API to push in commits/pull requets

(Please correct me if I am wrong)

But someday we’ll be there.

Meanwhile, we have something what we call in Finnish a “fur cap solution” (karvalakkiratkaisu). It means solving the problem in the least complex way, but still achieving the solution somehow. (If someone would like to clarify the etymology of this proverb or has a translation for it I’d like to hear that also)

In Sphinx HTML templates we simply include backlinks to the source code file on GitHub and have bulleted instructions how to edit the file:

Future future

After struggling several years of getting any developer documentation for Plone I am think I am finally quite happy and my work here is done. It’s not perfect, but compared to other CMS’s we are not definitely the worst anymore.

What still is needed is more interactivity – where people can ask questions in the context of documentation, like happens on php.net API documentation. Then store the question and the answer in the context, so the future generations do not need to wonder the same thing again. This far I have been manually linking stackoverflow.com questions direcly to the documentation when I am too lazy busy to start writing anything. Sadly, the orignal plone.org documentation had discussion feature until it was disabled, due to lack of good notification system. Maybe we could do something akin the lines using DISQUS and Sphinx and post questions to the mailing list?

Ps.

The edit pop-up note text, due to CSS3 rotation, looks like crap on Firefox 10 / OSX. Chrome does not suffer from this. A bug in FF text rendering?

 Subscribe to this blog in a reader Follow me on Twitter

Name servers are part of any serious hosting infrastructure. They have been traditionally run on BIND or similar UNIX daemons. The usual hosting tasks including mapping domain names and subdomains (A, CNAME, et. al records) to servers and delivering email (MX and SPF records).

Usually a basic hosting plan is only good for setting A records for few subdomains like www, but on the instant you need something more advanced like routing MX records to Google Apps email you need to start considering a real DNS service. GoDaddy and other big name registrants provide tools for this, but they are usually specific only for top level domains (TLDs) the registrant supports. If you come from a cold country like .fi you cannot centralize your DNS management to big popular domain registrants.

The bad thing with running your own name servers is that when the service goes down the whole your infrastructure becomes virtually inaccessible – web sites, email, anything. Thus you usually run at least two BIND daemons on two physically separated servers just in case one blows up the other can still keep DNS records running. Also DNS, being core part of internet infrastructure, is subject for many kind of attacks and keeping your skills and knowledge up-to-date with BIND may be time consuming.

Like with email servers, hosting name servers is a thing you really don’t want to do. Enter Amazon Route 53. (tha name Route 53 probably comes from the famous U.S. road  Route 67 and DNS port 53?).

Route 53 is ridiculous easy to use. Just go to the site and register one for yourself. However if you have legacy BIND servers running around there exist ways to import data automatically instead of manually reading through zone files and typing them in to Route 53 by hand.

Setting up cli53

cli53 is a Python based command line interface for Route 53. It uses buildout for installation (of Python package dependencies).

These instructions have been tested with Ubuntu 8.04 server. Prerequisitements for using this stuff is knowing basic Ubuntu server management from the command line.

Run everything as root.

Install wget and unzip

apt-get install wget unzip

Download ZIP copy of cli53 from Github (we don’t do git clone here – no need to install git on the server):

wget --no-check-certificate https://github.com/barnybug/cli53/zipball/master

Extract

unzip master

Install as user local under the extracted folder (note that the folder varies across GitHub exports)

cd barnybug-cli53-3b468b7
python bootstrap.py
bin/buildout

This will generate the local bin/cli53 command.

Now you need to set up AWS credentials. Route 53, and other AWS service, are operated using access keys which you can get from Security Credentials page in AWS web console. Set the access keys for your shell/SSH session using enviroment variables:

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Side note: as this point you need to check that your server clock is in correct time your AWS remote operations will fail (300 seconds thresold).. Use date command to see the server clock time and correct it with the same date command using this obscure date command syntax.

Now you can run a test by listing your Amazon Route 53 Hosted Zones (Hosted Zone = zonefile = one per top level domain usually) using list command:

bin/cli53 list

The proper output for one domain which has been created throught-the-web in Route 53 control panel is something like:

HostedZones:
  - CallerReference: 9265CCC3-9C41-98CE-8820-E26C0356C478
    Config:
      Comment: test
    Id: /hostedzone/Z2L3NT0WCS8OWA
    Name: xxx.fi.

Importing zone files

Now when cli53 is running and working it is time to rock’n'roll.

On our server configuration BIND zonefiles lived in /srv/dns/zones though this is not the default location for Ubuntu BIND. There is one file per each zone and the filename is the domain name.

[root@foobar][23:02][/srv/dns/zones]# ls -l
total 176
-rw-r--r-- 1 root root  425 Dec 30  2008 abc.com
-rw-r--r-- 1 root root 1713 Dec 30  2008 abcabc.com
-rw-r--r-- 1 root root 1719 Dec 30  2008 abcabcabc.com

We can simple create an one-liner shell script which will loop through all the files and import them to Amazon Route 53. However – there is a trick. You’ll get unknown origin exception if your zone files lack a certain line:

Traceback (most recent call last):
  File "/root/barnybug-cli53-3b468b7/bin/cli53", line 22, in <module>
    cli53.cli53.main()
  File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 495, in main
    args.func(args)
  File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 268, in cmd_import
    raise Exception, 'Could not find origin'
Exception: Could not find origin

This is because the zone file doesn’t have $ORIGIN directive. it’s optional with BIND as BIND uses the filename as $ORIGIN if it’s not defined, but the directive is mandatory  for cli53 (cli53 doesn’t yet synthetize $ORIGIN).

You can simple add $ORIGIN to the beginning of each file and copy files to a working directory with the following shell one-liner (it’s always safe to make copies of processed files than try to fix them in place):

mkdir ../transfer
for i in * ; do echo "\$ORIGIN $i." > ../transfer/$i ; cat $i >> ../transfer/$i ; done

Then do a test run for one import:

~/barnybug-cli53-3b468b7/bin/cli53 create xxx.fi
~/barnybug-cli53-3b468b7/bin/cli53 import xxx.fi --file xxx.fi --replace --wait

Note: deleting zones in Route 53 control panel is painful difficult, so make sure you import zones you only really need.

If everything looks good we are ok to upload everything to Route 53. Again, a shell one-liner does the trick for us:

for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 create $i ; ~/barnybug-cli53-3b468b7/bin/cli53 import $i --file $i --wait ; done

This will take abour 15-20 seconds per domain.

Note: if you need to re-import add –replace flag.

The ugly part

Each zone has Source of Authority record with authoritative nameserves (NS records). These will be change from your BIND server IP addresses to Amazon ones. Route 53 will re-assign its own name servers for each imported Hosted Zone. However, you cannot know beforehand what name servers the dice has chosen for your Hosted Zone, so automatizing this process is little bit difficult. NS records servers are overriden when the zone file is uploaded to Route 53.

You can see new SOA and NS records when you choose the domain in Route 53 control panel and press Go to Record Sets.

To make things easier later on here is a script which will dump all the name servers of all domains you have in Route 53 and collect are info to domain-info.txt file:

for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 info $i >> /tmp/domain-info.txt ; done

Testing Route 53 DNS

Browser through imported domain records in Route 53 control panel and see if they look ok.

You can also test the actual DNS functionality. When you use ping command, your web browser or any other “normal” mean which queries DNS records they use the Source of Authority name servers as set by in your domain registrant settings. When in the middle of migration to Route 53, the authorative name servers are still pointing to the old name servers in this point.

dig command can be used to query a specific name server for DNS records.

• In Route 53 control panel pick any domain and any of its name servers
• Use dig command to query the records of this domain

Example:

dig -t ANY @ns-1340.awsdns-39.org yourdomainname.com

Output should be something like this:

; <<>> DiG 9.4.2-P1 <<>> -t ANY @ns-1340.awsdns-39.org yourdomain.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9352
;; flags: qr aa rd; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;yourdomain.com.            IN    ANY

;; ANSWER SECTION:
yourdomain.com.        172800    IN    NS    ns-470.awsdns-58.com.
yourdomain.com.        172800    IN    NS    ns-553.awsdns-05.net.
yourdomain.com.        172800    IN    NS    ns-1340.awsdns-39.org.
yourdomain.com.        172800    IN    NS    ns-1706.awsdns-21.co.uk.
yourdomain.com.        900    IN    SOA    ns-1340.awsdns-39.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
yourdomain.com.        0    IN    MX    10 aspmx.l.google.com.
yourdomain.com.        0    IN    MX    20 alt1.aspmx.l.google.com.
yourdomain.com.        0    IN    MX    20 alt2.aspmx.l.google.com.
yourdomain.com.        0    IN    TXT    "v=spf1 a mx a:smtp.something.fi a:auth-smtp.something.fi include:aspmx.googlemail.com ~all"
yourdomain.com.        0    IN    A    84.20.128.49

;; Query time: 32 msec
;; SERVER: 205.251.197.60#53(205.251.197.60)
;; WHEN: Wed Nov 23 00:04:53 2011
;; MSG SIZE  rcvd: 413

(subdomains are not listed)

Finalizing it

Go to your domain name registrant and update the domain name servers point to your Route 53 servers.

• For each domain in domain-info.txt
• Go to the corresponding registrant from whom you obtained the domain (e.g. domain.ficora.fi)
• Update the authoritative name servers to be as stated in the file
• Wait 24-48 hours or whatever time-to-live time you have for your name servers – usually this is longish period
• See that your services still run
• Take down your old name servers

More info

• http://hathology.com/2011/02/how-to-import-bind-zone-files-into-amazon-route-53/
• http://docs.amazonwebservices.com/Route53/latest/DeveloperGuide/index.html?MigratingDNS.html#Step_UpdateRegistrar

 Subscribe to this blog in a reader Follow me on Twitter

The short history of reloading

The web developer community takes reload-code-on-development as granted. You edit your source file, hit refresh and poof: your changed are there. It is a good way and the only real web development way and one of the sane things PHP community has taught to us. Things could be different: for example if you are developing an embedded code on mobile platforms you might need to build the ROM image for hours before you see your changed source code line in the action.

Side note: To check your code changes it is often faster the hit enter on the browsee address bar line as this does not reload CSS and JS files for the HTTP request

For PHP the reloading is easy as the software stack parses every PHP file again on every request. There is no reload – only load. In fact there is a whole “optimizer” open source business model spawned around PHP just to make it faster. PHP processes are stateless.

For Python the things are not so simple. Python web frameworks have separate process lifespan and HTTP request handling lifespans. Often a server process is started, it spawns N threads and each thread handles one HTTP request at a time. Python processes are stateful.

As being stateful, Python processes may start slowly. You need to parse all source code and create initial memory objects, etc. Frameworks do not optimize for fast start-up because it really doesn’t matter on the production service as you spawn Python processes only once, when the server is booted.

Plone CMS, coming with choking 250 MB of source code (Linux kernel has 400-500 MB) is the extreme of this slow initialization. Plone CMS is the biggest open source Python project out there (does anyone dare to challenge my clain?).  When Plone starts it loads most of source code to memory and parsing and initializing Python itself, plus various XML files, is quite an achievement.

What Tornado, Django, Paster etc. Python frameworks do that when they reload they simply zap the process dead and reinitialize 100% virgin process. This is ok as these low level frameworks have little overhead. Though it becomes painful slow too when your Django service grows for many applications and the code starts piling up…

For Plone 100% start up this is no go. Plone people care about the start up time, but there is not much they can do about it… even the smallest sites have start up times of dozens of seconds.

Python and module reload

Plone used to have (has) a helper package called plone.reload. It works by reloading Python modules. Python interpreter allows you to reload a module again in the memory. The code is actually based on xreload.py written by Guido van Rossum himself.

However, there is a catch. Naive code reload does not work. As xreload.py comments state there are issues. Global stateful objects, other initialization specific code paths, etc. are ignored. So with plone.reload you ended up with a broken Python process as many times you ended up with a successful reload. It’s so frustrating to use that you don’t actually want to use it.

From Finland with Love

sauna.reload was initially created in Sauna Sprint 2011 event, arranged by EESTEC and Plone Community. The event name comes from the abundance of sauna in Finland and the package name comes from the fact that a lot of sauna was put into the effort of creating it.

 Going trident

A fork. (The orignal image)

A spoon (The orignal image)

sauna.reload takes a different strategy of reloading

The idea behind sauna.reload goes back to the days as I was working with Python for Nokia Series 60 phones. Symbian seriously sucks, peace on its memory (though not sure if it’s dead yet or just a walking corpse). If you did any Symbian development you saw how it could never fly – such a monster it was.

One of the issues was start-up time of the apps, especially Python start up time. Symbian IO was slow and simple fact to running the application to the point where all imports had been run took several seconds. For all of this time you could just show some happy loading screen for the user. It was very difficult to deliver a Python application which would have acceptable user experience on Symbian.

Far back in the day I chatted with Jukka Laurila who gave me some tips. Emacs used to have unexec() command which simply dumped the application state to the disk and resumed later. This works nicely if your application is an isolated binary and does not have any references to DLLs, so it doesn’t work so nicely on any contemporary platform.

On the other hand, Maemo folks had something called PyLauncher. PyLauncher loaded a daemon to memory and this daemon had Python interpreter and pygtk library loaded. When Python script had to be run, the PyLauncher daemon simply fork()’ed a new child and this new child inherited already loaded Python interpreter and pygtk for free. Fork() is a copy-on-write operation on modern operating systems.

This gave me a cunning idea.

Taking a shortcut

When you develop a web application and you want to reload changes you usually don’t want to reload the whole stack. In fact, your custom code tends to sit up on the top of the framework stack and unless you are a framework core developers yourself, you never poke into those files.

What if we run Plone initialization process to the point it is just about to load your custom code, freeze the process and then just always go back to this point when we want to load our (changed) code?

Plone process is in fact a series of Python egg loads. Your custom eggs are loaded last.

With sofware only your imagination is the limit

So I presented the idea to Asko Soukka and Esa-Matti Suuronen from University of Jyväskylä (one of the largest Plone users in Finland) in Sauna Sprint 2011. If I recall correctly their first reaction was “wow dude that definitely is not going to work”.

But at the end of the tunnel was the freedom of Plone web developers – being released from the chains of Zope start-up times forever. Even if it had turned out sauna.reload was not possible we would have been stranded in the middle of the forest having nothing to do. So the guys decided to give it a crack. All the time they spent to develop sauna.reload would be paid back later with gracious multiplier.

sauna.reload is using awesome cross-platform Watchdog Python package to monitor the source code files. Plone is frozen in the start-up process before any custom source code eggs from src/ folder are being loaded. When there is a file-system change event sauna.reload kills the forked child process and reforks a new child from the frozen parent process, effectively continuing the loading from the clean table where none of your source code files were present.

Going uphill

San Francisco, where we are hosting Plone Conference 2011 and sprints, is a city of hills. They feel quite steep for a person who comes from a place where maps don’t have countours.

With Zope 2, the technology the ancients left to us, things were not that straightforward.

ZODB, the database used in Plone, happens in-process for the development instances (single process mode). When you go back to the orignal frozen copy of Plone process you’ll also travel back in append-only database time.

The sauna.reload devs managed to fix this. However this code is prone to ZODB internal changes as happened with Plone 4.1.

Another funky issues were how to monkey-patch Zope loading process early enough and reordering egg loading process so that we hold loading the custom code till very end.

For The Win

sauna.reload works on all Python packages which declare z3c.autoinclude loading in setup.py. Just go and use it. This, of course, is based on the assumption you use an operating system which supports fork() operation and unfortunately Windows is not privileged for this.

Note: OSX users might need to use trunk version of Watchdog until a bug fix is released

Your Plone restart time goes down to 1 second from 40 seconds. Also, the restart happens automatically every time you save your files. It works with Grok, Zope component architecture, everything… ZCML is reloaded too. You should feel an unsurmountable boost in your productivity.

We still have some quirks in sauna.reload. If you try hard enough you rarely might be able to corrupt database. We don’t know how, so keep trying and report your findings pack to GitHub issue tracker.  Also as sauna.reload is meant for the development only you shouldn’t really worry about breaking  things. We also have an old issue tracker which was not migrated to collective – how one can migrate a issue tracker on Github?

The Future

Who knows? Maybe someone gets inspiration of this work and ports it to other Python frameworks.

Our plan is to release a Plone developer distribution which comes with sauna.reload and other goodies by default. This way new Plone developers could instantly start hacking their code by just copy-pasting examples and keep hitting refresh.

 Subscribe to this blog in a reader Follow me on Twitter

The presentation was originally aimed for Python developers, but suits for anyone with some programming experience and not much Javascript experience.

Discussed topics include: require.js, ECMAScript 5, strict mode, JSLint, this scoping and bind(), binding event handlers using jQuery

Ps. Always when searching Javascript related information in Google et. al. use MDN (Mozilla Developer Network) prefix as otherwise you’ll get misinformation and spammy results. E.g. search “MDN javascript bind” instead of “javascript bind”.

Pps. Put w3cschool.com domain on the black list in your search engine. It’s bad.

Ppps. PDF is available here.

 Subscribe to this blog in a reader Follow me on Twitter

This blog post is a long flow of mind try to pinpoint how to define if a framework is helpful for you or not. Though I use a Python framework as an example these thoughts can be applied for any programming language, framework, CMS, etc. out there. I’d be especially interest how Joomla!, Drupal etc. popular PHP platforms solve problems mentioned here.

The framework in the discussion was towards high level of web development, contentish things. To make tailored CMS, and stuff. This framework was on higher level than Pyramid, little bit higher than Django. It is named after an egyptian god and because Egyptian gods have difficult names spelled with symbols, my Finnish mind already lost it.

So the friend of mine was trying to convince me about the need and superiority of this new frame and asked the question: “Do you like it?”

And I answered: “Does it help me to get my work done?”

My question was the right question, but wrong in so many ways. We were discussing about the same thing, but from different viewpoints…. and in this point, the discussion got off the rail. My friend tried to justify the existence of the new framework and I was trying to help him understand what kind of needs I hope a framework would solve for me.

As we put a lot of energy in seemingly fruitless conversation, I become sad. This discussion and thoughts about it keep me awake during the night and I tried to come up a simple way to measure the usefulness of a high level web framework.

Now I have a new question, which in the end is the same question as above, but more communicative. This question was inspired by Douglas Crockford’s talk about Javascript metaperformance and especially the part where he tells that there are a lot of smart people doing web – sometimes they focus on wrong things, but  if you give them a proper target to shoot at they can get amazing things done.

Next time, when someones needs my opinion about a web framework, I’ll ask “How does it do a date picker widget?”

Modern Times

To see how my little question works you need to dive into the history of web development a bit.  The web user interaction is much about forms. The forms have input fields and these input fields are wrapped into user interact-able widgets which have additional properties besides simply handling punching of keyboard keys: input validation, typing auto completion, mouse interaction, nice bling bling decoration and effects  and so on… especially when creating things like CMS where the users are expected to input stuff through the browser you need to pay careful attention for the user interaction details.

The purpose of a software development framework is to make the life of a software developer simple and get the thing done.  Because web is much about forms a lot of effort has been put to make form development simple. My question could have been a less form specific; I chose a date picker because it is familiar and everyone has come across one e.g. when booking flight tickets. Thus we should have quite unified idea what we are discussing about. The date picker also tends to be one of the most complex form widgets available in your toolbox, so were are not setting the bar too low here.

Last Tango in Paris

Web itself was not originally designed to have or to do rich widgets for easy user interaction. We have a HTML mark-up language which tries to separate content from the presentation. We have a layout language, CSS,  which tries to separate layout from the content, but in the end fails doing so. Then we have a Javascript programming language which pulls strings in your web browser to make HTML and CSS co-operate and  do the widgets using Document Object Model. A model, which was not designed Javascript programing language in mind.

(sidenote: on the day I see bbc.co.uk level site delivering both web and mobile version from the same HTML code  just using different CSS stylesheets I can say that CSS actually delivers it promise to separate the layout from the content)

Now is now and web is steering more and more to the direction where it can do more for the user… or the interaction is more efficient because of richer controls and paradigms, enabled by Javascript and HTML5, can be applied to solve the same old problems. In this world if you do a modern web framework for and expect it to remain on the isolation on the server-side from all the gory details of Javascript and CSS you are doing it wrong.

Doing a web widget is like a dance of several participants who need to co-ordinate their actions so that the individual performances are in harmony and the actors are not stepping on each other toes. A web framework is like premade step guide telling you how to execute your dance. The result is magnificent performance of  0.5 seconds of your site visitor time when he or she subconsciously picks a date in the web form by clickety-clicky, or as on the mobile era, by thumbity-thumbly.

Missing in Action

Now, I could have asked “Does it do a date picker widget?” The answer would probably be “yes”, as of course you can put a date picker widget on every HTML page out there. Then I could asked “how?” and then the answer would be something like:

• Download your favorite Javascript widget library (e.g. JQuery UI)
• Toss CSS files somewhere, Javascript files somewhere in your project
• Edit your  main template <head> to have these files included – if you are smart include them only on the pages where the user interaction is actually needed
• Bootstrap <input> fields doing date picking using a CSS class markers or similar markers in the source code somewhere, or have a bootstrap JS running by <input> ids
• Convert between Javascript widget framework date model and your web framework date model in somewhere of the form layers
• etc. etc.

This is what I call a blunt approach. It’s solving the problem of dancing like this:

This all is tolerable if you do it once for your web appy. But what about if you need different widgets on the page too? How about checkbox groups with internal dependencies? Time picker?  Yoyo? Again you need to toss files somewhere, edit head, etc.

When your web application grows this all quickly leads to a mess where you have tens of different files included in the main template. Different modules start to clash with their CSS classes and you need to start resolving the issues by your own hand written CSS overrides. Suddenly the page load times are through the roof, because you are doing too many HTTP requests per page. You need to start thinking about the cache. You need to think about merging static media files. Which leads you to think solving dependency orders between media files.

Does the framework help you solve the problems like above to get the thing done?

Notting HIll

I studied industrial engineering and management in University of Oulu (home of IRC – never forget to mention this when I have a chance).  We had a course called Quality which was about quality. You can study subjects like this from very broad perspective especially if your plan is to become a pointy haired boss who doesn’t know crap about what he or she is doing.

The definition of quality is simple. Quality is not the same thing as high value brand (e.g. Mercedes Benz). Quality is not the same thing as faultiness (no bugs in the delivered software). Quality is about whether the customer is satisfied what he or she gets.

The customer of a software framework is a developer. What the developer is buying is a tool to make delivering software more easy – to get the thing done by the end of the day.

My friend was telling how high test coverage the framework has. This is a nice trait for a framework. Higher the test coverage, the less  bugs you are likely to encounter. Essentially this is faultiness.

Now a bad, but thoughtful, analogue: in industrial engineering faultiness comes with a price. On your production line you’ll pay a high cost of 100% faultiness products. You usually want to do testing by sampling and simply let few bad products slip through instead of having the high cost of testing every single manufactured item. In the end, the testing cost of the product is away from the other areas of product development and manufacturing (features, delivery options, etc.)

Faultiness is important, but it does not make high quality software framework alone. Inspired by the leaked rant of Steve Yegge if you dial test coverage 100% but have no accessibility you have no product.

Which brings me to the punch line of software framework quality: accessibility. To enable developers get the thing done with the framework it must have documentation telling how you get the thing done.

Django folks get it. Django has always had very detailed documentation and no surprise it won the most of the hearts of Python developers back in 2006ish when the framework situation was different with Turbogears et. al. being still in their strength.  Anti examples would be packages z3c.form or plone.portlets (chosen by a fair dice roll – guaranteed to be random). Both the packages are frameworkish and have near perfect test coverage. Too bad it seems that only 0.0001% of people are smart enough to use these packages and mostly get frustrated whilst trying to get the thing done.

(sidenote: even though z3c.form has plentiful documentation does not primarily address the needs of the framework users)

Diamonds are forever

Another trait of quality is that it must be measurable.  The absolute measurement of the quality is the customer satisfaction: how happy people are with your product. However the level of satisfaction is difficult to measure directly. Indirect, easy numbers, are measured instead.

One easy quality measurement  is that you go around and ask random users whether they are happy with the framework. If you do not know any of these developers users yet you can simply follow the chat on a related IRC channel:  friendlier atmosphere usually correlates with happiness which correlates with quality.

But being an engineer I hope we could use something more tangible than random chat around a pint or two.

The most important indirect number of your framework quality could be API documentation coverage. How many of the functions of your API have sane helpful documentation or documentation in the first place? As this number  translates to how easily people can pick up your framework and simply read how to get things done with it. Especially with Python a good API documentation (parameters in, results out) is important because Python has poor statistical code analysis or auto-completion support, something  what Java and C# IDEs enjoy.

I Am Legend

The another nice thing you can learn about the quality on the university courses is that less numbers you need express the quality of your product the better. It’s over simplifying of things to say how good or bad your product ultimately is. Your pointy haired boss might be able to process few percent scale numbers a day, but less he needs to think anything besides golf, the better. You suggest a new framework to be used for the new software project and the boss takes a decision if it’s good or not, essentially the boolean value or quality. On the rare day the PHB is on his/her brightest the PHB might think one step forward and ask why the framework is any good and then you can answer that the goodness has been tested with the scientific method found from a random blog in the internets.

.. and to shine at the very end of quality simplification spectrum I came up with the following objective decision tree to measure the quality of a web framework:

Is a web framework, CMS, tool etc. any good for modern development needs? Yes / No

Yes if…

• You take one average developer, like me, who has wide general programming experience, but not yet insight how the framework works (don’t pick any super smart guy like the framework developer itself might be).
• You give the developer four hours of time (I am generous about my time today), a hello world style tutorial and API reference manual.
• The developer is able to create a simple form / content type / page  containing a date picker widget with some customizations like template changes or non-default options. The customizations are important because that’s what you eventually will do – you won’t be mass producing hello world applications from the conveyor belt with the framework.
• The resulting Javascript and CSS widget bindings to HTML are handled in maintainable manner and media files are merged, compressed and HTTP cache headers are set  properly.

…else no.

This is just a litmus test. It doesn’t necessary tell how good a framework is, but if the test fails then it might be a sign that the framework is either not a developer friendly or is not solving the needs of modern web frameworks. The most possible failure mode is that there are no API documentation at all -  which means that the project is not disciplined to communicate what problems the framework should be solving in the first place and is worth as much as any random bunch of code you can find from the open source archives.

Also, this blog post is just guidelining: no need to nitpick the details unless they advance the humankind somehow. As mentioned before, the blog post tries to give a good target to shoot at and highlights what kind of facts you might need to communicate out to success as a software development framework. Please feel free to challenge me, but as far as I am aware, none of current web frameworks or CMSs passes this test.

Finally, how the proper dance of HTML, CSS and Javascript should look like, Chuck Norris style:

 Subscribe to this blog in a reader Follow me on Twitter

Track 1, auditorium Lambda 9:45 Opening notes 10:00 “from __magic__ import wtf” by Tommie Gannert from Spotify 11:00 “Replacing Spreadsheets with Django” by Teemu Kurppa from Huikea Experience 12:00 Lunch 13:00 “High-Performance Computing with Python” by Jussi Enkovaara from CSC 14:00 “Understanding Encodings” by Ezio Melotti, a CPython core developer 15:00 Lightning talks 16:00 Keynote by Mike Bradshaw Closing notes

Track 2, classroom B2033 11:00 “Python for Data Science” by Harri Hämäläinen, a semantic data scientist 12:00 Lunch 13:00 “Building Your Own Cloud” by Andy Smith from OpenStack 14:00 “Non-blocking QA” by Anthony Long

 Subscribe to this blog in a reader Follow me on Twitter

We just need a sponsor organization and some 200 000 USD for the evaluation fee.

Anyone?

I know you want __init__.py (though underscore is not an allowed character in home name).

(thanks for Petri for the orignal idea)

 Subscribe to this blog in a reader Follow me on Twitter

After the headscratching was over we had figured out that renaming filenames to use latin-1 encoding makes the web server serve them fine. If it were a fresh start I’d rather configure the server itself to use now prevailent UTF-8, but because the infrastructure was shared with other, older, projects this was out of the question.

Thus, the script below was created to address the filename encoding problem (syntax highlighting available on the orignal publication):

#!/usr/bin/python
"""

    Recursively fix filename encoding problems

    http://www.opensourcehacker.com, MIT licensed

"""

import os

# Source filename encoding
FROM="utf-8"

# Target filename encoding
TO="latin-1"

# Current working directory
PATH=os.getcwd()

for root, dirs, files in os.walk(PATH):

    # Assume files are 8-bit strings in the native encoding
    # of the system which we cannot know
    for f in files:
        try:
            decoded = f.decode(FROM)
        except UnicodeDecodeError:
            print "Cannot decode:" + f
            continue

        fullpath = os.path.join(root, f)

        newpath = os.path.join(root, decoded.encode(TO))

        if newpath != fullpath:
            print "Renaming:" + fullpath + " to:" + newpath
            os.rename(fullpath, newpath)

I still don’t know how to verify / check / guess which encoding Linux file systems are using (if not UTF-8…), so someone please enlight me. In this case we learnt this by a smart guess and testing: we created two filenames, one name encoded with UTF-8 and one encoded with Latin-1 and see which file the web server was giving for us. I am also unsure whether the fact that files were orignally kept in Subversion repository and committed there from Windows had anything to do with the problem.

 Subscribe to this blog in a reader Follow me on Twitter

1. For constant innovation choose Python

To stay competitive it’s not enough for you to do the same thing as yesterday. It’s a deceivingly comfortable choice: no one has been fired for doing the same choices as done on the year before. But we have learn something dearly in Finland: the comfortability of no change comes with a terrible price. You’ll soon follow the fate of 00s mobile operating system: when you keep clinging on the old way the competitors will soon show the better way.

Python has proven its reputation as an agile technology platform. How about doing a small or medium sized project with Python instead of using (too) heavy enterprise tools? Think about if you could get all the time saved back in your next annual bonus? Here is a good related essay “Beating the Averages” from Paul Graham.

PyCon is the best place to see and learn the latest secret weapons Python has to offer. Many of open souce hackers, who write the development tools, are present in person. For example, in PyCon Finland we have Andy Smith from Rackspace (cloooouds). Be at the event and pump all the knowledge out of the brains of the masterminds.

2. Be global, act local

Spotify will be present with a considerable delegation. For those who live in a barrel (no offense US) Spotify is (was?) a real hot Swedish start-up which seeks to dethrone iTunes in the game of global digital music consumption.

Besides Spotify from Sweden, we have plenty of other international guests. The window of new opportunities is open when organizations having similar competency background meet (read: Python). I have been personally participating in ventures which started as  chit-chat in a conference and later developed into full blown businesses just because people and companies with the same mindset where given a chance.

SMEs might have developed technologies for which they seek new channels of marketing. A trending website might need a new localization. You can find new clients for whom to subcontract your Python talent.. or anything in between. When you are dealing with international people already familiar with the core technology you have much more trust for being able to work together. There is no fear of conflict of interest as market area geographies are isolated (especially true in EU).

3. Best of the best

I have heard rumors that there is a shortage of Python programmers in Finland to hire. There exists a simple explanation: The Python programmers are the best programmers out there and thus they already have the best jobs.

PyCon Finland is the place to meet Python programmers as they have been nicely gathered into one place for you to pick. This is your once in a year chance to verify the fact whether all the best programmers are already hired. (note: if you manage to disprove me I am not offering beer prize or anything)

 Subscribe to this blog in a reader Follow me on Twitter

I was migrating blog posts from one WordPress site to another. I wanted to have the old site still run the front page and few other static pages and I wanted to move the blog posts only. A best practice for content migrations is set HTTP 301 Moved Permantently redirects on the orignal web server so that users and search engines following old links will automatically find the new content.

WordPress sites were and are hosted on Apache web server. Since writing Apache .htaccess redirects by hand is something no living soul wants to do, I wrote a little Python script using Requests (a.k.a. urllib done right) and lxml libraries.

This script will

• Read sitemap.xml.gz file
• Extract page URLs out of it
• (add optional filtering step here)
• Output Apache .htaccess redirect rules so that old links will point to the new site

The beauty of scraping the information from sitemap.xml: Even though this script was written for WordPress it is valid for any CMS supporting Sitemaps protocol. (hint: Plone, Drupal, Joomla!, you name it.. they all support it)

In theory one could run this script in cron so that all new WordPress content gets automatically redirected continuously.

In additional for setting up .htaccess WordPress provides easy export and import tool for blog posts through its admin interface.

And the script:

# Python stdlib
from StringIO import StringIO
import urlparse

# 3rd party
import requests
from lxml import etree

# Sitemap of the source site
SITEMAP_URL = "http://mobilizejs.com/wp-content/plugins/simple-google-sitemap-xml/sitemap.xml"

# Where posts will be moved
NEW_DOMAIN="http://opensourcehacker.com"

# Apache .htaccess line template
# http://httpd.apache.org/docs/2.0/mod/mod_alias.html#redirectmatch
REDIRECT_TEMPLATE="RedirectMatch permanent %(path)s$ %(new_domain)s%(path)s"

response = requests.get(SITEMAP_URL)

def create_apache_redirect_to_other_domain(url, new_domain):
    """
    Creates .htaccess redirect entry.

    Create HTTP 301 Permanent Redirect.

    Output is something along lines::

        RedirectMatch permanent /tag/cache-control/$ http://opensourcehacker.com/tag/cache-control/

    """
    parts = urlparse.urlparse(url)

    vars = {
        "new_domain" : new_domain,
        "path" : parts.path          
    }    

    # Simple Python string template substituion
    return REDIRECT_TEMPLATE % vars

# Parse sitemap
tree = etree.parse(StringIO(response.content))
root = tree.getroot()

locs = root.xpath("//*[name()='loc']")

# Iterate through each sitemap <loc> entry and creating corresponding redirect
for loc in locs:
    url = loc.text
    redirect_line = create_apache_redirect_to_other_domain(url, NEW_DOMAIN)
    print redirect_line

 Subscribe to this blog in a reader Follow me on Twitter