How does it do a date picker widget?

Python folks love frameworks. They put framework in your framework so that you can create a framework. A friend of mine was involved yet another of these framework ventures and was seriously excited when he was trying to explain to me how good the new web framework was and what problems it solves.

This blog post is a long flow of mind try to pinpoint how to define if a framework is helpful for you or not. Though I use a Python framework as an example these thoughts can be applied for any programming language, framework, CMS, etc. out there. I’d be especially interest how Joomla!, Drupal etc. popular PHP platforms solve problems mentioned here.

The framework in the discussion was towards high level of web development, contentish things. To make tailored CMS, and stuff. This framework was on higher level than Pyramid, little bit higher than Django. It is named after an egyptian god and because Egyptian gods have difficult names spelled with symbols, my Finnish mind already lost it.

So the friend of mine was trying to convince me about the need and superiority of this new frame and asked the question: “Do you like it?”

And I answered: “Does it help me to get my work done?”

My question was the right question, but wrong in so many ways. We were discussing about the same thing, but from different viewpoints…. and in this point, the discussion got off the rail. My friend tried to justify the existence of the new framework and I was trying to help him understand what kind of needs I hope a framework would solve for me.

As we put a lot of energy in seemingly fruitless conversation, I become sad. This discussion and thoughts about it keep me awake during the night and I tried to come up a simple way to measure the usefulness of a high level web framework.

Now I have a new question, which in the end is the same question as above, but more communicative. This question was inspired by Douglas Crockford’s talk about Javascript metaperformance and especially the part where he tells that there are a lot of smart people doing web – sometimes they focus on wrong things, but  if you give them a proper target to shoot at they can get amazing things done.

Next time, when someones needs my opinion about a web framework, I’ll ask “How does it do a date picker widget?”

1. Modern Times

To see how my little question works you need to dive into the history of web development a bit.  The web user interaction is much about forms. The forms have input fields and these input fields are wrapped into user interact-able widgets which have additional properties besides simply handling punching of keyboard keys: input validation, typing auto completion, mouse interaction, nice bling bling decoration and effects  and so on… especially when creating things like CMS where the users are expected to input stuff through the browser you need to pay careful attention for the user interaction details.

The purpose of a software development framework is to make the life of a software developer simple and get the thing done.  Because web is much about forms a lot of effort has been put to make form development simple. My question could have been a less form specific; I chose a date picker because it is familiar and everyone has come across one e.g. when booking flight tickets. Thus we should have quite unified idea what we are discussing about. The date picker also tends to be one of the most complex form widgets available in your toolbox, so were are not setting the bar too low here.

2. Last Tango in Paris

Web itself was not originally designed to have or to do rich widgets for easy user interaction. We have a HTML mark-up language which tries to separate content from the presentation. We have a layout language, CSS,  which tries to separate layout from the content, but in the end fails doing so. Then we have a Javascript programming language which pulls strings in your web browser to make HTML and CSS co-operate and  do the widgets using Document Object Model. A model, which was not designed Javascript programing language in mind.

(sidenote: on the day I see bbc.co.uk level site delivering both web and mobile version from the same HTML code  just using different CSS stylesheets I can say that CSS actually delivers it promise to separate the layout from the content)

Now is now and web is steering more and more to the direction where it can do more for the user… or the interaction is more efficient because of richer controls and paradigms, enabled by Javascript and HTML5, can be applied to solve the same old problems. In this world if you do a modern web framework for and expect it to remain on the isolation on the server-side from all the gory details of Javascript and CSS you are doing it wrong.

Doing a web widget is like a dance of several participants who need to co-ordinate their actions so that the individual performances are in harmony and the actors are not stepping on each other toes. A web framework is like premade step guide telling you how to execute your dance. The result is magnificent performance of  0.5 seconds of your site visitor time when he or she subconsciously picks a date in the web form by clickety-clicky, or as on the mobile era, by thumbity-thumbly.

3. Missing in Action

Now, I could have asked “Does it do a date picker widget?” The answer would probably be “yes”, as of course you can put a date picker widget on every HTML page out there. Then I could asked “how?” and then the answer would be something like:

  • Download your favorite Javascript widget library (e.g. JQuery UI)
  • Toss CSS files somewhere, Javascript files somewhere in your project
  • Edit your  main template <head> to have these files included – if you are smart include them only on the pages where the user interaction is actually needed
  • Bootstrap <input> fields doing date picking using a CSS class markers or similar markers in the source code somewhere, or have a bootstrap JS running by <input> ids
  • Convert between Javascript widget framework date model and your web framework date model in somewhere of the form layers
  • etc. etc.

This is what I call a blunt approach. It’s solving the problem of dancing like this:

This all is tolerable if you do it once for your web appy. But what about if you need different widgets on the page too? How about checkbox groups with internal dependencies? Time picker?  Yoyo? Again you need to toss files somewhere, edit head, etc.

When your web application grows this all quickly leads to a mess where you have tens of different files included in the main template. Different modules start to clash with their CSS classes and you need to start resolving the issues by your own hand written CSS overrides. Suddenly the page load times are through the roof, because you are doing too many HTTP requests per page. You need to start thinking about the cache. You need to think about merging static media files. Which leads you to think solving dependency orders between media files.

Does the framework help you solve the problems like above to get the thing done?

4. Notting HIll

I studied industrial engineering and management in University of Oulu (home of IRC – never forget to mention this when I have a chance).  We had a course called Quality which was about quality. You can study subjects like this from very broad perspective especially if your plan is to become a pointy haired boss who doesn’t know crap about what he or she is doing.

The definition of quality is simple. Quality is not the same thing as high value brand (e.g. Mercedes Benz). Quality is not the same thing as faultiness (no bugs in the delivered software). Quality is about whether the customer is satisfied what he or she gets.

The customer of a software framework is a developer. What the developer is buying is a tool to make delivering software more easy – to get the thing done by the end of the day.

My friend was telling how high test coverage the framework has. This is a nice trait for a framework. Higher the test coverage, the less  bugs you are likely to encounter. Essentially this is faultiness.

Now a bad, but thoughtful, analogue: in industrial engineering faultiness comes with a price. On your production line you’ll pay a high cost of 100% faultiness products. You usually want to do testing by sampling and simply let few bad products slip through instead of having the high cost of testing every single manufactured item. In the end, the testing cost of the product is away from the other areas of product development and manufacturing (features, delivery options, etc.)

Faultiness is important, but it does not make high quality software framework alone. Inspired by the leaked rant of Steve Yegge if you dial test coverage 100% but have no accessibility you have no product.

Which brings me to the punch line of software framework quality: accessibility. To enable developers get the thing done with the framework it must have documentation telling how you get the thing done.

Django folks get it. Django has always had very detailed documentation and no surprise it won the most of the hearts of Python developers back in 2006ish when the framework situation was different with Turbogears et. al. being still in their strength.  Anti examples would be packages z3c.form or plone.portlets (chosen by a fair dice roll – guaranteed to be random). Both the packages are frameworkish and have near perfect test coverage. Too bad it seems that only 0.0001% of people are smart enough to use these packages and mostly get frustrated whilst trying to get the thing done.

(sidenote: even though z3c.form has plentiful documentation does not primarily address the needs of the framework users)

5. Diamonds are forever

Another trait of quality is that it must be measurable.  The absolute measurement of the quality is the customer satisfaction: how happy people are with your product. However the level of satisfaction is difficult to measure directly. Indirect, easy numbers, are measured instead.

One easy quality measurement  is that you go around and ask random users whether they are happy with the framework. If you do not know any of these developers users yet you can simply follow the chat on a related IRC channel:  friendlier atmosphere usually correlates with happiness which correlates with quality.

But being an engineer I hope we could use something more tangible than random chat around a pint or two.

The most important indirect number of your framework quality could be API documentation coverage. How many of the functions of your API have sane helpful documentation or documentation in the first place? As this number  translates to how easily people can pick up your framework and simply read how to get things done with it. Especially with Python a good API documentation (parameters in, results out) is important because Python has poor statistical code analysis or auto-completion support, something  what Java and C# IDEs enjoy.

6. I Am Legend

The another nice thing you can learn about the quality on the university courses is that less numbers you need express the quality of your product the better. It’s over simplifying of things to say how good or bad your product ultimately is. Your pointy haired boss might be able to process few percent scale numbers a day, but less he needs to think anything besides golf, the better. You suggest a new framework to be used for the new software project and the boss takes a decision if it’s good or not, essentially the boolean value or quality. On the rare day the PHB is on his/her brightest the PHB might think one step forward and ask why the framework is any good and then you can answer that the goodness has been tested with the scientific method found from a random blog in the internets.

.. and to shine at the very end of quality simplification spectrum I came up with the following objective decision tree to measure the quality of a web framework:

Is a web framework, CMS, tool etc. any good for modern development needs? Yes / No

Yes if…

  • You take one average developer, like me, who has wide general programming experience, but not yet insight how the framework works (don’t pick any super smart guy like the framework developer itself might be).
  • You give the developer four hours of time (I am generous about my time today), a hello world style tutorial and API reference manual.
  • The developer is able to create a simple form / content type / page  containing a date picker widget with some customizations like template changes or non-default options. The customizations are important because that’s what you eventually will do – you won’t be mass producing hello world applications from the conveyor belt with the framework.
  • The resulting Javascript and CSS widget bindings to HTML are handled in maintainable manner and media files are merged, compressed and HTTP cache headers are set  properly.

…else no.

This is just a litmus test. It doesn’t necessary tell how good a framework is, but if the test fails then it might be a sign that the framework is either not a developer friendly or is not solving the needs of modern web frameworks. The most possible failure mode is that there are no API documentation at all –  which means that the project is not disciplined to communicate what problems the framework should be solving in the first place and is worth as much as any random bunch of code you can find from the open source archives.

Also, this blog post is just guidelining: no need to nitpick the details unless they advance the humankind somehow. As mentioned before, the blog post tries to give a good target to shoot at and highlights what kind of facts you might need to communicate out to success as a software development framework. Please feel free to challenge me, but as far as I am aware, none of current web frameworks or CMSs passes this test.

Finally, how the proper dance of HTML, CSS and Javascript should look like, Chuck Norris style:

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Best prepaid voice and data plan for internet addicted US visitors

On my way to Plone Conference I found this sweet deal from T-Mobile.

For 3 USD / day you’ll get unlimited local voice, text and 200 MB data / day. You can prepay it for arbitrary number of days.

They didn’t know the deal in the store, but found it on T-mobile website and eventually figured out, so be prepared to wait in the store. But it seems to be a real deal. I get EDGE connection in Los Angeles, no HDSPA though 🙁 However this is enough to use SSH and Maps reliably thus making life little bit easier for the country boy in this big city.

For the comparison, ATT offered 500 MB data for 25 USD, no voice, no SMS.

If you have locked phone…. well then you can just blame yourself for making a bad deal 🙂 T-Mobile does not seem to offer i-stuff.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Set-up PPTP virtual private network (VPN) server on Ubuntu Linux

This is short guide how to set-up VPN service on Ubuntu Linux server. VPN is intended to secure your internet traffic against local snooping by encrypting the traffic and tunneling it out from known good server. Possible use cases are

  • Secure your organization internal traffic between home office, mobile devices an the office
  • Hide internet traffic from the big brother
  • Spoof your location for TV and video streams: you are renting a server in a different country, like US, allowing you to watch US only video streams
  • Spoof your location for iPhone / iPad / iPod / Android apps
  • Visit sites which are otherwise unreachable (though in this case an SSH tunnel + SOCKS proxy is easier to set-up, more lightweight, alternative)
  • Test mobile applications against local servers during mobile app development

These instructions set-up VPN using PPTP protocol which is widely supported. After the server has been set-up you can use any modern OS or device to connect to the VPN: Windows, OSX, Linux, iOS, Android, etc.

1. Prerequisitements

  • Basics of UNIX command line (the hacker typing thing)
  • Basics of Linux server administration

2. Setting up PPTP daemon

We are using (old) Ubuntu 8.04 server though instructions should apply for later Ubuntu versions too. These installation instructions are restart free – however few commands must be placed into /etc/rc.local file to make the changed settings persist over Linux restarts.

First install some free software

sudo apt-get install pptpd telnet iptables

Pick up your server public fixed IP address. Usually this is what you see in ethernet 0 adapter.

ipconfig

Then edit /etc/pptpd.conf

sudo nano /etc/pptpd.conf

(Use sudo nano for editing other files too later in this tutorial)

Uncomment and change the localip line to your server public IP address

localip 70.164.204.106

Then edit /etc/ppp/pptpd-options

Make PPTP service to give DNS name resolving servers for the VPN clients. In optimal case you use the DNS server from your server provider, but Google’s public DNS servers (below) work fine:

sudo nano /etc/ppp/pptpd-options

Uncomment and change ms-dns lines to

ms-dns 8.8.8.8
ms-dns 8.8.4.4

Edit /etc/ppp/chap-secrets as sudo and add your PPTP username and password (plaintext)

sudo nano /etc/ppp/chap-secrets 

# Secrets for authentication using CHAP
# client    server    secret            IP addresses
opensource  pptpd     hacker            *

Run the following commands to enable IP NAT routing on your server (allows piping traffic through the public IP of the server):

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo sysctl net.ipv4.ip_forward=1

These settings above are effective only until the server restart. Put them into /etc/rc.local start-up script file to make sure Linux runs them on the start-up.

Restart PPTP daemon to make it read settings again

sudo /etc/init.d/pptpd restart

3. Setting VPN PPTP client

For the VPN client (iPad, Windows, OSX, Ubuntu Network Manager) the settings would be for the example above:

  • Protocol: PPTP
  • Server: your server public IP or DNS name
  • Login: opensource
  • Password: hacker
  • Encryption level: auto (or it should pick secure MPPE 128 by default)
  • Send all traffic: Yes (will forward all traffic through VPN, not just one inside your VPN devices)

4. Troubleshooting

If the VPN traffic is not going through, i.e. your internet stops working when VPN client is active, here are some instructions to pinpoint where the error happens.

Inspect the server syslog while connecting to the VPN service

sudo tail -f /var/log/syslog

These lines are “normal”:

Oct 22 18:34:05 ubuntu pppd[19129]: Plugin /usr/lib/pptpd/pptpd-logwtmp.so loaded.
Oct 22 18:34:05 ubuntu pppd[19129]: pppd 2.4.4 started by root, uid 0
Oct 22 18:34:05 ubuntu pppd[19129]: Using interface ppp0
Oct 22 18:34:05 ubuntu pppd[19129]: Connect: ppp0 <--> /dev/pts/1
Oct 22 18:34:05 ubuntu pptpd[19128]: GRE: Bad checksum from pppd.
Oct 22 18:34:08 ubuntu pppd[19129]: MPPE 128-bit stateless compression enabled
Oct 22 18:34:09 ubuntu pppd[19129]: Cannot determine ethernet address for proxy ARP
Oct 22 18:34:09 ubuntu pppd[19129]: local  IP address x.x.x.x
Oct 22 18:34:09 ubuntu pppd[19129]: remote IP address 192.168.0.234

On the client side, try to use ping command to reach servers by IP address to determine if you have problems with DNS set-up

  • Your server public IP
  • 8.8.8.8 (Internet IP, Google’s public DNS server)
  • google.com

E.g. run the command

ping 8.8.8.8

.. and see if the packets go through.

If you manage to ping numeric IPs, but not google.com the problem is with DNS settings.

After the traffic is flowing through check the public IP address your device using duckduckgo.com:

https://duckduckgo.com/?q=!what+is+my+ip

It should give your VPN server IP address instead of your lcoal public IP address.

5. Random notes

  • iOS devices and Android devices also may use crowdsourced WLAN / 3G station data for determining yout location, so you might need to spoof your local WLAN connection somehow
  • To determine if WLAN is being used just open Google Maps, show your location and if it doesn’t go to the location of VPN server then it is using your WLAN position
  • Practical workaround: Just set a temporary tethering WLAN access point using your mobile phone (e.g. Android) and walk to somewhere where your device cannot hear WLAN to work around this – most of apps check the location only once during the launch

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

PyCon Finland live streams running

Here, today!

Track 1, auditorium Lambda 9:45 Opening notes 10:00 “from __magic__ import wtf” by Tommie Gannert from Spotify 11:00 “Replacing Spreadsheets with Django” by Teemu Kurppa from Huikea Experience 12:00 Lunch 13:00 “High-Performance Computing with Python” by Jussi Enkovaara from CSC 14:00 “Understanding Encodings” by Ezio Melotti, a CPython core developer 15:00 Lightning talks 16:00 Keynote by Mike Bradshaw Closing notes

Track 2, classroom B2033 11:00 “Python for Data Science” by Harri Hämäläinen, a semantic data scientist 12:00 Lunch 13:00 “Building Your Own Cloud” by Andy Smith from OpenStack 14:00 “Non-blocking QA” by Anthony Long

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+