Monthly Archives: 2007-10

Tuning file system performance for Plone development

Posted on by
3

I recently read this article about tuning Ext3 file system for better performance. I was doing a fresh Ubuntu 7.10 install on my laptop, so I decided to see how much this would affect to my every day Plone development.

On Linux, every time a file is read, its access time attribute is rewritten. This causes a lot of unnecessary writes to file system. Since there are only few rare application needing this feature, turning of the feature can give a nice performance boost on systems dealing with large amount of files.

Plone 3.0 has 10000 files. A lot of them are read during the start-up. Maybe I am getting somewhere here…

When you are doing Plone development, you need to restart Plone often. I used this highly scientific method to measure Plone start-up time from issuing zopectl fg to getting the front page load completed in Firefox. I warmed the file system cache beforehand by doing two dry runs.

I also did some simple front page bombing with ab tool.

System setup

  • HP nx9420 laptop (5400 RPM hard disk)
  • Plone 3.0.2/Zope 2.10.4
  • Intel Core 2 Duo, 2 Ghz
  • Ubuntu 7.10
  • Applied following Ext3 options: noatime, data=writeback

Out-of-the-box filesystem

Lap 1: 23s

Lap 2: 22s

Lap 3: 22s

ab stats:

Concurrency Level: 10 Time taken for tests: 11.805239 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 2058700 bytes HTML transferred: 2030600 bytes Requests per second: 8.47 [#/sec] (mean)

Tuned file system

Lap 1: 21s

Lap 2: 22s

Lap 3: Didn’t bother to do it…

ab stats:

Concurrency Level: 10 Time taken for tests: 12.102054 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 2058700 bytes HTML transferred: 2030600 bytes Requests per second: 8.26 [#/sec] (mean)

Conclusion

“Hooray.”

Though Plone/Zope crawls through of thousands of files during the start up (and thus touches their access times), the slow start-up process seem to be CPU bound. Magic file system tricks won’t make your everyday Plone development more effective.

Plone business, part II, conference in Shangai

Posted on by
Reply

Let me first frame this blog entry: I was Shangai’ed last night. I picked up this new wonderful term when I was eavesdropping a conversation in a Plone conference. “Did you Shangai’ed last night?” “What do you mean?” “Drink till 5 am.” I must admit that I Shangai’ed quite badly. But you don’t need to worry about me. Friends from my hotel, they were totally Finnish’ed.

So, I am sitting at Fiumicino airport on my way back from a Plone conference, enjoying a well deserved hangover. Since I really don’t have anything I could do, I finally find time for updating my blog. 6-7 months ago I wrote a blog entry about what it feels like starting one’s own software sweatshop. Things have progressed quite nicely and I’d like to give a status update.

First about the conference. Plone conference 2007 (Plone being the best open source content management system out there with a strong business oriented community) was held in Naples, Italy. It was my first big open source gathering – 350 participants were present from all over the world. My home city, Oulu, is a bit off from the international routes and travelling was expensive: my wallet got Finnish’ed too. Still, I believe the marvellous conference was worth every euro.

I finally met one of my US clients face-to-face. They are a nice senior couple, who run a family web business. We had a lot of wonderful discussions, and got some humane depth into our business relationships, which I believe greatly helps getting possible joint ventures done more smoothly in the future.

Getting faces and voices for IRC nicks was a little confusing, in a positive way. You were hit with thoughts like “Oh my, this guy was so tall” or “I expected a younger one”. I also noticed I have started referring the Plone scene as “we” instead of “them” – a clear sign that the community has stolen my soul.

What comes to presentations and workshop sessions, Plone developers are, quoting a sentence from Limi, “- geniuses”. Plone/Zope is currently very hard to master software-development-wise, but this is a recognized problem. People are working on it to make it easier and more like other agile web frameworks out there (Django, Ruby on Rails or TurboGears).

By the way, writing on my Macbook feels like being the Sex and The City main character. 90% of people in the Plone scene are using Macs. I wonder why we don’t ask for sponsorship from Apple, since being in touch with Plone folks is an ultra effective mouth-to-mouth marketing channel for Apple. I didn’t have a Macintosh laptop in the previous Plone event I participated. The social pressure to get one is enormous – I had to order a Macbook just for this trip =)

1. The business part

Currently, Red Innovation incorporates four employees and one contractor. I am managing things, doing less coding every day. Three of my friends work as flexible part timers until they finish their studies. One freelancer is doing contracting through me for bigger clients. Plone is not the only thing in our boat: we have been doing everything from embedded mobile phone Linux to video and social sites on every known programming language.

In the conference, there were really cool presentations on how to become a Plone consultant and how to work with Plone consultants. I hope I could send the latter presentation video for all of my clients. Two of its main points were to be open and to have a budget for consultants. It’s really hard to work outside the information loop and all feature requests come as granted from the Heaven without explanation of the logic behind them. You can implement the feature, but the result is what the client told you, not what the client was thinking. Small fixes and adjustments follow after the initial implementation when the client clarifies his/her wishes. Even more small adjustments follow and micro-management ensures. It’s stressful to work this way, since you have zero degree of creative freedom and the same thing must be fixed repeatedly.

In the conference presentation of consulting business they told that the first few years are hard. You are not paid well and you don’t have any cash reserve, which drives you to accept all the work you can get. This slowly drifts you away from developing a sustainable growing business and your original goal as an open source entrepreneur. Sadly, I can confirm this to be very true. Currently I am working hard just to keep the company running. Contracts are too short. The continuance is missing. Average hourly fee is somewhere between 40€-50€/hour. As they mentioned in the presentation, in the UK Plone hour price level was more like 80€/hour, and this was for contracts shorter than a year. I can only dream about getting a Plone contract longer than a year. The most horrible thing to reveal is that our company is accepting PHP jobs whilst missing suitable projects for our core technology platform.

However, this is just a start. It was expected. On the bright side, our average hour price is rising, slowly but steadily. I have achieved almost everything I dreamed of when I finished high school. I have my own luxury top floor office and I can decide my own working hours. Most importantly, I am not doing Dilbert job as being a code monkey transforming design documents to code implementation whilst boring myself to death of insignificance in a grey corporate mass.

In the conference, a friend of mine asked something along the lines: “Mikko, How high you think you are in Plone developer ranking?” This sputtered the following truth: I won’t ever wave a flag at the very top of Mt. Plone-developers. When you want to run a business, be prepared to sacrifice the happiness of coding. You will write Word documents, emails, talk on the phone, and so on. Technology goes forward; you don’t have time to catch the train. Sooner or later you’ll see only the backlights of your prior superiour technical skills.

But, my dear Service Buying Friend, this doesn’t matter from your point of view as a client. Nowadays I offer something better over my coding thumbs. Instead being able to solve your problem, I can point a person to do it (and take my little slice in the middle). I want to change the world and make it happier place for all of us. I believe I achieve a far greater effect by establishing a role model of running open source business instead of just code, code and code. Even if I code very cool stuff.

2. User group Finland?

Finland lacks open source business scene. Contracting work business is generally very conservative here. Companies, with the exception of Nokia, neither have a clue how to be a part of a community nor why to contribute back to the community. Our public sector is enjoying high Microsoft-corruption where ASP and Sharepoint are the words.

There must be a change in this. Even for the sake of the national budget: I want my tax money spent on cost effective Finnish open source solutions instead of the crap produced in Redmond.

Maybe the Plone-Python user group, working with patience, could have an effect here. The evangelistic work of teaching customers, teaching public sector and keeping noise in media would eventually convert the hearts of non-techies. Besides Red Innovation, there is one known active Plone consulting shop in Finland. Two universities are using Plone in large scale. The great presentation of Roberto Allende about starting a user group just might have motivated me enough to gather some leadership and accept the task of bringing pieces together.

P.S. Plone 3.0 compatible DataGridField release will come soon! I promise!

P.P.S Posting this blog with WordPress update and all took 7 days. DGF 1.6 is already out there…

Wrong swap UUID after hibernation in Feisty

Posted on by
Reply

I’ve been using uswsusp for suspending/hibernating my Ubuntu Feisty laptop but suddenly it failed to resume from disk hibernation (blank screen with blinking cursor). I booted up in restoration mode and Ubuntu reported that it couldn’t restore the snapshot. After pressing enter to continue, the system booted up just fine, skipping the snapshot restoration as supposed. The startup problem vanished, but it brought up a new one when trying to hibernate:

>>> sudo s2disk
Could not use the resume device (try swapon -a)

Of course, swapon also gives a problem:

>>> sudo swapon -a
swapon: cannot stat
/dev/disk/by-uuid/4a815ae8-fa5b-4265-826c-d777a723e87b:
No such file or directory

It seems that the UUID reference for swap is broken. Or is it the swap? At this point I did some Google research and it turned out the behaviour was because of an Ubuntu Feisty bug, which causes the swap UUID change occasionally. It is closely related to hibernation, yet the cause remains unclear. To fix it, let’s do:

>>> free -m | grep -i swap
Swap:            0          0          0

Which indicates that the system doesn’t find swap at all (because of wrong UUID). To find correct one:

>>> sudo fdisk -l | grep swap
/dev/sda6      10669   10917  2000061 82  Linux swap / Solaris

Find your swap there and go for:

>>> sudo vol_id /dev/sda6
ID_FS_UUID=083d41f0-de57-48d4-92eb-aefde8fd6ec9

Then you’ll just have to edit it in /etc/fstab and hibernation should work again. You could also try restoring the original snapshot by editing:

>>> sudo nano /etc/initramfs-tools/conf.d/resume

and correcting the reference there also. I didn’t test this myself, however.

That should do it, but don’t get too excited: some report that the UUID keeps changing even after the fix, and it has to be manually changed over and over again. Luckily, I haven’t experienced such behaviour (yet!) and it’s quick to fix (though you might lose ability to hibernate, which is a really bad thing).