Making Plone easy again (and Sauna Sprint topics)

Posted on by

This blog post is about making Plone CMS to accessible for more developers. This is a major topic on which we are working on in Sauna Sprint 2011 (see the previous blog post about Sauna Sprint and motivation to come to Finland).

1. Preface

Please do not take this post too seriously if I am ranting tongue-in-cheek. This thinking does not reflect the official opinion of Sauna Sprint team any way, it is just my personal rambling.

2. Background

Plone is having popularity issues. This can be deducted by:

  • PHP CMS are more popular (Google trends and so on)
  • Plone lacks love
  • By discussing with Python programmers outside Plone community: they see Plone as monster

Plone is a superior CMS product (feature wise, polish wise, tech wise, and so on). So how did this happen?

My theory, you are free to disagree, is that the community has been driving “wrong” ideologies

  • Plone’s “buyers” are not users, but developers
  • There has been no driving community factor to make Plone accessible for more developers. Plone community has been working to make Plone work for the existing community members, not new members. The consulting companies have been too busy to extract money from their existing big clients. It might even be that it’s the best interest of these companies to keep the barrier of entry high, as this way they protect the Plone business for themselves. Note that this strategy will fail in the point there is no longer Plone business.

The big boys are telling that customers are abandoning Plone, because there are no developers. I know a lot of Plone developers, looking for more Plone work.  I’ll translate this that there are not enough affordable Plone developers entering the market.

I am also not buying arguments like

  • “don’t compare apples to oranges (Django vs. Plone)”
  • “its history and evolved that way”
  • “big tool must be complex”

3. The result

Plone is easy to use, but hard to develop. If you need to use code generators and boiler-plate in Python code, you are doing it wrong.

Plone is a soup of different, non-mainstream, technologies. To be a Plonista you need master: Python (ok docs), Plone user mode (good docs), Zope component architecture (some docs), ZCML (no useful docs), GenericSetup (no docs), TAL (some docs), Archetypes (basically no docs), buildout (aaaaaarghs) … You have too much to learn from material which does not exist. Today the material effectively does not exist if it’s not your first Google hit – I am against book.

New developers are not getting in. It is easier to duplicate basic Plone functionality in a hacky in-house Django CMS than customize Plone for your use case. Plone does not give any more added value to the new developers. There is no business motivation to adapt Plone as a technology platform in new ventures.

4. The quest

Let’s fix it, as I see the developer acceptance the most crucial challenge Plone community is facing if the community wants to survive.

What we can do here is

  • Make Plone easier for developers
  • Make more, better, learning material available
  • Eliminate and protect against factors in Plone which hinder the goals of two above bullets (do not include new technologies or modules to core if they make developers sad)

To accomplish this we can

  1. Promote “easy” technologies inside and outside Plone community (Dexterity over Archetypes, Grok over ZCML)
  2. Making a standard workflow available for them (Django style tutorial that do 1, 2, 3 and you are a certified Plone developer)
  3. Kill technologies which make the stack unnecessary complex and are not necessarily needed in newbie use cases (buildout, I am looking at you)
  4. Replace old, badly documented and difficult solutions with easier ones (could we actually replace TAL with something more popular?)

5. Operational plan

Here are some concrete tasks what we could tackle in Sauna Sprint.

Make it easy department

  • Plone auto-restart team: provide working and fast auto-restart á la Django or Tornado (hint: plone.reload doesn’t cut it – you need real restarts process-wise)
  • Default custom product team: ship Plone with a file-system based custom product skeleton, so that when you need to include your first custom Python module you don’t need to learn about buildout and paster first . You actually can copy-paste in Python code and it just works.
  • Through-the-web is happy again team: Make it possible to code and execute Python through-the-web so that it actually works.: a working replacement for RestrictedPython

Fix user-wise broken things department

  • Buildout team: make buildout more user friendly: spit out error messages which are actually useful, have newbie buildout tutorial from the developer perspective. Nothing bad should happen when buildout is run. Make buildout to confirm version updates, etc.
  • Anti-buildout team: how to install eggs without buildout. Why did we need the buildout in the first place?

This is how you do it department

  • Tutorial team: Make “gapless” hand-holding tutorial for creating your first view, form and content-type. Make it so that you do not need to look outside the tutorial or learn anything unnecessary (don’t make me think)
  • Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily
  • Document generator team: Integrate collective.developermanual with Sphinx tool that generate the reference  documentation from the source code: views, viewlets, templates, CSS classes, portlets, etc.

Hasta la vista babe department

  • Kill it with fire team: Hunt down all bad learning material and make it redirect to new learning material (no ExternalMethods, no Install.py, no plone_skins, etc.)
  • Nuke it from orbit team: I am not sure what this team could do, but I like the feeling of the name

We hope that novice  developers attend the sprint. This way we can use you as guinea pigs and document the pain points on a road to become Plone developer. Don’t worry – we’ll inject you with proper medicine to numb the pain and forget bad memories.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

20 thoughts on “Making Plone easy again (and Sauna Sprint topics)”

  1. Through-the-web coding support is useless for production environments. Web interface itself is fine, but the code stored in the database is not maintainable. Production needs version control, release management and automated deployment, and all that is hard as hell with TTW code.

  2. TTW is hell for actual development.

    However I was more thinking of it like for running migration scripts etc.

    Let’s have a proper “Default custom add-on” on file system, so people don’t need to things throug-the-web 🙂

  3. You have most things right, but some things wrong.

    * Fast auto-restart is impossible with Plone, at least while it’s still based on Zope 2, probably forever. It is too big and has too much code, restarts will always be slow.

    * TTW development is a mistake and should die. It’s a major reason Plone is so complicated and developer-unfriendly. The replacement for this is Dexterity, with TTW definable content types, and a new workflow tool that has a nice UI for making workflows.

    (* And I don’t understand what you mean with “file-system based custom product skeleton”.)

    The nuke it from orbit team has an important mission:

    Step one: Get rid of all CMF dependencies. CMF is, has always been and will always be evil. This is a big step, of course.

    Step two: Write our own publisher, to get rid of the Zope 2 publisher. It has accumulated cruft over the years and is not a good option anymore. (I wonder if it would be possible to use the Pyramid publisher? I honestly do not have a clue).

    Step three: Remove all other remains of Zope 2.

    This will take at least five years, but then we can port it to Python 3 and make it run on PyPy. 🙂

  4. For auto-restart I have been thinking the following strategy which takes the shortcut that we are mostly interest reloading our own add-ons, not the whole zope:

    Create a parent process

    Run it until Plone egg has been loaded (after this add-on loading beging)

    Suspend the process

    Fork

    Continue loading in the child process

    When files change, kill child, fork parent again, continue loading from the plug-in load start point

    This way there is no overhead of loading Zope cruft – it is outside the effect of auto-restart.

  5. file-system based custom product skeleton:

    Have something in src/ prepared when you install Plone.

    Then when you need your new view class, you just toss it directly to the folder. No buildout needed, no egg creation needed, no paster needed, etc.

    Something like old “custom” plone_skins folder, but on file system.

  6. I would really love to see done the “Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily”.

    I’ve been programming Plone since 2 years and I still have hard time getting everything. ZCA, ZTK, five, all form generators …

    It would be extremely useful to know where we stand now (i.e. the picture as of 4.0/4.1 and the future transitions/recommended usage of technologies).

  8. I don’t use Plone, but I did give a try. I think you nailed a lot of the problems I encountered: a huge up-front learning curve, and many non-mainstream Python technologies (buildout and Zope). I don’t know if better documentation will help with this – there are plenty of books for Zope and Plone. Plone is large and complicated, so I don’t think recruiting from Python or programming newbies should be your goal. Making it much simpler to deploy and develop for would be great, but it’s also a lot of work. I think there are two things you can do to find more developers without a lot of code changes:

    1. Give mainstream Python developers a reason to use Plone, and then provide a clear path to it. What does Plone do better than Django? How is Plone similar to and different from Django? Here’s how you can move from Django development to Plone …

    2. Give experienced developers from non-Python frameworks a reason to move to Plone. These people are using expensive proprietary solutions or solutions built on a crappy language (PHP). Advertise how Plone is better and cheaper to the commercial segment. Advocate for Plone to people who already know Drupal but are tired of PHP’s silliness.

  9. @Mikki Ohtmaa
    Take a look at the latest plone.app.theming. We now support template customization and adding of view based templates in themes which can be put in a simple resources directory.

  10. @Mikko

    I don’t know if a good point for the kind of forking you are thinking of exists: I never managed to understand exactly what happens when Zope starts up, but as far as I remember it’s pretty damn spaghettoed (translation: it has lots of side-effect loading).

    There are also a number of other issues I’m quite scared of, such as:
    1. If you fork a process that has a zodb connection open, what happens if you try to use the connection in the child and in the parent? Does mayhem happen?
    2. There is a lot of “chain-effect” in ZCML loading, especially when z3c.autoinclude is used (aka, pretty much everywhere): this doesn’t make it easy to load some zcml in some process and some in another. See why plone.app.testing disables the z3c.autoinclude entrypoint.

    However, why not just add an autoreload behaviour on plone.reload? Namely, just watch the files in the develop eggs and trigger a code/zcml reloading using the plone.reload API? That would be faster to implement and, IMHO, cover 99% of cases (plone.reload tends to fail only when you are doing “clever” things, which you shouldn’t be doing anyway ;))

    Regarding buildout: it is annoying at times, but so is developing with virtualenv. I gave a look at this document https://github.com/zookeepr/zookeepr/wiki/Creating-a-development-environment-with-pip-and-virtualenv and it doesn’t make it sound all that easier compared to buildout (having to run setup.py develop for every package I have is outright annoying). Of course, you are very welcome to explore the possibility of using just virtualenv and pip to develop Plone (or develop with Plone) but do consider the number of steps one has to perform to:
    * Bootstrap a development environment
    * Set up more than one Plone instance (in a zeo setup, maybe)
    * Make modifications to zope.conf that can be somehow kept in a versioning system (changing ports, for example, or even more complex stuff)
    And keep in mind those have to be at least equals to the steps (intended as “command I type in the shell”) that one has to perform if doing the same thing with buildout.

  11. “There has been no driving community factor to make Plone accessible for more developers.”

    This is complete BS. *So* much community energy has gone into this for *years*. I won’t pretend there isn’t more to do. But initiatives like Diazo and Dexterity would not exist if it weren’t for a very real desire to make Plone more accessible to more developers.

    Martin

  12. Remove tal/metal, zcml, buildout, archetypes, any theming support besides xdv (or whatever is the name of the beast ian bicking created), zeo (to use relstorage). Replacing zope2 with pyramid seems nice but too far fetched to me.

  13. “This is complete BS. *So* much community energy has gone into this for *years*.”

    Yes. But it has not been top priority, has it?

    Dexterity, Diazo are steps to right direction and I very much like them. However, they are lost in the big picture, as people can’t engage into Dexterity when they still need to fight with a lot of buildouting and ZCML’ing and setup.py before getting into something useful.

  14. plone.reload and Python module reloading generally has too many corner cases and simply does not work reliably.

    If you want to see how add-on installation should be done please see WordPress 🙂

  15. Two years ago I needed a kind a CMS that could be extended newly developed content types. Being quite familiar with Pyhton I selected Plone 3. I bought the book Professional Plone development from M. Aspelli. After 1 month I gave up. It was too complex. ZCML is over engineered, the skiniing was a mess, the mix between Zope 3 and Zope 2 technologies was a nightmare and Plone was unacceptably slow. I switched to Pylons as framework and wrote my own small CMS on top of it. This main reason that I switch was not the defencies mentioned above, but the lack of internationalization. Plone is avaialable in multiple languages but it does not support multilingual content. Nevertheless the development experience with Pylons is so much better than with Plone. The mako tenplate language is much more powerfull than TAL/METAL, On my development machine at that time it took 15 seconds to restart Plone after changes were made, it took less than a second to reload Pylons. Perhaps Plone 4 is better, but I won’t switch

  17. Pingback: ILRT python / KISS djangocon.eu

  18. Mikko, thanks not only for pointing this out, but helping improve the documentation on plone.org. I’ve come across a lot of material you’ve done/edited.

    You’ve address a lot of problems that really happens. Plone IS difficult, PERIOD. In the long run, in the long term, this complexity is going to save your butt, but to get there, everything is going to be a PAIN IN THE ASS.

    The problem with Plone is that mixes everything is really hard to a developer to understand, and ALWAYS changing.

    Seriously, I just use Plone if needing complex workflows and need a really secure website. I can learn AND create a full working WordPress site in a week, but Plone I spent a f*cking month just to understand where everything fits.

    But I don’t agree about excluding old entries in documentation about Install.py and such – a lot of people still run legacy code, it’s not possible to upgrade/migrate, and they will be really left in the dark. Not everyone is using plone 4, can’t use, and we still need that documentation.

  19. Pingback: Mikko Ohtamaa: Making Plone easy again (and... | Python | Syngu

  20. Two sentences sum up my experience with Plone so far (I started developing with Plone in August 2012):

    1 – “Plone is easy to use, but hard to develop.”
    2 – “TTW is hell for actual development.”

    Programming by ZMI, extending page templates by ZMI (big, really big “tal:define”s. Where is the separation between presentation and logic?), and all those “context” and “here” in “Zope Python Script” is simply sad for a developer today. It’s difficult understand all the stack. There are good new tecnologies like Dexterity and Grok. But, for example, all the standard content types in Plone 4 is still using the Archetypes framework.

    I don’t have real hopes that Plone will be good as plataform/framework for developers. Concerns about backwards compatibility are big. It’s a choice, and I respect. There motivations (reasons) for this.

    Perhaps the Plone developers could create a new CMS, using Pyramid, as a successor of Plone. I think that Plone, product-box, is good. But developers suffer…

Leave a Reply

Your email address will not be published. Required fields are marked *