Monthly Archives: 2012-1

Mobilizing websites with responsive design and HTML5 tutorial

Posted on by
11

This is a blog post series tutorial for adapting your existing websites for mobile devices without building a separate mobile site. It shows, with examples, how with little changes in your HTML, CSS and Javascript code you can deliver much nicer user experience for small screen and mobile devices. You can make existing HTML designs more mobile friendly with selective CSS loading and HTML5 tags. Selective CSS loading with CSS3 media queries allow you to change layout depending on the browser screen size: this kind of layout is called responsive design.

The tutorial is divided to several, functionality specific, blog posts, each with screenshot examples, explanations and links for more in-depth information.

The tutorial was written in a conjunction with a consulting project for a Finnish public sector organization. As the one of the funding sources is Finnish tax money, including some of my very own pennies, it was a common interest to get the information born in the consultation project published.

Below is an example what one can accomplish.

The site landing page before any mobilization was done; the site is using the default desktop styles on mobile devices:

The site after HTML and CSS adjustments:

1. Prerequisites

Prerequisites for understanding this tutorial are

2. Table of contents

The tutorial contents is outlined below. I’ll keep linking to new blog posts as I finish writing them. Stay tuned, by following the RSS feed or Twitter.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Fixing WordPress bad new lines (extra br tag) when copy-pasting HTML by modifying wpautop() behavior

Posted on by
1

When you copy-paste arbitary HTML code into HTML view in WordPress post editor, you’ll get HTML soft newlines converted to hard newlines (<br>)  by default. This is utterly annoying for me, who writes posts offline in reStructuredText and then copies the content to WordPress thru rst.ninjs.org or rst2html.py.

I finally found some time to workaround this issue. The culprint is wpautop() filter.

Add the following in functions.php thru the theme editor:

remove_filter( 'the_content', 'wpautop' );
remove_filter( 'the_excerpt', 'wpautop' );

// Preserve <p> tag and do not insert <br> when transforming HTML for display
function wpautop2($pee) {
 return wpautop($pee, false);
}

add_filter( 'the_content', 'wpautop2' );
add_filter( 'the_excerpt', 'wpautop2' );

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

readthedocs.org, GitHub edit backlink and short history of Plone documentation

Posted on by
8

TL; DR: To encourage contributions to developer documentation you can maintain Sphinx based documentation on GitHub, deploy it via readthedocs.org and have this nice “Edit this document” pop-up note on the deployed docs.

The source code for the edit pop-up is here. I hope you can see my cool CSS3 effects!

Also this blog post is a story how you gain 180k words worth of developer documentation almost from the scratch for your favorite Python project. A Story worth of telling, I hope.

1. History

To understand the context and the level of complexity.

Plone CMS, mature (as in your great-great-great-grand-aunt) Python based content management system, has not been very developer  friendly. The goal of “to make it easy to customize and extend” has never appeared in Plone roadmap or conference keynotes. Instead, the most of the developers have generally been busy of doing more interesting stuff and pushing the technology forward.

Plone is also geared toward more heterogeneous end of codebases out there, due to its long history (>10 years – a long time for a Python product) and adoption of many third party-ish packages in the codebase (248 Python packages) . All these packages, including critical packages you should understand for extending Plone for your own purpose, come with less or even less documentation. Sometimes packages are not Plone related and authors may not be interested how to apply the re-usable components in the context of Plone.

It’s all sucks if you are a developer and simply want to get your little customization done into Plone. You need to drill through a stack and try to guess which piece of history to apply in which point of source code.

2. From closed official documentation to open-ended unofficial documentation

There exists an official Plone developer documentation – official in the sense Plone Foundation has copyright over it. However this level of ownership, right to relicense, comes with a price: it’s “hard” to contribute for it – you need to sign a contribution agreement (by hand) and snailmail it somewhere. Due to lack of content and contributions, the official documentation does not carry you far.

Somewhere in the past “unofficial” Plone developer documentation was started. The goal of this project was to have more open-ended documentation process to get some developer documentation for Plone in the first place and the fix some of the problems mentioned before. The spirit was “wiki-style” though the tools were different. The community went through some mailing list wars about the future of the documentation. This was the future then (and now is the current past):

  • First the unofficial documentation was bunch of Plone pages (WYSIWYG) managed in plone.org. We were living the time when Sphinx was still a dream in its authors minds.
  • Then it was converted to reStructured Text pages, still managed in plone.org
  • Then it was pulled out from plone.org, converted to Sphinx, put into Plone “collective” Subversion control (no contrib agreement needed). We were living the time when first Sphinx release was put out into PyPi.
  • Sphinx becomes de facto method of creating Python developer documentation
  • Then it was hosted as standalone HTML files, under collective.developermanual Python package, on random hosts
  • Then it was hosted on plone.org again. This time Sphinx generated HTML was imported to plone.org using a tool called Funnelweb. Funnelweb was originally created for importing arbitary sites into Plone as Plone content (it has been known to work for Liferay, WordPress, Drupal, etc.) Howver, this was still bad idea as Funnelweb does not have really have a concept of updating docs – just one time imports and data dumps.
  • And, da-da, it’s 2010s, readthedocs.org is launched. The documentation management is moved to GitHub as, with readthedocs.org, this solves the need for automatic documentation deployments after a commit.
  • GitHub creates a button Fork and edit this file
  • readthedocs.org gains edit back link when issue 152 is worked around
  • The documentation title is renamed from Plone community maintained blaablaablaa to more simple Plone Developer Documentation (as opposite to official Plone Developer Manual)
  • This blog post is written – hoping to encourage even more open-ended developer documentation contributions

Plone Developer Documentation now contains 183 792 words. It is comprehensive, but not coherent. It also crucially lacks polished, hand holding, Hello world tutorial needed to lure in new developers (someone please take a ball on this). Even with these little hindrances, based on the readers’ comments, it’s slowly becoming really really good.

The unofficial documentation is licensed under GPL license. The rationale for this is simply the fact that when collective.developermanual package was created this was the default license coming in Paster templates. Also, Plone Foundation’s official stance is that when you import code from Plone your product should be GPL. The documentation contains a lot of code starting with import plone… This little gem does not have any practical impact, but I wish to live to see some arguments about this.

3. Fork and edit this file

I love GitHub; I hate Git. Git is the anti-thesis of user interfaces – what a version control system would look like if it were developed by persons knowing only C and Perl and who’d love to scare away all other developers how are not equally smart. But luckily with GitHub, everyone, even without theoretical version management course studies, can work with Git.

One of GitHub’s recent innovations is Fork and edit this file button. It allows anyone to suggest changes to source file. No prior relationship between the author and the contributor is needed – the suggestion goes to the pull request queue and you’ll get a notification about the contribution.

This was a holy grail for me who cares less about politics and more about getting in more contributions.

  • Any person can edit the file, as Git/SVN checkout is not needed
  • You don’t need to maintain user list of “authored editors”
  • The GitHub editor itself (ACE based?) is actually nice: selecting text and hitting Tab key actually indents a block of text. It’s way beyond crappy <TEXTAREA>.
  • You get pull request notifications
  • You can post-edit proposed changes easily and communicate with the contributor

4.  Linking it back

A perfect documentation update workflow would be that when you want to edit a piece of text you simply click it in the place, edit in place and press Save. This is what Wikipedia was before it grew too large.

With readthedocs.org, GitHub and Javascript  magic we cannot reach such fidelity yet. We

  • Don’t know if user is logged into GitHub, or his/her GitHub username
  • Cannot edit individual Sphinx paragraphs in ACE
  • Don’t have GitHub Javascript API to push in commits/pull requets

(Please correct me if I am wrong)

But someday we’ll be there.

Meanwhile, we have something what we call in Finnish a “fur cap solution” (karvalakkiratkaisu). It means solving the problem in the least complex way, but still achieving the solution somehow. (If someone would like to clarify the etymology of this proverb or has a translation for it I’d like to hear that also)

In Sphinx HTML templates we simply include backlinks to the source code file on GitHub and have bulleted instructions how to edit the file:

Edit this document

The source code of this file is hosted on GitHub. Everyone can update and fix errors in this document with few clicks – no downloads needed.

  1. Go to Plone Developer Documentation on GitHub.
  2. Press Fork and edit this file button
  3. Edit file contents using GitHub’s text editor in your web browser
  4. Press Propose file change at the bottom of the page

For basic information about updating this manual and Sphinx format please see Writing and updating the manual guide.

5. Future future

After struggling several years of getting any developer documentation for Plone I am think I am finally quite happy and my work here is done. It’s not perfect, but compared to other CMS’s we are not definitely the worst anymore.

What still is needed is more interactivity – where people can ask questions in the context of documentation, like happens on php.net API documentation. Then store the question and the answer in the context, so the future generations do not need to wonder the same thing again. This far I have been manually linking stackoverflow.com questions direcly to the documentation when I am too lazy busy to start writing anything. Sadly, the orignal plone.org documentation had discussion feature until it was disabled, due to lack of good notification system. Maybe we could do something akin the lines using DISQUS and Sphinx and post questions to the mailing list?

6. Ps.

The edit pop-up note text, due to CSS3 rotation, looks like crap on Firefox 10 / OSX. Chrome does not suffer from this. A bug in FF text rendering?

 

 

 

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Fixing POSKeyError: ‘No blob file’ content in Plone

Posted on by
6

Plone CMS 4.x onwards stores uploaded files and images in ZODB as BLOBs. They exist in var/blobstorage folder structure on the file system, files being named after persistent object ids (non-human-readable). The objects themselves, without file payload, are stored in an append-only database file called filestorage and usually the name of this file is Data.fs.

If you copy Plone site database object data (Data.fs) and forget to copy blobstorage folder, or data gets out of the sync during the copy, various problems appear on the Plone site

  • You cannot access a content item which has a corresponding blob file missing on the file system
  • You cannot rebuild the portal_catalog indexes
  • Database packing may fail

Instead, you’ll see something like this – an evil POSKeyError exception (Persistent Object Storage):

Traceback (most recent call last):
  File "/fast/xxx/eggs/ZODB3-3.10.3-py2.6-macosx-10.6-i386.egg/ZODB/Connection.py", line 860, in setstate
    self._setstate(obj)
  File "/fast/xxx/eggs/ZODB3-3.10.3-py2.6-macosx-10.6-i386.egg/ZODB/Connection.py", line 922, in _setstate
    obj._p_blob_committed = self._storage.loadBlob(obj._p_oid, serial)
  File "/fast/xxx/eggs/ZODB3-3.10.3-py2.6-macosx-10.6-i386.egg/ZODB/blob.py", line 644, in loadBlob
    raise POSKeyError("No blob file", oid, serial)
POSKeyError: 'No blob file'

The proper solution is to fix this problem is to

  • Re-copy blobstorage folder
  • Restart Plone twice in foreground mode (sometimes freshly copied blobstorage folder does not get picked up – some kind of timestamp issue?)
  • Plone site copy instructions

However you may have failed. You may have damaged or lost your blobstorage forever. To get the Plone site to a working state all content having bad BLOB data must be deleted (usually meaning losing some of site images and uploaded files).

Below is Python code for Grok view which you can drop in to your own Plone add-on product. It creates an admin view which you can call directly thru URL. This code will walk thru all the content on your Plone site and tries to delete bad content items with BLOBs missing.

The code handles both Archetypes and Dexterity subsystems’ content types.

Note: Fixing Dexterity blobs with this code have never been tested – please feel free to update the code in collective.developermanual on GitHub if you find it not working properly.

The code, fixblobs.py:

"""

    A Zope command line script to delete content with missing BLOB in Plone, causing
    POSKeyErrors when content is being accessed or during portal_catalog rebuild.

    Tested on Plone 4.1 + Dexterity 1.1.

    http://stackoverflow.com/questions/8655675/cleaning-up-poskeyerror-no-blob-file-content-from-plone-site

    Also see:

    http://pypi.python.org/pypi/experimental.gracefulblobmissing/

"""

# Zope imports
from ZODB.POSException import POSKeyError
from zope.component import getMultiAdapter
from zope.component import queryUtility
from Products.CMFCore.interfaces import IPropertiesTool
from Products.CMFCore.interfaces import IFolderish, ISiteRoot

# Plone imports
from five import grok
from Products.Archetypes.Field import FileField
from Products.Archetypes.interfaces import IBaseContent
from plone.namedfile.interfaces import INamedFile
from plone.dexterity.content import DexterityContent

def check_at_blobs(context):
    """ Archetypes content checker.

    Return True if purge needed
    """

    if IBaseContent.providedBy(context):

        schema = context.Schema()
        for field in schema.fields():
            id = field.getName()
            if isinstance(field, FileField):
                try:
                    field.get_size(context)
                except POSKeyError:
                    print "Found damaged AT FileField %s on %s" % (id, context.absolute_url())
                    return True

    return False

def check_dexterity_blobs(context):
    """ Check Dexterity content for damaged blob fields

    XXX: NOT TESTED - THEORETICAL, GUIDELINING, IMPLEMENTATION

    Return True if purge needed
    """

    # Assume dexterity contennt inherits from Item
    if isinstance(context, DexterityContent):

        # Iterate through all Python object attributes
        # XXX: Might be smarter to use zope.schema introspection here?
        for key, value in context.__dict__.items():
            # Ignore non-contentish attributes to speed up us a bit
            if not key.startswith("_"):
                if INamedFile.providedBy(value):
                    try:
                        value.getSize()
                    except POSKeyError:
                        print "Found damaged Dexterity plone.app.NamedFile %s on %s" % (key, context.absolute_url())
                        return True
    return False

def fix_blobs(context):
    """
    Iterate through the object variables and see if they are blob fields
    and if the field loading fails then poof
    """

    if check_at_blobs(context) or check_dexterity_blobs(context):
        print "Bad blobs found on %s" % context.absolute_url() + " -> deleting"
        parent = context.aq_parent
        parent.manage_delObjects([context.getId()])

def recurse(tree):
    """ Walk through all the content on a Plone site """
    for id, child in tree.contentItems():

        fix_blobs(child)

        if IFolderish.providedBy(child):
            recurse(child)

class FixBlobs(grok.CodeView):
    """
    A management view to clean up content with damaged BLOB files

    You can call this view by

    1) Starting Plone in debug mode (console output available)

    2) Visit site.com/@@fix-blobs URL

    """
    grok.name("fix-blobs")
    grok.context(ISiteRoot)
    grok.require("cmf.ManagePortal")

    def disable_integrity_check(self):
        """  Content HTML may have references to this broken image - we cannot fix that HTML
        but link integriry check will yell if we try to delete the bad image.

        http://collective-docs.readthedocs.org/en/latest/content/deleting.html#bypassing-link-integrity-check "
        """
        ptool = queryUtility(IPropertiesTool)
        props = getattr(ptool, 'site_properties', None)
        self.old_check = props.getProperty('enable_link_integrity_checks', False)
        props.enable_link_integrity_checks = False

    def enable_integrity_check(self):
        """ """
        ptool = queryUtility(IPropertiesTool)
        props = getattr(ptool, 'site_properties', None)
        props.enable_link_integrity_checks = self.old_check

    def render(self):
        #plone = getMultiAdapter((self.context, self.request), name="plone_portal_state")
        print "Checking blobs"
        portal = self.context
        self.disable_integrity_check()
        recurse(portal)
        self.enable_integrity_check()
        print "All done"
        return "OK - check console for status messages"

More info

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+