Monthly Archives: 2011-3

Google App Engine: issues with dynamic instances and DeadlineExceededErrors

Posted on by
1

1. Dynamic instances and processing time

This Google App Engine feature came me as a surprise, though it makes perfect sense. Your site is slow if it has low traffic.

Google App Engine runs Python code on instances. By default, instances are dynamic. Instances are shutdown if they do not have enough traffic (requests per minute). Thus, when you get the individual hits to App Engine now and then, App Engine must restart your instance every time for each hit.

When this happens, you see the following in App Engine console logs for every request on low volume traffic:

This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.

It is not always ok to add 500 – 2000 milliseconds processing delay on the top of the normal processing time. Google’s own recommendation was that each page should be served within 200 milliseconds.

There are three ways to optimize this issue

  • Use App Engine premium feature “Always on” 0,30 $ / day which keeps your instance always running
  • Use cron job or such to keep your instance alive (polling once in a minute seems to do the job)
  • Optimize your imports and split your code to several modules with light amount of imports, so that start up is fast (modules are imported only once)

We are using Zabbix software to monitor our sites (sidenote: I don’t recommend Zabbix as the first monitoring software choice as it is very difficult to use and has bad user experience, alienating both sysadmins and developers away from it). This is what we had before optimizations – App Engine was starting a new process for every request:

… and this is output we got after optimizations:

Here is the corresponding diagram after optimizations from App Engine dashboard itself. These processing times are without network latency. As far as I know Google does not expose the endpoints of App Engine hosting, so you don’t know from which site of the world your responses come from. By comparing this diagram to the diagram above, you can see how Internet traffic is affecting to your App Engine application.

2. The PITA of dying instances

For some reason, App Engine instances misbehave sometimes. This causes the HTTP requests die ungracefully.

Normally it is not a problem as you lost few page loads now and then. People are used to “Internet grade” service and can hit the refresh button if they have problems opening a page.

However if you are monitoring your site and the site gives an unnecessary alarm in the middle of the night, waking up your bastard operator from Hell, he will be very angry next morning and tell you to migrate the crappy software from unreliable Python / App Engine to more reliable PHP servers 🙁

This is what you see in App Engine logs:

A serious problem was encountered with the process that handled this request, causing it to exit.
This is likely to cause a new process to be used for the next request to your application.
If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

After digging in deeper, you see that it is a problem of instating a new object in the database, exceeding 30 seconds hard limit for processing a HTTP request:

2011-03-09 05:06:20.794 / 500 30094ms 86cpu_ms 40api_cpu_ms
0kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
.NET CLR 2.0.50727),gzip(gfe),gzip(gfe),gzip(gfe)

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
  File "/base/data/home/apps/mfabrikkampagne/1.347249742610459821/main.py", line 494, in main
    run_wsgi_app(application)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 97, in run_wsgi_app
    run_bare_wsgi_app(add_wsgi_middleware(application))
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
    result = application(env, _start_response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 515, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/mfabrikkampagne/1.347249742610459821/main.py", line 296, in get
    try: self.session = Session()

So it looks like there is a temporary hick-up in Google App Engine’s Data Store (Big Table?). In the example above the error comes from gaeutilities‘s Session model, but it could be any other model.

It is possible to catch DeadlineExceededError and temporarily work-around it, as shown in App Engine documentation.

The best way to handle this situation is to adjust your monitoring software – Zabbix in our case. Zabbix allows you to configure triggers so that they don’t alarm on every bad item state change. Instead, you can use min() function and trigger the alarm after the trigger condition has failed every time during a monitoring period. Just make sure that the trigger period is at least twice long as the update interval of your web scenario: this way Zabbix can logs at least two item state changes and allows one of them to be failed one.

For example if

  • Update interval of web scenario is 60 seconds
  • Trigger function must check minimal failures of 1 during 2*60 seconds + some buffer = 150 seconds.
{xxx.fi:web.test.fail[de.mfabrik.com].min(150)}=1

This will allow one failed response before triggering the alarm.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Visual Studio and Microsoft go Python

Posted on by
Reply

Microsoft Technical Computing Group has released a beta version for its Python integration for Visual Studio.

This is, indeed, interesting development, as it clearly shows that Python has reached a new level of  programming language maturity.  Receiving this much of attention from mighty Microsoft means that Python is no longer a mere prospect member in the cabin of enterprise solutions.

Python Tools for Visual Studio are not focused only on Microsoft’s own .NET run-time: even Jython and PyPy are partially supported, claims the spec sheet. Looks like some kind of cloud integration is on its way – maybe Microsoft wants to challenge Google App Engine by providing even better cloud development tools?

Also there seems to be more information coming in PyCon…

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

How to render a portlet in Plone

Posted on by
Reply

It’s easy 🙂 It took me only two years to figure this out.

Below is an example how to render a portlet in Plone programmatically. This is useful when you want to have special page layouts and you need to include portlet output from another part of the site.

  • Portlet machinery uses Zope’s adapter pattern extensively. This allows you to override things based on the content context, HTTP request, etc.
  • A portlet is assigned to some context in some portlet manager
  • We can dig these assignments up by portlet assignment id (not user visible) or portlet type (portlet assignment interface)
  • Each portlet has its own overrideable renderer class

This all makes everything flexible, though still not flexible enough for some use cases (blacklisting portlets). The downside is that accessing things through many abstraction layers and plug-in points (adaptions) is little cumbersome.

Here is sample code for digging up a portlet and calling its renderer:

        import Acquisition
        from zope.component import getUtility, getMultiAdapter, queryMultiAdapter
        from plone.portlets.interfaces import IPortletRetriever, IPortletManager, IPortletRenderer

        def get_portlet_manager(column):
            """ Return one of default Plone portlet managers.

            @param column: "plone.leftcolumn" or "plone.rightcolumn"

            @return: plone.portlets.interfaces.IPortletManagerRenderer instance
            """
            manager = getUtility(IPortletManager, name=column)
            return manager

        def render_portlet(context, request, view, manager, interface):
            """ Render a portlet defined in external location.

            .. note ::

                Portlets can be idenfied by id (not user visible)
                or interface (portlet class). This method supports look up
                by interface and will return the first matching portlet with this interface.

            @param context: Content item reference where portlet appear

            @param manager: IPortletManagerRenderer instance

            @param view: Current view or None if not available

            @param interface: Marker interface class we use to identify the portlet. E.g. IFacebookPortlet 

            @return: Rendered portlet HTML as a string, or empty string if portlet not found
            """    

            retriever = getMultiAdapter((context, manager), IPortletRetriever)

            portlets = retriever.getPortlets()

            assignment = None

            for portlet in portlets:

                # portlet is {'category': 'context', 'assignment': , 'name': u'facebook-like-box', 'key': '/isleofback/sisalto/huvit-ja-harrasteet
                # Identify portlet by interface provided by assignment
                if interface.providedBy(portlet["assignment"]):
                    assignment = portlet["assignment"]
                    break

            if assignment is None:
                # Did not find a portlet
                return ""

            #- A special type of content provider, IPortletRenderer, knows how to render each
            #type of portlet. The IPortletRenderer should be a multi-adapter from
            #(context, request, view, portlet manager, data provider).

            renderer = queryMultiAdapter((context, request, view, manager, assignment), IPortletRenderer)

            # Make sure we have working acquisition chain
            renderer = renderer.__of__(context)

            if renderer is None:
                raise RuntimeError("No portlet renderer found for portlet assignment:" + str(assignment))

            renderer.update()
            # Does not check visibility here... force render always
            html = renderer.render()

            return html

This is how you integrate it to your view class:

    def render_slope_info(self):
        """ Render a portlet from another page in-line to this page 

        Does not render other portlets in the same portlet manager.
        """
        context = self.context.aq_inner
        request = self.request
        view = self

        column = "isleofback.app.frontpageportlets"

        # Our custom interface marking a portlet
        from isleofback.app.portlets.slopeinfo import ISlopeInfo

        manager = get_portlet_manager(column)

        html = render_portlet(context, request, view, manager, ISlopeInfo)
        return html

…and this is how you call your view helper method from TAL page template:

        <div tal:replace="structure view/render_slope_info" />

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Lazily load elements becoming visible using jQuery

Posted on by
5

It is a useful trick to lazily load comments or such elements at the bottom of page. Some elements may be loaded only when they are scrolled visible.

  • All users are not interested in the information and do not necessary read the article long enough to see it
  • By lazily loading such elements one can speed up the initial page load time
  • You save bandwidth
  • If you use AJAX for the dynamic elements of the page you can more easily cache your pages in static page cache (Varnish) even if the pages contain personalized bits

For example, Disqus is doing this (see comments in jQuery API documentation).

You can achieve this with in-view plug-in for jQuery.

Below is an example for Plone triggering productappreciation_view loading when our placeholder div tag becomes visible.

...
<head>
  <script type="text/javascript" tal:attributes="src string:${portal_url}/++resource++your.app/in-view.js"></script>
</head>
...
<div id="comment-placefolder">

 <!-- Display spinning AJAX indicator gif until our AJAX call completes -->

 <p>
 <!-- Image is in Products.CMFPlone/skins/plone_images -->
 <img tal:attributes="src string:${context/@@plone_portal_state/portal_url}/spinner.gif" /> Loading comments
 </p>

 <!-- Hidden link to a view URL which will render the view containing the snippet for comments -->                       
 <a rel="nofollow" style="display:none" tal:attributes="href string:${context/absolute_url}/productappreciation_view" />

 <script>

 jq(document).ready(function() {

   // http://remysharp.com/2009/01/26/element-in-view-event-plugin/                                        
   jq("#comment-placeholder").bind("inview", function() {

     // This function is executed when the placeholder becomes visible

     // Extract URL from HTML page
     var commentURL = jq("#comment-placeholder a").attr("href");

     if (commentURL) {
     // Trigger AJAX call
       jq("#comment-placeholder").load(commentURL);
     }

   });                                     

 });     
 </script>
</div>

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+