Monthly Archives: 2011-11

Creating a viewlet manager in Plone

Posted on by
1

This blog post is a part of recent Plone developer manual update (the free to edit version). It tries to explain how to create viewlet managers in Plone. A viewlet manager is core part of Plone page rendering mechanism.

Sadly, viewlet managers lacked enough documented how they should be used or what kind of features they actual have. Also, I think this is an examplatory case of one of the worse parts of Plone process: some code gets pushed into the mainstream without proper documentation and then people need to spend three years to figure out how it works. I hope I think I finally got it how it works after reading the code. My hope is that in the future we could get less packages like this and if there is something frameworkish then please think your fellow developers and all the wasted work hours put into to the attempt trying to figure out how they work as it could have been solved with few lines of proper examples written in five minutes. I know you can do it.

Not to mention than viewlet manager mechanism is one of the spoiled parts of Plone architecture as you need to change and override and edit excessive amount of stuff just to change a bit of HTML. So I hope this helps you to get it done.

Viewlet managers contains viewlets. Viewlet manager itself is a Zope 3 interface which contains an OrdereredViewletManager implementation. OrderedViewletManager handles saving the order of the viewlets in the site database and provides the fancy /@@manage-viewlets output.

Viewlet manage can be rendered in a page template code using the following expression:

<div tal:replace="structrure provider:viewletmanagerid" />

Each viewlet manager allows you to shuffle viewlets inside a viewlet manager. This is done by using /@@manage-viewlets view. These settings are stored in the site database, so a good practice is to export viewlets.xml using portal_setup and then include the necessary bits of this viewlets.xml with your add-on installer so that when your add-on is installed, the viewletconfiguration is changed accordingly.

Note: You cannot move viewlets between viewlet managers. I know it sucks, but life is hard and Plone is harder. Hide viewlets in one manager using /@@manage-viewlets and viewlets.xml export, then re-register the same viewlet to a new manager.

Viewlet managers are based on zope.viewlet.manager.ViewletManager and plone.app.viewletmanager.manager.OrderedViewletManager.

More info

Creating a viewlet manager: Grok way

Recommended if you want to keep the number of files and lines of XML and Python minimum.

An example here for related Python code:

* http://code.google.com/p/plonegomobile/source/browse/gomobiletheme.basic/trunk/gomobiletheme/basic/viewlets.py#80

Creating a viewlet manager: ZCML way

For those who want to write XML.

Usually viewlet managers are dummy interfaces and the actual implementation comes from plone.app.viewletmanager.manager.OrderedViewletManager.

In this example we put two viewlets in a new viewlet manager so that we can properly CSS float then and close this float.

Note: This example uses extensive Python module nesting: plonetheme.yourtheme.browser.viewlets is just too deep. You really don’t need to do some many levels, but the orignal plone3_theme paster templates do it in bad way.  One of Python golden rules is that flat is better than nested. You can just dump everything to the root of your plonetheme.yourtheme package.

In your browser/viewlets/manager.py or similar file add:

<browser:viewletManager
 name="plonetheme.yourtheme.headerbottommanager"
 provides="plonetheme.yourtheme.browser.viewlets.manager.IHeaderBottomViewletManager"
 class="plone.app.viewletmanager.manager.OrderedViewletManager"
 layer="plonetheme.yourtheme.browser.interfaces.IThemeSpecific"
 permission="zope2.View"
 template="headerbottomviewletmanager.pt"
 />

Then in browser/viewlets/configure.zcml:

<browser:viewletManager
 name="plonetheme.yourock.browser.viewlets.MyViewletManager"
 provides=".viewlets.MyViewletManager"
 class="plone.app.viewletmanager.manager.OrderedViewletManager"
 layer="plonetheme.yourock.interfaces.IThemeLayer"
 permission="zope2.View"
 />

Optionally you can include a template which renders some wrapping HTML around viewlets. browser/viewlets/headerbottomviewletmanager.pt:

<div id="header-bottom">
    <tal:comment replace="nothing">
        <!-- Rendeder all viewlets inside this manager.

             Pull viewlets out of the manager and render then one-by-one
         -->
    </tal:comment>

    <tal:viewlets repeat="viewlet view/viewlets">
           <tal:viewlet replace="structure python:viewlet.render()" />
    </tal:viewlets>

    <div style="clear:both"><!-- --></div>
</div>

And then re-register some stock viewlets against your new viewlet manager in browser/viewlets/configure.zcml:

 <!-- Re-register two stock viewlets to the new manager -->

 <browser:viewlet
   name="plone.path_bar"
   for="*"
   manager="plonetheme.yourtheme.browser.viewlets.manager.IHeaderBottomViewletManager"
   layer="plonetheme.yourtheme.browser.interfaces.IThemeSpecific"
   class="plone.app.layout.viewlets.common.PathBarViewlet"
   permission="zope2.View"
   />

<!-- This is a customization for rendering the a bit different language selector -->
<browser:viewlet
   name="plone.app.i18n.locales.languageselector"
   for="*"
   manager="plonetheme.yourtheme.browser.viewlets.manager.IHeaderBottomViewletManager"
   layer="plonetheme.yourtheme.browser.interfaces.IThemeSpecific"
   class=".selector.LanguageSelector"
   permission="zope2.View"
  />

Now, we need to render our viewlet manager somehow. One place to do it is in a main_template.pt, but because we need to add this HTML output to a header section which is produced by another viewlet manager, we need to create a new viewlet just for rendering our viewlet manager. Yo dawg – we put viewlets in your viewlets so you can render viewlets!

browser/viewlets/headerbottom.pt:

<tal:comment replace="nothing">
    <!-- Render our precious viewlet manager -->
</tal:comment>
<tal:render-manager replace="structure provider:plonetheme.yourtheme.headerbottommanager" />

Only six files needed to change a bit of HTML code – welcome to the land of productivity! On the top of this you also need to create a new viewlets.xml export for your theme.

After all this ZCML typing you probably should just look the grok example above.

More info

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Migrating from BIND DNS servers to Amazon Route 53 by using cli53

Posted on by
4

Amazon Route 53 offers DNS as a hosted service. They maintain robust DNS service for you with easy to user web interface and API.

Name servers are part of any serious hosting infrastructure. They have been traditionally run on BIND or similar UNIX daemons. The usual hosting tasks including mapping domain names and subdomains (A, CNAME, et. al records) to servers and delivering email (MX and SPF records).

Usually a basic hosting plan is only good for setting A records for few subdomains like www, but on the instant you need something more advanced like routing MX records to Google Apps email you need to start considering a real DNS service. GoDaddy and other big name registrants provide tools for this, but they are usually specific only for top level domains (TLDs) the registrant supports. If you come from a cold country like .fi you cannot centralize your DNS management to big popular domain registrants.

The bad thing with running your own name servers is that when the service goes down the whole your infrastructure becomes virtually inaccessible – web sites, email, anything. Thus you usually run at least two BIND daemons on two physically separated servers just in case one blows up the other can still keep DNS records running. Also DNS, being core part of internet infrastructure, is subject for many kind of attacks and keeping your skills and knowledge up-to-date with BIND may be time consuming.

Like with email servers, hosting name servers is a thing you really don’t want to do. Enter Amazon Route 53. (tha name Route 53 probably comes from the famous U.S. road  Route 67 and DNS port 53?).

Route 53 is ridiculous easy to use. Just go to the site and register one for yourself. However if you have legacy BIND servers running around there exist ways to import data automatically instead of manually reading through zone files and typing them in to Route 53 by hand.

1. Setting up cli53

cli53 is a Python based command line interface for Route 53. It uses buildout for installation (of Python package dependencies).

These instructions have been tested with Ubuntu 8.04 server. Prerequisitements for using this stuff is knowing basic Ubuntu server management from the command line.

Run everything as root.

Install wget and unzip

apt-get install wget unzip

Download ZIP copy of cli53 from Github (we don’t do git clone here – no need to install git on the server):

wget --no-check-certificate https://github.com/barnybug/cli53/zipball/master

Extract

unzip master

Install as user local under the extracted folder (note that the folder varies across GitHub exports)

cd barnybug-cli53-3b468b7
python bootstrap.py
bin/buildout

This will generate the local bin/cli53 command.

Now you need to set up AWS credentials. Route 53, and other AWS service, are operated using access keys which you can get from Security Credentials page in AWS web console. Set the access keys for your shell/SSH session using enviroment variables:

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Side note: as this point you need to check that your server clock is in correct time your AWS remote operations will fail (300 seconds thresold).. Use date command to see the server clock time and correct it with the same date command using this obscure date command syntax.

Now you can run a test by listing your Amazon Route 53 Hosted Zones (Hosted Zone = zonefile = one per top level domain usually) using list command:

bin/cli53 list

The proper output for one domain which has been created throught-the-web in Route 53 control panel is something like:

HostedZones:
  - CallerReference: 9265CCC3-9C41-98CE-8820-E26C0356C478
    Config:
      Comment: test
    Id: /hostedzone/Z2L3NT0WCS8OWA
    Name: xxx.fi.

2. Importing zone files

Now when cli53 is running and working it is time to rock’n’roll.

On our server configuration BIND zonefiles lived in /srv/dns/zones though this is not the default location for Ubuntu BIND. There is one file per each zone and the filename is the domain name.

[root@foobar][23:02][/srv/dns/zones]# ls -l
total 176
-rw-r--r-- 1 root root  425 Dec 30  2008 abc.com
-rw-r--r-- 1 root root 1713 Dec 30  2008 abcabc.com
-rw-r--r-- 1 root root 1719 Dec 30  2008 abcabcabc.com

We can simple create an one-liner shell script which will loop through all the files and import them to Amazon Route 53. However – there is a trick. You’ll get unknown origin exception if your zone files lack a certain line:

Traceback (most recent call last):
  File "/root/barnybug-cli53-3b468b7/bin/cli53", line 22, in <module>
    cli53.cli53.main()
  File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 495, in main
    args.func(args)
  File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 268, in cmd_import
    raise Exception, 'Could not find origin'
Exception: Could not find origin

This is because the zone file doesn’t have $ORIGIN directive. it’s optional with BIND as BIND uses the filename as $ORIGIN if it’s not defined, but the directive is mandatory  for cli53 (cli53 doesn’t yet synthetize $ORIGIN).

You can simple add $ORIGIN to the beginning of each file and copy files to a working directory with the following shell one-liner (it’s always safe to make copies of processed files than try to fix them in place):

mkdir ../transfer
for i in * ; do echo "\$ORIGIN $i." > ../transfer/$i ; cat $i >> ../transfer/$i ; done

Then do a test run for one import:

~/barnybug-cli53-3b468b7/bin/cli53 create xxx.fi
~/barnybug-cli53-3b468b7/bin/cli53 import xxx.fi --file xxx.fi --replace --wait

Note: deleting zones in Route 53 control panel is painful difficult, so make sure you import zones you only really need.

If everything looks good we are ok to upload everything to Route 53. Again, a shell one-liner does the trick for us:

for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 create $i ; ~/barnybug-cli53-3b468b7/bin/cli53 import $i --file $i --wait ; done

This will take abour 15-20 seconds per domain.

Note: if you need to re-import add –replace flag.

3. The ugly part

Each zone has Source of Authority record with authoritative nameserves (NS records). These will be change from your BIND server IP addresses to Amazon ones. Route 53 will re-assign its own name servers for each imported Hosted Zone. However, you cannot know beforehand what name servers the dice has chosen for your Hosted Zone, so automatizing this process is little bit difficult. NS records servers are overriden when the zone file is uploaded to Route 53.

You can see new SOA and NS records when you choose the domain in Route 53 control panel and press Go to Record Sets.

To make things easier later on here is a script which will dump all the name servers of all domains you have in Route 53 and collect are info to domain-info.txt file:

for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 info $i >> /tmp/domain-info.txt ; done

4. Testing Route 53 DNS

Browser through imported domain records in Route 53 control panel and see if they look ok.

You can also test the actual DNS functionality. When you use ping command, your web browser or any other “normal” mean which queries DNS records they use the Source of Authority name servers as set by in your domain registrant settings. When in the middle of migration to Route 53, the authorative name servers are still pointing to the old name servers in this point.

dig command can be used to query a specific name server for DNS records.

  • In Route 53 control panel pick any domain and any of its name servers
  • Use dig command to query the records of this domain

Example:

dig -t ANY @ns-1340.awsdns-39.org yourdomainname.com

Output should be something like this:

; <<>> DiG 9.4.2-P1 <<>> -t ANY @ns-1340.awsdns-39.org yourdomain.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9352
;; flags: qr aa rd; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;yourdomain.com.            IN    ANY

;; ANSWER SECTION:
yourdomain.com.        172800    IN    NS    ns-470.awsdns-58.com.
yourdomain.com.        172800    IN    NS    ns-553.awsdns-05.net.
yourdomain.com.        172800    IN    NS    ns-1340.awsdns-39.org.
yourdomain.com.        172800    IN    NS    ns-1706.awsdns-21.co.uk.
yourdomain.com.        900    IN    SOA    ns-1340.awsdns-39.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
yourdomain.com.        0    IN    MX    10 aspmx.l.google.com.
yourdomain.com.        0    IN    MX    20 alt1.aspmx.l.google.com.
yourdomain.com.        0    IN    MX    20 alt2.aspmx.l.google.com.
yourdomain.com.        0    IN    TXT    "v=spf1 a mx a:smtp.something.fi a:auth-smtp.something.fi include:aspmx.googlemail.com ~all"
yourdomain.com.        0    IN    A    84.20.128.49

;; Query time: 32 msec
;; SERVER: 205.251.197.60#53(205.251.197.60)
;; WHEN: Wed Nov 23 00:04:53 2011
;; MSG SIZE  rcvd: 413

(subdomains are not listed)

5. Finalizing it

Go to your domain name registrant and update the domain name servers point to your Route 53 servers.

  • For each domain in domain-info.txt
    • Go to the corresponding registrant from whom you obtained the domain (e.g. domain.ficora.fi)
    • Update the authoritative name servers to be as stated in the file
  • Wait 24-48 hours or whatever time-to-live time you have for your name servers – usually this is longish period
  • See that your services still run
  • Take down your old name servers

6. More info

 

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Varnish, caching and HTTP cookies

Posted on by
11

These short notes related on caching and HTTP cookies are based on my experience with Varnish and Plone CMS and WordPress.

Sanifying cookies for caching

Any cookie set on the server side (session cookie) or on the client-side (e.g. Google Analytics Javascript cookies) is poison for caching the anonymous visitor content.

Common cookies for all CMS systems are usually

  • Session cookie (anonymous user session): ZopeId,  PHPSESSID
  • Logged in user cookie: __ac (Plone)
  • Active language cookie: I18N_LANGUAGE (Plone)
  • Analytics cookies (Google Analytics et. al.): various ugly cookies
  • Some other status information e.g. status message: statusmessages (Plone)

HTTP caching needs to deal with both HTTP request and response cookie handling

  • HTTP request Cookie header. The browser sending HTTP request with Cookie header confuses Varnish cache look-up. This header can be set by Javascript also, not just by the server. Cookie can be preprocessed in vcl_recv.
  • HTTP response Set-Cookie header. This is server-side cookie set. If your server is setting cookies Varnish does not cache these responses by default. Howerver, this might be desirable behavior if e.g. multi-lingual content is served from one URL with language cookies. Set-Cookie can be post-processed in vcl_fetch.

Example how remove all Plone related cookies besides ones dealing with the logged in users (content authors):

sub vcl_recv {

  if (req.http.Cookie) {
      # (logged in user, status message - NO session storage or language cookie)
      set req.http.Cookie = ";" req.http.Cookie;
      set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
      set req.http.Cookie = regsuball(req.http.Cookie, ";(statusmessages|__ac)=", "; \1=");
      set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
      set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

      if (req.http.Cookie == "") {
          remove req.http.Cookie;
      }
  }
  ...

# Let's not remove Set-Cookie header in VCL fetch
sub vcl_fetch {

    # Here we could unset cookies explicitly,
    # but we assume plone.app.caching extension does it jobs
    # and no extra cookies fall through for HTTP responses we'd like to cache
    # (like images)

    if (!obj.cacheable) {
        return (pass);
    }
    if (obj.http.Set-Cookie) {
        return (pass);
    }
    set obj.prefetch =  -30s;
    return (deliver);
}

Another example how to purge Google Analytics cookies only and leave other cookies intact:

sub vcl_recv {

         # Remove Google Analytics cookies - will prevent caching of anon content
         # when using GA Javascript. Also you will lose the information of
         # time spend on the site etc..
         if (req.http.cookie) {
            set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
            if (req.http.cookie ~ "^ *$") {
                remove req.http.cookie;
            }
          }

          ....

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

/join #javascript.fi @ IRCNet

Posted on by
Reply

If you like or hate Javascript programming and want to share your thoughts in Finnish in the oldest social network of the history then type the following command in your IRC client:

/join #javascript.fi

(on IRCNet network).

We welcome all the followers of Brendan and Douglas to /join our ranks.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+