Autodiscovering log files for logrotate

Logrotate is the UNIX application responsible for removing old (website) log file entries and preventing your server disk filling up. Logrotate is configured by writing config files where you specify one entry per file/file collection.

When you install applications like Apache or Nginx from your operating system package manager they usually come with a preconfigured logrotate entries. However, when you write your own custom software you need to take care of logrotation set up yourself.

If you are hosting a several website processes on the same server, e.g. Plone CMS sites, writing the log rotate entries for all log files manually can be pain. Below is a simple Python script which will auto discover all log files under a certain folder and its subfolders and generate logrotate entries for them.

The script is easy to adopt for other systems.

With Plone, since Plone uses buildout based installation process, you could alternatively use buildout configuration entries to maintain the log rotation for Plone. However, some sysadmins may prefer the system-wide logrotate configuration.

The script is hosted on Github for internal maintenance.

See also the related superuser.com question.

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""

    Auto-discover log files and generate logrorate config based on it.

    - Find any folders containing log files

    - Add logrotate entry for *.log files in that folder

    - Try to signal Zope application server after log rotation to release the file
      handle and actually release the disk space associated with it

    http://collective-docs.readthedocs.org/en/latest/hosting/zope.html#log-rotate

    http://man.cx/logrotate

    Usage example for Ubuntu / Debian::

        sudo -i -u  # root
        # Generate a log rotate config which is automatically
        # picked up by a log rotate on the next run
        discover-log-rotate /etc/logrotate.d/plone-all /srv/plone

    Confirm that it works by running logrotate manualy.

    Dry run, outputs a lot but doesn't touch the files::

        logrotate -f -d /etc/logrotate.conf

    Test run, rotates the files::

        logrotate -f /etc/logrotate.conf

    Also see that your processes can re-open log files after logrotate run
    by restating a sample process.

"""

__license__ = "Public domain"
__author__ = "Mikko Ohtamaa <http://opensourcehacker.com>"

import sys

import fnmatch
import os

#: Logroate config template applied for each folder.
#: Modify for your own needs.
TEMPLATE = """
%(folder)s/*.log {
        weekly
        missingok
        # How many days to keep logs
        # In our cases 3 months
        rotate 12
        compress
        delaycompress
        notifempty

        # THE FOLLOWING IS ZOPE SPECIFIC BEHAVIOR

        # This signal will tell Zope to reopen the file-system inode for the log file
        # so it doesn't keep reserving the old log file handle for even if the file is deleted
        # We guess some possible process and PID file names.
        # The process here is little wasfeful, but we don't need to try match a log file to a running process name.
        # TODO: Ideas how to get a process name from the log file name?
        postrotate
            [ ! -f %(installation)s/var/instance.pid ] || kill -USR2 `cat %(installation)s/var/instance.pid`
            [ ! -f %(installation)s/var/client1.pid ] || kill -USR2 `cat %(installation)s/var/client1.pid`
            [ ! -f %(installation)s/var/client2.pid ] || kill -USR2 `cat %(installation)s/var/client2.pid`
            [ ! -f %(installation)s/var/client3.pid ] || kill -USR2 `cat %(installation)s/var/client3.pid`
            [ ! -f %(installation)s/var/client4.pid ] || kill -USR2 `cat %(installation)s/var/client4.pid`
        endscript
}

"""

def run():
    """
    Execute the script.
    """
    if len(sys.argv) < 2:
        sys.exit("Usage: discover-log-rotate [generated config file] [directory]")

    config = sys.argv[1]
    folder = sys.argv[2]

    out = open(config, "wt")

    print "Running log rotate discovery on: %s" % folder

    # Find any folders containing log files
    matches = []
    for root, dirnames, filenames in os.walk(folder):
        for filename in fnmatch.filter(filenames, '*.log'):

            full = os.path.join(root, filename)
            dirpath = os.path.dirname(full)

            if not dirpath in matches:
                print "Adding log rotate entry for: %s" % dirpath
                matches.append(dirpath)

    for match in matches:

        folder = os.path.abspath(match)

        # Assume the main Zope installation folder is format /srv/plone/xxx
        # and the underlying log folder /srv/plone/xxx/var/log
        installation = os.path.abspath(os.path.join(folder, "../.."))
        cooked = TEMPLATE % dict(folder=match, installation=installation)
        out.write(cooked)

    out.close()

    print "Wrote %s" % config

if __name__ == "__main__":
    run()

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Responsive design workflow using two browsers

I am building a new mobilization add-on for Plone CMS. I’d like to share the following workflow which I found greatly to speed up CSS tuning tasks for responsive design

This way you have CSS debugger for desktop and mobile versions open simultaneously and you still use your screen real estate effectively.

A mobile version and Chrome’s CSS debugger on the top of desktop Firefox.

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Accessing priviledged Javascript APIs from your web page in Firefox with Selenium

Firefox is internally built on Javascript components using native C++ bindings. These components are normally exposed only for add-ons. The functionality includes high level stuff like bookmark management and low level functionality like direct disk access.

You might have special situations when you want to directly access these APIs from a  web page you control. One such situation is automated testing where your Javascript test framework might, for the performance reasons, write some test data directly to disk or socket for efficient communication with your host testing process. Another use case is when you are building a Very Special Web Applications and you need to give the admin users some extra special functionality which would be cumbersome to develop using traditional HTML / HTTP platforms.

Note: Enabling XPCOM access globally in your everyday browser is not something you should do due to security implications.

Here is a quick example how you can enable privileged APIs in Selenium Test Driver in Python:

source_uri = "ADDRESS_TO_YOUR_WEB_PAGE_ON_TEST_SERVER"

profile = webdriver.firefox.firefox_profile.FirefoxProfile()

parsed = urlparse.urlparse(source_uri)
host_base = "%s://%s" % (parsed.scheme, parsed.netloc)

print "Enabling XPCOM for codebase", host_base

set_pref = profile.set_preference
set_pref("signed.applets.codebase_principal_support", True)
set_pref("capability.principal.codebase.p0.granted", "UniversalXPConnect");
set_pref("capability.principal.codebase.p0.id", host_base);
set_pref("capability.principal.codebase.p0.subjectName", "");

# This opens a new Firefox browser where the settings
# have been modified to allow XPCom access from URL
# you specified above
self.browser = webdriver.Firefox(firefox_profile=profile)
self.browser.get(source_uri) # Load page

Then you can whitelist Javascript functions for XPCOM access:

/** Decorate the target function to have special XP priviledges.
 * You need to do this for each individual Javascript function
 */
function UseXPCOM(f) {
    return function() {
        netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
        return f.apply(this, arguments);
    };
};

And here is an example how you now can have raw disk access for writing files directly from Javascript:

       /** Open file for binary writing access */
       openFileForBinaryWrite : UseXPCOM(function(filename) {
            this.outputFilename = filename;
            this.outputFile = Components.classes["@mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
            this.outputFile.initWithPath(filename);
            var stream = Components.classes["@mozilla.org/network/file-output-stream;1"].createInstance(Components.interfaces.nsIFileOutputStream);

            // PR_WRONLY + PR_CREATE_FILE + PR_APPEND, default mode, no behaviour flags
            stream.init(this.outputFile, 0x02 | 0x10, 00600, false);
            this.stream = Components.classes["@mozilla.org/binaryoutputstream;1"].createInstance(Components.interfaces.nsIBinaryOutputStream);

            this.stream.setOutputStream(stream);
        }),

And now you can write there:

this.stream.writeBytes("my bytes");

More about XPCOM APIs.

See the orignal post for syntax highlighted examples.

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Making the world to love Plone again

This is my happy rant about the sad state of Plone adoption and bootstrapping your Plone the development project. Again – you may have heard this before. But this time I draw pretty pictures 🙂

Each new Plone release has brought less happiness to the developers. We are not in free fall anymore, but the happiness curve remains flat (source: rants of Plone IRC channel)

Developers are the consumers of Plone the project. The effective result on the first diagram is that it is very hard to consider Plone as a platform for your new project unless you are “in the Plonealready. There is lack of “start building my site on Plone from the scratch” option.

Eventually this will marginalize Plone. People prefer to work on easier tools, tools they can get started easily even those tools would not be the best fit for the solving the problem (PHP, anyone?) When I learnt Plone I was still in a school. Can a student anymore learn Plone? (source: mailing list questions and on-line discussion)

This will eventually lead to a situation that we all die to the old age, alone with our Plone sites. That would be sad (source: Plone Conference picture galleries)

I don’t believe this is a result of any person doing things wrong. It is just a general oversight, the tradegy of commons. People are too excited to work on the fresh things and forget those who are still learning to drink our champagne (choosing a bottle is difficult if they lack labels). There hasn’t been focus to make Plone easier to adopt. This is slowly changing with plone.api and things. However, based on e.g. the sprint topic list of last Plone conference the message is going through slow. (source: past sprints until summer 2012)

What we can do?

  • On the Plone roadmap and Plone conference keynote I want to see “Plone easier for developers”. Looks like it has not made appearance on any keynote or roadmap for a while. The highlight have been features, features and features. Let’s make this year different.
  • For every Plone release, let’s require that there is at least 1-2 PLIPs which actually state “this is how Plone will become easier for developers”
  • How we can gently guide community members to take the tasks which make adopting Plone easier? We need to staff the tasks with people who actually know things. People how need to dig up things by turning stones (code) upside down don’t tend to work there very long.
  • Every time there is a new feature in Plone the feature sponsor should think: “How, with this new cool feature, I can make people more happy when they start using it?” I don’t want to see any feature which after the landing will lead to a situation where people come to ask to IRC “How this works” and I cannot point them to any up-to-date functional manual about the things. Lack of on-line tutorial = the feature is effective not functional. It makes me feel so sad: “How did this happen, again?”
  • How can we reduce the steps the developer needs to take to get his Hello world running? Can we make a tutorial which fits into a single screenful of text, because your dev tools would be provided out of the box? The first impression counts the most.
  • plone.org needs love too. Documentation team no longer effectively exists. Let’s give candy for those who volunteer.

How can we make these things happen? It seems everyone accepts this is the state of the things, but looks like there is no remedy coming or it is happening slowly

  • Should we set core repositories to read-only until the documentation can catch up the things?
  • Should we lock core developers and their friends to a room full of beer with disabled PyPi access, but give functional plone.org edit access?
  • Let’s take some community members to a river boat ride on Arnheim and don’t let them land until they post out a solid story of functional ZopeSkel, unified installer and Hello World tutorial?
  • Make sure that each Plone Conference sprint topics proposals has a bullet “this is how we actually make Plone adoption easier, not more difficult”

Let’s meet at Plone Conference (now with less offensive talk topic)

Cheers,

 

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+