Monthly Archives: 2012-12

Sevabot – Skype bot with UNIX scripting and HTTP webhooks integration

Posted on by
1

Skype is the most popular IM client with over 200 million monthly users. It’s easy and reliable: anyone can use it anywhere, even on mobile. When you are working with virtual teams, Skype group chats become vital tool of communication.

Skype lacks good built-in support for external service messages, or robots, sending and receiving automated messages. Sevabot, a friendly Skypebot, addresses this situation by allowing you to script Skype easily, and even remotely, over simple HTTP interface and with any UNIX scripting language. You can even send Skype messages to your group chat directly from a web browser Javascript.

Sevabot loves you.

1. Use cases

Sevabot is geared to be an AI support person for a virtual development and operations teams.

  • Receive development status information to Skype chat (continuous integration like Jenkins, Travis, Github and Subversion commits)
  • Get operational alerts like overloaded servers, service down (e.g. Zabbix monitoring alerts)
  • Add your own group chat commands to make Sevabot perform automatized tasks for you (you can write scripts in *any* UNIX supported programming language)

The scope of Sevabot does not need to be limited to software development: You could, e.g., make Sevabot interact with Google Docs or Salesforce.

2. Installation

Sevabot can be run on Linux server, Linux desktop and OSX desktop. Due to how Skype is built you need to run a Skype GUI client inside a virtual X server to deploy Skype on a production server. Vagrant automatized server configuration is supported for automatically creating and deploying a virtual machine running Sevabot.

Sevabot can be run Windows, in theory, but the authors have not had inspiration to explore this cunning option.

3. Programming

Sevabot is written in Python. Scripting Sevabot does not, however, need Python knowledge as

  • Sevabot modules are normal UNIX scripts which you can write in Bash, Python, Perl, Ruby, Javascript, Haskel or whatever is your daily drug
  • External services call Sevabot over HTTP webhooks and any programming language can do HTTP requets

You can even script Sevabot without programming knowledge, as there exist HTTP webhook middleman services like Zapier, which allow you to connect event sources (e..g Github API) and sinks (Sevabot HTTP interface).

In fact Sevabot is a middleman between Flask web server framework (enables HTTP interface) and Skype4Py API (control Skype GUI client with Python).

Here is a simple example how to send a message to a Skype chat from a Bash shell script. It’s a Subversion post-commit hook which displays the commit message in a specific Skype group chat (full example):

#!/bin/bash

repo="$1"

rev="$2"

svnlook="/usr/bin/svnlook"

# Get last commit author
author=`$svnlook author $repo`

# Get last commit message
commit_message=`$svnlook log $repo`

# List of changed files
changed=`$svnlook changed $repo`

# Chat id
chat="YOUR-CHAT-ID-HERE"

# Sevabot endpoint informaiton
# Shared secret
secret="YOUR-SHARED-SECRET-HERE"
msgaddress="http://YOURSERVER.COM:5000/msg/"

# Create chat message
msg="★ $author - $commit_message $changed"

# Sign the message with MD5
md5=`echo -n "$chat$msg$secret" | md5sum`

#md5sum command prints a '-' to the end. Let's get rid of that.
for m in $md5; do
    break
done

# Call Sevabot's HTTP interface
curl $msgaddress --data-urlencode chat="$chat" --data-urlencode msg="$msg" --data-urlencode md5="$m"

exit 0

4. You want it – come for us

Go to Sevabot Github project page for more information. Also check the community information; we have around five active contributors currently.

Feliz Natal e boas festas, Mikko and Sevabot

 

 

 

 

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Configuring your Python application using environment variables

Status

2

This blog post is a short explanation how to give and consume environment variables in your (Python) code. It’s a nice little trick, but if you are not well-versed in UNIX systems you might not know it.

Table Of Content

1. Configuring your application

2. Using environment variables in UNIX

3. Another trick: socket.gethostname()

Please see the original blog post for syntax color examples.

1. Configuring your application

An application can be configured usually in three ways. Through

  • command-line arguments (I prefer simplistic plac library with Python)
  • configuration files (I prefer YAML or Python standard library ConfigParser for INI style files)
  • environment variables

The two first ones are the most well-known methods. However, the last option is the easiest for quick and dirty hacks.

2. Using environment variables in UNIX

Note: Environment variables work alike in UNIX and Windows. However, I have no longer valid experience about Windows and how to set environment variables in the latest cmd.exe incarnations. (I used to write AUTOEXEC.bats for DOS, but that’s like long time ago). So if there are Windows gurus around please leave a comment.

Environment variables are

  • Easy way to make deployment specific changes in your application: run application differently on different computers by starting the app with a different command
  • Pass parameters for your application when you cannot parse command line yourself. E.g. your application is using Django or Zope launcher script and poking the launch script arguments is not easy.
  • Can be consumed everywhere inside your application, whether it is module level code (run on import), inside function, inside class and so on…
  • Needs only two lines to set up (import os ; os.environ.get())
  • Environment variables are especially useful if you want to have, in your scripts, some secret variables (username, password) which must not be committed on a public code repositories like Github (example). Note: the command lines are public for the all users of the UNIX system, so don’t do this on a shared server if you cannot expose the information to other server users.

Whether and how using environment variables is a good practice is debatable. But it definitely saves your butt when you need to make something quick and dirty.

You can give environment variables to your application on UNIX command line by simply prefixing your command with them (bash, sh, etc. shells):

DEMO_MODE=true python myapp.py

Or if you want to make the effect persistent for the current shell session use export (bash style)

export DEMO_MODE=true
python myapp.py
# Again
python myapp.py

Then you can read out for this environment variable easily using os.environ dictionary (pseudo code)

import os

# Just to check for the existing of DEMO_MODE environment variable,
# but you could also compare its value, pass it forward and so on
DEMO_MODE = os.environ.get("DEMO_MODE", None)

# We make some of the class members conditional 
# by given environment vaiables
class MyForm(object):

    name = StringField()

    if not DEMO_MODE:
       secret = PasswordField()

If you are using Python as the configuration language of your application (Django’s settings.py, Pyramid’s Configurator) you can also use this trick there to make some settings conditional which you normally hardcode in Python. Example settings.py:

import os

# Set Django debug mode based on environment variable
DEBUG = "DEBUG_MODE" in os.environ

Then, on the staging server, you would launch your Django application as

DEBUG_MODE=xxx python manage.py runserver

You can also give several environment variables once:

DEBUG_MODE=xxx API_SECRET=yyy python manage.py runserver

3. Another trick: socket.gethostname()

This is a way to bind certain settings to certain computers in your Python code by checking the name of the computer as returned by gethostname() call. This is useful if you don’t want to forget giving a specific launcher command on a specific server.

import socket

# Temporary hack to run hidden fields on a demo server
if socket.gethostname() in ['mikko-laptop', 'demoserver.example.com']:
     import special_config as config 
else:
     import normal_config as config

The only downside is that sometimes the hostname is not stable: I have noticed this behavior on OSX when connecting different networks on my laptop and apparently the name is given by the network.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Timeouting commands in shell scripts

Posted on by
Reply

Often you want to automatize something using shell scripting. In a perfect world your script robot works for you without getting tired, without hick-ups, and you can just sit at the front of your desk and sip coffee.

Then we enter the real world: Your network is disconnected. DNS goes downs. Your HTTP hooks and downloads stall. Interprocess communication hangs. Effectively this means that even if your script is running correctly from the point of operating system it won’t finish its work before you finish your cup of coffee.

Below is an example how to create timeouts and notifications in a shell script.

1. Never gonna give you up

(c) Rick Astley

Automation must work 100% and you must always know if it doesn’t do that. Otherwise if you cannot trust the automated systems you could have this glorious moment of “oh it has been broken for three months now”. No amount of coffee makes your day after that. Then you spend your nights pondering whether your scripting is running instead of playing Borderlands 2.

Some safety guards around your coffee cup include

  • Sane timeout thresholds for commands to avoid hang situations. E.g. your automated “make smallapp” should probably not run for 50 hours straight.
  • Get a notification if a fully automatized process does not end up as expected. So that you find out the problem right away, not three months later. Do not use email as the communication channel, it is unreliable and has awful signal-to-noise ratio. Better solutions include instant messengers (Skype), SMS (check twilio.com)
  • Get a notification also when a service, which should be running all the time, is restarted.

2. Timeout.bash

The shell scripting (sh, zsh, bash)  does not offer built-in tools for timeouting commands by default (please correct me if I am wrong, but stackoverflow.com et. al folks did not know any better). In Bash cookbook, there exist a timeout wrapper script example (The orignal SO.com answer).

The trick is that when the timeout wrapper terminates the command, you’ll get the exit code of SIGTERM (143) or SIGKILL (137) to the parent script. This might not be entirely clear from reading the script.

3. Using timeout wrapper and sending failure notifications to Skype

Below is an example how you can hook timeout to your own script. Here we have a simple continous integration script (ghetto-ci) which polls version control repositories and on any change executes the test suite. A test failure is reported back to Skype by ghetto-ci using Sevabot Skype bot. The loop script uses a server specific (Ubuntu) way to set up a headless X server, so that the tests can run Firefox on the server (can one call a headless Firefox server “mittens”?) The continous integration loop is deployed simply as leaving it running on the screen‘ed terminal on a server.

Some protection we do: The script has extra checks to protect against hung processes by killing them using pkill before starting each test run. Selenium’s WebDriver seem to often cause situations where the Python tests don’t quit cleanly, leaving around all kind of zombie processes. In this case, the test command timeouts using the timeout wrapper and we get a notification to Skype. Based on this we can refine the test running logic, timeout delay and such to make the script more robust allowing us to focus more on drinking the coffee and less watching a looping process running in a UNIX screen for potential failures.

Pardon me for possibly ugly shell scripting code. SH is not my primary language. The example code is also on Github. Please see the orignal blog post for syntax colored example.

#!/bin/bash
#
# Run CI check for every 5 minutes and
# execute tests if svn has been updated.
#
# The script sets up xvfb (X window framebuffer)
# which is used to run the headless Firefox.
#
# We will post a Skype message if the Selenium WebDriver
# has some issues (it often hangs)
#
# We also signal the tests to use a special static Firefox build
# and do not rely (auto-updated) system Firefox
#
# NOTE: This script is NOT sh compliant (echo -n),
# bash needed

# Timeouting commands shell script helper
# http://stackoverflow.com/a/687994/315168
TIMEOUT=timeout.sh

# Which FF binary we use to run the tests
FIXED_FIREFOX=$HOME/ff16/firefox/firefox

# It's me, Mariooo!
SELF=$(readlink -f "$0")

# Skype endpoint information
SKYPE_CHAT_ID="1234567890"

SKYPE_SHARED_SECRET="toholampi"

SEVABOT_SERVER="http://yourserver.com:5000/msg/"

#
#  Helper function to send Skype messages from help scripts.
#  The messages are signed with a shared seceret.
#
#  Parameter 1 is the message
#
function send_skype_message() {
    msg="$1"
    md5=`echo -n "$SKYPE_CHAT_ID$msg$SKYPE_SHARED_SECRET" | md5sum`

    #md5sum pads a '-' to the end of the string. We need to get rid of that.
    for m in $md5; do
        break
    done

    result=`curl --silent --data-urlencode chat="$SKYPE_CHAT_ID" --data-urlencode msg="$msg" --data-urlencode md5="$m" $SEVABOT_SERVER`
    if [ "$result" != "OK" ] ; then
        echo "Error in HTTP communicating to Sevabot: $result"
    fi
}

# Tell the tests to use downgraded FF!6
# which actually works with Selenium
if [ -e $FIXED_FIREFOX ] ; then
    FIREFOX_PATH=$FIREFOX_PATH
    export FIREFOX_PATH
    echo "Using static Firefox 16 build to run the tests"
fi

send_skype_message "♫ ci-loop.sh restarted at $SELF"

while true
do
    # Kill hung testing processes (it might happen)
    pkill -f "bin/test"

    # Purge existing xvfb just in case
    pkill Xvfb

    sleep 5

    echo "Opening virtual X11"

    # Start headless X
    Xvfb :15 -ac -screen 0 1024x768x24 &

    # Tell FF to use this X server
    export DISPLAY=localhost:15.0

    # Run one cycle of continous integration,
    # give it 15 minutes to finish
    echo "Starting test run"
    $TIMEOUT -t 900 continous-integration.sh
    result=$?

    if [ "$result" == "143" ] ; then
        echo "------- CI TIMEOUT OOPS --------"
        send_skype_message "⚡ Continuous integration tests timed out - check the ci-loop.sh screen for problems"
    fi

    sleep 300
done

Voila. Back to the coffee.

 

 

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

psutil – advanced OS process management utilities for Python

Posted on by
Reply

psutil is a Python library and API for UNIX system administration utilities. It provides Pythonic API e.g. for top, lsof, netstat and kill like tasks.

The benefits of using this library include

  • Because psutil API resembles UNIX process management command line utilities the developers find themselves home
  • Pythonic, easy to use, API
  • Cross-platform
  • Good documentation

psutil is maintained by Giampaolo Rodola and Jay Loden.

Below is a short example what you can do with psutil. It checks if the current operating system is running Apache (web server) process and this process is listening to particular TCP/IP ports. The use case could be e.g. per-requirements check that the software can be deployed against a particular system.

import psutil

def is_apache_running_in_ports(process_name="apache", ports=(80, 443)):
    """
    Check if a local Apache instance is running and listening to certain ports.

    :param ports: List of ports Apache should be listening to (all ports must be included)
    """

    ports_to_go = list(ports)

    # Iterate over all system processes (ps)
    for proc in psutil.process_iter():
        if proc.name != process_name:
            # Not target process
            continue

        # Iterate over all ports this process is listening to 
        for con in proc.get_connections():
            # Tuple ip, port
            port = con.local_address[1]
            if port in ports_to_go:
                ports_to_go.remove(port)

    # Did we find all ports we wanted to
    return len(ports_to_go) == 0
Note: When installing psutil be sure that you follow the Python best practices: sudo-free virtualenv installation.

More examples at psutil homepage.

 

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+