Inspecting thread dumps of hung Python processes and test runs

Sometimes, moderately complex Python applications with several threads tend to hang on exit. The application refuses to quit and just idles there waiting for something. Often this is because if any of the Python threads are alive when the process tries to exit it will wait any alive thread to terminate, unless Thread.daemon is set to true.

In the past, it use to be little painful to figure out which thread and function causes the application to hang, but no longer! Since Python 3.3 CPython interpreter comes with a faulthandler module. faulthandler is a mechanism to tell the Python interpreter to dump the stack trace of every thread upon receiving an external UNIX signal.

Here is an example how to figure out why the unit test run, executed with pytest, does not exit cleanly. All tests finish, but the test suite refuses to quit.

First we run the tests and set a special environment variable PYTHONFAULTHANDLER telling CPython interpreter to activate the fault handler. This environment variable works regardless how your Python application is started (you run python command, you run a script directly, etc.)


And then the test suite has finished, printing out the last dot… but nothing happens despite our ferocious sipping of coffee.


How to proceed:

Press CTRL-Z to suspend the current active process in UNIX shell.

Use the following command to send SIGABRT signal to the suspended process.

kill -SIGABRT %1

Voilá – you get the traceback. In this case, it instantly tells SQLAlchemy is waiting for something and most likely the database has deadlocked due to open conflicting transactions.

Fatal Python error: Aborted

Thread 0x0000000103538000 (most recent call first):
  File "/opt/local/Library/Fra%                                                                                                                                                                     meworks/Python.framework/Versions/3.4/lib/python3.4/", line 154 in _eintr_retry
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/", line 236 in serve_forever
  File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/", line 40 in run
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/", line 921 in _bootstrap_inner
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/", line 889 in _bootstrap

Current thread 0x00007fff75128310 (most recent call first):
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/engine/", line 442 in do_execute
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/sql/", line 3638 in drop_all
  File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/", line 124 in teardown
  File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/_pytest/", line 41 in main
  File "/Users/mikko/code/trees/venv/bin/py.test", line 9 in <module>

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

Almost free Netflix VPN on Amazon EC2 set up in 30 minutes using Ansible

This is a hacker/developer/sysadmin oriented guide to set up on demand VPN on Amazon EC2 server. Amazon EC2 t2.micro instance costs 0.013 USD per hour, so the deal is very light for your purse, beating commercial VPN pricing by an order of magnitude. Further, Amazon provides 750 hours of free of t2.micro instance usage for new users.

We automatize the cumbersome server configuration tasks using Ansible automatization tool. With Ansible, you’ll get your own VPN service up’n’running in 30 minutes. The instructions work on OSX and Linux systems and Ansible won’t run on Windows.


We set up PPTP VPN. The VPN is good for watching Netflix where US region enjoys wider availability of entertainment. The VPN is also good defeating other US geofencing services and protecting your privacy on open Wi-Fi hotspots. Android, iOS (iPhone / iPad), OSX, Linux and Windows all support PPTP protocol out of the box, so no additional software is needed on your device.


Setting up EC2

Go to AWS and sign up for an account. The payment options include all common credit cards.

Sign up for t2.micro on demand instance using Ubuntu 14.04 image. Pick an instance from US East Coast (N. Virginia) availability zone for the maximum speed from Europe. AWS prompts yoy to download SSH keys to access the instance. Store them safely. Below we assume you download keys named amazon and store them in your SSH folder:

mv ~/Downloads/amazon.pem ~/.ssh
chmod 400 ~/ssh/amazon.pem

By default, EC2 instances are firewalled. In your EC2 instance list, click Security Group for the instance (Security group name should be like launch-wizard-1). Go to Inboud > Edit and choose All traffic. This allows access to PPTP service and Squid proxy which will be installed on the server.

Screen Shot 2015-04-12 at 19.03.25

Installing PPTP using Ansible

Install Ansible. Packages available for Ubuntu, OSX Homebrew and others. Ansible is a tool installed on your local computer and it will connect the server over SSH.

Clone the Ansible playbook which automates PPTP installation on your server.:

git clone

Replace hosts file in the playbook. Change the IP address to your instance IP address as shown in Amazon AWS console:

[amazon] ansible_ssh_user=ubuntu ansible_sudo=true ansible_ssh_private_key_file=~/.ssh/amazon.pem

You need to wait until instance state says Running in AWS console. Then proceed to install and configure PPTP on the server using Ansible:

ansible-playbook -i hosts site.yml --extra-vars '{"hosts":"","username":"vpn","password":"changeme"}'

Now you can connect to your VPN using the instance IP address, username vpn and password changeme.

Connecting to VPN with Android

Test your VPN on Android by going to menu Settings > Wireless & networks > More > VPN. Hit + to add new entry. Choose PPTP, enter your server IP, username and password. Choose the created entry to connect. In few seconds you should see Connected status and a key icon appears in Android status bar.

Please note that the instance IP address changes on every start and stop. You can update the VPN IP address by doing a long press on the entry in Android VPN menu.

Starting and stopping EC2 instance on demand

Download AWS Console app from Google Play Store or from App Store for your mobile. The app takes your AWS credentials and allows you to start and stop the EC2 instances  with a single click.

After starting you’ll see the new IP address of the instance in Public IP field of AWS Console app.



Please note that PPTP does not work on cheap OpenVZ virtual machines due to lack of kernel support – a full virtual machine is required. The Ansible playbook is kindly provided by Liang Shan.

\"\" Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+