- Gearman and Django
Gearman is server and application framework for distributing and processing jobs between processes and machines.
At CodeScale we use Gearman quite extensively therefore we’ve created library to simplify development of Gearman related modules in Django.
You are welcome to read more, download and contribute on GitHub.
- Emacs and Python – The Definitive Answer
Over the past year or so, I’ve used Emacs pretty extensively for Python/Django development.
During that time, I’ve customized my .emacs file a lot, mostly with .el packages found on the net
related to Python development in Emacs – rope, pymacs and friends.
If you google python+emacs, you’ll find a lot of packages and recipes how to glue it all together,
but if you are Emacs Lisp newbie like me, having all that stuff in one easy-to-install, tested package
would be priceless.
I was pretty sure my hacks were not best in the world and someone must have done it better.
And I was right
Gabriele Lanaro put it all into emacs-for-python package which is really easy to setup
and to be impressed in under a minute.
If you are on Debian/Ubuntu, just install the dependencies:
$ sudo apt-get install pymacs pyflakes
Then extract emacs-for-python somewhere
and add it to your .emacs file:
(load-file "/path/to/epy-init.el")
Reload or restart Emacs and you are ready to go.There are a several pre-configured emacs/python packages out of the box and it all works together flawlessly.
Thank you Gabriele, great work.
- Using SQLite in Django for Unit Testing
Great thing about Django is that configuration – settings.py is pure Python file.
This gives you power and expressiveness allowing to use conditions and expressions normally available in python script.
On past few projects, I’ve used ‘trick’ to automatically determine if ‘manage.py test’ was invoked
and use SQLite for unit testing instead of standard database used for production.
I thought it’s worth sharing, so here it is
The point is that if you have unit tests in your application(s) (which you should, believe me), running them on top of ‘standard’ database used for development or production (PostgreSQL, MySQL…) may be not the fastest thing around.
Considering you have _a lot_ of unit tests or initial_data fixtures, running them in memory would be great thing.
Let’s customize settings.py a little bit.import sys # is ./manage.py test ? TEST = 'test' in sys.argv if TEST: # in-memory SQLite used for testing DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': ':memory:', } } else: # devevelopment/production db DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'yourproductiondb', 'USER': 'yourproductiondbuser', 'PASSWORD': "secret_password", 'HOST': 'localhost', 'PORT': '', }, }
So what we did:
- automatically determine if ‘./manage.py test’ was invoked and store that in TEST variable
- if TEST is True, use SQLite in-memory database instead of your production/development databaseAssuming your tests and models are fine with SQLite, you can save some time when periodically running unit tests,
which is worth some hacking
- log4net MongoDB appender
I have some good news for anyone interested in logging from log4net into MongoDB.
I have created log4net appender for Mongo, so it’s now easily possible to use MongoDB as a storage for the logs.
Details, sources, download, everything is on GitHub:
log4mongo-net at GitHub
More info, quickstart: log4mongo.org
- Getting Started with Memcached
The goal of this article
What is caching
What is Memcached
Installation (Linux, Windows)
Basic get/set operations
Where to go next
Conclusion
The goal of this article
This article is aiming to provide basic introduction into server-side
caching using open-source memcached system. If you are developer looking for caching
solution for your (web)applications, this article is going to give you the overview
of how to install and use memcached from several programming environments.
The goal of this article is to provide quick overview for developers with zero or minimal experience
with memcached, not to provide comprehensive reference of all memcached features.What is caching
Performance matters. This fact becomes more and more important in the era of the web sites
with thounsands, millions or hundreds of milions of users.
Caching is one of the most important performance strategy applicable independently from the type of your application.
The basic idea is pretty simple – reduce the number of expensive tasks and put the results into the storage which is accesible faster.You can find implementation of caching in many software products you use on daily-basis.
- Web browser cache images, scripts, or whole pages retrieved from server, storing them on client disk
- Text processor reads content of file from disk into memory to provide better performance when working with document
- Server-side application store results read from database in memory for faster accessWhat is Memcached
Memcached is free, open-source memory caching system with focus on high-performance,
distributed deployment and scalability.
Memcached started as a project for LiveJournal with very clear goal – speed up website by caching frequently
used items loaded from database into memory. Today, memcached is the core infrastructure component for the biggest players on web field, including Google/YouTube, Digg, Facebook, Wikipedia, Amazon and many others.The design concept is that memcached acts as an key-value store, running separately from your application.
The independence from the memcached consumer is very important for easy deployment and the fact
that memcached stays generic from the consumer’s perspective.
Memcached runs as a separate service/daemon listening for incoming connection on TCP/IP.
Clients opens TCP connection to memcached, sends storage or retrieval command, gets response and closes the connection. Clients usually use socket pools to eliminate the need to open separate connection for every memcached request and returns connection into pool after performing the operation instead of closing it.Installation (Linux, Windows)
This chapter is going to guide you through the basic installation procedure on Linux and Windows platforms
with special focus on common pitfals of Windows memcached port.Linux
Linux is probably the most commonly used OS for memcached with wide support of binary, ready to use packages for various distributions.
Memcached has dependency on libevent library, therefore you need also libevent to get memcached up&running.
On Debian/.deb-based distros, installation is pretty easy:$ sudo apt-get install memcached
This will install memcached and required dependencies (libevent).
On Debian, the config file is /etc/memcached.conf. There is a list of command-line options with comments.For installation from source, get the latest source from memcached homepage and proceed
with standard ./configure && make commands. Note that libevent (and libevent-dev on Debian) must be available.Windows
Installation on Windows can be tricky due to the fact that there are several versions of memcached compiled for Windows. The main difference is their age and the level of potentional issues. I recommend using the most recent build from jellycan which is basically memcached 1.2.6 compiled for Windows.
Get it, unzip, execute memcached.exe and you have the most minimal memcached setup ready.
Windows is not officially supported as a target platform for which memcached is continuously built and tested.
If you are more interested on the reasons for this, this thread is quite interesting description
of actual problems.The very funny part about memcached on Windows is running it as a service with non-default settings.
As the default configuration of memcached allows only 64MB of RAM to be used for items, the very
common requirement for running memcached is to run it as a service with more than 64MB of memory allowed.
Default memcached switch for installing as a service on Windows is ‘-d’, which ignores additional command-line options.
This problem can be resolved using ‘sc.exe’ for service installer.@echo off set ramSize=512 sc create memcached binPath= "c:\memcached\memcached.exe -d runservice -m %ramSize%" start= auto DisplayName= "memcached" sc start memcached
Save this to the install.bat file, change the path to memcached.exe and ramSize if needed, then run the bat file.
Test if memcached is running and accepting connections
Memcached by default listen on TCP port 11211.
After installation, you can test if memcached is accepting incoming connection just with telnet.$ telnet localhost 11211 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. version VERSION 1.2.8 quit Connection closed by foreign host. $
Note that on Windows 2008 Server, telnet is not installed by default and must be added as additional component.
However, I recommend using PuTTY on Windows instead.Basic get/set operations
Before we jump into real code, there are few things which you should keep in mind from the start:- maximum size per item in memcached is 1MB. This is by-design limitation, mostly done because of memory allocation optimizations. This problem can be partially resolved using compression on client-side before putting item into cache, but not for all cases. Memcached is not the right place to cache large images or documents. Memcached is optimized to store small chunks of data (strings, objects etc). However stored objects can be anything as memcached follows ‘What you set is what you get’ rule, keep the size limitation in mind.
- memcached does not provide security/authentication mechanism. Again, this is by design. Memcached is designed to be as fast as possible and you should delegate security to different layer. The simplest way is to firewall memcached server(s) which would allow only machines connecting to memcached directly (web servers) to open the connection. In common scenarios, memcached servers are not a part of publicly visible network and only web servers know IPs of memcached servers in internal network.
- memcached is not a database, it’s a key-value pair store. You can’t make something like ‘select * from items’ on memcached to dump all items in cache or so. Normally, you can access values in cache only by knowing their key(s).
The most common operations with memcached are getting items from cache and setting them to cache.
If you have code which query database for anything and then send the results out, you can basically
rewrite it to query the memcache first. If item is found in the cache, you send this out – avoiding DB query.
If item is not yet in the cache (which can be for several reasons we’ll describe later), you get results from DB query first
and then set them into the cache. On next request, only cache will be hit and you get the performance boost.Speaking in code, if you have something like:
result = query("select a,b,c from x"); return result;
You can rewrite it into:
result = get_from_memcache("mykey1"); if (result == NULL) { result = query("select a,b,c from x"); // get result from DB set_to_memcache("mykey1", result); // set result into memcache } return result;
It’s that simple. Note that there are several reasons client may return NULL/null/undefined/None value for get operation:
- item is not yet in the cache under specified key
- item may be expired. Memcached supports expiration of items which is very useful if you need automatically invalidate item after some period of time. You can also set expiry parameter to ’0′ and tell memcache not to expire item.
- item may be deleted
- memcached has been restarted, or ‘flush_all’ command was sent to memcached, cleaning up all cachesI’d recommend following libraries for Python, Java and .NET.
Python
python-memcached package, available though apt-get or easy_install.
You can also use this package with Django.>>> import memcache >>> mc = memcache.Client(['127.0.0.1:11211'], debug=0) >>> mc.set("a", "X1") True >>> mc.get("a") 'X1'
Java
spymemcached is library developed and maintaned by memcached core contributor.
It’s also good choice if you use Hibernate for you projects, because hibernate-memcached second level cache provider is based on spymemcached. I would recommend using spymemcached even you don’t use Hibernate..NET
enyim.com Memcached Client seems to be the best option for .NET client and is under active development.
However, if you would like to use NHibernate with memcached, NHibernate memcached provider ships with different library (far older and not under active development).
If you do not use NHibernate, I would recommend go with Enyim.Where to go next
Except the basic get/set operations, there is a lot of other magic available in memcached.
Very detailed description of all commands and options can be found in protocol documentation, for example statistics, increment/decrement statements, multi-get retrieval, UDP protocol usage and other built-in features.There is a large community around memcached and a lot of useful resources.
Following links are good starting point:
Memcached official site
Memcached support mailing list
Memcached wiki, FAQ, HowTos
List of client APIs/libraries
Forks/reimplementations and other projects related to memcachedConclusion
Memcached may be the silver bullet for many scalability problems of webapplications where frequent read operations from slower store (database, disk..) occurs.
However memcached can help you ‘make things faster’, keep in mind that the optimization rule #1 is to optimize application logic where possible.
Sooner or later, nothing will save you in case of poorly designed DB queries or suboptimal algorithms.
If you pass this ‘optimize yourself’ routine and decide you need caching, memcached can be very handy because
of its stability, strong adoption, excellent community support and distributed nature.Slovak translation of this article at Zdrojak.
- WebSockets and pywebsocket – quick & dirty playground
Playing with HTML 5 WebSockets
One of the interesting features proposed for HTML 5 is The Web Sockets API.
Web Sockets API define bidirectional communication between the web browser (using Javascript) and server. The use of Web Sockets may be very interesting replacement of frequent browser-to-server calls, especially for ‘check if something has changed’ or ‘is there something new ?’ tasks. Web Sockets API also support tunneling through HTTP CONNECT statements, therefore firewalls and proxy servers would not be a problem.Talk is cheap, show me the code !
The easiest option that worked for me was checking out pywebsocket which claims to be Web Sockets extension for Apache. The pywebsockets distribution contains standalone version of server and some samples so you don’t need to do any magic with Apache to see ‘the real thing’.
First, checkout the pywebsocket, build it and run standalone.py:$ svn checkout http://pywebsocket.googlecode.com/svn/trunk/ pywebsocket-read-only $ cd pywebsocket-read-only/src $ python setup.py build $ sudo python setup.py install $ cd mod_pywebsocket/ $ sudo python standalone.py -d ../example/
Now you have standalone version of pywebsocket up&running, using ‘../example’ as document root.
There is ‘echo_wsh.py’ script in example directory which provides basic ECHO service.
Now, create the HTML file with the JavaScript which we’ll use as ECHO client:<script> function sendMessage() { var msg = document.getElementById('message').value; if ("WebSocket" in window) { var ws = new WebSocket("ws://localhost/echo"); ws.onopen = function() { ws.send(msg); }; ws.onmessage = function (evt) { var received_msg = evt.data; showResponse(received_msg); }; ws.onclose = function() { alert('socket closed'); }; } else { alert('no websockets'); } } function showResponse(msg) { document.getElementById('response').innerHTML = document.getElementById('response').innerHTML + msg + '</br>' } function clearResponse() { document.getElementById('message').value = ''; document.getElementById('response').innerHTML = ''; } </script> <input type="text" id="message" name="message" /> <input type="button" value="Send" onclick="sendMessage()" /> <hr /> <input type="button" value="Clear" onclick="clearResponse()" /><br /> <span id="response"> </span>
Note that HTML/Javascript is really quick&dirty, but it will work for the test.
Open Chrome browser (version 4.0.249.0 and higher implements WebSockets API) and browse HTML test file.
Enter something into input field and voilà, server will echo your message and you will see the response:
You will also see alert when you interrupt standalone.py.
I hope the WebSockets API will get some attention from other browser vendors (support in Firefox in progress, AFAIK) as this may be lifesaver for many cases where bidirectional messaging between browser and server comes to game.
- Django 1.2 is coming
…and seems to be impressive.
I just read What’s coming in Django 1.2 and new features are very promising. Django 1.2 final should be released on March 2010 and according to the roadmap feature list is already frozen.
Here are some of them I’m very happy with:
Integrating logging with Python – huge advantage for debugging
QuerySet.raw() – mapping to model instances using raw SQL, automatically
Multi-DB support – big step forward for sharding/clustering scenariosCan’t wait !
- Skype on Linux and green screen instead of video
And solution for Ubuntu
If you encounter seeing sluggish broken green screen in your Skype instead of the video, try to run skype like this:
LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so skype
For 64-bit Ubuntu, the path may vary (probably /usr/lib32/libv4l/v4l1compat.so).
If your skype video works using this trick, you can create shell alias and put it into .bashrc.
This fixed my issue on Ubuntu 9.10 with Skype beta for Ubuntu, hope that helps.
- Install Chrome/Chromium in Ubuntu – quick & easy
How to install Chromium daily builds on Ubuntu:
UPDATE: Chrome on Linux is not officialy supported.
sudo echo "deb http://ppa.launchpad.net/chromium-daily/ppa/ubuntu jaunty main" >>/etc/apt/sources.list sudo echo "deb-src http://ppa.launchpad.net/chromium-daily/ppa/ubuntu jaunty main" >>/etc/apt/sources.list sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 4E5E17B5 sudo apt-get update sudo apt-get install chromium-browser
- Reusable django.wsgi and settings.py with Apache
Reusable solution for deploying Django in Apache
Django plays with Apache very well. Using mod_wsgi is recommended
to host Django apps in Apache and by my (short) experience it works fine.
There are few things I did not like on all the deployment docs/manuals
and it was the way how they handle the paths to application.
I was OK with entering fixed, absolute path to my django app in host config of Apache, but it made no sense for me to insert the path to application in django.wsgi or settings.py files.
Here is the solution which worked fine.
Configuration of VirtualHost in Apache:<VirtualHost *> ServerName www.mysite.com WSGIDaemonProcess mysite processes=2 maximum-requests=500 threads=1 WSGIProcessGroup mysite WSGIScriptAlias / /var/www/mysite/django.wsgi Alias /site-media "/var/www/mysite/site-media" Alias /admin-media /usr/lib/python2.6/site-packages/django/contrib/admin/media <Directory /var/www/mysite> Order deny,allow Allow from all </Directory> </VirtualHost>
This is the only place you need to set fixed path to django app.
As you can see, WSGIScriptAlias directive points to django.wsgi file in project directory,
which does handle requests and forward them to django app.
Here is how django.wsgi look like:import os, sys sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/..') sys.path.append(os.path.dirname(os.path.abspath(__file__))) os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings' os.environ['PYTHON_EGG_CACHE'] = '/tmp/python-eggs' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
As you can see, django.wsgi handler will append path of project directory into PythonPath automatically.
Now we want to do something similar in settings.py for MEDIA_ROOT.
Relevant fragments of settings.py:import os BASE = os.path.dirname(__file__) PROJECT_DIR = os.path.dirname(os.path.abspath(__file__)) MEDIA_ROOT = os.path.join(BASE, 'site-media') MEDIA_URL = '/site-media/' ADMIN_MEDIA_PREFIX = '/admin-media/'
That’s all folks. You can now reload apache with command ‘apache2ctl graceful’ run as root and your great app should be up&running. Oh yes, don’t forget to install mod_wsgi for Apache first
