Jannis Leidel

In my last post I showed you how to write your own widget for django-tagging form fields that uses the nifty jQuery autocomplete plugin to simplify entering tags. This time I’d like to take on a suggestion PatricK made in the comments:

my question is, if this could be done on a more abstract level for enhancing relations. when having a foreignkey, you get the browse-icon (lens) right near the input-field. it´d be awesome to search the relations just by typing something into the field … the autocomplete-functionality should then search the related entries on the basis of the defined search-fields.

Well, this is what I’m going to do today :)

Preface

Thanks to the way Django’s forms work, the following implementation should be applicable to your own Django code — but please bare in mind that this isn’t a copy and paste how-to. I rather encourage you to learn about the flexibility of forms, fields and widgets and how it’s used in the automatic Admin interface. You’ll be able to — and sometimes required — to customize the code posted below.

As an example I’m going to use the message system that is included in Django’s auth contrib app. As far as I know Django doesn’t provide an admin representation for the Message class by default because it’s supposed to be used programmatically (and is used by the admin itself, too). The model consists of two fields, a ForeignKey to a User and a text field for the message.

Use case

Since ForeignKey fields are rendered as select fields by default there is a good chance that they become unusable if there are a lot of possible choices. So in case you have a lot of users in your database for example it might be a bit cumbersome to choose a user from the ForeignKey dropdown menu. Django provides an option that allows specifying a raw_id_fields attribute in your ModelAdmin subclass that will tell Django to render the ForeignKey as a simple input taking the primary key of the refered object. The actual selection is done by clicking the browse icon next to the field and browsing the model with the standard admin interface in a popup. What I want to provide here though is a way to do that directly in place, in the original form.

Widget

We are going to use the jQuery and the great Autocomplete plugin again which does all the hard work. Please download the sources of the autocomplete plugin if you don’t have it already. It includes everything we need to get started. Copy all files and folders of the extracted jQuery Autocomplete zip file to the directory you specified in the MEDIA_ROOT setting. It should then contain: ‘jquery.autocomplete.js’, ‘jquery.autocomplete.css’ and a ‘lib’ directory with the other necessary files. In case you arrange your files differently don’t hesitate to change the paths below in the inner Media class.

It has been proven to be a good idea to put something like the follwing widget in the widgets.py of your Django app, but any place will do if you change the import paths.

from django import forms
from django.conf import settings
from django.utils.safestring import mark_safe
from django.utils.text import truncate_words

class ForeignKeySearchInput(forms.HiddenInput):
    """
    A Widget for displaying ForeignKeys in an autocomplete search input 
    instead in a <select> box.
    """
    class Media:
        css = {
            'all': ('jquery.autocomplete.css',)
        }
        js = (
            'lib/jquery.js',
            'lib/jquery.bgiframe.min.js',
            'lib/jquery.ajaxQueue.js',
            'jquery.autocomplete.js'
        )

    def label_for_value(self, value):
        key = self.rel.get_related_field().name
        obj = self.rel.to._default_manager.get(**{key: value})
        return truncate_words(obj, 14)

    def __init__(self, rel, search_fields, attrs=None):
        self.rel = rel
        self.search_fields = search_fields
        super(ForeignKeySearchInput, self).__init__(attrs)

    def render(self, name, value, attrs=None):
        if attrs is None:
            attrs = {}
        rendered = super(ForeignKeySearchInput, self).render(name, value, attrs)
        if value:
            label = self.label_for_value(value)
        else:
            label = u''
        return rendered + mark_safe(u'''
            <style type="text/css" media="screen">
                #lookup_%(name)s {
                    padding-right:16px;
                    background: url(
                        %(admin_media_prefix)simg/admin/selector-search.gif
                    ) no-repeat right;
                }
                #del_%(name)s {
                    display: none;
                }
            </style>
<input type="text" id="lookup_%(name)s" value="%(label)s" />
<a href="#" id="del_%(name)s">
<img src="%(admin_media_prefix)simg/admin/icon_deletelink.gif" />
</a>
<script type="text/javascript">
            if ($('#lookup_%(name)s').val()) {
                $('#del_%(name)s').show()
            }
            $('#lookup_%(name)s').autocomplete('../search/', {
                extraParams: {
                    search_fields: '%(search_fields)s',
                    app_label: '%(app_label)s',
                    model_name: '%(model_name)s',
                },
            }).result(function(event, data, formatted) {
                if (data) {
                    $('#id_%(name)s').val(data[1]);
                    $('#del_%(name)s').show();
                }
            });
            $('#del_%(name)s').click(function(ele, event) {
                $('#id_%(name)s').val('');
                $('#del_%(name)s').hide();
                $('#lookup_%(name)s').val('');
            });
            </script>
        ''') % {
            'search_fields': ','.join(self.search_fields),
            'admin_media_prefix': settings.ADMIN_MEDIA_PREFIX,
            'model_name': self.rel.to._meta.module_name,
            'app_label': self.rel.to._meta.app_label,
            'label': label,
            'name': name,
        }
</select>

This widget renders ForeignKeys as hidden inputs and adds a second input field to search in the related model using asynchronous requests, sometimes refered to as AJAX. Once the user selects a result the widget will automatically set the hidden field to the primary key. When editing an existing entry this will automatically populate the search field with the correct verbose value.

Admin integration

As already stated we are going to create an admin class for the Message model to enable admins to create messages for other users that will be displayed the next time they use the admin (or any other site that uses User.get_and_delete_messages).

The best place for the admin class is an admin.py file in one of your own app directories because it’s then picked up by Django’s autodiscover() function.

import operator
from django.db import models
from django.contrib.auth.models import Message
from django.http import HttpResponse, HttpResponseNotFound
from django.contrib import admin
from django.db.models.query import QuerySet
from django.utils.encoding import smart_str

from yourapp.widgets import ForeignKeySearchInput

class MessageAdmin(admin.ModelAdmin):
    list_display = ('user', 'message')
    related_search_fields = {
        'user': ('username', 'email'),
    }

    def __call__(self, request, url):
        if url is None:
            pass
        elif url == 'search':
            return self.search(request)
        return super(MessageAdmin, self).__call__(request, url)

    def search(self, request):
        """
        Searches in the fields of the given related model and returns the 
        result as a simple string to be used by the jQuery Autocomplete plugin
        """
        query = request.GET.get('q', None)
        app_label = request.GET.get('app_label', None)
        model_name = request.GET.get('model_name', None)
        search_fields = request.GET.get('search_fields', None)

        if search_fields and app_label and model_name and query:
            def construct_search(field_name):
                # use different lookup methods depending on the notation
                if field_name.startswith('^'):
                    return "%s__istartswith" % field_name[1:]
                elif field_name.startswith('='):
                    return "%s__iexact" % field_name[1:]
                elif field_name.startswith('@'):
                    return "%s__search" % field_name[1:]
                else:
                    return "%s__icontains" % field_name

            model = models.get_model(app_label, model_name)
            qs = model._default_manager.all()
            for bit in query.split():
                or_queries = [models.Q(**{construct_search(
                    smart_str(field_name)): smart_str(bit)})
                        for field_name in search_fields.split(',')]
                other_qs = QuerySet(model)
                other_qs.dup_select_related(qs)
                other_qs = other_qs.filter(reduce(operator.or_, or_queries))
                qs = qs & other_qs
            data = ''.join([u'%s|%s\n' % (f.__unicode__(), f.pk) for f in qs])
            return HttpResponse(data)
        return HttpResponseNotFound()

    def formfield_for_dbfield(self, db_field, **kwargs):
        """
        Overrides the default widget for Foreignkey fields if they are
        specified in the related_search_fields class attribute.
        """
        if isinstance(db_field, models.ForeignKey) and \
                db_field.name in self.related_search_fields:
            kwargs['widget'] = ForeignKeySearchInput(db_field.rel,
                                    self.related_search_fields[db_field.name])
        return super(MessageAdmin, self).formfield_for_dbfield(db_field, **kwargs)
admin.site.register(Message, MessageAdmin)

So this code does several things:

• It inherits from the default ModelAdmin to be able to override the default widget.
• A new related_search_fields class attribute that is a mapping between lowercase model names and a list of fields to searched in. It uses the same syntax as the search_fields attribute to speed up the lookups or restrict them if needed. The following example would only match the beginning of the first and last name of users:

related_search_fields = {
    'user': ('^first_name', '^last_name'),
}

• It overrides the __call__ method to add additional URL handler to the admin to be used by the widget for searching. It will be waiting for requests on /<root-admin-path>/auth/message/search/.
• The search view that constructs the querysets depending on the provided model and search fields and returns the data as a simple string. The Autocomplete plugin can parse that and will display it in the selection.
• A formfield_for_dbfield method that will override the widget of any ForeignKey field whose name is specified in the related_search_fields class attribute.

Once assembled the result will mostly look like that:

Please note that I use a different stylesheet for the Autocomplete plugin here.

Today while working on django-page-cms I received a feature request by an user to enable the admin form field for tags of a page to automatically suggest input options — just like YouTube‘s or Google search field for example. Go ahead and try it if you don’t know what I mean.

Like my last article about a custom WYMEditor widget I will show you how such a autocomplete widget could be implemented by using jQuery and the great Autocomplete plugin which does all the hard work for us. Please download the sources of the autocomplete plugin if you don’t have it already. It includes everything we need to get started.

On the server side I’m going to use the well-known django-tagging which luckily also makes handling the tags a piece of cake.

The model

Although the whole process is entirely applicable to your own models, I’m going to use the following model as a matter of example.

from django.db import models
from tagging.fields import TagField

class Article(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()
    tags = TagField()

It’s a rather simple model with a field that uses django-tagging‘s backend. It will later be rendered with your autocomplete widget.

The widget

Copy all files and folders of the extracted jQuery Autocomplete zip file to the directory you specified in the MEDIA_ROOT setting. It should then contain: 'jquery.autocomplete.js', 'jquery.autocomplete.css' and a 'lib' directory with the other necessary files. In case you arrange your files differently don’t hesitate to change the paths below in the inner Media class. It has been proven to be a good thing to put something like this widget in the widgets.py of a Django app.

from django import forms
from django.db.models import get_model
from django.utils import simplejson
from django.utils.safestring import mark_safe
from tagging.models import Tag

Article = get_model('yourapp', 'article')

class AutoCompleteTagInput(forms.TextInput):
    class Media:
        css = {
            'all': ('jquery.autocomplete.css',)
        }
        js = (
            'lib/jquery.js',
            'lib/jquery.bgiframe.min.js',
            'lib/jquery.ajaxQueue.js',
            'jquery.autocomplete.js'
        )

    def render(self, name, value, attrs=None):
        output = super(AutoCompleteTagInput, self).render(name, value, attrs)
        page_tags = Tag.objects.usage_for_model(Article)
        tag_list = simplejson.dumps([tag.name for tag in page_tags],
                                    ensure_ascii=False)
        return output + mark_safe(u'''<script type="text/javascript">
            jQuery("#id_%s").autocomplete(%s, {
                width: 150,
                max: 10,
                highlight: false,
                multiple: true,
                multipleSeparator: ", ",
                scroll: true,
                scrollHeight: 300,
                matchContains: true,
                autoFill: true,
            });
            </script>''' % (name, tag_list))

The render method queries all tags saved in other articles instances and passes them as a JavaScript array to the autocomplete initializer in the HTML.

The form

Next the widget needs to be hooked up with the admin interface. One way to do that is to override the default form Django is using when generating the admin site. This form inherits from the default ModelForm class and has a tags field that uses your custom AutoCompleteTagInput widget and is used for the model field with the same name. Please save the following code in the 'forms.py' of your app.

from django import forms
from django.db.models import get_model
from tagging.forms import TagField
from yourapp.widgets import AutoCompleteTagInput

class ArticleAdminModelForm(forms.ModelForm):
    tags = TagField(widget=AutoCompleteTagInput(), required=False)

    class Meta:
        model = get_model('yourapp', 'article')

Assembling

Now all what’s left to do is to connect your new ModelForm class with the admin interface by creating a custom ModelAdmin to override the default. At the end just register your model with the default admin site.

from django.contrib import admin
from django.db.models import get_model
from yourapp.forms import ArticleAdminModelForm

class ArticleAdmin(admin.ModelAdmin):
    # ..
    form = ArticleAdminModelForm

admin.site.register(get_model('yourapp', 'article'), ArticleAdmin)

Recently I’ve been looking for a useful WYSIWYM editor to be used in Django’s admin interface. Thanks to the awesome admin being backed by the forms module it’s rather easy to replace a form field widget with a custom widget.

Although it’s more or less the defacto standard, I’m not going to use TinyMCE this time, partly because it feels a bit like bloatware and partly because it’s not based on jQuery — the JavaScript framework of choice of the project where I also want to use it: django-page-cms. The editor I’m going to use is WYMeditor because it provides authors some helpers to write sane (X)HTML instead of hiding it away while not leaving them with a blank textarea.

The widget

To use the following widget you need to download the source files of WYMedtior first and put them in the directory you specified in the MEDIA_ROOT setting of your Django-based site. So your MEDIA_ROOT directory should now contain a 'jquery' and a 'wymeditor' directory.

Also, copy the following code to a place of your convenience, e.g. the widgets.py of the Django app you are going to add the WYMeditor functionality to.

from django import forms
from django.conf import settings
from django.utils.safestring import mark_safe

class WYMEditor(forms.Textarea):
    class Media:
        js = (
            'jquery/jquery.js',
            'wymeditor/jquery.wymeditor.pack.js',
        )

    def __init__(self, language=None, attrs=None):
        self.language = language or settings.LANGUAGE_CODE[:2]
        self.attrs = {'class': 'wymeditor'}
        if attrs:
            self.attrs.update(attrs)
        super(WYMEditor, self).__init__(attrs)

    def render(self, name, value, attrs=None):
        rendered = super(WYMEditor, self).render(name, value, attrs)
        return rendered + mark_safe(u'''<script type="text/javascript">
            jQuery('#id_%s').wymeditor({
                updateSelector: '.submit-row input[type=submit]',
                updateEvent: 'click',
                lang: '%s',
            });
            </script>''' % (name, self.language))

The WYMEditor widget inherits most of its features from the default forms.Textarea widget and only adds a small bit of JavaScript to initialize in the browser. The inner Media class does all the hard work of adding the correct path to WYMEditor’s source files to the admin template.

The form

Next you need to create a ModelForm subclass that will later be used by the admin as the default form while rendering the edit page. You are again free to put it anywhere in your files as long as you are able to import it. In this case let’s add it to the forms.py of your app, a good place to store form classes.

Please note you need to customize some parts of the following snippet to match your app:

• The second line needs to successfully import the WYMEditor widget
• Line five should have the name of the field you want to apply the WYMeditor to, in this example the model field body would get it.
• The last line uses a nifty tool of Django that loads a model without the need to import it. In fact you just need to give it the lowercase name of the app and the lowercase name of the model.

from django import forms
from django.db.models import get_model
from yourapp.widgets import WYMEditor

class ArticleAdminModelForm(forms.ModelForm):
    body = forms.CharField(widget=WYMEditor())

    class Meta:
        model = get_model('yourapp', 'article')

Assembling

Now open the 'admin.py' of your app to hook that ModelForm up with the admin interface. First import your ModelForm subclass. Then set the class attribute of the ModelAdmin subclass called form to the form subclass you just imported. Finally register your custom admin class and your model with the default admin site.

from django.contrib import admin
from django.db.models import get_model
from yourapp.forms import ArticleAdminModelForm

class ArticleAdmin(admin.ModelAdmin):
    # ..
    form = ArticleAdminModelForm

admin.site.register(get_model('yourapp', 'article'), ArticleAdmin)

That’s it for today, I hope everything works for you. Let me know what you think.

Given my long abstinence from this very weblog I would like to take the opportunity to post some stuff I’ve been doing the last couple of months. That’s especially easy since I have some people to stay on the wheel with (Brian Rosner, James Tauber, Justin Lilly, Greg Newman, Eric Holscher and Eric Florenzano) — all pals from the wider Django community, most of them also connected to the project I’ve been spending my nights with in the last six months: Pinax. They are all doing the one-post-a-day marathon in November. Maby I do, too.

dbtemplates 0.5.0

This time I want to talk a bit about some changes I recently did to a small reusable Django app of mine, that provides an easy way to save Django templates in the database, instead on the file system.

With the release of django-dbtemplates 0.5 it includes changes, mostly inspired by a short conversation with Alexander Artemenko on Github in which he proposed to add the ability to load the contents of a new database template from disk if the content field is left empty. That speeds up the process of moving filesystem templates a bit because users don’t need to copy and paste every thing if the templates are in reach of other template loaders anyway.

version control

We also talked about adding version control to dbtemplates to enable users to jump back and forth in case a change to a database template didn’t work out so well. We first considered a project of Arne Brodowski: django-rcsfield, which is a pretty nifty piece of software that uses “real” version control systems to versionize database field contents, while always keeping the HEAD version in the database. Although I trust the Git backend I wrote for it to handle different versions of the database templates, we figured it would be not too useful for end users like editors and authors who for various reasons don’t have access to the filesystem of the server.

So I actually ended up adding support for django-reversion, a third party app that saves different versions of a database entry in a other table, everytime Django’s post_save signal is fired. Very useful and during my tests also very stable. dbtemplates supports it out of the box — once django-reversion is installed properly, it will versionize every change of your database templates — accessible in the usually unchallenged “history” section of each database template.

caching backends

The third big change is the addition of a customizable caching backend. Until now you were only able to let dbtemplates cache every database template on the file system to speed things up and reduce the number of database queries.

I replaced that mechanism with a pluggable backend system to allow other ways of handling template loading and provide a simple API to build your own.

dbtemplates comes with two backends by default: FileSystemBackend, which is a port of the old way of doing things and DjangoCacheBackend, which is a thin wrapper around Django’s insanely useful caching framework. Be sure to checkout dbtemplates’ documentation to learn more about it.

Writing an own backend is as easy as pie:

from dbtemplates.cache import BaseCacheBackend
class AwesomeBackend(BaseCacheBackend):
    def load(self, name):
        """Loads a template from the cache with the given name"""
        return 'I wanted this template: %s' % name

    def save(self, name, content):
        """Saves the given template content with the given name.""" 
        print 'Saving the template %s with the content %s.' % (name, content)    

    def remove(self, name):
        """Removes the template with the given name from the cache."""
        print 'Bye bye, %s' % name

That’s it. Put that code somewhere in your code base and point dbtemplates to it by specifying the DBTEMPLATES_CACHE_BACKEND setting, e.g. 'awesome_app.cache.AwesomeBackend'.

Feel free to hop on IRC to #django-hotclub or visit code.google.com/p/django-dbtemplates/ to learn more.

Im Zuge des diesjährigen Linuxtages würden wir uns gern am Freitag, 30. Mai 2008, 17:00 Uhr mit anderen Django-Benutzern und -Entwicklern treffen und gemeinsam gekühlte Getränke genießen. Treffpunkt ist vor dem Haupteingang.

We’d like to take the opportunity of this year’s Linuxtag to meet on Friday, May 30, 2008 5:00 PM with other Django users and developers to have an informal meetup and some cold beverages. The gathering place is in front of the main entrance.

UPCOMING, VENTERIA

After visiting this year’s awesome Pycon in Chicago I’m very looking forward to attend another major event of the Python community: Europython 2008 in Vilnius, Lithuania. The flight and accommodation is surprisingly inexpensive, so if you are not sure about attending here are some other good reasons.

The call for participation went out in April and promises to bring us all a great conference. If you have a good idea for a talk or a lightning talk, want to be in touch with other developers of your favorite Python software in an open space or just like to hang out with nice people please consider participating. Since Europython is a community conference you decide what will happen!

Django sprint?

As you may know EuroPython is divided in a conference and a sprint part. Like other occasions in the past (like Pycon) I think there is a good chance we can hold a Django sprint if enough people join. And I think on the way to Django 1.0 it’s a good time to fix bugs, work on documentation, test edge cases, discuss new features, work on reusable apps and do other fantastic things with Django. We all have seen what the queryset-refactor brought us, so let’s meet to work on the rest of list.

If you are interested please leave a comment here or come to the #django-sprint IRC channel to discuss things further.

UPDATE

Yay, apparently Europython’s registration opened today!

Since version 3.56 Virtualmin Pro is slowly gaining support for Python based web applications because it includes an official install script for Django. This makes my version of a Django install script slightly unnecessary, even though the official version only supports stable releases (0.96.1) and not the SVN trunk (which we all know is quite common in Djangoland). Anyway, I think this is great news.

Other Python based web applications also benefit from the official Django install script of course because there are some helpful features in Virtualmin’s codebase now. Which brings us to:

Trac

Virtualmin already does a great job managing Subversion repositories for virtual servers but misses an install scripts for a web application that makes working with these repositories easier later in daily work. Since I really like Trac for its ease of use, powerful features and ubiquity in the developer scene I wrote an install script for it: trac.pl.

Just put it in /etc/webmin/virtual-server/scripts/ and go to the “Install script” section. Keep in mind that you need to create the repositories (important: with “Allow anonymous read access?” set to yes) and give the users the appropriate rights to access the SVN repositories before.

I used a server running Debian Etch and Virtualmin Pro 3.56 to build that and would love to hear what experience you have with it.

As a little side project I hacked together a Django and xmpppy based Jabber bot for Pownce. It’s implemented as a reusable Django app, using Django’s signal framework and database backend.

So, without further ado I present: pownce-jabber-bot

Please don’t hesitate to send feature requests, bug reports or become a project member!

See the Google Code project page for installation and setup instructions.

Currently available commands:

• help COMMAND […]
• message NOTE [RECEIVER]
• link URL [NOTE] [RECEIVER]
• register USERNAME PASSWORD
• unregister PASSWORD

Just a small script I got from python-nose.googlecode.com and modified to fix a strange NoneType bug: rst2wiki.py. All thanks to Jason H. Pellerin for this great script! Use it like this:

$ python rst2wiki.py < my_document.txt > my_document.wiki