Jannis Leidel

Dear Djangonauts,

I’m a little baffled right now and I guess you all have a lot to do, so please forgive me if I’m disturbing. Going back to the begin of the Google summer of code, my mentor James and I discussed the best strategy to start implementing reusable Django-apps and the so-called “Hotclub of France” conventions (e.g. a public web repository for reusable apps). Unfortunately, a real discussion did not really take off, neither on the discussion list nor on IRC.

It now seems to me as it could just be to early for Django to have such a repository. Everybody, including me, loves to fiddle with Django in some degree and is suspicious of automatic and ready-made products. After creating ticket #6080 I’m a little tired of bringing up the same arguments again and again on the net. That’s why I wanted to ask you, if you generally see a chance for this to come before Django 1.0 and how you handle this at your projects if at all.

To continue my last post about reusable Django apps, I’d like to talk about setuptools, show you a way to install Django by using setuptools and propose the reintroduction into Django’s code, even if setuptools have been dumped by the developers not long ago due to manageability reasons. Features like entry points, egg-files, the develop command, automatic versioning and tight integration with the Python Package Index make it worth installing.

Reusable Django apps

I started django-reusableapps considering the widespread use of “setuptools” in highly dynamic Python web projects like Turbogears and Pylons and the still unresolved problem of a smart “Django apps” API. Being the responsible Google Summer of Code student for this topic, I now decided (some months after my GSoC) to go with “setuptools”, dump my GSoC code and rethink the whole reason why Django apps should be pluggable: DRY.

It’s very easy to develop a custom Django app nowadays, but quite hard to quickly create meshed up projects, manage production sites and share the involved app with a larger audience in a semi-automatic manner like the Python Package Index.

The entry points for example, enable Python applications (such as Django or any Django-based app) to provide hooks to other applications in an automatic but unobstrusive, hence powerful fashion – a perfect choice for an Application API, from my point of view.

django-reusableapps resembles the plugin loading mechanism of Trac and should be considered a very temporary solution, until Django supports app-loading from eggs natively.

Using setuptools with Django aps

To start with a “setuptools”-based Django installation you need a new setup.py file to replace the version provided by Django (note this currently only works with a current subversion checkout). Please delete the symbolic link or the django directory you might have created in the site-packages directory by following Django’s installation instructions.

Then, add the following text to the “setup.cfg” file in the django source directory:

[egg_info]
   tag_build = dev
   tag_svn_revision = 1
   

This tells “setuptools” to generate version numbers like 0.97dev-r6706 during the installation, marking it as a developer snapshot which would be superseded by a 0.97 release, once it’s released.

You can then use virtually any command of setup.py: install, develop, bdist_egg, bdist_mpkg, bdist_wininst, bdist_rpm etc.

One bonus of the new setup.py file is the instant availablitiy of a Django command, the same utility known as django-admin.py and manage.py - just by declaring a “console_scripts” entry point in the “setup()” function:

entry_points = {
       'console_scripts': 'django = django.core.management:execute_from_command_line',
   },
   

There is an easy way with “setuptools” to keep track of the changes in Django’s svn tree. Use the develop command to avoid installing Django over and over again after each subversion update:

$ cd django_src
   $ sudo python setup.py develop
   ...
   Creating /Library/Python/2.5/site-packages/Django.egg-link (link to .)
   Adding Django 0.97dev-r6704 to easy-install.pth file
   Installed /Users/Jannis/Code/django_src
   ...
   

This tells “setuptools” to create a link in the “site-packages” directory to the current directory with the Django checkout. Updating Django with subversion is also easy:

$ cd django_src
   $ svn update
   U    django/contrib/admin/templatetags/admin_list.py
   U    django/template/defaultfilters.py
   U    tests/regressiontests/templates/filters.py
   Updated to revision 6708.
   $ sudo python setup.py develop
   ...
   Creating /Library/Python/2.5/site-packages/Django.egg-link (link to .)
   Removing Django 0.97dev-r6704 from easy-install.pth file
   Adding Django 0.97dev-r6706 to easy-install.pth file
   Installed /Users/Jannis/Code/django_src
   ...
   

You need to run “python setup.py develop” after every update in order to keep the version string of Django in sync with the checkout. The already existing link to the “old” version (0.97dev-r6704) is then replaces by a link to the “new” version (0.97dev-r6706).

Django can be a dependency of a Django-based app

While you build a Django app, you might consider to include information about dependencies which are required to be installed by using setuptools’ install_requires keyword. Fortunately this also applies to Django developer snapshots, if originally installed with “setuptools” like I described above. Just add a keyword to “setup()” of your app’s “setup.py”:

install_requires = ['Django >= 0.97dev-r6706,==dev',],

This tells “setuptools” to install Django (if necessary) either by using a current subversion checkout or the newest release if available.

Django can have dependencies and extras

Django tries hard to have as less dependencies as possible by bundling some essential packages (e.g. simplejson) and informing users to install packages if required (e.g. Flup, PIL, PyYaml and all database adapters). These dependencies could also be defined in a setup.py file in the extras_require keyword:

extras_require = {
    'MySQL':  ["MySQLdb>=1.2.1p2"],
    'SQLite': ["pysqlite>=2.0.3"],
    'PostgreSQL': ["psycopg>=1.1.21"],
    'PostgreSQL2': ["psycopg2>=2.0.5"],
    'Oracle': ["cx_Oracle>=4.3.1"],
    'PyYaml': ["PyYaml"],
}

Later, while installing Django, extras could be specified in square brackets — for example one or more extras:

$ easy_install Django[PostgreSQL]
$ easy_install Django[SQLite, PyYaml]
$ easy_install http://code.djangoproject.com/svn/django/trunk/

Now imagine the same functionality with Django-based apps. Think of installing a weblog and all its recommended dependencies by running:

$ easy_install ColtraneBlog[DefaultThemes]

I hereby propose the reintroduction of “setuptools” because of the great advantages while designing a future “Django Applications” API.

Why not add the app-loading mechanism of django-reusableapps to Django’s core. Specifically add it to django.db.models.loading or another module which looks for qualified apps, either depending on INSTALLED_APPS or the “django.apps” entry point.

django-reusableapps is yet another approach on enabling Django to load reusable, pluggable, egg-based applications without changing the Django sourcecode. Think of plugins or components, e.g. django-registration, django-voting, django-tagging etc.

It uses setuptools for finding, handling and loading egg-based Python modules with a certain “entry point” (

'django.apps'

Egg-based Python modules (a.k.a. eggs) are compressed packaged Python modules like Django apps (and “[..] to Pythons as Jars are to Java…”). Every Django app can be converted to an egg distribution by using a special setup.py file.

I moved my code repository from code.google.com/p/django-package/ to websushi.org/trac/

Please also feel free to test django-package.diff with your Django trunk checkout:

1. wget http://websushi.org/bzr/django-package/django-package.diff
   

2. cp django-package.diff /path/to/django_src
   

3. cd /path/to/django_src
   

4. patch -p0 < django-package.diff
   

django-admin.py startapp

django-admin.py editapp

django-admin.py startapp --noskeleton

I also started working on the Django application repository application itself, wondering what Ross is doing at djangoapps.org?

I’m calling for your ideas about the name of the Django application repository: My proposals:

• djangopackages.com
• djangoapps.com
• djangoapps.org
• djangoapplications.com
• djangocontrib.com
• djangoforge.com
• djangoshop.com
• djangoproject.com/packages/
• djangoproject.com/apps/

just dumping ideas about the realization of a public application repository:

goals:

• central Django application repository
• integration with setuptools via django-admin.py
• easy download and searchable

requirements:

• file management (aka upload)
• community features
• style
• application based (eat your own..)

future:

• Paste hooks
• SVN/Bazaar/.. hooks

implementation:

1. standard setuptools procedures are used to register & upload the release of a django app to the Cheesshop
2. uploaded files have all the same keyword “django.contrib”
3. the homepage URL in the release.py of each django app is a sluggified url to the public repository entry (e.g. http://djangopackages.com/p/django-registration/)
4. a standalone Django application scans periodically the Cheesshop and adds new applications with the keyword “django.contrib” to the repository database (can be cron, signals and/or something else) see: cheeseshop_import.py (cheeserater.com, thanks to Jakob)
5. built with the common django-* application to provide standard community features (now on Google Code): django-registration
   django-openid
   django-voting or django-score-voting
   django-tagging
   django-utils
   django-contact-form
   typogrify
   cab (?)
   django-captcha (?)
   django-discussion (?)
   django-profile-images (?)
   
6. needed models: Package, Owner

questions

• would it make sense to wrap the register and upload process? (e.g. django-admin.py uploadapp)
• encourage BestPracticesToWorkWith3rdPartyAppsAndMakingYoursPortable !/?
• should dependencies be shown in the repository? are owners allowed to pick?
• if needed, what is the best way to administrate the repository editorially?

I’m catching up right now with my Summer of Code project after some horrible exams and have now the time to read the very interesting discussions on the Django developers list.

First, as you may know I wrote some code which changes django.core.management (django-admin.py/manage.py) to ask for basic metadata as soon as the user creates a new django application with “startapp”. The metadata gets stored in a release.py for easy manipulation. A setup.py is also provided to use any functionality of the setuptools.

Unfortunately the problem isn’t just a matter of the actual process of packaging. As pointed out in the discussion about best practices and default application layouts it’s mere the problem of giving the not-so-pro-developers the chance to use pre-built applications without forgetting the needs of the pros.

One proposal was a better documentation of best practices to help beginners understand why “projects” are good to dive into django but are awkward when it comes to distribution and deployment. A great default application scheme was written by Sebastian Macias, even if it confused me a little at first because it imitates Python’s “site-packages”..

I’m sure some of you already have solutions for distribution/deployment because it is obligatory for high traffic sites (or is it?). Server arrangements are pretty cool too, but there must be more. Django applications should be treated like applications not like websites.

Anyway the fact remains: a standard system to manage Django-based modules is required, whether we call them apps, snippets, components, extensions or plugins (think of Wordpress). I proposed to use setuptools even if a lot of people seem to be annoyed about it (why exactly?).

Turbogears and Pylons are using Paste and Setuptools in combination to provide reusability and deployment - considering the different code structure and software strategy a smart decision. Django may need some own handlers to solve the different aspects of this whole topic. Why not rethink the way Django gets deployed?

UPDATE: You may be interested in reading this: Ian Bicking’s view on the failures of setuptools and python packaging. Well then.. so then just don’t do for Django either? Naah..

This week I continued to work on the

startapp

django- 
admin.py startapp myapp

release.py, 
setup.py, MANIFEST.in, docs/, test/, myapp/, myapp/templates/myapp

Besides editing release.py manually you can now edit the meta information by running inside an application dir:

django-admin.py 
editapp

You can find a

ReleaseWrapper

django.utils.package

>> import os 
   >>> from django.utils.package import ReleaseWrapper, DEFAULT_DIRECTORIES, DEFAULT_FILES 
   >>> release = ReleaseWrapper(os.getcwd()) 
   >>> print (release.NAME, release.VERSION) 
   ('myapp', '0.1') 
   >>> release.previous_name = release.NAME 
   >>> release.NAME = "myapp2" 
   >>> release['VERSION'] = "0.2" 
   >>> release.update(os.getcwd(), DEFAULT_DIRECTORIES, DEFAULT_FILES)
   Moved: /Users/Jannis/Desktop/test_app/myapp to /Users/Jannis/Desktop/ 
   test_app/myapp2 
   Created: /Users/Jannis/Desktop/test_app/myapp2/templates/myapp2 
   Written: /Users/Jannis/Desktop/test_app/release.py 
   Written: /Users/Jannis/Desktop/test_app/setup.py 
   Written: /Users/Jannis/Desktop/test_app/MANIFEST.in
   

The application is ready to be used by setuptools, e.g. creating a zip- like “

egg

python setup.py bdist_egg

easy_install

Creating a stripped down project-based application (

__init__.py, models.py, 
views.py

django-admin.py --noskeleton startapp mysimpleapp

Please have a look at http://code.google.com/p/django-package/ for further details, full installation instructions and of course the patch.

And please, tell me what you think :)

Here is the first update on my Google Summer of Code 2007 project-a simple rip-off of my mail to the developer list. Here you can find the new django-package project with the current patch:

http://code.google.com/p/django-package/

For now the patch adds a command line switch to the startapp command of django.core.management:

# django-admin.py --package startapp myapp

It creates a basic skeleton application in the current directory, ready to be used by setuptools. For testing purposes you can use setuptools standard functions like running

python setup.py develop

python setup.py develop --uninstall

python setup.py bdist_egg

The application skeleton

I would like to collect some of your ideas about the common file names and directories which should be created by default. This is also needed when it comes to packaging non-python files like template html-files, sql-files and so on. What are the common filenames? How do you manage your django projects?

As suggested by Neil there is already a release.py for easier manipulation of the meta data. The current code creates this:

# ls
   MANIFEST.in
   docs/
   release.py
   setup.py
   test/
   myapp
   myapp/templates/
   

When introducing something like package management we also have to think about changing the workflow.

Current workflow:

1. startproject
2. startapp
3. development and testing
4. semi-automatic deployment

Proposed workflow:

1. startapp (with package skeleton and meta information)
2. develop application and test with “python setup.py develop”
3. egg deployment with “python setup.py bdist_egg” and “easy_install myapp-0.1.egg”

Generic applications like django-registration, django-tagging, django-voting, django-utils, django-contact-form, django-forum.. (more) can be installed system wide and therefore live on the python path (in python’s standard site-packages directory).

I’d like to encourage new developers to follow my workflow proposal as it makes development, packaging and deployment much easier.

Hence I suggest making it the default behaviour of

startapp

One more thing..

One hypothetic idea is to introduce a whole new management command

package

startproject

startapp

package project

package app

package egg

package upload

UPDATE: Err, wOOOt! My application was today officially accepted by Google and Django’s main developers. Thank you for your support and the confidence you have in me. – Hi. This weblog is about my summer of code 2007. Please look at my curriculum vitae to get to know me better.

My formal application for Google’s Summer of Code is as follows:

Django is one of the major web frameworks that emerged within the last years and created a solid user and developer base. It simplifies the development process by providing tools to reduce repetition, abstracting common web paradigms and still being hackable. Until today that attracted a wide section of programmers, from novices to pros.

As the code base is constantly moving towards the 1.0 release, more and more people are using Django to build small projects and websites, just as well as commercial applications like intranet services and content managment systems are realized with it, too. Good for them:

Reinventing the wheel with Django is perfectly easy – every Django beginner wrote a todo list or weblog applicaton :)

The implementation of a package system would lower the threshold for Django and Python beginners significantly because it reduces the hassle of the current installation procedure.

My task is to do the groundwork for an application which manages Django application packages and integrates tightly with the Django code.

The following steps should illustrate my considerations:

1. Infrastructure (6 weeks)

   Define hooks in the Django code to standardized python modules (e.g. setuptools, distutils, py2exe, py2app etc.) because they solve a lot of software management problems such as dependency tracking and version management. Then implement generic packaging functionality for use via django-admin.py/manage.py: CRUD, upload, download, install package.

Example of a first-time installation of a Django app:

 <strong>tichy:~#</strong> django-admin.py package search Satchmo
    Found 1 package on djangoapps.com:
    1. satchmo - The webshop for perfectionists with deadlines.
<strong>tichy:~#</strong> django-admin.py package install satchmo
    Installation directory for satchmo? [/home/jannis/django/satchmo/]
    Installing satchmo (/home/jannis/django/satchmo/)........Done.
    Syncing sqlite database with initial data...Done.
    Running tests.... 0 errors found.
    Development server is running at http://127.0.0.1:8000/
    Quit the server with CONTROL-C.