I first started using Django when I started at VendAsta and this issue didn’t creep up there because we are use Google App Engine and its DataStore on the backend instead of a traditional database. With no database there was no issue with creating/dropping anything. It was only when I started using Django on my own that I came across that problem. There was one more issue with Django that I came across only when I started doing things outside of work: Django expects all of your models to be in a single file called models.py. I hate that. A lot.

I did some googling about the problem and found that there were a bunch of other people who didn’t like it much either. There were a few solutions but they all involved manually mucking about with the __all__ attribute in the __init__.py file of your models directory. Sure, it works, but I don’t like having to define a model in two places to use it. Oddly enough it was in some test code at work that I found the answer.

For our project at VendAsta we are using the Django App Engine Helper to provide a nice easy interface to the DataStore on the backend. A part of this is a small __init__.py file which allows you to spread your tests out in multiple files. Django likes its tests to be in a single tests.py file just like it likes its models to be in models.py. The wee bit of Google code is shown in Listing 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#!/usr/bin/python2.4
#
# Copyright 2008 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
 
 
 
"""Loads all the _test.py files into the top level of the package.
 
This file is a hack around the fact that Django expects the tests "module" to
be a single tests.py file and cannot handle a tests package inside an
application.
 
All _test.py files inside this package are imported and any classes derived
from unittest.TestCase are then referenced from this file itself so that they
appear at the top level of the tests "module" that Django will import.
"""
 
import os
import re
import types
import unittest
 
PACKAGE = 'appname.tests'
TEST_RE = r"^.*_test.py$"
 
# Search through every file inside this package.
test_names = []
test_dir = os.path.dirname( __file__)
for filename in os.listdir(test_dir):
  if not re.match(TEST_RE, filename):
    continue
  # Import the test file and find all TestClass clases inside it.
  test_module = __import__('%s.%s' % (PACKAGE, filename[:-3]), 
                           {}, {},
                           filename[:-3])
  for name in dir(test_module):
    item = getattr(test_module, name)
    if not (isinstance(item, (type, types.ClassType)) and
            issubclass(item, unittest.TestCase)):
      continue
    # Found a test, bring into the module namespace.
    exec "%s = item" % name
    test_names.append(name)
 
# Hide everything other than the test cases from other modules.
__all__ = test_names

Listing 1: __init__.py file from Google for a tests module

Essentially what this file does is loop through all files in the same directory and for each on that matches the pattern listed on line 40 (in this case, files that end in ‘_test.py’) and import all of the Classes from those files that are subclasses of the TestCase class. Finally, the code on line 62 does the magic with __all__ that I wanted to avoid doing manually for my models. It was this line that caught my attention and started me wondering if this code could be used for spreading models across multiple files.

In order for this to work for models there are a few things that need to be changed. First of all, on line 40, the pattern should be changed to match on *.py to grab all of the python files in the directory. Then, on line 46, add

or filename == “__init__.py”

to ensure that we don’t try to import the file we are in. Finally, remove the subclass requirement from line 55, as we are just concerned with importing classes here not what they inherit from. Putting it all together results in Listing 2. Place that in the __init__.py of a directory called models and you’ll not have to manually manage the __all__ list for your models in separate files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/usr/bin/python2.4
 
import os
import re
import types
import unittest
 
PACKAGE = 'my_app.models'
MODEL_RE = r"^.*.py$"
 
# Search through every file inside this package.
model_names = []
model_dir = os.path.dirname( __file__)
for filename in os.listdir(model_dir):
  if not re.match(MODEL_RE, filename) or filename == "__init__.py":
    continue
  # Import the model file and find all clases inside it.
  model_module = __import__('%s.%s' % (PACKAGE, filename[:-3]),
                           {}, {},
                           filename[:-3])
  for name in dir(model_module):
    item = getattr(model_module, name)
    if not isinstance(item, (type, types.ClassType)):
      continue
    # Found a model, bring into the module namespace.
    exec "%s = item" % name
    model_names.append(name)
 
# Hide everything other than the classes from other modules.
__all__ = model_names

Listing 2: __init__.py file modified for a models module

Now, there is one last thing that you are going to need to do in order for everything to run properly in Django. It is a very simple addition to the end of each of your classes. I have seem some people complaining about the need to do this in various forums, but honestly, if you are using any form of modern text editing application, it should only take you about 2 minutes to create a new template for a model that has the little piece of code shown in Listing 3 at the bottom of the class (replace your_app_name with your actual app name).

1
2
3

class Meta:
        """Meta Class for your model."""
        app_label = 'your_app_name'

Listing 3: Meta class required at the end of each model for proper functioning.

The first hurdle is actually getting a Django site running on Dreamhost. Luckily, there is this blog post which details exactly how to get your site up and running on Dreamhost. I have used this method a couple of times (once modified to get a Satchmo install running) and have met with success each time.

Once the site is running, setting up tests poses another problem. The default test behaviour for a Django site creates and then destroys a whole test database on each test run. Normally this is not a problem, but the way Dreamhost has their MySQL setup running, the commands to create and drop databases are not available outside of their administration panel. What follows is the solution I have come up with and am currently using to run Django unittests with my Dreamhost MySQL databases.

First of all, it is important to understand a little about the default testing setup that Django uses. When you run the command python manage.py test, Django looks at the TEST_RUNNER setting to determine what to do. By default this setting points to django.test.simple.run_tests. A quick look and we can see that this is the functionality we want to alter. Specifically, lines 2 and 4 shown in listing #1 are what we need to replace.

1
2
3
4

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)
result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
connection.creation.destroy_test_db(old_name, verbosity)

Listing 1: Excerpt from django.test.simple.run_tests

It is these lines which make the calls to create and then drop the test database. The functionality needed, however, in order to work with Dreamhost’s setup, is to “flush” and existing test database both before and after the test run is complete. I have accomplished this through the creation of a method called flush_test_db, shown in Listing #2.

1
2
3
4
5
6
7
8
9
10
11

def flush_test_db(test_db):
    """Method to flush out the test database, removing tables, before
    and after test runs"""
 
    from django.db import connection 
    cursor = connection.cursor() 
 
    current_tables =  connection.introspection.table_names()
    for table in current_tables:
        cursor.execute("use %s" % test_db)
        cursor.execute("drop table %s" % table)

Listing 2: New method to remove all tables from the test db

With this method, we can drop all of the tables in the test db both before and after a test run. It is necessary to do both just in case something goes horribly awry during a test run and we end up with leftover data from a previous test run. With this new method, we can then create a new version of the run_tests method and place both methods into a new file. I called my file test_runner.py and placed it in a tests directory within my app. The complete test_runner.py is shown in Listing 3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

"""Override of the simple tests module from django in order to skip
database creation and use an existing db"""
import unittest
from django.conf import settings
from django.db.models import get_app, get_apps
from django.db.backends.creation import TEST_DATABASE_PREFIX
from django.test import simple
from django.test.utils import setup_test_environment, teardown_test_environment
from django.core import management 
 
def flush_test_db(test_db):
    """Method to flush out the test database, removing tables, before
    and after test runs"""
 
    from django.db import connection 
    cursor = connection.cursor() 
 
    current_tables =  connection.introspection.table_names()
    for table in current_tables:
        cursor.execute("use %s" % test_db)
        cursor.execute("drop table %s" % table)
 
def run_tests(test_labels, verbosity=1, interactive=True, extra_tests=[]):
    """
    Replacement for the run_tests in Django.
    """
    setup_test_environment()
 
    settings.DEBUG = False    
    suite = unittest.TestSuite()
 
    if test_labels:
        for label in test_labels:
            if '.' in label:
                suite.addTest(simple.build_test(label))
            else:
                app = get_app(label)
                suite.addTest(simple.build_suite(app))
    else:
        for app in get_apps():
            suite.addTest(simple.build_suite(app))
 
    for test in extra_tests:
        suite.addTest(test)
 
    old_name = settings.DATABASE_NAME
 
    if settings.TEST_DATABASE_NAME:
        test_database_name = settings.TEST_DATABASE_NAME
    else:
        test_database_name = TEST_DATABASE_PREFIX + settings.DATABASE_NAME
 
    #replace the normal DATABASE_NAME with the name of the test db
    settings.DATABASE_NAME = test_database_name
 
    #flush the test db
    flush_test_db(test_database_name)
    #load the test db with models
    management.call_command('syncdb')
 
    result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
 
    #flush the test db again
    flush_test_db(test_database_name)
 
    #reset the value of DATABASE_NAME in settings
    settings.DATABASE_NAME = old_name
    teardown_test_environment()
 
    return len(result.failures) + len(result.errors)

Listing 3: The complete test_runner.py

The final step is to point the test framework to your new test runner by placing a new line in your settings.py. The exact value will vary based on where in your project you have placed the new test runner. In this example I have placed the test_runner.py file, containing the run_tests method in the tests directory of an app called ‘your_app’. This is shown in listing 4.

1

TEST_RUNNER = 'your_app.tests.test_runner.run_tests'

Listing 4: Settings.py setting pointing the test framework towards your new test_runner

Finally a few words of caveat. I have omitted any comments I am using to tell pylint to ignore certain things and I have done little in this sample code to properly handle Exceptions that might arise. Nevertheless, this should get you a good start on running unit tests in Django if you host your site with Dreamhost.