Saturday, November 17, 2012

AppEngine: Intermittent DJANGO_SETTINGS_MODULE is undefined error on Django with Python 2.7 on Google App Engine

Ever since moving from webapp to Django on Python 2.7, I was plagued with the following error, albeit intermittently. Whenever, I would flush my memcache or start a new instance or run into an unhandled exception, my appengine based app wouldn't start. Logs gave me the following error,

Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined. 

I searched far and wide for a solution, looked up topics on stackoverflow. The DJANGO_SETTINGS_MODULE is a common problem for django novices like yours truly and stackoverflow had umpteen questions and answers regarding the issue, but I couldn't find the right answer.

Well, let me start off with a piece of my bad code.

I looked up the official documentation for pointers when I started the migration from webapp templates to purer Django templates on my appengine project. The Django Documentation on Settings page mentions the following two options to configure django settings on a project(apart from using django-admin.py, if you are running a fuller django stack, instead of just django templates)

Option One: 

In your script file(the file which handles your urls as mentioned in your app.yaml) for eg. main.py, add this at the very top before calling anything Django. The Django Documentation on settings page describes this in context of mod_wsgi.

import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'


Option Two: 

In your script file add this at the top before using anything django
from django.conf import settings

settings.configure(DEBUG=True, TEMPLATE_DEBUG=True,
    TEMPLATE_DIRS=('/home/web-apps/myapp', '/home/web-apps/base'))

Now, my settings file looks like this, as you may know from my previous posts. This settings.py is loaded when you set the DJANGO_SETTINGS_MODULE to the string 'settings'
settings.py
INSTALLED_APPS = ('myapp')
Now, all this does is asks django to find a folder named myapp in my root application folder, for all the files that django templates needs.
Folder structure
applicationfolder(root)/
--myapp/
  __init__.py
  --templates/
    --template.html
--app.yaml
--main.py
--settings.py


For eg. all of the template files should go under the folder myapp, in a folder called templates, the tags, in a folder called templatetags and so on. So far, so good.

As I didn't know how to configure the second option for appengine, as we have no idea what the absolute folder paths will be for our appengine applications, I decided to use option one.

After I had just moved to django templates from webapp, templates wouldn't render as I had not defined the settings module and I would get the DJANGO_SETTINGS_MODULE is undefined error 100 percent of the time and naturally so.

So I added the settings as per option one in my main.py.

main.py
import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

#followed by other imports

This worked rather well in the beginning. My templates started to render and I had no issues, till I ran into an unhandled exception when testing my site on the sdk's development server. After I fixed the code which was causing the exception, I got the DJANGO_SETTINGS_MODULE is undefined error which I would then go on to dread for the next day or so.

I restarted the server and the error disappeared and my templates were rendering again, without changing the code.

Then, when I flushed my memcache while working on my app, I once again got the DJANGO_SEETINGS_MODULE is undefined error. I restarted my server and the error disappeared.

Then, I deployed the code on appengine and noticed my app would run for sometime, but would eventually stop serving with the aforementioned error.

Here is a stack trace from appengine's log, with appname as my application name

Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/apps/s~appname/1.766876876868686/main.py", line 807, in dispatch
    webapp2.RequestHandler.dispatch(self)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~appname/1.766876876868686/main.py", line 1417, in get
    self.template_out(tmpl, tvals)
  File "/base/data/home/apps/s~appname/1.766876876868686/main.py", line 885, in template_out
    dtemplate=get_template(path)
  File "/base/python27_runtime/python27_lib/versions/third_party/django-1.4/django/template/loader.py", line 145, in get_template
    template, origin = find_template(template_name)
  File "/base/python27_runtime/python27_lib/versions/third_party/django-1.4/django/template/loader.py", line 127, in find_template
    for loaader_name in settings.TEMPLATE_LOADERS:
  File "/base/python27_runtime/python27_lib/versions/third_party/django-1.4/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/base/python27_runtime/python27_lib/versions/third_party/django-1.4/django/conf/__init__.py", line 40, in _setup
    raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.

I didn't understand much, other than that the code mentions a line from my main.py,

    dtemplate=get_template(path)

which is how I get the template file's path.

A solution to this error on the live environment was to shutdown the instance, the new instance would mostly start serving normally, until the error would pop up again.

I have now(apparently) fixed this, which I will describe below, but I am not sure about the why's and how's of the error.

Finding a solution:

I looked up questions on stackoveflow ( which is now the official place to ask questions regarding google appengine (use the official google-app-engine tag) ), but couldn't find anything that fit. I went back to the official documentation, which gave the following way to check for whether the environment variable DJANGO_SETTINGS_MODULE is set or not

from django.conf import settings
if not settings.configured:
    settings.configure(myapp_defaults, DEBUG=True)

The syntax used is the one from option two, so I modified it to,

import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

from django.conf import settings
if not settings.configured:
  os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

Which did not make any sense to me and did not work. The app would work for sometime and then would get into an error again.

Deleting the compiled ( .pyc ) files and restarting the app, gave me some more running time, but the fix was not permanent. I started searching for code samples on code.google.com/codesearch which could give me insights on how other users use this on their appengine apps, I found some code samples, but no solution.

Searching for working Djnago implementations on appengine, brought me to this post by Ed Crewe on migrating his app from Python 2.5 to Python 2.7 which led me to the fix.

The Fix:

In Ed Crewe's post, he is using the same,

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'


which is option one above, to set his settings in his new python 2.7 app, which was causing me all the trouble(I wonder if it is causing him any...), but in the code for his Python 2.5 version of the app he does this

import os
from google.appengine.dist import use_library

use_library('django', '1.2')
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

from django.conf import settings

# Force Django to reload its settings.
settings._target = None

#code adapted from http://edcrewe.blogspot.com/2012/06/upgrading-google-app-engine-django-app.html,

There is no use_library in Python 2.7(The django version is specified using the libraries handler in app.yaml), so the code for Python 2.7 looked like this on my app

main.py (fixed)

import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

from django.conf import settings

settings._target = None

The line in bold is what works the wonder. It, apparently, forces django to reload the settings.

The application now works and the dreaded error has not been heard from again. I did not know about settings._target, but searching for the term on google leads me to plenty of results, some like this post from Thomas Brox Rost on setting up django on the then young appengine in 2008

tl;dr
see main.py (fixed)

Please post a comment if you spot an error in the code.
Sources:
1. Ed Crewe: Upgrading a Google Django App
2. Django Documentation: Settings

No comments:

Post a Comment