AppEngine: Python 2.7, Django Custom Template Filters without webapp, errors and more

-
Part of the move from Python 2.5 to Python 2.7 on Google App Engine, included moving from Django 0.96 to Django > 1.2. Another part of moving, for me atleast, included leaving webapp behind. I moved to Django 1.3, but leaving webapp is a work in progress and a part of the topic of this post.

Webapp Django vs Django

AppEngines's Internal Django:

Webapp uses django to handle templating, however, this django version is an internal version(Let's call this "AppEngine's Internal Django"). This internal version may not support some of the features that a regular installation of Django templates would.

As of SDK 1.7.3, webapp is located in google.appengine.ext.webapp and the internal django is located at google.appengine._internal.django.

To use templates from webapp, one is expected to do the following 
from google.appengine.ext.webapp import template

The webapp.template module is just a wrapper for Appengine's Internal Django templates. which are located at google.appengine._internal.django.template

To understand this, let us look at how Django Templates work in general.

How django.template works?
(sample code which shows how django.template works. For demonstration only. Not required for appengine. To test - Download and extract django, cd into django using terminal/cmd, run python. At the python prompt type in the following.)
from django.template import Template, Context

t = Template("<html><body>This is static data. This is {{ dynamicdata }}</body></html>")
somevariable="some value" 
c = Context({'dynamicdata':somevariable})
t.render(c)

This will output the following
This is static data. This is some value

There is support for providing external files(like "view.html"), to hold your template html, using django.template.loader.get_template.

This however involves, defining DJANGO_SETTINGS_MODULE, defining INSTALLED_APPS, TEMPLATE_DIRS, restructuring your project's folder structure etc.

To make things easier webapp.template acts as a wrapper for the above.

Using webapp.template
(sample code for demonstration, doesn't work at the command line) 
from google.appengine.ext.webapp import template
somevariable="some value" 
output = template.render('templates/view.html', {'dynamicdata': somevariable })
print output

and templates/view.html would contain the required html.

 google.appengine.ext.webapp.template uses the internal django version at google.appengine._internal.django , to do your work. This means you do not have to directly import google.appengine._internal.django.

Library Django

Apart from the partial internal version of Django, App engine also includes various libraries. As of SDK 1.7.3, Django 0.96 to Django 1.4 versions are included in the library. These versions of Django, apparently include the entire Django framework. Let us call these versions as "Library Django".
(Django is not fully compatible with Google's Datastore, but is with Google Cloud SQL.)

This library version of django, can be used by calling

import django

Which version of Django is used, depends upon app.yaml.

If the file app.yaml includes the following

app.yaml 
...
libraries:
- name: django
  version: "1.3"
...

In the above case, Django version 1.3 is used. The version number can be changed.

( Another method to import is using use_library

 from google.appengine.dist import use_library
use_library('django', '1.3')
import django

This works in Python 2.5, but not  in Python 2.7 

 On using in python 2.7, I get the following error 

 File "google_appengine/google/appengine/dist/_library.py", line 190, in DjangoVersion
    import django
ImportError: No module named django

)

The topic of this post is using Custom Template Filters with Django Templates on Google App Engine.

The ways in which this can be done are.

1. Using AppEngine's Internal Django templates (webapp.template)

2. Using Django templates from Library Django (django.template)

Example One: Using AppEngine's Internal Django templates (webapp.template)

 (My webapp implementation of custom template filters was based on an example from this 2008 blog post.)

Directory structure:
appname-root-directory/
    -filterdir/
        -customfilters.py
        -__init__.py
    -main.py
    -app.yaml
    -view.html
    -base.html  
 
app.yaml

application: appname
version: 1
runtime: python27
api_version: 1

threadsafe: true

handlers:
- url: /.*
  script: main.app

#end of app.yaml#
main.py should load webapp.template and register the filter by webapp.template.register_template_library('filterdir.customfilters'), where filterdir, is the directory where customfilters.py is located
main.py

from google.appengine.ext import webapp
from google.appengine.ext.webapp import template

webapp.template.register_template_library('filterdir.customfilters')


class Manager(webapp.RequestHandler):

  def template_print(self, template_file, template_values):
    self.response.headers['Content-Type'] = 'text/html; charset=utf-8'

    template_file_path = template_file
    res = template.render(template_file_path, template_values)
    self.response.out.write(res)


class HomePage(Manager):
  def get(self):
    template_values = {
      'hello' : "Hello World!"
    }
    self.template_print("view.html", template_values)

app = webapp.WSGIApplication(
                                     [('/', HomePage)],
                                     debug=True)

#end of main.py#

customfilters.py should import webapp, have a variable named register and register the filter 
customfilters.py

from google.appengine.ext import webapp

register = webapp.template.create_template_register()

@register.filter('addTuring') 
def addTuring(element):
  return str("Alan Turing says " + (element)) 

#end of customfilters.py#


__init__.py is an empty file
base.html

 
{% block content %}
Default Content
{% endblock content %}
 

#end of base.html#

view.html
{% extends "base.html" %}
{% block content %}
{{hello|addTuring}}
{% endblock content %}
#end of view.html#

Example Two: Using Django templates from Library Django (django.template) 

Directory structure:
appname-root-directory/
    -default/
        -templates/
            -base.html
            -view.html
         -templatetags/
             -customfilters.py
            -__init__.py
        -__init__.py
     -main.py

    -app.yaml
    -view.html
    -base.html
    -settings.py
app.yaml should mention libraries and load the required version of django. 
app.yaml
application: appname
version: 1
runtime: python27
api_version: 1

threadsafe: true

handlers:
- url: /.*
  script: main.app

libraries:
- name: django
  version: "1.3" 

# end of app.yaml #
main.py should load django.template and set os.environ['DJANGO_SETTINGS_MODULE'] to the name of the file which will contain django settings. In this case the string "settings" is used to refer to file settings.py 
main.py
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from google.appengine.ext import webapp
from django.template import Template, Context
from django.template.loader import get_template

class Manager(webapp.RequestHandler):

  def template_print(self, template_file, template_values):
    self.response.headers['Content-Type'] = 'text/html; charset=utf-8'

    t=get_template(template_file)
    c = Context(template_values)
    output = t.render(c)
    self.response.out.write(output)



class HomePage(Manager):
  def get(self):
    template_values = {
      'hello' : "Hello World!"
    }
    self.template_print("view.html", template_values)


app = webapp.WSGIApplication(
                                     [('/', HomePage)],
                                     debug=True)

#end of main.py#
settings.py should mention INSTALLED_APPS. The value should include the name of the folder containing the templatetags folder.Blank __init__.py files are required to avoid getting a "No module named" error
settings.py
INSTALLED_APPS = ('default')
#end of settings.py#
customfilters.py should import django,and register the filter using djnago.template.Library(). 
customfilters.py

from django import template

register = template.Library()

@register.filter('addTuring') 
def addTuring(element):
  return str("Alan Turing says " + (element)) 

#end of customfilters.py#
__init__.py are blank files
base.html(same in all examples)

 
{% block content %}
Default Content
{% endblock content %}
 
# end of base.html#
view.html is the view/template(which extends a base template in this case. You could paste all the html in view.html with the {{}} tags, to avoid using the extends line. ) In view.html, load the filter library using {% load library_name %, where library_name in our case is customfilters, the name of the .py file containing the filter code.
view.html 

{% extends "base.html" %}
{% load customfilters %}
{% block content %}
{{hello|addTuring}}
{% endblock content %}

#end of view.html#

There was one other way(a mix of the above two ways), which I tried, which actually made me write this post.

webapp.template with technique used in Library Django above.

If you go through, the documentation for Django Custom Tags and Filters for 1.3 or the current version of Django or the old Djnago 0.96 Documentation (put up by someone kind, on appspot), they all mention calling {% load filter_or_tag_name %} in the template. However doing that, when you are using webapp.template to call the internal django, does not seem to work. 

TemplateSyntaxError: 'customfilter' is not a valid tag library: Template library customfilter not found, tried google.appengine._internal.django.templatetags.customfilter

A google search of "tried google.appengine._internal.django.templatetags", lead me to this bug report on code.google.com .

The bad source code which lead to the error, and a poor solution follow.

Bad Code:

Directory Structure mimics the structure of Library Django example above, However, running the app returned TemplateDoesNotExist error
TemplateDoesNotExist: view.html
which meant that the application folder "default", was not being detected by the app, and hence the templates were not found. Appengine's webapp framework, searches the root of the application for templates unlike Django which searches applicationname/templates for templates and applicationname/templatetags for tags and filters. So, view.html and base.html were copied to the root folder too. 

Directory structure:
appname-root-directory/
    -default/
 -templates/
         -base.html
         -view.html
    -templatetags/
         -customfilters.py
         -__init__.py 
    -__init__.py
    -main.py
    -app.yaml
    -view.html
    -base.html
    -settings.py
    -appengine_config.py
    -base.html
    -view.html

app.yaml:
same as Library Django example

appengine_config.py

webapp_django_version = '1.3'
#end of appengine_onfig.py#

main.py is an amalgamation of different versions. One of them is below. The os.environ line is used to pointto settings.py, but it seems to have no effect.
main.py

import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

class Manager(webapp.RequestHandler):

  def template_print(self, template_file, template_values):
    self.response.headers['Content-Type'] = 'text/html; charset=utf-8'

    template_file_path = template_file
    output = template.render(template_file_path, template_values)
    self.response.out.write(output)


class HomePage(Manager):
  def get(self):
    template_values = {
      'hello' : "Hello World!"
    }
    self.template_print("view.html", template_values)

app = webapp.WSGIApplication(
                                     [('/', HomePage)],
                                     debug=True)

#end of main.py

customfilters.py is like the one from Library Django example above. Both file locations default/templatetags/customfilters.py and default/costumfilters.py( both with settings.py pointing to default )  and customfilters.py in the root folder were tried. 
customfilters.py

from django import template

register = template.Library()

@register.filter('addTuring') 
def addTuring(element):
  return str("Alan Turing says " + (element))

#end of customfilters.py#
view.html contains the {% load library_name %} tag, like the Library Django example above 
view.html


{% extends "base.html" %}
{% load customfilters %}
{% block content %}
{{hello|addTuring}}
{% endblock content %}

# end of view.html #

Explaining the error: 
TemplateSyntaxError: 'customfilters' is not a valid tag library:
Template library customfilters not found, tried google.appengine._internal.django.templatetags.customfilters

On removing the {% load customfilters %} tag from view.html,the error would change to

TemplateSyntaxError: Invalid filter: 'addTuring'
Which is expected. This means
1. The error is because of no load tag in the beginning or
2. The error is beacuse the filter is not registered.

While developing on appengine with python 2.7, I noticed, that .py files which were used by the server, would get compiled. In this example, main.pyc and appengine_config.pyc files were created. But, the settings.py file and customfilters.py(and its __init__.py file) were not compiled. Perhaps, the files were not being read at all.

If the files are not read at all, the filter would remain unregistered.

On reinstating the {% load customfilters %} tag in view.html, the original error returns.

app.yaml -> asks for django 1.3 from library. and sends control to main.py.

main.py->imports webapp.template, sets os.environ["DJANGO_SETTINGS_MODULE"]="settings"

settings.py->sets INSTALLED APPLICATIONS to default, but is apprently not read at all.

The problem, thus seems to be settings.py having no effect on webapp's internal django version.

The Fix:

The fix was to use

1. Use webapp.template to point to the tag/filter folder(filterdir.customfilters like in example one) in main.py  

main.py
...
webapp.template.register_template_library('filterdir.customfilters')  

....

2. Remove {% load library_name %} from view.html and place view.html and base.html in the root

3. Remove os.environ from main.py and remove settings.py.

4. Keep customfilters.py in a folder named filterdir, like in example one. The contents of customfilters.py is the same as example two.

This essentially meant using webapp for pointing to the filter/tag file, but registering the filter using the library version of django.

If the filter were registered using webapp, then the fix would become the same as example one.

tl;dr:
use example two

Post a comment, if you find errors in the code.

Sources:
1. Custom Template Tags and Filters
2. Third Party libraries in Python 2.5
3. Using Django Custom Helper Elements 
4. Using Custom Helper Elements in Django - daily.profeth.de

Comments

  1. UnknownJanuary 15, 2013 at 11:44 PM

    I was able to load my custom filters for all templates for solution #2 via:

    template.base.add_to_builtins('default.templatetags')

    thanks for the writeup!

    ReplyDelete

Post a Comment

Popular posts from this blog

CPU Temperature Guide for Intel Core 2 Duo : Range of Normal CPU Temperatures

-
Is your CPU Temperature too High? How high is high enough? Is 60 degree Celsius high enough or is 70 degree Celsius the threshold? Hipatic covers your temperature problems right here. Below we discuss Intel Core 2 Duo. If your CPU is overheating, visit our CPU Overheating Guide and CPU Overheating: Symptoms, Causes, & Solutions To measure CPU Temperatures you may need Real Temp Intel Core 2 Duo: (If you wish to access the PDF related to your processor, please first follow the link given under sources at the end of the article instead of directly accessing the pdf link given under each sub heading. This will help you access the latest information on your processor from Intel's website & will also prevent hotlinking.) 1. Intel® Core™2 Duo Processor E8000 Series Intel ® Core™2 Duo Processor E8600, E8500, E8400, E8300, E8200, E8190 Thermal Profile Range: 45.1 degree Celsius (Power - 0W) to 72.4 degree Celsius (Power 65W) Source: http://download.intel.com/design/processor/dat
Read more

CS50: "undefined reference to `GetString' " and other errors

-
Image
I was going through the CS50 lectures of Fall 2011. CS50 recommends using the CS50 Appliance which is a VirtualBox installation of Fedora with libraries and software needed for the CS50 course. However, if you are already running a flavour of Linux (Ubuntu 12.04 in the case of this article) and want to do the examples natively, read on. On Linux, the CS50 manual gives instructions on how to install the CS50 Library for C  for Linux ( Fedora / Ubuntu ). After installing the CS50 Library for C, I tried running some samples. While making pointers.c in the Week 5 lecture, I came across this error $make pointers cc     pointers.c   -o pointers pointers.c: In function ‘main’: pointers.c:27:5: error: ‘for’ loop initial declarations are only allowed in C99 mode pointers.c:27:5: note: use option -std=c99 or -std=gnu99 to compile your code make: *** [pointers] Error 1 If instead of using make, you use gcc pointers .c you will get a similar error. To correct this, I ran $gcc pointers.
Read more

Appengine: Custom Error Handlers using webapp2 - Python 2.7

-
Google Developers has a writeup on Custom Error Responses using app.yaml Here is a sample app.yaml app.yaml application: appname version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /static static_dir: static http_headers: Vary: Accept-Encoding - url: /articles/.* script: articles.app On the Development Environment, I get the following error when I visit http://localhost:8080/ i.e the home page Not found error: / did not match any patterns in application configuration. On Appengine, a standard 404 Error is shown Error: Not Found The requested URL / was not found on this server. This is because the app.yaml file does not specify any handler for "/" Here is the articles.py file articles.py import webapp2 class HomePage(webapp2.RequestHandler): def get(self): self.response.write.out("Hello Birds!") app = webapp2.WSGIApplication( [ (r'/articles/', HomePage) ], debug=True) Specify
Read more