KodoBlog | tech practice & philosophy

Dynamic ModelForm creation

Sat, 30 Jul 2011 18:57:00 +0400

That looks amazing!

def get_model_form_class(model_class, fields_list=None, exclude_list=None):
    class form_class(forms.ModelForm):
        class Meta:
            model = model_class
            fields = fields_list
            exclude = exclude_list
    return form_class

The idea taken from: http://stackoverflow.com/questions/297383/dynamically-update-modelforms-meta-class/297478#297478

Shortcut for printing user's full name or username in template

Tue, 20 Jul 2010 02:25:59 +0400

That’s it:

from django import template

@register.filter
def nice_name(user):
    """
    Example::

        Hi, {{ user|nice_name }}
    """
    return user.get_full_name() or user.username

Great setup example for Nginx + FastCGI + Mercurial hgwebdir

Tue, 13 Jul 2010 14:04:29 +0400

Great setup example for Nginx + FastCGI + Mercurial hgwebdir:

It also uses fabfile for automation of start / restart process.

Seems like i’m in trend. The idea to build decentralized network comes to me again and...

Wed, 07 Jul 2010 10:10:00 +0400

Seems like i’m in trend. The idea to build decentralized network comes to me again and again.

And it seems, we’ll see first p2p social networks soon. Wuala is running. Diaspora is coming. Actually, their start is amazing: they’ve raised investments on Kickstarter and made friends with Pivotal Labs.

I’m working on social network prototype at the moment. The key feature although is not p2p at all. Another major trend is: there’s too much information. So, we need filter and customize information flows completely. So, custom design matters :)

thecodefarm team

Sat, 19 Jun 2010 23:13:00 +0400

thecodefarm team:

I’m going to play with Dajax.

It’s created by Jorge Bastida from thecodefarm team. I like them. :)

How to extract html page title by URL

Fri, 04 Jun 2010 16:09:00 +0400

Actually the subject can be divided into two tasks:

retreive data
extract information from it

There’s standard library urllib2 in Python for retreiving data over HTTP and a number of libraries for parsing HTML data. I’ll use html5lib in this example.

First iteration of retrieving data

import urllib2

def read_url(url):
    try:
        response = urllib2.urlopen(url)
    except urllib2.URLError:
        return u''
    encoding = get_charset(response.headers)
    return unicode(data, encoding)

We need extra utility function get_charset:

def get_charset(headers, default='utf-8'):
    try:
        content_type = headers['content-type'].lower()
        if content_type.find('charset=') > 0:
            return content_type.split('charset=')[-1].lower()
    except KeyError:
        pass
    return default

Now we can get data!

>>> d = read_url('http://python.org')
>>> d[:50]
u'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Trans'

Seems like that’s what wee need.

Extracting title with html5lib

There are examples for it: http://www.sal.ksu.edu/faculty…

Here’s extractor function based on that examples:

from html5lib import HTMLParser, treebuilders, treewalkers

parser = HTMLParser(tree=treebuilders.getTreeBuilder("dom"))
walker = treewalkers.getTreeWalker("dom")

def extract_title(html):
    domtree = parser.parse(html)
    titleNode = False
    title = u''
    for token in walker(domtree):
        if token['type'] == 'StartTag' and token['name'] == 'title':
            titleNode = True
        elif titleNode:
            if token['type'] == 'EndTag' and token['name'] == 'title':
                break
            elif token.has_key('data'):
                title += token['data']
    return title.strip()

Let’s try!

>>> extract_title(d)
u'Python Programming Language -- Official Website'

Amazing! That’s working!

Optimization, possibly

The one drawback of extraction method above is that page has to be completely downloaded and parsed for title extraction. I’ve tried to optimize it: read HTTP data just until title data is read.

Here’s read_url function revisited. It’s designed to read data by chunks until specified string is met.

import re
import urllib2

def read_url(url, until=None, chunk=100):
    try:
        response = urllib2.urlopen(url)
    except urllib2.URLError:
        return u''

    encoding = get_charset(response.headers)

    if until:
        next, data, trunk_at = True, '', None
        while next:
            next = response.read(chunk)
            data += next
            until_match = re.search(until, data, re.IGNORECASE)
            if until_match:
                response.close()
                data = unicode(data, encoding)
                return data[:data.find(until) + len(until)]
    else:
        data = response.read()
    return unicode(data, encoding)

So, we can now read until </title>!

>>> d = read_url('http://python.org/', until='</title>')
>>> d
u'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html xmlns
="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n\n<head>\n  <meta http-equiv="content-type" content="text/html; charset=utf-8" />\
n  <title>Python Programming Language -- Official Website</title>'

Let’s test perfomance. The very-very basic test looks like:

def test():
    from time import time

    t1 = time()
    d1 = read_url('http://python.org/', until='</title>')
    t2 = time()

    t3 = time()
    d2 = read_url('http://python.org/')
    t4 = time()

    print t2-t1
    print t4-t3

Results:

>>> test()
0.131000041962
0.31500005722
>>> test()
0.12700009346
0.318000078201
>>> test()
0.125999927521
0.31299996376

Optimized extractor shown considerable faster results.

That’s it

Other HTML parsing libraries are mentioned here.

Django Widget for CommaSeparatedIntegerField with pre-defined choices

Fri, 04 Jun 2010 14:27:00 +0400

Django Widget for CommaSeparatedIntegerField with pre-defined choices:

Assume, we need a registration form for some event with time frame for 3 days, starting at Monday. User can select any of 3 days, so we need to show 3 checkboxes.

Here’s the basic example for such form:

class EventRegistrationForm(forms.Form):
    days = forms.CharField(widget=NumbersSelectionWidget(
                    ['Mon', 'Tue', 'Wed'], range(1, 4)))

Unicode 'funny characters'

Sun, 23 May 2010 10:16:00 +0400

There’re characters that sometimes cause strange behaviour, when trying to print them to console.

It seems that depends on environment and Python compilation. I’ve tested it on Windows Vista and it failed, than it worked on some *nix machines and failed on others. I used Python 2.6.x version. So, it’s possible you will be unable to rebpoduce it!

An example

Russian character ‘ы’ has code U+044B, symbol ‘©’ has code U+00A9.

    >>> a = u'ы'
    >>> a
    u'\u044b'
    >>> b = u'\u00a9'
    >>> b
    u'\xa9'

Trying to print:

    >>> print a
    ы
    >>> print b
    ...
    UnicodeEncodeError: 'charmap' codec can't encode character u'\xa9' in position 0: character maps to <undefined>
    >>>

Try to write to file:

    >>> f = open('test.txt', 'w')
    >>> f.write((a + b).encode('utf-8'))
    >>> f.close()

That’s working ok!

What to do

Walk wide.

I’ve found a suggestion to remove such charecters from text: http://code.activestate.com/recipes/546517-accent2htmlcodepy-convert-accents-and-special-char/

Possibly that’s not best solution, but may be nesseccary, if you want to print text with ‘funny characters’ to console.

_spec_chars = [u'\xc1',u'\xe1',u'\xc0',u'\xc2',u'\xe0',u'\xc2',u'\xe2',u'\xc4',u'\xe4',u'\xc3',u'\xe3',u'\xc5',u'\xe5',u'\xc6',u'\xe6',u'\xc7',u'\xe7',u'\xd0',u'\xf0',u'\xc9',u'\xe9',u'\xc8',u'\xe8',u'\xca',u'\xea',u'\xcb',u'\xeb',u'\xcd',u'\xed',u'\xcc',u'\xec',u'\xce',u'\xee',u'\xcf',u'\xef',u'\xd1',u'\xf1',u'\xd3',u'\xf3',u'\xd2',u'\xf2',u'\xd4',u'\xf4',u'\xd6',u'\xf6',u'\xd5',u'\xf5',u'\xd8',u'\xf8',u'\xdf',u'\xde',u'\xfe',u'\xda',u'\xfa',u'\xd9',u'\xf9',u'\xdb',u'\xfb',u'\xdc',u'\xfc',u'\xdd',u'\xfd',u'\xff',u'\xa9',u'\xae',u'\u2122',u'\u20ac',u'\xa2',u'\xa3',u'\u2018',u'\u2019',u'\u201c',u'\u201d',u'\xab',u'\xbb',u'\u2014',u'\u2013',u'\xb0',u'\xb1',u'\xbc',u'\xbd',u'\xbe',u'\xd7',u'\xf7',u'\u03b1',u'\u03b2',u'\u221e']

def cleanspec(s, cleaned=_spec_chars):
    return ''.join([(c in cleaned and ' ' or c) for c in s])

Try print cleaned text:

    >>> print cleanspec(b + a)
     ы

That is workaround. May be, that’s an issue for Python’s print, I’m not quite sure about that.

That’s it :)

Next order value for Django model instance

Sat, 22 May 2010 01:39:00 +0400

Assume, we have Django model with special weight integer field for ordering. We may want to assign its value automatically on save. Here’s the snippet implementing such behaviour:

class MyModel(models.Model):
    # some fields...
    weight = models.IntegerField(default=0)

    class Meta:
        ardering = ('weight',)

    def save(self, *args, **kwargs):
        self.weight = get_next_value(self, field_name='weight')
        super(MyModel, self).save(*args, **kwargs)

Here’s the get_next_value implementation:

def get_next_value(instance, field_name='order', step=10, **filter):
    model = instance.__class__
    qs = model.objects.order_by('-%s' % field_name)
    if filter:
        qs = qs.filter(**filter)
    try:
        max_value = getattr(qs[:1][0], field_name, 0)
    except IndexError:
        max_value = 0
    return (max_value / step + 1) * step

The implementation above can handle any field name with specified step. It can also provide next field value for filtered queryset.

Provided snippet is a part of halfbit-web-helpers collection.

Smarter Django cart

Mon, 17 May 2010 16:57:00 +0400

Introduction

django-carting is a basic online store application for Django. It is designed in a sketchy manner to be like a rewritable application.

Demo is available:
http://carting-demo.05bit.com

Concept

It’s conceptually differs both from Satchmo and LFS projects. Basically, that’s just a “cart application” with utilities, which can used to build full-featured online store.

The two points are:

products catalog is always too custom to be customized
design and html layouts are usually totally new for commercial project

So, that parts should be rewritten from scratch each time.

Feature

Cart is smartly binded to user or session. It binds to authenticated user or stores in session for anonymous user. So, cart is remebered for authenticated user and does not dissappear when session expires.

Projects powered by

Smarter Django project configuration

Sun, 09 May 2010 11:45:00 +0400

Two points

Generally we need different settings for Django project in development environment on localhost and on production environment. Settings may differ also across different development environments when application is developed by many programmers.
Project source is good to be stored in some VCS repository, so settings should also be stored there. Settings modifications that are common for different environments should be automatically applied after updating from repository.

How to

Here’s a couple of recipes on how to configure Django project in a bit smarter way.

1. Define settings template

Just make a copy of settings.py to settings_template.py and then import settings_template from settings:

from settings_template import *

Then you may override some settings, i.e. DUBUG flag or database settings etc.:

from settings_template import *

DEBUG = False

Note: now settings.py should not be stored in repository, but settings_template.py should be stored there.

2. Define paths based on project directory

I know many good programmers do that :) Add somewhere at start of settings, I mean settings_template.py:

import os

PROJECT_ROOT = os.path.realpath(os.path.dirname(__file__))

Now you can define media and template paths like this:

MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')

MEDIA_URL = '/media/'

ADMIN_MEDIA_PREFIX = '/media/admin/'

TEMPLATE_DIRS = (
    os.path.join(PROJECT_ROOT, 'templates'),
)

3. Configure serving static files

Django documentation offers to serve on not to serve static files depenging on DEBUG flag. Sometimes that may not be correct, as we may need debug mode on production environment, where static files are served by webserver.

I offer to use separate SERVE_STATIC flag in settings for this. So, here is urls.py example:

from django.conf import settings

# some urls config ...

if getattr(settings, 'SERVE_STATIC', False):
    urlpatterns += patterns('',
        (r'^' + settings.MEDIA_URL[1:] + r'(?P<path>.*)$',
         'django.views.static.serve',
         {'document_root': settings.MEDIA_ROOT})
    )

So, when I need to serve static files by development server, I just add SERVE_STATIC = True to settings.py.

Feedbacks

Please give a feedback, if it was useful or not. I hope it was :) If you have your own recipes on subject, you’re welcome to share links or snippets.

Python logging example and helper

Tue, 27 Apr 2010 09:40:00 +0400

When trying to find simple example on how to use Python logging module for writing logs to file, I became frustrated, as there are only a few useful examples. Here’s one that really works and covers most of use cases: see it on mechanicalcat.net.

And here’s simple helper to open log file, based on that example:

import re
import logging

def openlog(filename, logger_name=None, level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s'):
    if not logger_name:
        logger_name = filename[filename.rfind('/')+1:]
    logger = logging.getLogger(logger_name)
    handler = logging.FileHandler(filename)
    formatter = logging.Formatter(format)
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    logger.setLevel(level)
    return logger

So, you can simply use this helper like this:

logger = openlog('/var/log/just_a_test.log')
logger.debug("hello, i'm debug message!")

Actually, basic examples are provided in docs. I missed them first time, because reference seems to be complicated at start point.

Flavoured Markdown

Mon, 26 Apr 2010 19:47:00 +0400

Markdown is good, but it has 2 features that’s hard to explain to inexperienced user:

it doesn’t convert urls to links automatically
it doesn’t convert line breaks to html <br> tags

It seems, for 80% cases this would better be done. Python Markdown supports extensions and thus can be “flavoured”.

So, there are extensaions:

autobr for converting line breaks and
autolink for converting urls

Automatically converting urls to html links

Sun, 25 Apr 2010 19:13:00 +0400

Here’s simple Python snippet for converting urls to links in single text line:

import re

link_re = re.compile('(\s+\(?|^)((http|ftp|https)://[-\w\#$%&~/.;:=,?@+]+)', re.IGNORECASE)

def autolinks(line):
    return link_re.sub(r'\1<a href="\2" target="_blank">\2</a>', line)

It’s simple yet smart enough:

doesn’t touch urls inside html links <a href=”some url”>some text</a>
can be used with Markdown, as it doesn’t touch Markdown links

To enable auto links replacement in Markdown, you may use autolink extension.

Highway to brilliant wiki, part 1. Brillixy.

Sat, 24 Apr 2010 19:13:00 +0400

Hey!

I am unsatisfied.

It’s 21st century, the age of perfect web-frameworks and rapid development. But there’s no good wiki-like engine to satisfy reasonable requirements.

There are Trac, DokuWiki and many others and others. By the way, I really appreciate Trac and DokuWiki. But I need less. And more.

So, I just need an engine which:

has wiki-like basis: users can create, edit and delete pages,
has hierarchical pages structure,
can be easily themed, design is really matters,
is written in Python / Django and uses best web-development practices.

Brillixy

I started project to make such engine. I called it “Brillixy”. It’s hosted here: http://bitbucket.org/rudi/brillixy

Some design goals are defined:

keep specific part as tiny as possibe, reuse proven Django applications, write new ones for generic tasks
it should has multiple-project structure, so single installation may be used by some team for a number of projects

And some design decisions are done:

use django-treebeard for storing pages hierarchically
use django-tagging for tagging
use markdown as a basis for markup language, use flavours for it, use autotext to provide specific syntax

What’s next?

I’m planning to release v0.1 with most basic functions:

creating, editing, deleting pages
commenting pages
simple tagging, no AJAX
simple permissions scheme, allow everything to authenticated users
no changesets view at this point

It should be enougth to start use it for some project. I’m planning to complete it before 9 May 2010.

Rewritable Django apps

Fri, 23 Apr 2010 15:24:00 +0400

The common good approach in designing applications for Django is write them to be reusable. Another approach is to build rewritable applications on top of reusable ones.

What’s the point?

There’re number of specific tasks, which can be splitted to generic blocks. But it seems, some “generic” blocks better not to be generic by theirs nature, because they need “a little bit specific” usage for every project.

We can build such apps in two ways:

make big reusable application with customization settings or
build rewritable application

Reusable vs Rewritable

Here’s comparison table of approaches.

Reusable	Rewritable
Try to cover most generic use cases “out-of-box” and provide settings	Set of “sketches” to provide basic use cases with examples for usages and possible extendings
Source code is redundant as it covers different use cases	Source code is minimalistic
Usually depends only on Django and extra generic Python libs	Usually are build on top of other proven applications
You’d better not modify application source	Application source can be rewritten or extended using provided basis

Examples

Reusable:

django-tagging
sorl-thumbnail
and many other Django apps

Rewritable:

django-accounts-basic
(… actually, it’s hard to find examples, as that’s not common practice to publish such apps, but they MAY be useful! Here may be: basic e-shop application, basic wiki application, etc. Yes. I know, there ARE e-shops and wikis, but they are designed to be reusable, not to be rewritable).

Conclusions

Both approaches are for applications made by programmers for programmers.

Reusable applications are good for providing “out-of-box” solutions for generic tasks, they are rather “solid”, so fine customization is hard or can cause overcomplication.

Rewritable applications are good for providing code basis on top of reusable apps. They must be clean and as small as possible and it’s critical to provide working examples.