How To Start a Scalable Django Project

Some people have this idea that Django is a slow and unscalable framework. Having built more than 15 Django applications with the majority of the daily active users exceeding more than 50k, we beg to defer. Yes, Django is a framework and yes there are downsides to every framework but there are multiple ways of giving your project the ability to scale.

We don’t believe in the start-up terminology build something decent now and fix/make it better later. If we have the one or two extra hours we need to make the app scalable than we will make it happen. We like to think “measure twice cut one”.

Alright, let’s get down to business. Just a heads up this tutorial will only cover these topics and is made for developers with some experience.

1. Starting a new Django 2.2.6 project.

2. Configuring your database scheme (Django models).

3. Enable backend caching data with Redis.

We don’t want to overwhelm you and quite frankly it will be too much to write all in one post but if you are interest in other topics with Django here are some you might be interested in:

1. Creating a Scaleable API with Django Rest Framework

2. Using Docker with Django, Celery, Redis, and Postgres (Local & Production)

3. Connecting a front-end framework like React to Django

4. Realtime updates and tracking with Django Channels

5. Configuring Django testing and flake for cleaner code.

Starting the project

Not going to dive into creating a Django project because the current Django documentation is great way to learn how to start a project (https://www.djangoproject.com/start/).

Here are the dependencies we use in almost every application because why should you spend time building something that has already been built. You can use pip to install all of these:

1. Django-cacheops (A slick app that supports automatic or manual queryset caching and automatic granular event-driven invalidation.) https://github.com/Suor/django-cacheops
2. Django-cleanup (The django-cleanup app automatically deletes files for FileField, ImageField and subclasses.) https://github.com/un1t/django-cleanup
3. Django-storages (django-storages is a collection of custom storage backends for Django.) https://github.com/jschneier/django-storages
4. Celery (Celery is an asynchronous task queue/job queue based on distributed message passing.) https://www.celeryproject.org/
5. Redis (Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.) https://redis.io/

Configuring your database scheme (Django models).

Now let’s create a user app and model:

python manage.py startapp users

In your users/models.py file:

import os
from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils.translation import ugettext_lazy as _
class CustomUser(AbstractUser):
username = None
email = models.EmailField(_(‘Email address’), unique=True)
USERNAME_FIELD = ‘email’
REQUIRED_FIELDS = []
def __str__(self):
return self.email

Let’s break this down. We are extending the current AbstractUser model that Django provides and removing the username field while creating an email field.

If you need the username if then just delete these lines:

username=None
USERNAME_FIELD=’email’
REQUIRED_FEIDS=[]

In order for an app to scale we like to make the scheme similar to a tree branch. Where the majority of the models are connected to a user by foreign keys or ManyToMany.

The reason for having the models set up like this is because Django’s ORM is extremely powerful and simple. For example, if we wanted to get the King model instance while we only currently have Vassal0, instead of doing a whole new call to get King with the King ID:

i_king = King.objects.get(id=king_id)

we can just do something like this

i_king = vassal0.lord0.king

Another positive is while querying a structure like this you can make it even faster and take up even fewer queries by having something like this.

I_king = Vassals.objects.select_related(‘lord0’, ‘lord0__king’)

And if you were trying to get lord0 or vassal0 from king:

I_king = King.objects.prefetch_related(‘lord_set’, ‘lord_set__vassal_set’)

Enable backend caching data with Redis.

The main dependency to help with this is django-cachops. It is by far the simplest and customizable cache dependency for Django I have come across. (https://github.com/Suor/django-cacheops) Make sure you follow the documentation to get the best out of it. You must also install Redis in order for this to work.

Automatic caching

It’s automatic you just need to set it up.

Manual caching

You can force any queryset to use cache by calling it’s .cache() method:

Article.objects.filter(tag=2).cache()

Here you can specify which ops should be cached for queryset, for example, this code:

qs = Article.objects.filter(tag=2).cache(ops=[‘count’])
paginator = Paginator(objects, qs)
articles = list(pager.page(page_num))

This just one of the ways you can speed up Django and your project overall. We suggest to always be sure to check the number of queries you are running on call. A great tool for doing this is Django Debug Toolbar. https://github.com/jazzband/django-debug-toolbar

Hopes this helps you understand the possibility of scaling and optimizing Django a bit more.