Learn how to integrate ElasticSearch into a Django (v1.9.3) codebase.

The tutorial uses a custom project template that is no loger available, so recreating manually to match tutorial structure.

Target Structure

This is the iproject structure the tutorial uses (not a typical setup for Django).

.
├── README.md
├── project/
│   ├── apps/
│   │   └── core/
│   │       ├── admin.py
│   │       ├── apps.py
│   │       ├── management
│   │       │   └── commands
│   │       │       ├── dummy-data.py
│   │       │       └── push-to-index.py
│   │       ├── migrations
│   │       │   └── 0001_initial.py
│   │       ├── models.py
│   │       ├── urls.py
│   │       └── views.py
│   ├── conf/
│   │   ├── __init__.py
│   │   └── base.py
│   ├── manage.py
│   ├── static/
│   │   ├── bootstrap-theme.min.css
│   │   ├── bootstrap.min.css
│   │   ├── bootstrap.min.js
│   │   ├── dashboard.css
│   │   ├── jquery-ui.css
│   │   ├── jquery-ui.min.css
│   │   ├── jquery-ui.min.js
│   │   └── jquery.min.js
│   ├── templates/
│   │   ├── base.html
│   │   ├── index.html
│   │   └── student-details.html
│   ├── urls.py
│   └── wsgi.py
└── requirements.txt

Starting Structure

1. create a VirtualEnvironment to isolate the project and activate it.

2. Pip install these dependencies for the project:

   
   $ pip install Django==1.9.3 elasticsearch requests django_extensions debug_toolbar django-debug-toolbar
   
   

3. Generate a new Django project: $ django-admin startproject project

4. This is the default project structure generated by manage.py when starting a new project.

   
   .
   |_____project/
   |_____|_______init__.py
   |_____|_____settings.py
   |_____|_____urls.py
   |_____|_____wsgi.py
   |_____manage.py
   
   

Now that we have a project, the base files will need LOTS of tweaking.

Refactoring Project Structure

1. Rename the nested project/project folder to project/project.bak. We will not be using this directory (thus we break with django best practices).

2. Create the following directory structures:

   
   $ mkdir project/apps/core
   $ mkdir conf
   $ mkdir db
   $ mkdir log
   $ mkdir media
   $ mkdir static
   $ mkdir templates
   
   

3. Copy the default settings.py to the new project root: $ cp project/project.bak/settings.py project/conf/base.py

4. Copy the default urls.py to the project root: $ cp project/project.bak/urls.py project/urls.py

5. Copy the default wsgi.py to the project root: $ cp project/project.bak/wsgi.py project/wsgi.py

6. Edit project/manage.py to match this:

   
   #!/usr/bin/env python
   import os
   import sys
   
   if __name__ == "__main__":
       # os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
       os.environ.setdefault("DJANGO_SETTINGS_MODULE", "conf")
   
       from django.core.management import execute_from_command_line
   
       execute_from_command_line(sys.argv)
   
   

7. Edit project/wsgi.py to match this:

   
   """
   WSGI config for project project.
   
   It exposes the WSGI callable as a module-level variable named ``application``.
   
   For more information on this file, see
   https://docs.djangoproject.com/en/1.9/howto/deployment/wsgi/
   """
   
   import os
   from os.path import abspath, dirname
   from sys import path
   
   SITE_ROOT = dirname(dirname(abspath(__file__)))
   path.append(SITE_ROOT)
   
   os.environ.setdefault("DJANGO_SETTINGS_MODULE", "src.conf")
   
   from django.core.wsgi import get_wsgi_application
   application = get_wsgi_application()
   
   

8. Edit project/urls.py to match this:

   
   """project URL Configuration
   
   The `urlpatterns` list routes URLs to views. For more information please see:
       https://docs.djangoproject.com/en/1.9/topics/http/urls/
   Examples:
   Function views
       1. Add an import:  from my_app import views
       2. Add a URL to urlpatterns:  url(r'^$', views.home, name='home')
   Class-based views
       1. Add an import:  from other_app.views import Home
       2. Add a URL to urlpatterns:  url(r'^$', Home.as_view(), name='home')
   Including another URLconf
       1. Import the include() function: from django.conf.urls import url, include
       2. Add a URL to urlpatterns:  url(r'^blog/', include('blog.urls'))
   """
   
   from django.conf import settings
   from django.conf.urls import include, url
   from django.contrib import admin
   # from core.views import autocomplete_view, student_detail, HomePageView
   
   urlpatterns = [
       url(r'^admin/', admin.site.urls),
       # url(r'^autocomplete/', autocomplete_view, name='autocomplete-view'),
       # url(r'^student', student_detail, name='student-detail'),
       # url(r'^$', HomePageView.as_view(), name='index-view'),
   ]
   
   if settings.DEBUG:
       from django.conf.urls.static import static
       urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
       urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
   
       import debug_toolbar
       urlpatterns = [
           url(r'^__debug__/', include(debug_toolbar.urls)),
       ] + urlpatterns
   
   

9. Edit project/conf/base.py to math this:

   
   """
   Django settings for project project.
   
   Generated by 'django-admin startproject' using Django 1.9.3.
   
   For more information on this file, see
   https://docs.djangoproject.com/en/1.9/topics/settings/
   
   For the full list of settings and their values, see
   https://docs.djangoproject.com/en/1.9/ref/settings/
   """
   
   import os
   import sys
   
   from elasticsearch import Elasticsearch, RequestsHttpConnection
   
   # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
   BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
   ROOT_DIR = os.path.dirname(BASE_DIR)
   sys.path.append(
       os.path.join(BASE_DIR, 'apps')
   )
   
   # Quick-start development settings - unsuitable for production
   # See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/
   
   # SECURITY WARNING: keep the secret key used in production secret!
   SECRET_KEY = 'alun^uucv3t=(9%z=mt5q085x^xn$n5+h=tbjta-lfm5qmikra'
   
   # SECURITY WARNING: don't run with debug turned on in production!
   DEBUG = True
   
   # ALLOWED_HOSTS = []
   ALLOWED_HOSTS = ['*']
   
   
   # Application definition
   
   INSTALLED_APPS = [
       'django.contrib.admin',
       'django.contrib.auth',
       'django.contrib.contenttypes',
       'django.contrib.sessions',
       'django.contrib.messages',
       'django.contrib.staticfiles',
       # ADDED:
       'core',
       'django_extensions',
       'debug_toolbar',
   ]
   
   MIDDLEWARE_CLASSES = [
       'django.middleware.security.SecurityMiddleware',
       'django.contrib.sessions.middleware.SessionMiddleware',
       'django.middleware.common.CommonMiddleware',
       'django.middleware.csrf.CsrfViewMiddleware',
       'django.contrib.auth.middleware.AuthenticationMiddleware',
       'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
       'django.contrib.messages.middleware.MessageMiddleware',
       'django.middleware.clickjacking.XFrameOptionsMiddleware',
       # ADDED:
       'debug_toolbar.middleware.DebugToolbarMiddleware',
   ]
   
   # ROOT_URLCONF = 'project.urls'
   ROOT_URLCONF = 'urls'
   
   TEMPLATES = [
       {
           'BACKEND': 'django.template.backends.django.DjangoTemplates',
           # 'DIRS': [],
           'DIRS': [
               os.path.normpath(os.path.join(BASE_DIR, 'templates')),
           ],
           'APP_DIRS': True,
           'OPTIONS': {
               'context_processors': [
                   'django.template.context_processors.debug',
                   'django.template.context_processors.request',
                   'django.template.context_processors.media', # Added
                   'django.template.context_processors.static', # Added
                   'django.contrib.auth.context_processors.auth',
                   'django.contrib.messages.context_processors.messages',
               ],
           },
       },
   ]
   
   # WSGI_APPLICATION = 'project.wsgi.application'
   WSGI_APPLICATION = 'wsgi.application'
   
   
   # Database
   # https://docs.djangoproject.com/en/1.9/ref/settings/#databases
   
   DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.sqlite3',
           # 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
           'NAME': os.path.join(BASE_DIR, 'db/development.db'),  # Added.
       }
   }
   
   
   # Password validation
   # https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
   
   AUTH_PASSWORD_VALIDATORS = [
       {
           'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
       },
       {
           'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
       },
       {
           'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
       },
       {
           'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
       },
   ]
   
   
   # Internationalization
   # https://docs.djangoproject.com/en/1.9/topics/i18n/
   
   LANGUAGE_CODE = 'en-us'
   
   TIME_ZONE = 'UTC'
   
   USE_I18N = True
   
   USE_L10N = True
   
   USE_TZ = True
   
   
   # Static files (CSS, JavaScript, Images)
   # https://docs.djangoproject.com/en/1.9/howto/static-files/
   
   STATIC_URL = '/static/'
   
   # EVERYTHING BELOW HERE ADDED.
   
   # STATIC FILE CONFIGURATION
   # See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
   STATIC_ROOT = os.path.join(ROOT_DIR, 'assets')
   
   # See:
   # https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
   STATICFILES_DIRS = (
       os.path.join(BASE_DIR, 'static'),
   )
   
   STATICFILES_FINDERS = (
       'django.contrib.staticfiles.finders.FileSystemFinder',
       'django.contrib.staticfiles.finders.AppDirectoriesFinder',
   )
   # END STATIC FILE CONFIGURATION
   
   # MEDIA CONFIGURATION
   # See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
   MEDIA_ROOT = os.path.normpath(os.path.join(ROOT_DIR, 'media'))
   
   # See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
   MEDIA_URL = '/media/'
   # END MEDIA CONFIGURATION
   
   LOGGING = {
       'version': 1,
       'disable_existing_loggers': False,
       'filters': {
           'require_debug_true': {
               '()': 'django.utils.log.RequireDebugTrue',
           },
           'require_debug_false': {
               '()': 'django.utils.log.RequireDebugFalse',
           },
       },
       'formatters': {
           'verbose': {
               'format': "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
               'datefmt': "%d/%b/%Y %H:%M:%S"
           },
           'simple': {
               'format': '%(levelname)s %(message)s'
           },
       },
       'handlers': {
           'console': {
               'level': 'DEBUG',
               'filters': ['require_debug_true'],
               'class': 'logging.StreamHandler',
           },
           'file': {
               'level': 'ERROR',
               'class': 'logging.FileHandler',
               'filters': ['require_debug_false'],
               'filename': 'log/error.log',
               'formatter': 'verbose'
           },
       },
       'loggers': {
           'django.db.backends': {
               'level': 'DEBUG',
               'handlers': ['console'],
           },
           'django.request': {
               'handlers': ['file'],
               'level': 'ERROR',
               'propagate': True,
           },
       }
   }
   
   # from elasticsearch import Elasticsearch, RequestsHttpConnection
   
   ES_CLIENT = Elasticsearch(
       ['http://127.0.0.1:9200/'],
       connection_class=RequestsHttpConnection
   )
   ES_AUTOREFRESH = True
   
   

10. Create an init file for the conf app: $ touch project/apps/conf/__init__.py

11. Edit the conf/__init__.py file to match this:

    
    from .base import *
    
    try:
        from .local import *
    except ImportError:
        pass
    
    

12. Create the project/apps/core/models.py file: $ touch project/apps/core/models.py

13. Populate the new models.py file as follows:

    
    from django.db import models
    from django.core.validators import MinValueValidator, MaxValueValidator
    
    class University(models.Model):
        name = models.CharField(max_length=255, unique=True)
    
    class Course(models.Model):
        name = models.CharField(max_length=255, unique=True)
    
    class Student(models.Model):
        YEAR_IN_SCHOOL_CHOICES = (
            ('FR', 'Freshman'),
            ('SO', 'Sophomore'),
            ('JR', 'Junior'),
            ('SR', 'Senior'),
        )
        # note: incorrect choice in MyModel.create leads to creation of incorrect record
        year_in_school = models.CharField(
            max_length=2, choices=YEAR_IN_SCHOOL_CHOICES)
        age = models.SmallIntegerField(
            validators=[MinValueValidator(1), MaxValueValidator(100)]
        )
        first_name = models.CharField(max_length=50)
        last_name = models.CharField(max_length=50)
        # various relationships models
        university = models.ForeignKey(University, null=True, blank=True)
        courses = models.ManyToManyField(Course, null=True, blank=True)
    
    

14. Make migrations for the new models: $ python project/manage.py makemigrations core

15. This results in a new file: project/apps/core/migrations/0001_initial.py

16. Make a directory for the db: $ mkdir project/db

17. Build the db tables from the migration: $ python project/manage.py migrate

18. Create a new file for the admin interface: $ tpuch project/apps/core/admin.py

19. Edit the new admin.py file to match this:

    
    from django.contrib import admin
    from .models import University, Course, Student
    
    admin.site.register(University)
    admin.site.register(Course)
    admin.site.register(Student)
    
    

20. Create a superuser for the admin interface: $ python project/manage.py createsuperuser

    • Provide a username and password as prompted.

21. Test the project: $ python project/manage.py runserver

    • You can see the running project in a browser at: http://localhost:8000/admin/
    • If you access http://localhost:8000 you will get a 404 error because we have not define any urls to handle that route.

22. Login to the admin panel to verify everything is working.

Working Django Application Structure

The state of the project structure after adapting the default django project to match tutorial, installing dependencies, configuring middleware, etc.

Note: I am intentionally ommitting the __Pycache__, .git and project.bak dirs from this tree output.

$ tree -I ".git|__pycache__|project.bak"
.
├── README.md
├── project
│   ├── apps
│   │   └── core
│   │       ├── admin.py
│   │       ├── migrations
│   │       │   ├── 0001_initial.py
│   │       │   └── __init__.py
│   │       └── models.py
│   ├── conf
│   │   ├── __init__.py
│   │   └── base.py
│   ├── db
│   │   └── development.db
│   ├── log
│   │   └── error.log
│   ├── manage.py
│   ├── media
│   ├── static
│   ├── templates
│   ├── urls.py
│   └── wsgi.py
└── requirements.txt

Create A Reusable Command To Populate DB Tables

...

Tutorial

• Part 1, Making a Basic Django App
• Part 2, Populating the Database
• Part 3, add data to the elasticsearch index in bulk, write a basic command, and add a mapping to the elasticsearch index
• Part 4, add functional frontend items, write queries, allow the index to update
• Tutorial Code Repo

Additional Refs

• https://docs.djangoproject.com/en/1.9/
• https://github.com/elastic/elasticsearch
• https://www.elastic.co/guide/index.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
• https://www.codingforentrepreneurs.com/blog/django-virtualenv-python-gitignore-file/
• https://pixabay.com/en/blog/posts/django-search-with-elasticsearch-47/
• https://model-mommy.readthedocs.io/en/latest/
• https://pypi.python.org/pypi/names/0.3.0

Troubleshooting Refs

• http://elasticsearch-py.readthedocs.io/en/master/helpers.html#bulk-helpers
• http://www.django-rest-framework.org/api-guide/serializers/#modelserializer
• https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-store.html
• https://www.elastic.co/guide/en/elasticsearch/reference/current/suggester-context.html
• https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docs-bulk.html