Monitor Django App With Prometheus and Grafana

Django, a high-level Python web framework, is widely used for building robust web applications. Monitoring Django applications is essential for ensuring optimal performance, identifying bottlenecks, and maintaining system reliability.

Prometheus is a powerful monitoring system that collects real-time metrics from various sources, offering efficient time-series data storage. Grafana provides customizable dashboards for visualizing metrics, enabling teams to monitor system performance effectively.

Integrating Prometheus and Grafana with Django allows teams to monitor key metrics such as request response times, database queries, cache hits, and resource utilization. This setup supports proactive issue identification, performance optimization, and ensures smooth operation of Django applications.

In this guide, we'll demonstrate how to set up Prometheus to collect Django metrics and utilize Grafana for insightful metric visualization, empowering you to effectively monitor and manage Django applications.

1- Architecture

This document outlines the architecture for monitoring Django applications using Prometheus and Grafana. This setup allows you to visualize key metrics and gain insights into the health and performance of your Django application.

Architecture Overview

The architecture consists of three main components:

Django Exporter

This tool acts as a bridge between Django and Prometheus. It captures the Django Events and translates them into quantifiable metrics suitable for monitoring.

Prometheus

Prometheus is an open-source monitoring application that scrapes (collects) metrics from various sources, including the Django Exporter. It stores these metrics in a time-series database.

Grafana

Grafana is an open-source tool, utilizes Prometheus as a data source to create informative dashboards that visualize the collected Django metrics.

2- Django Configuration

install the following package:

pip install django-prometheus

Add the following configurations to your Django settings file:

INSTALLED_APPS = [
    ...
    'django_prometheus',
]

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    ...
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

In your Django project's urls.py file, add the following URL pattern:

urlpatterns = [
    ...
    path('', include('django_prometheus.urls')),
]

Monitor databasese queries by adding the following configuration to your Django settings file:

DATABASES = {
    'default': {
        ...
        'ENGINE': 'django_prometheus.db.backends.postgresql',
    }
}

Monitor cache hits by adding the following configuration to your Django settings file:

CACHES = {
    'default': {
        'BACKEND': 'django_prometheus.cache.backends.locmem.LocMemCache',
        "LOCATION": "unique-snowflake",
    }
}

To monitor django models in your models.py file, Just add the ExportModelOperationsMixin as such:

from django_prometheus.models import ExportModelOperationsMixin

class MyModel(ExportModelOperationsMixin('my_model'), models.Model):
    attr = models.CharField(max_length=255, blank=True, null=True)
    ...

3- Prometheus Configuration

Add job for django app in prometheus.yml file with the following configuration:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: django-app
    static_configs:
      - targets: ['django-app:8000']

rule_files:
  - "/etc/prometheus/prometheus_alerts.yaml"

Add prometheus_alerts.yaml file contains 4 alerts rules : 4xx , 5xx , database and migration errors

"groups":  
- "name": "django"  
 "rules":  
- "alert": "DjangoMigrationsUnapplied"  
  "annotations":  
    "dashboard_url": "http://grafana:3000//d/django-overview-jkwq/django-overview?var-namespace={{ $labels.namespace }}&var-job={{ $labels.job }}"  
    "description": "The job {{ $labels.job }} has unapplied migrations."  
    "summary": "Django has unapplied migrations."  
  "expr": |  
    sum(  
      django_migrations_unapplied_total{  
        job=~"core-django-exporter"  
      }  
    ) by (namespace, job)  
    > 0  
  "for": "15m"  
  "labels":  
    "severity": "warning"  
- "alert": "DjangoDatabaseException"  
  "annotations":  
    "dashboard_url": "http://grafana:3000//d/django-overview-jkwq/django-overview?var-namespace={{ $labels.namespace }}&var-job={{ $labels.job }}"  
    "description": "The job {{ $labels.job }} has hit the database exception {{ $labels.type }}."  
    "summary": "Django database exception."  
  "expr": |  
    sum (  
      increase(  
        django_db_errors_total{  
          job=~"core-django-exporter"  
        }[10m]  
      )  
    ) by (type, namespace, job)  
    > 0  
  "labels":  
    "severity": "info"  
- "alert": "DjangoHighHttp4xxErrorRate"  
  "annotations":  
    "dashboard_url": "http://grafana:3000//d/django-requests-by-view-jkwq/django-requests-by-view?var-namespace={{ $labels.namespace }}&var-job={{ $labels.job }}&var-view={{ $labels.view }}"  
    "description": "More than 5% HTTP requests with status 4xx for {{ $labels.job }}/{{ $labels.view }} the past 5m."  
    "summary": "Django high HTTP 4xx error rate."  
  "expr": |  
    sum(  
      rate(  
        django_http_responses_total_by_status_view_method_total{  
          job=~"core-django-exporter",  
          status=~"^4.*",  
          view!~"<unnamed view>|health_check:health_check_home|prometheus-django-metrics"  
        }[5m]  
      )  
    )  by (namespace, job, view)  
    /  
    sum(  
      rate(  
        django_http_responses_total_by_status_view_method_total{  
          job=~"core-django-exporter",  
          view!~"<unnamed view>|health_check:health_check_home|prometheus-django-metrics"  
        }[5m]  
      )  
    )  by (namespace, job, view)  
    * 100 > 5  
  "for": "1m"  
  "labels":  
    "severity": "warning"  
- "alert": "DjangoHighHttp5xxErrorRate"  
  "annotations":  
    "dashboard_url": "http://grafana:3000//d/django-requests-by-view-jkwq/django-requests-by-view?var-namespace={{ $labels.namespace }}&var-job={{ $labels.job }}&var-view={{ $labels.view }}"  
    "description": "More than 5% HTTP requests with status 5xx for {{ $labels.job }}/{{ $labels.view }} the past 5m."  
    "summary": "Django high HTTP 5xx error rate."  
  "expr": |  
    sum(  
      rate(  
        django_http_responses_total_by_status_view_method_total{  
          job=~"core-django-exporter",  
          status=~"^5.*",  
          view!~"<unnamed view>|health_check:health_check_home|prometheus-django-metrics"  
        }[5m]  
      )  
    )  by (namespace, job, view)  
    /  
    sum(  
      rate(  
        django_http_responses_total_by_status_view_method_total{  
          job=~"core-django-exporter",  
          view!~"<unnamed view>|health_check:health_check_home|prometheus-django-metrics"  
        }[5m]  
      )  
    )  by (namespace, job, view)  
    * 100 > 5  
  "for": "1m"  
  "labels":  
    "severity": "warning"

4- Grafana Configuration

Add Prometheus as a data source in Grafana:
- Navigate to the Grafana dashboard.
- Click on the gear icon to access the configuration menu.
- Select "Data Sources" and click on "Add data source."
- Choose "Prometheus" as the data source type.
- Enter the Prometheus URL (e.g., http://prometheus:9090) and save the configuration.
Import the Django dashboard in Grafana:
- Navigate to the Grafana dashboard.
- Click on the "+" icon to add a new dashboard.
- Select "Import" and enter the dashboard ID (e.g., 14529).
- Click on "Load" to import the Django dashboard.
- Customize the dashboard as needed and save the changes.

5- Docker Configuration

Create a docker-compose.yml file with the following configuration:

version: "3"
services:
  django-app:
    image: django-app:latest
    ports:
      - "8000:8000"
    depends_on:
      - prometheus
    networks:
      - monitoring

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_alerts.yaml:/etc/prometheus/prometheus_alerts.yaml
    ports:
      - "9090:9090"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - monitoring
  
  postgres:
    image: postgres:latest
    environment:
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb
    volumes:
      - postgres-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    networks:
      - monitoring

volume:
  postgres-data:
  grafana-data:

networks:
  monitoring:

Conclusion

Monitoring Django applications with Prometheus and Grafana provides valuable insights into system performance, resource utilization, and potential issues. By setting up Prometheus to collect Django metrics and Grafana to visualize these metrics, teams can proactively monitor and manage Django applications effectively.

This setup enables teams to identify bottleneneck, optimize performance, and ensure the smooth operation of Django applications. By leveraging the power of Prometheus and Grafana, teams can enhance the reliability, scalability, and efficiency of Django-based projects.

In summary, monitoring Django applications with Prometheus and Grafana is a critical practice for maintaining application health, optimizing performance, and delivering a seamless user experience.