Real-time Emotion Detection with OpenCV: How DJoz Works

The Idea That Started It

It was late at night during a college hackathon. Someone put on a sad playlist and the room felt heavier. I wondered: what if the music player could sense the mood of the room itself — not from what you clicked, but from what your face was saying?

That question became DJoz (Dynamic Jukebox). An AI system that reads your face via webcam, classifies your emotional state in real time, and curates a playlist that matches — or intentionally contrasts — your mood. No buttons. No search. Just look at the camera.

Here's a complete breakdown of how the computer vision pipeline works under the hood.

The Full Pipeline at a Glance

Before diving into each component, here's the end-to-end flow:

Webcam frame
    ↓
Grayscale conversion
    ↓
Haar Cascade face detector  →  face bounding box (x, y, w, h)
    ↓
Crop + resize to 48×48px
    ↓
CNN (trained on FER-2013)   →  emotion probabilities [angry, disgust, fear, happy, neutral, sad, surprise]
    ↓
Rolling mode (last 30 frames)  →  stable emotion label
    ↓
MySQL playlist lookup
    ↓
Render recommendation to browser

Step 1: Face Detection with Haar Cascades

The first challenge is finding where the face is in each video frame. OpenCV's Haar Cascade classifier is fast enough to run in real-time on a CPU — critical for a webcam-based system that needs to process 30 frames per second:

import cv2

face_cascade = cv2.CascadeClassifier(
    cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
)

cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(
        gray,
        scaleFactor=1.1,
        minNeighbors=5,
        minSize=(48, 48)
    )
    for (x, y, w, h) in faces:
        face_roi = gray[y:y+h, x:x+w]
        face_roi = cv2.resize(face_roi, (48, 48))
        # pass to CNN classifier

Step 2: Emotion Classification with a CNN

Once we have the face region, we pass it through a Convolutional Neural Network trained on the FER-2013 dataset — 35,887 labeled facial images across 7 emotion classes: angry, disgust, fear, happy, neutral, sad, and surprise.

The architecture is deliberately lightweight: 4 convolutional blocks (Conv2D → BatchNorm → MaxPool → Dropout) followed by two dense layers and a softmax output. The small size keeps inference fast enough for real-time use.

Training for 50 epochs on FER-2013 achieves approximately 65% validation accuracy. That number might sound low, but context matters: emotion classification is inherently subjective. Even human labelers disagreed on 20–30% of FER-2013 images. For a recommendation system where "close enough" is sufficient, 65% works well in practice.

Step 3: Emotion-to-Playlist Mapping

Each detected emotion maps to a curated content category stored in MySQL. The mapping reflects both intuitive matching and deliberate therapeutic contrast:

• Happy → Upbeat pop, feel-good playlists, comedy shorts
• Sad → Lo-fi / acoustic (comforting), then gradually uplifting content
• Angry → Two tracks: high-energy metal (release) or guided meditation (calm)
• Neutral → Ambient / focus music — the default working state
• Fear / Surprise → Discovery playlist — unfamiliar, interesting content
• Disgust → Palate cleanser — highly-rated, universally-liked content

Step 4: Temporal Smoothing

Raw frame-by-frame predictions are noisy. Your expression changes dozens of times per second — a single surprised blink shouldn't trigger a playlist switch.

The fix is a rolling mode: collect the predicted emotion for the last 30 frames (~1 second at 30fps), and only trigger a recommendation change if the majority emotion in that window differs from the current one:

from collections import deque
from statistics import mode

emotion_buffer = deque(maxlen=30)

def get_stable_emotion(raw_emotion):
    emotion_buffer.append(raw_emotion)
    if len(emotion_buffer) == 30:
        return mode(emotion_buffer)
    return None  # wait for buffer to fill

This single change made the system feel natural instead of jittery — the difference between a prototype and something you'd actually want to use.

Step 5: Streaming to the Browser via Flask

The processed video feed (with emotion overlay annotations) is streamed to the browser using Flask's multipart/x-mixed-replace MIME type — the simplest way to push live video over HTTP without WebSockets:

from flask import Response

def gen_frames():
    while True:
        frame = capture_and_annotate_frame()  # detect + classify + draw
        _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
        yield (
            b'--frame\r\n'
            b'Content-Type: image/jpeg\r\n\r\n'
            + buffer.tobytes()
            + b'\r\n'
        )

@app.route('/video_feed')
def video_feed():
    return Response(
        gen_frames(),
        mimetype='multipart/x-mixed-replace; boundary=frame'
    )

What I'd Do Differently Today

Building DJoz taught me more than any textbook. With two more years of experience, here's how I'd redesign it:

Replace Haar Cascade with MediaPipe Face Mesh. It handles non-frontal faces, varying lighting, and partial occlusion — all the conditions where DJoz currently struggles. It also provides 468 facial landmarks for free, enabling much richer feature extraction.

Use transfer learning instead of training from scratch. Starting from a pretrained ResNet-50 or EfficientNet and fine-tuning on FER-2013 would likely push accuracy from 65% to 75%+ while training significantly faster.

Add a confidence threshold. If the model is less than 60% confident in its emotion prediction, don't trigger a recommendation change. This would dramatically reduce false positives from ambiguous expressions.

DJoz is open source — check it out on GitHub.