We actually need to implement this via Middleware, as the Locale Middleware comes later in the process and hinders us from removing the header. Anyway, it's cleaner, since we're not duplicating code

config.py

@@ -31,7 +31,7 @@ INSTALLED_APPS.extend(
 
 MIDDLEWARE.extend(
     [
-        "django.middleware.locale.LocaleMiddleware",
+        "ivatar.middleware.RemoveVaryForImagesMiddleware",
     ]
 )
 MIDDLEWARE.insert(