mirror of
https://git.linux-kernel.at/oliver/ivatar.git
synced 2025-11-11 10:46:24 +00:00
File upload security (iteration 1), security enhancements and OpenTelemetry (OTEL) implementation (sending data disabled by default)
This commit is contained in:
Oliver Falk
240
.cursorrules
Normal file
240
.cursorrules
Normal file
@@ -0,0 +1,240 @@
|
||||
# ivatar/libravatar Project Rules
|
||||
|
||||
## Project Overview
|
||||
ivatar is a Django-based federated avatar service that serves as an alternative to Gravatar. It provides avatar images for email addresses and OpenID URLs, with support for the Libravatar federation protocol.
|
||||
|
||||
## Core Functionality
|
||||
- Avatar service for email addresses and OpenID URLs
|
||||
- Federated compatibility with Libravatar protocol
|
||||
- Multiple authentication methods (OpenID, OpenID Connect/Fedora, Django auth)
|
||||
- Image upload, cropping, and management
|
||||
- External avatar import (Gravatar, other Libravatar instances)
|
||||
- Bluesky handle integration
|
||||
- Multiple theme support (default, clime, green, red)
|
||||
- Internationalization (15+ languages)
|
||||
|
||||
## Technical Stack
|
||||
- **Framework**: Django 4.2+ with Python 3.x
|
||||
- **Database**: SQLite (development), MySQL/MariaDB, PostgreSQL (production)
|
||||
- **Image Processing**: PIL/Pillow for image manipulation
|
||||
- **Authentication**: django-openid-auth, social-auth-app-django
|
||||
- **Caching**: Memcached and filesystem caching
|
||||
- **Email**: Mailgun integration via django-anymail
|
||||
- **Testing**: pytest with custom markers
|
||||
|
||||
## Key Models
|
||||
- `Photo`: Stores uploaded avatar images with format detection and access counting
|
||||
- `ConfirmedEmail`: Verified email addresses with assigned photos and Bluesky handles
|
||||
- `ConfirmedOpenId`: Verified OpenID URLs with assigned photos and Bluesky handles
|
||||
- `UserPreference`: User theme preferences
|
||||
- `UnconfirmedEmail`: Email verification workflow
|
||||
- `UnconfirmedOpenId`: OpenID verification workflow
|
||||
|
||||
## Security Features
|
||||
- File upload validation and sanitization
|
||||
- EXIF data removal (ENABLE_EXIF_SANITIZATION)
|
||||
- Malicious content scanning (ENABLE_MALICIOUS_CONTENT_SCAN)
|
||||
- Comprehensive security logging
|
||||
- File size limits and format validation
|
||||
- Trusted URL validation for external avatar sources
|
||||
|
||||
## Development Workflow Rules
|
||||
|
||||
### External Resources & Libraries
|
||||
- **Web search is always allowed** - use web search to find solutions, check documentation, verify best practices
|
||||
- **Use latest library versions** - always prefer the latest stable versions of external libraries
|
||||
- **Security first** - outdated libraries are security risks, always update to latest versions
|
||||
- **Dependency management** - when adding new dependencies, ensure they're actively maintained and secure
|
||||
|
||||
### Testing
|
||||
- **MANDATORY: Run pre-commit hooks and tests before any changes** - this is an obligation
|
||||
- Use `./run_tests_local.sh` for local development (skips Bluesky tests requiring API credentials)
|
||||
- Run `python3 manage.py test -v3` for full test suite including Bluesky tests
|
||||
- **MANDATORY: When adding new code, always write tests to increase code coverage** - never decrease coverage
|
||||
- Use pytest markers appropriately:
|
||||
- `@pytest.mark.bluesky`: Tests requiring Bluesky API credentials
|
||||
- `@pytest.mark.slow`: Long-running tests
|
||||
- `@pytest.mark.integration`: Integration tests
|
||||
- `@pytest.mark.unit`: Unit tests
|
||||
|
||||
### Code Quality
|
||||
- Always check for linter errors after making changes using `read_lints`
|
||||
- Follow existing code style and patterns
|
||||
- Maintain comprehensive logging (use `logger = logging.getLogger("ivatar")`)
|
||||
- Consider security implications of any changes
|
||||
- Follow Django best practices and conventions
|
||||
- **Reduce script creation** - avoid creating unnecessary scripts, prefer existing tools and commands
|
||||
- **Use latest libraries** - always use the latest versions of external libraries to ensure security and bug fixes
|
||||
|
||||
### Database Operations
|
||||
- Use migrations for schema changes: `./manage.py migrate`
|
||||
- Support multiple database backends (SQLite, MySQL, PostgreSQL)
|
||||
- Use proper indexing for performance (see existing model indexes)
|
||||
|
||||
### Image Processing
|
||||
- Support multiple formats: JPEG, PNG, GIF, WEBP
|
||||
- Maximum image size: 512x512 pixels (AVATAR_MAX_SIZE)
|
||||
- Maximum file size: 10MB (MAX_PHOTO_SIZE)
|
||||
- JPEG quality: 85 (JPEG_QUALITY)
|
||||
- Always validate image format and dimensions
|
||||
|
||||
## Configuration Management
|
||||
- Main settings in `ivatar/settings.py` and `config.py`
|
||||
- Local overrides in `config_local.py` (not in version control)
|
||||
- Environment variables for sensitive data (database credentials, API keys)
|
||||
- Support for multiple deployment environments (development, staging, production)
|
||||
|
||||
## Authentication & Authorization
|
||||
- Multiple backends: Django auth, OpenID, Fedora OIDC
|
||||
- Social auth pipeline with custom steps for email confirmation
|
||||
- User account creation and management
|
||||
- Email verification workflow
|
||||
|
||||
## Caching Strategy
|
||||
- Memcached for general caching
|
||||
- Filesystem cache for generated images
|
||||
- 5-minute cache for resized images (CACHE_IMAGES_MAX_AGE)
|
||||
- Cache invalidation on photo updates
|
||||
|
||||
## Internationalization
|
||||
- Support for 15+ languages
|
||||
- Use Django's translation framework
|
||||
- Template strings should be translatable
|
||||
- Locale-specific formatting
|
||||
|
||||
## File Structure Guidelines
|
||||
- Main Django app: `ivatar/`
|
||||
- Account management: `ivatar/ivataraccount/`
|
||||
- Tools: `ivatar/tools/`
|
||||
- Static files: `ivatar/static/` and `static/`
|
||||
- Templates: `templates/` and app-specific template directories
|
||||
- Tests: Co-located with modules or in dedicated test files
|
||||
|
||||
## Security Considerations
|
||||
- Always validate file uploads
|
||||
- Sanitize EXIF data from images
|
||||
- Use secure password hashing (Argon2 preferred, PBKDF2 fallback)
|
||||
- Implement proper CSRF protection
|
||||
- Use secure cookies in production
|
||||
- Log security events to dedicated security log
|
||||
|
||||
## Performance Considerations
|
||||
- Use database indexes for frequently queried fields
|
||||
- Implement proper caching strategies
|
||||
- Optimize image processing operations
|
||||
- Monitor access counts for analytics
|
||||
- Use efficient database queries
|
||||
|
||||
## Production Deployment & Infrastructure
|
||||
|
||||
### Hosting & Sponsorship
|
||||
- **Hosted by Fedora Project** - Free infrastructure provided due to heavy usage by Fedora community
|
||||
- **Scale**: Handles millions of requests daily for 30k+ users with 33k+ avatar images
|
||||
- **Performance**: High-performance system optimized for dynamic content (CDN difficult due to dynamic sizing)
|
||||
|
||||
### Production Architecture
|
||||
- **Redis**: Session storage (potential future caching expansion)
|
||||
- **Monitoring Stack**:
|
||||
- Prometheus + Alertmanager for metrics and alerting
|
||||
- Loki for log aggregation
|
||||
- Alloy for observability
|
||||
- Grafana for visualization
|
||||
- Custom exporters for application metrics
|
||||
- **Apache HTTPD**:
|
||||
- SSL termination
|
||||
- Load balancer for Gunicorn containers
|
||||
- Caching (memory/socache and disk cache - optimization ongoing)
|
||||
- **PostgreSQL**: Main production database
|
||||
- **Gunicorn**: 2 containers running Django application
|
||||
- **Containerization**: **Podman** (not Docker) - always prefer podman when possible
|
||||
|
||||
### Development Environment
|
||||
- **Dev Instance**: dev.libravatar.org (auto-deployed from 'devel' branch via Puppet)
|
||||
- **Limitation**: Aging CentOS 7 host with older Python 3.x and Django versions
|
||||
- **Compatibility**: Must maintain backward compatibility with older versions
|
||||
|
||||
### CI/CD & Version Control
|
||||
- **GitLab**: Self-hosted OSS/Community Edition on git.linux-kernel.at
|
||||
- **CI**: GitLab CI extensively used
|
||||
- **CD**: GitLab CD on roadmap (part of libravatar-ansible project)
|
||||
- **Deployment**: Separate libravatar-ansible project handles production deployments
|
||||
- **Container Management**: Ansible playbooks rebuild custom images and restart containers as needed
|
||||
|
||||
### Deployment Considerations
|
||||
- Production requires proper database setup (PostgreSQL, not SQLite)
|
||||
- Static file collection required: `./manage.py collectstatic`
|
||||
- Environment-specific configuration via environment variables
|
||||
- Custom container images with automated rebuilds
|
||||
- High availability and performance optimization critical
|
||||
|
||||
## Common Commands
|
||||
```bash
|
||||
# Development server
|
||||
./manage.py runserver 0:8080
|
||||
|
||||
# Run local tests (recommended for development)
|
||||
./run_tests_local.sh
|
||||
|
||||
# Run all tests
|
||||
python3 manage.py test -v2
|
||||
|
||||
# Database migrations
|
||||
./manage.py migrate
|
||||
|
||||
# Collect static files
|
||||
./manage.py collectstatic -l --no-input
|
||||
|
||||
# Create superuser
|
||||
./manage.py createsuperuser
|
||||
```
|
||||
|
||||
## Code Style Guidelines
|
||||
- Use descriptive variable and function names
|
||||
- Add comprehensive docstrings for classes and methods
|
||||
- **MANDATORY: Include type hints for ALL new code** - this is a strict requirement
|
||||
- Follow PEP 8 and Django coding standards
|
||||
- Use meaningful commit messages
|
||||
- Add comments for complex business logic
|
||||
|
||||
## Error Handling
|
||||
- Use proper exception handling with specific exception types
|
||||
- Log errors with appropriate levels (DEBUG, INFO, WARNING, ERROR)
|
||||
- Provide user-friendly error messages
|
||||
- Implement graceful fallbacks where possible
|
||||
|
||||
## API Compatibility
|
||||
- Maintain backward compatibility with existing avatar URLs
|
||||
- Support Libravatar federation protocol
|
||||
- Ensure Gravatar compatibility for imports
|
||||
- Preserve existing URL patterns and parameters
|
||||
|
||||
## Monitoring & Logging
|
||||
- Use structured logging with appropriate levels
|
||||
- Log security events to dedicated security log
|
||||
- Monitor performance metrics (access counts, response times)
|
||||
- Implement health checks for external dependencies
|
||||
- **Robust logging setup**: Automatically tests directory writeability and falls back gracefully
|
||||
- **Fallback hierarchy**: logs/ → /tmp/libravatar-logs → user-specific temp directory
|
||||
- **Permission handling**: Handles cases where logs directory exists but isn't writable
|
||||
|
||||
## GitLab CI/CD Monitoring
|
||||
- **MANDATORY: Check GitLab pipeline status regularly** during development
|
||||
- Monitor pipeline status for the current working branch (typically `devel`)
|
||||
- Use `glab ci list --repo git.linux-kernel.at/oliver/ivatar --per-page 5` to check recent pipelines
|
||||
- Verify all tests pass before considering work complete
|
||||
- Check pipeline logs with `glab ci trace <pipeline-id> --repo git.linux-kernel.at/oliver/ivatar` if needed
|
||||
- Address any CI failures immediately before proceeding with new changes
|
||||
- Pipeline URL: https://git.linux-kernel.at/oliver/ivatar/-/pipelines
|
||||
|
||||
## Deployment Verification
|
||||
- **Automatic verification**: GitLab CI automatically verifies dev.libravatar.org deployments on `devel` branch
|
||||
- **Manual verification**: Production deployments on `master` branch can be verified manually via CI
|
||||
- **Version endpoint**: `/deployment/version/` provides commit hash, branch, and deployment status
|
||||
- **Security**: Version endpoint uses cached git file reading (no subprocess calls) to prevent DDoS attacks
|
||||
- **Performance**: Version information is cached in memory to avoid repeated file system access
|
||||
- **SELinux compatibility**: No subprocess calls that might be blocked by SELinux policies
|
||||
- **Manual testing**: Use `./scripts/test_deployment.sh` to test deployments locally
|
||||
- **Deployment timing**: Dev deployments via Puppet may take up to 30 minutes to complete
|
||||
- **Verification includes**: Version matching, avatar endpoint, stats endpoint functionality
|
||||
|
||||
Remember: This is a production avatar service handling user data and images. Security, performance, and reliability are paramount. Always consider the impact of changes on existing users and federated services.
|
||||
@@ -11,6 +11,7 @@ cache:
|
||||
variables:
|
||||
PIP_CACHE_DIR: .pipcache
|
||||
|
||||
# Test with OpenTelemetry instrumentation (always enabled, export disabled in CI)
|
||||
test_and_coverage:
|
||||
stage: build
|
||||
coverage: "/^TOTAL.*\\s+(\\d+\\%)$/"
|
||||
@@ -23,6 +24,10 @@ test_and_coverage:
|
||||
POSTGRES_HOST: postgres
|
||||
DATABASE_URL: "postgres://django_user:django_password@postgres/django_db"
|
||||
PYTHONUNBUFFERED: 1
|
||||
# OpenTelemetry instrumentation always enabled, export controlled by OTEL_EXPORT_ENABLED
|
||||
OTEL_EXPORT_ENABLED: "false" # Disable export in CI to avoid external dependencies
|
||||
OTEL_SERVICE_NAME: "ivatar-ci"
|
||||
OTEL_ENVIRONMENT: "ci"
|
||||
before_script:
|
||||
- virtualenv -p python3 /tmp/.virtualenv
|
||||
- source /tmp/.virtualenv/bin/activate
|
||||
@@ -33,7 +38,6 @@ test_and_coverage:
|
||||
- pip install coverage
|
||||
- pip install pycco
|
||||
- pip install django_coverage_plugin
|
||||
|
||||
script:
|
||||
- source /tmp/.virtualenv/bin/activate
|
||||
- echo 'from ivatar.settings import TEMPLATES' > config_local.py
|
||||
@@ -43,7 +47,8 @@ test_and_coverage:
|
||||
- echo "CACHES['default'] = CACHES['filesystem']" >> config_local.py
|
||||
- python manage.py sqldsn
|
||||
- python manage.py collectstatic --noinput
|
||||
- coverage run --source . manage.py test -v3 --noinput
|
||||
- echo "Running tests with OpenTelemetry instrumentation enabled..."
|
||||
- coverage run --source . scripts/run_tests_with_coverage.py
|
||||
- coverage report --fail-under=70
|
||||
- coverage html
|
||||
artifacts:
|
||||
@@ -106,23 +111,42 @@ pages:
|
||||
# fi
|
||||
# - docker build --pull -t "$CI_REGISTRY_IMAGE${tag}" .
|
||||
# - docker push "$CI_REGISTRY_IMAGE${tag}"
|
||||
semgrep:
|
||||
stage: test
|
||||
allow_failure: true
|
||||
image: registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:latest
|
||||
|
||||
# Deployment verification jobs
|
||||
verify_dev_deployment:
|
||||
stage: deploy
|
||||
image: python:3.11-alpine
|
||||
only:
|
||||
- master
|
||||
- devel
|
||||
variables:
|
||||
CI_PROJECT_DIR: "/tmp/app"
|
||||
SECURE_LOG_LEVEL: "debug"
|
||||
DEV_URL: "https://dev.libravatar.org"
|
||||
MAX_RETRIES: 30
|
||||
RETRY_DELAY: 60
|
||||
before_script:
|
||||
- apk add --no-cache curl
|
||||
- pip install Pillow
|
||||
script:
|
||||
- rm -rf .virtualenv
|
||||
- /analyzer run
|
||||
artifacts:
|
||||
paths:
|
||||
- gl-sast-report.json
|
||||
- semgrep.sarif
|
||||
- echo "Waiting for dev.libravatar.org deployment to complete..."
|
||||
- python3 scripts/check_deployment.py --dev --max-retries $MAX_RETRIES --retry-delay $RETRY_DELAY
|
||||
allow_failure: false
|
||||
|
||||
verify_prod_deployment:
|
||||
stage: deploy
|
||||
image: python:3.11-alpine
|
||||
only:
|
||||
- master
|
||||
when: manual
|
||||
variables:
|
||||
PROD_URL: "https://libravatar.org"
|
||||
MAX_RETRIES: 10
|
||||
RETRY_DELAY: 30
|
||||
before_script:
|
||||
- apk add --no-cache curl
|
||||
- pip install Pillow
|
||||
script:
|
||||
- echo "Verifying production deployment..."
|
||||
- python3 scripts/check_deployment.py --prod --max-retries $MAX_RETRIES --retry-delay $RETRY_DELAY
|
||||
allow_failure: false
|
||||
|
||||
include:
|
||||
- template: Jobs/SAST.gitlab-ci.yml
|
||||
|
||||
229
FILE_UPLOAD_SECURITY.md
Normal file
229
FILE_UPLOAD_SECURITY.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# File Upload Security Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The ivatar application now includes comprehensive file upload security features to protect against malicious file uploads, data leaks, and other security threats.
|
||||
|
||||
## Security Features
|
||||
|
||||
### 1. File Type Validation
|
||||
|
||||
**Magic Bytes Verification**
|
||||
|
||||
- Validates file signatures (magic bytes) to ensure uploaded files are actually images
|
||||
- Supports JPEG, PNG, GIF, WebP, BMP, and TIFF formats
|
||||
- Prevents file extension spoofing attacks
|
||||
|
||||
**MIME Type Validation**
|
||||
|
||||
- Uses python-magic library to detect actual MIME types
|
||||
- Cross-references with allowed MIME types list
|
||||
- Prevents MIME type confusion attacks
|
||||
|
||||
### 2. Content Security Scanning
|
||||
|
||||
**Malicious Content Detection**
|
||||
|
||||
- Scans for embedded scripts (`<script>`, `javascript:`, `vbscript:`)
|
||||
- Detects executable content (PE headers, ELF headers)
|
||||
- Identifies polyglot attacks (files valid in multiple formats)
|
||||
- Checks for PHP and other server-side code
|
||||
|
||||
**PIL Image Validation**
|
||||
|
||||
- Uses Python Imaging Library to verify file is a valid image
|
||||
- Checks image dimensions and format
|
||||
- Ensures image can be properly loaded and processed
|
||||
|
||||
### 3. EXIF Data Sanitization
|
||||
|
||||
**Metadata Removal**
|
||||
|
||||
- Automatically strips EXIF data from uploaded images
|
||||
- Prevents location data and other sensitive metadata leaks
|
||||
- Preserves image quality while removing privacy risks
|
||||
|
||||
### 4. Enhanced Logging
|
||||
|
||||
**Security Event Logging**
|
||||
|
||||
- Logs all file upload attempts with user ID and IP address
|
||||
- Records security violations and suspicious activity
|
||||
- Provides audit trail for security monitoring
|
||||
|
||||
## Configuration
|
||||
|
||||
### Settings
|
||||
|
||||
All security features can be configured in `config.py` or overridden in `config_local.py`:
|
||||
|
||||
```python
|
||||
# File upload security settings
|
||||
ENABLE_FILE_SECURITY_VALIDATION = True
|
||||
ENABLE_EXIF_SANITIZATION = True
|
||||
ENABLE_MALICIOUS_CONTENT_SCAN = True
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
The security features require the following Python packages:
|
||||
|
||||
```bash
|
||||
pip install python-magic>=0.4.27
|
||||
```
|
||||
|
||||
**Note**: On some systems, you may need to install the libmagic system library:
|
||||
|
||||
- **Ubuntu/Debian**: `sudo apt-get install libmagic1`
|
||||
- **CentOS/RHEL**: `sudo yum install file-devel`
|
||||
- **macOS**: `brew install libmagic`
|
||||
|
||||
## Security Levels
|
||||
|
||||
### Security Score System
|
||||
|
||||
Files are assigned a security score (0-100) based on validation results:
|
||||
|
||||
- **90-100**: Excellent - No security concerns
|
||||
- **80-89**: Good - Minor warnings, safe to process
|
||||
- **70-79**: Fair - Some concerns, review recommended
|
||||
- **50-69**: Poor - Multiple issues, high risk
|
||||
- **0-49**: Critical - Malicious content detected, reject
|
||||
|
||||
### Validation Levels
|
||||
|
||||
1. **Basic Validation**: File size, filename, extension
|
||||
2. **Magic Bytes**: File signature verification
|
||||
3. **MIME Type**: Content type validation
|
||||
4. **PIL Validation**: Image format verification
|
||||
5. **Security Scan**: Malicious content detection
|
||||
6. **EXIF Sanitization**: Metadata removal
|
||||
|
||||
## API Reference
|
||||
|
||||
### FileValidator Class
|
||||
|
||||
```python
|
||||
from ivatar.file_security import FileValidator
|
||||
|
||||
validator = FileValidator(file_data, filename)
|
||||
results = validator.comprehensive_validation()
|
||||
```
|
||||
|
||||
### Main Validation Function
|
||||
|
||||
```python
|
||||
from ivatar.file_security import validate_uploaded_file
|
||||
|
||||
is_valid, results, sanitized_data = validate_uploaded_file(file_data, filename)
|
||||
```
|
||||
|
||||
### Security Report Generation
|
||||
|
||||
```python
|
||||
from ivatar.file_security import get_file_security_report
|
||||
|
||||
report = get_file_security_report(file_data, filename)
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Validation Errors
|
||||
|
||||
The system provides user-friendly error messages while logging detailed security information:
|
||||
|
||||
- **Malicious Content**: "File appears to be malicious and cannot be uploaded"
|
||||
- **Invalid Format**: "File format not supported or file appears to be corrupted"
|
||||
|
||||
### Logging Levels
|
||||
|
||||
- **INFO**: Successful uploads and normal operations
|
||||
- **WARNING**: Security violations and suspicious activity
|
||||
- **ERROR**: Validation failures and system errors
|
||||
|
||||
## Testing
|
||||
|
||||
### Running Security Tests
|
||||
|
||||
```bash
|
||||
python manage.py test ivatar.test_file_security
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
|
||||
The test suite covers:
|
||||
|
||||
- Valid file validation
|
||||
- Malicious content detection
|
||||
- Magic bytes verification
|
||||
- MIME type validation
|
||||
- EXIF sanitization
|
||||
- Form validation
|
||||
- Integration tests
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Usage
|
||||
|
||||
- Files are processed in memory for validation
|
||||
- Large files (>5MB) may impact performance
|
||||
- Consider increasing server memory for high-volume deployments
|
||||
|
||||
### Processing Time
|
||||
|
||||
- Basic validation: <10ms
|
||||
- Full security scan: 50-200ms
|
||||
- EXIF sanitization: 100-500ms
|
||||
- Total overhead: ~200-700ms per upload
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **python-magic Import Error**
|
||||
|
||||
- Install libmagic system library
|
||||
- Verify python-magic installation
|
||||
|
||||
2. **False Positives**
|
||||
- Review security score thresholds
|
||||
- Adjust validation settings
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging to troubleshoot validation issues:
|
||||
|
||||
```python
|
||||
LOGGING = {
|
||||
"loggers": {
|
||||
"ivatar.security": {
|
||||
"level": "DEBUG",
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Deployment Recommendations
|
||||
|
||||
1. **Enable All Security Features** in production
|
||||
2. **Monitor Security Logs** regularly
|
||||
3. **Keep Dependencies Updated**
|
||||
4. **Regular Security Audits** of uploaded content
|
||||
|
||||
### Monitoring
|
||||
|
||||
- Monitor security.log for violations
|
||||
- Track upload success/failure rates
|
||||
- Alert on repeated security violations
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential future improvements:
|
||||
|
||||
- Virus scanning integration (ClamAV)
|
||||
- Content-based image analysis
|
||||
- Machine learning threat detection
|
||||
- Advanced polyglot detection
|
||||
- Real-time threat intelligence feeds
|
||||
461
OPENTELEMETRY.md
Normal file
461
OPENTELEMETRY.md
Normal file
@@ -0,0 +1,461 @@
|
||||
# OpenTelemetry Integration for ivatar
|
||||
|
||||
This document describes the OpenTelemetry integration implemented in the ivatar project, providing comprehensive observability for avatar generation, file uploads, authentication, and system performance.
|
||||
|
||||
## Overview
|
||||
|
||||
OpenTelemetry is integrated into ivatar to provide:
|
||||
|
||||
- **Distributed Tracing**: Track requests across the entire avatar generation pipeline
|
||||
- **Custom Metrics**: Monitor avatar-specific operations and performance
|
||||
- **Multi-Instance Support**: Distinguish between production and development environments
|
||||
- **Infrastructure Integration**: Works with existing Prometheus/Grafana stack
|
||||
|
||||
## Architecture
|
||||
|
||||
### Components
|
||||
|
||||
1. **OpenTelemetry Configuration** (`ivatar/opentelemetry_config.py`)
|
||||
|
||||
- Centralized configuration management
|
||||
- Environment-based setup
|
||||
- Resource creation with service metadata
|
||||
|
||||
2. **Custom Middleware** (`ivatar/opentelemetry_middleware.py`)
|
||||
|
||||
- Request/response tracing
|
||||
- Avatar-specific metrics
|
||||
- Custom decorators for operation tracing
|
||||
|
||||
3. **Instrumentation Integration**
|
||||
- Django framework instrumentation
|
||||
- Database query tracing (PostgreSQL/MySQL)
|
||||
- HTTP client instrumentation
|
||||
- Cache instrumentation (Memcached)
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default | Required |
|
||||
| ----------------------------- | ------------------------------------ | -------------- | -------- |
|
||||
| `OTEL_ENABLED` | Enable OpenTelemetry | `false` | No |
|
||||
| `OTEL_SERVICE_NAME` | Service name identifier | `ivatar` | No |
|
||||
| `OTEL_ENVIRONMENT` | Environment (production/development) | `development` | No |
|
||||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector endpoint | None | No |
|
||||
| `OTEL_PROMETHEUS_ENDPOINT` | Prometheus metrics endpoint | `0.0.0.0:9464` | No |
|
||||
| `IVATAR_VERSION` | Application version | `1.8.0` | No |
|
||||
| `HOSTNAME` | Instance identifier | `unknown` | No |
|
||||
|
||||
### Multi-Instance Configuration
|
||||
|
||||
#### Production Environment
|
||||
|
||||
```bash
|
||||
export OTEL_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-production
|
||||
export OTEL_ENVIRONMENT=production
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
export OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
export IVATAR_VERSION=1.8.0
|
||||
export HOSTNAME=prod-instance-01
|
||||
```
|
||||
|
||||
#### Development Environment
|
||||
|
||||
```bash
|
||||
export OTEL_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-development
|
||||
export OTEL_ENVIRONMENT=development
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
export OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
export IVATAR_VERSION=1.8.0-dev
|
||||
export HOSTNAME=dev-instance-01
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
### Custom Metrics
|
||||
|
||||
#### Avatar Operations
|
||||
|
||||
- `ivatar_requests_total`: Total HTTP requests by method, status, path
|
||||
- `ivatar_request_duration_seconds`: Request duration histogram
|
||||
- `ivatar_avatar_requests_total`: Avatar requests by status, size, format
|
||||
- `ivatar_avatar_generation_seconds`: Avatar generation time histogram
|
||||
- `ivatar_avatars_generated_total`: Avatars generated by size, format, source
|
||||
- `ivatar_avatar_cache_hits_total`: Cache hits by size, format
|
||||
- `ivatar_avatar_cache_misses_total`: Cache misses by size, format
|
||||
- `ivatar_external_avatar_requests_total`: External service requests
|
||||
- `ivatar_file_uploads_total`: File uploads by content type, success
|
||||
- `ivatar_file_upload_size_bytes`: File upload size histogram
|
||||
|
||||
#### Labels/Dimensions
|
||||
|
||||
- `method`: HTTP method (GET, POST, etc.)
|
||||
- `status_code`: HTTP status code
|
||||
- `path`: Request path
|
||||
- `size`: Avatar size (80, 128, 256, etc.)
|
||||
- `format`: Image format (png, jpg, gif, etc.)
|
||||
- `source`: Avatar source (uploaded, generated, external)
|
||||
- `service`: External service name (gravatar, bluesky)
|
||||
- `content_type`: File MIME type
|
||||
- `success`: Operation success (true/false)
|
||||
|
||||
### Example Queries
|
||||
|
||||
#### Avatar Generation Rate
|
||||
|
||||
```promql
|
||||
rate(ivatar_avatars_generated_total[5m])
|
||||
```
|
||||
|
||||
#### Cache Hit Ratio
|
||||
|
||||
```promql
|
||||
rate(ivatar_avatar_cache_hits_total[5m]) /
|
||||
(rate(ivatar_avatar_cache_hits_total[5m]) + rate(ivatar_avatar_cache_misses_total[5m]))
|
||||
```
|
||||
|
||||
#### Average Avatar Generation Time
|
||||
|
||||
```promql
|
||||
histogram_quantile(0.95, rate(ivatar_avatar_generation_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
#### File Upload Success Rate
|
||||
|
||||
```promql
|
||||
rate(ivatar_file_uploads_total{success="true"}[5m]) /
|
||||
rate(ivatar_file_uploads_total[5m])
|
||||
```
|
||||
|
||||
## Tracing
|
||||
|
||||
### Trace Points
|
||||
|
||||
#### Request Lifecycle
|
||||
|
||||
- HTTP request processing
|
||||
- Avatar generation pipeline
|
||||
- File upload and processing
|
||||
- Authentication flows
|
||||
- External API calls
|
||||
|
||||
#### Custom Spans
|
||||
|
||||
- `avatar.generate_png`: PNG image generation
|
||||
- `avatar.gravatar_proxy`: Gravatar service proxy
|
||||
- `file_upload.process`: File upload processing
|
||||
- `auth.login`: User authentication
|
||||
- `auth.logout`: User logout
|
||||
|
||||
### Span Attributes
|
||||
|
||||
#### HTTP Attributes
|
||||
|
||||
- `http.method`: HTTP method
|
||||
- `http.url`: Full request URL
|
||||
- `http.status_code`: Response status code
|
||||
- `http.user_agent`: Client user agent
|
||||
- `http.remote_addr`: Client IP address
|
||||
|
||||
#### Avatar Attributes
|
||||
|
||||
- `ivatar.request_type`: Request type (avatar, stats, etc.)
|
||||
- `ivatar.avatar_size`: Requested avatar size
|
||||
- `ivatar.avatar_format`: Requested format
|
||||
- `ivatar.avatar_email`: Email address (if applicable)
|
||||
|
||||
#### File Attributes
|
||||
|
||||
- `file.name`: Uploaded file name
|
||||
- `file.size`: File size in bytes
|
||||
- `file.content_type`: MIME type
|
||||
|
||||
## Infrastructure Requirements
|
||||
|
||||
### Option A: Extend Existing Stack (Recommended)
|
||||
|
||||
The existing monitoring stack can be extended to support OpenTelemetry:
|
||||
|
||||
#### Alloy Configuration
|
||||
|
||||
```yaml
|
||||
# Add to existing Alloy configuration
|
||||
otelcol.receiver.otlp:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
otelcol.processor.batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 1024
|
||||
|
||||
otelcol.exporter.prometheus:
|
||||
endpoint: "0.0.0.0:9464"
|
||||
|
||||
otelcol.exporter.jaeger:
|
||||
endpoint: "jaeger-collector:14250"
|
||||
|
||||
otelcol.pipeline.traces:
|
||||
receivers: [otelcol.receiver.otlp]
|
||||
processors: [otelcol.processor.batch]
|
||||
exporters: [otelcol.exporter.jaeger]
|
||||
|
||||
otelcol.pipeline.metrics:
|
||||
receivers: [otelcol.receiver.otlp]
|
||||
processors: [otelcol.processor.batch]
|
||||
exporters: [otelcol.exporter.prometheus]
|
||||
```
|
||||
|
||||
#### Prometheus Configuration
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: "ivatar-opentelemetry"
|
||||
static_configs:
|
||||
- targets: ["ivatar-prod:9464", "ivatar-dev:9464"]
|
||||
scrape_interval: 15s
|
||||
metrics_path: /metrics
|
||||
```
|
||||
|
||||
### Option B: Dedicated OpenTelemetry Collector
|
||||
|
||||
For full OpenTelemetry features, deploy a dedicated collector:
|
||||
|
||||
#### Collector Configuration
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 1024
|
||||
resource:
|
||||
attributes:
|
||||
- key: environment
|
||||
from_attribute: deployment.environment
|
||||
action: insert
|
||||
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:9464"
|
||||
jaeger:
|
||||
endpoint: "jaeger-collector:14250"
|
||||
logging:
|
||||
loglevel: debug
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [batch, resource]
|
||||
exporters: [jaeger, logging]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [batch, resource]
|
||||
exporters: [prometheus, logging]
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Development Setup
|
||||
|
||||
1. **Install Dependencies**
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
2. **Configure Environment**
|
||||
|
||||
```bash
|
||||
export OTEL_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-development
|
||||
export OTEL_ENVIRONMENT=development
|
||||
```
|
||||
|
||||
3. **Start Development Server**
|
||||
|
||||
```bash
|
||||
./manage.py runserver 0:8080
|
||||
```
|
||||
|
||||
4. **Verify Metrics**
|
||||
```bash
|
||||
curl http://localhost:9464/metrics
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
|
||||
1. **Update Container Images**
|
||||
|
||||
- Add OpenTelemetry dependencies to requirements.txt
|
||||
- Update container build process
|
||||
|
||||
2. **Configure Environment Variables**
|
||||
|
||||
- Set production-specific OpenTelemetry variables
|
||||
- Configure collector endpoints
|
||||
|
||||
3. **Update Monitoring Stack**
|
||||
|
||||
- Extend Alloy configuration
|
||||
- Update Prometheus scrape configs
|
||||
- Configure Grafana dashboards
|
||||
|
||||
4. **Verify Deployment**
|
||||
- Check metrics endpoint accessibility
|
||||
- Verify trace data flow
|
||||
- Monitor dashboard updates
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Key Metrics to Monitor
|
||||
|
||||
#### Performance
|
||||
|
||||
- Request duration percentiles (p50, p95, p99)
|
||||
- Avatar generation time
|
||||
- Cache hit ratio
|
||||
- File upload success rate
|
||||
|
||||
#### Business Metrics
|
||||
|
||||
- Avatar requests per minute
|
||||
- Popular avatar sizes
|
||||
- External service usage
|
||||
- User authentication success rate
|
||||
|
||||
#### Error Rates
|
||||
|
||||
- HTTP error rates by endpoint
|
||||
- File upload failures
|
||||
- External service failures
|
||||
- Authentication failures
|
||||
|
||||
### Example Alerts
|
||||
|
||||
#### High Error Rate
|
||||
|
||||
```yaml
|
||||
alert: HighErrorRate
|
||||
expr: rate(ivatar_requests_total{status_code=~"5.."}[5m]) > 0.1
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High error rate detected"
|
||||
description: "Error rate is {{ $value }} errors per second"
|
||||
```
|
||||
|
||||
#### Slow Avatar Generation
|
||||
|
||||
```yaml
|
||||
alert: SlowAvatarGeneration
|
||||
expr: histogram_quantile(0.95, rate(ivatar_avatar_generation_seconds_bucket[5m])) > 2
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Slow avatar generation"
|
||||
description: "95th percentile avatar generation time is {{ $value }}s"
|
||||
```
|
||||
|
||||
#### Low Cache Hit Ratio
|
||||
|
||||
```yaml
|
||||
alert: LowCacheHitRatio
|
||||
expr: (rate(ivatar_avatar_cache_hits_total[5m]) / (rate(ivatar_avatar_cache_hits_total[5m]) + rate(ivatar_avatar_cache_misses_total[5m]))) < 0.8
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Low cache hit ratio"
|
||||
description: "Cache hit ratio is {{ $value }}"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### OpenTelemetry Not Enabled
|
||||
|
||||
- Check `OTEL_ENABLED` environment variable
|
||||
- Verify OpenTelemetry packages are installed
|
||||
- Check Django logs for configuration errors
|
||||
|
||||
#### Metrics Not Appearing
|
||||
|
||||
- Verify Prometheus endpoint is accessible
|
||||
- Check collector configuration
|
||||
- Ensure metrics are being generated
|
||||
|
||||
#### Traces Not Showing
|
||||
|
||||
- Verify OTLP endpoint configuration
|
||||
- Check collector connectivity
|
||||
- Ensure tracing is enabled in configuration
|
||||
|
||||
#### High Memory Usage
|
||||
|
||||
- Adjust batch processor settings
|
||||
- Reduce trace sampling rate
|
||||
- Monitor collector resource usage
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging for OpenTelemetry:
|
||||
|
||||
```python
|
||||
LOGGING = {
|
||||
"loggers": {
|
||||
"opentelemetry": {
|
||||
"level": "DEBUG",
|
||||
},
|
||||
"ivatar.opentelemetry": {
|
||||
"level": "DEBUG",
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
|
||||
- **Sampling**: Implement trace sampling for high-traffic production
|
||||
- **Batch Processing**: Use appropriate batch sizes for your infrastructure
|
||||
- **Resource Limits**: Monitor collector resource usage
|
||||
- **Network**: Ensure low-latency connections to collectors
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **Data Privacy**: Ensure no sensitive data in trace attributes
|
||||
- **Network Security**: Use TLS for collector communications
|
||||
- **Access Control**: Restrict access to metrics endpoints
|
||||
- **Data Retention**: Configure appropriate retention policies
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Custom Dashboards**: Create Grafana dashboards for avatar metrics
|
||||
- **Advanced Sampling**: Implement intelligent trace sampling
|
||||
- **Log Correlation**: Correlate traces with application logs
|
||||
- **Performance Profiling**: Add profiling capabilities
|
||||
- **Custom Exports**: Export to additional backends (Datadog, New Relic)
|
||||
|
||||
## Support
|
||||
|
||||
For issues related to OpenTelemetry integration:
|
||||
|
||||
- Check application logs for configuration errors
|
||||
- Verify collector connectivity
|
||||
- Review Prometheus metrics for data flow
|
||||
- Consult OpenTelemetry documentation for advanced configuration
|
||||
433
OPENTELEMETRY_INFRASTRUCTURE.md
Normal file
433
OPENTELEMETRY_INFRASTRUCTURE.md
Normal file
@@ -0,0 +1,433 @@
|
||||
# OpenTelemetry Infrastructure Requirements
|
||||
|
||||
This document outlines the infrastructure requirements and deployment strategy for OpenTelemetry in the ivatar project, considering the existing Fedora Project hosting environment and multi-instance setup.
|
||||
|
||||
## Current Infrastructure Analysis
|
||||
|
||||
### Existing Monitoring Stack
|
||||
|
||||
- **Prometheus + Alertmanager**: Metrics collection and alerting
|
||||
- **Loki**: Log aggregation
|
||||
- **Alloy**: Observability data collection
|
||||
- **Grafana**: Visualization and dashboards
|
||||
- **Custom exporters**: Application-specific metrics
|
||||
|
||||
### Production Environment
|
||||
|
||||
- **Scale**: Millions of requests daily, 30k+ users, 33k+ avatar images
|
||||
- **Infrastructure**: Fedora Project hosted, high-performance system
|
||||
- **Architecture**: Apache HTTPD + Gunicorn containers + PostgreSQL
|
||||
- **Containerization**: Podman (not Docker)
|
||||
|
||||
### Multi-Instance Setup
|
||||
|
||||
- **Production**: Production environment (master branch)
|
||||
- **Development**: Development environment (devel branch)
|
||||
- **Deployment**: GitLab CI/CD with Puppet automation
|
||||
|
||||
## Infrastructure Options
|
||||
|
||||
### Option A: Extend Existing Alloy Stack (Recommended)
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Leverages existing infrastructure
|
||||
- Minimal additional complexity
|
||||
- Consistent with current monitoring approach
|
||||
- Cost-effective
|
||||
|
||||
**Implementation:**
|
||||
|
||||
```yaml
|
||||
# Alloy configuration extension
|
||||
otelcol.receiver.otlp:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
otelcol.processor.batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 1024
|
||||
|
||||
otelcol.exporter.prometheus:
|
||||
endpoint: "0.0.0.0:9464"
|
||||
|
||||
otelcol.exporter.jaeger:
|
||||
endpoint: "jaeger-collector:14250"
|
||||
|
||||
otelcol.pipeline.traces:
|
||||
receivers: [otelcol.receiver.otlp]
|
||||
processors: [otelcol.processor.batch]
|
||||
exporters: [otelcol.exporter.jaeger]
|
||||
|
||||
otelcol.pipeline.metrics:
|
||||
receivers: [otelcol.receiver.otlp]
|
||||
processors: [otelcol.processor.batch]
|
||||
exporters: [otelcol.exporter.prometheus]
|
||||
```
|
||||
|
||||
### Option B: Dedicated OpenTelemetry Collector
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Full OpenTelemetry feature set
|
||||
- Better performance for high-volume tracing
|
||||
- More flexible configuration options
|
||||
- Future-proof architecture
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Deploy standalone OpenTelemetry Collector
|
||||
- Configure OTLP receivers and exporters
|
||||
- Integrate with existing Prometheus/Grafana
|
||||
|
||||
## Deployment Strategy
|
||||
|
||||
### Phase 1: Development Environment
|
||||
|
||||
1. **Enable OpenTelemetry in Development**
|
||||
|
||||
```bash
|
||||
# Development environment configuration
|
||||
export OTEL_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-development
|
||||
export OTEL_ENVIRONMENT=development
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
export OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
```
|
||||
|
||||
2. **Update Alloy Configuration**
|
||||
|
||||
- Add OTLP receivers to existing Alloy instance
|
||||
- Configure trace and metrics pipelines
|
||||
- Test data flow
|
||||
|
||||
3. **Verify Integration**
|
||||
- Check metrics endpoint: `http://dev-instance:9464/metrics`
|
||||
- Verify trace data in Jaeger
|
||||
- Monitor Grafana dashboards
|
||||
|
||||
### Phase 2: Production Deployment
|
||||
|
||||
1. **Production Configuration**
|
||||
|
||||
```bash
|
||||
# Production environment configuration
|
||||
export OTEL_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-production
|
||||
export OTEL_ENVIRONMENT=production
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
export OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
```
|
||||
|
||||
2. **Gradual Rollout**
|
||||
|
||||
- Deploy to one Gunicorn container first
|
||||
- Monitor performance impact
|
||||
- Gradually enable on all containers
|
||||
|
||||
3. **Performance Monitoring**
|
||||
- Monitor collector resource usage
|
||||
- Check application performance impact
|
||||
- Verify data quality
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
### Collector Resources
|
||||
|
||||
**Minimum Requirements:**
|
||||
|
||||
- CPU: 2 cores
|
||||
- Memory: 4GB RAM
|
||||
- Storage: 10GB for temporary data
|
||||
- Network: 1Gbps
|
||||
|
||||
**Recommended for Production:**
|
||||
|
||||
- CPU: 4 cores
|
||||
- Memory: 8GB RAM
|
||||
- Storage: 50GB SSD
|
||||
- Network: 10Gbps
|
||||
|
||||
### Network Requirements
|
||||
|
||||
**Ports:**
|
||||
|
||||
- 4317: OTLP gRPC receiver
|
||||
- 4318: OTLP HTTP receiver
|
||||
- 9464: Prometheus metrics exporter
|
||||
- 14250: Jaeger trace exporter
|
||||
|
||||
**Bandwidth:**
|
||||
|
||||
- Estimated 1-5 Mbps per instance
|
||||
- Burst capacity for peak loads
|
||||
- Low-latency connection to collectors
|
||||
|
||||
## Configuration Management
|
||||
|
||||
### Environment-Specific Settings
|
||||
|
||||
#### Production Environment
|
||||
|
||||
```bash
|
||||
# Production OpenTelemetry configuration
|
||||
OTEL_ENABLED=true
|
||||
OTEL_SERVICE_NAME=ivatar-production
|
||||
OTEL_ENVIRONMENT=production
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
OTEL_SAMPLING_RATIO=0.1 # 10% sampling for high volume
|
||||
IVATAR_VERSION=1.8.0
|
||||
HOSTNAME=prod-instance-01
|
||||
```
|
||||
|
||||
#### Development Environment
|
||||
|
||||
```bash
|
||||
# Development OpenTelemetry configuration
|
||||
OTEL_ENABLED=true
|
||||
OTEL_SERVICE_NAME=ivatar-development
|
||||
OTEL_ENVIRONMENT=development
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
|
||||
OTEL_PROMETHEUS_ENDPOINT=0.0.0.0:9464
|
||||
OTEL_SAMPLING_RATIO=1.0 # 100% sampling for debugging
|
||||
IVATAR_VERSION=1.8.0-dev
|
||||
HOSTNAME=dev-instance-01
|
||||
```
|
||||
|
||||
### Container Configuration
|
||||
|
||||
#### Podman Container Updates
|
||||
|
||||
```dockerfile
|
||||
# Add to existing Dockerfile
|
||||
RUN pip install opentelemetry-api>=1.20.0 \
|
||||
opentelemetry-sdk>=1.20.0 \
|
||||
opentelemetry-instrumentation-django>=0.42b0 \
|
||||
opentelemetry-instrumentation-psycopg2>=0.42b0 \
|
||||
opentelemetry-instrumentation-pymysql>=0.42b0 \
|
||||
opentelemetry-instrumentation-requests>=0.42b0 \
|
||||
opentelemetry-instrumentation-urllib3>=0.42b0 \
|
||||
opentelemetry-exporter-otlp>=1.20.0 \
|
||||
opentelemetry-exporter-prometheus>=1.12.0rc1 \
|
||||
opentelemetry-instrumentation-memcached>=0.42b0
|
||||
```
|
||||
|
||||
#### Container Environment Variables
|
||||
|
||||
```bash
|
||||
# Add to container startup script
|
||||
export OTEL_ENABLED=${OTEL_ENABLED:-false}
|
||||
export OTEL_SERVICE_NAME=${OTEL_SERVICE_NAME:-ivatar}
|
||||
export OTEL_ENVIRONMENT=${OTEL_ENVIRONMENT:-development}
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
|
||||
export OTEL_PROMETHEUS_ENDPOINT=${OTEL_PROMETHEUS_ENDPOINT:-0.0.0.0:9464}
|
||||
```
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### Collector Health Monitoring
|
||||
|
||||
#### Collector Metrics
|
||||
|
||||
- `otelcol_receiver_accepted_spans`: Spans received by collector
|
||||
- `otelcol_receiver_refused_spans`: Spans rejected by collector
|
||||
- `otelcol_exporter_sent_spans`: Spans sent to exporters
|
||||
- `otelcol_exporter_failed_spans`: Failed span exports
|
||||
|
||||
#### Health Checks
|
||||
|
||||
```yaml
|
||||
# Prometheus health check
|
||||
- job_name: "otel-collector-health"
|
||||
static_configs:
|
||||
- targets: ["collector.internal:8888"]
|
||||
metrics_path: /metrics
|
||||
scrape_interval: 30s
|
||||
```
|
||||
|
||||
### Application Performance Impact
|
||||
|
||||
#### Key Metrics to Monitor
|
||||
|
||||
- Application response time impact
|
||||
- Memory usage increase
|
||||
- CPU usage increase
|
||||
- Network bandwidth usage
|
||||
|
||||
#### Alerting Rules
|
||||
|
||||
```yaml
|
||||
# High collector resource usage
|
||||
alert: HighCollectorCPU
|
||||
expr: rate(otelcol_process_cpu_seconds_total[5m]) > 0.8
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High collector CPU usage"
|
||||
description: "Collector CPU usage is {{ $value }}"
|
||||
|
||||
# Collector memory usage
|
||||
alert: HighCollectorMemory
|
||||
expr: otelcol_process_memory_usage_bytes / otelcol_process_memory_limit_bytes > 0.8
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High collector memory usage"
|
||||
description: "Collector memory usage is {{ $value }}"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Network Security
|
||||
|
||||
- Use TLS for collector communications
|
||||
- Restrict collector access to trusted networks
|
||||
- Implement proper firewall rules
|
||||
|
||||
### Data Privacy
|
||||
|
||||
- Ensure no sensitive data in trace attributes
|
||||
- Implement data sanitization
|
||||
- Configure appropriate retention policies
|
||||
|
||||
### Access Control
|
||||
|
||||
- Restrict access to metrics endpoints
|
||||
- Implement authentication for collector access
|
||||
- Monitor access logs
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Data Retention
|
||||
|
||||
- Traces: 7 days (configurable)
|
||||
- Metrics: 30 days (configurable)
|
||||
- Logs: 14 days (configurable)
|
||||
|
||||
### Backup Strategy
|
||||
|
||||
- Regular backup of collector configuration
|
||||
- Backup of Grafana dashboards
|
||||
- Backup of Prometheus rules
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Sampling Strategy
|
||||
|
||||
- Production: 10% sampling rate
|
||||
- Development: 100% sampling rate
|
||||
- Error traces: Always sample
|
||||
|
||||
### Batch Processing
|
||||
|
||||
- Optimize batch sizes for network conditions
|
||||
- Configure appropriate timeouts
|
||||
- Monitor queue depths
|
||||
|
||||
### Resource Optimization
|
||||
|
||||
- Monitor collector resource usage
|
||||
- Scale collectors based on load
|
||||
- Implement horizontal scaling if needed
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Collector Not Receiving Data
|
||||
|
||||
- Check network connectivity
|
||||
- Verify OTLP endpoint configuration
|
||||
- Check collector logs
|
||||
|
||||
#### High Resource Usage
|
||||
|
||||
- Adjust sampling rates
|
||||
- Optimize batch processing
|
||||
- Scale collector resources
|
||||
|
||||
#### Data Quality Issues
|
||||
|
||||
- Verify instrumentation configuration
|
||||
- Check span attribute quality
|
||||
- Monitor error rates
|
||||
|
||||
### Debug Procedures
|
||||
|
||||
1. **Check Collector Status**
|
||||
|
||||
```bash
|
||||
curl http://collector.internal:8888/metrics
|
||||
```
|
||||
|
||||
2. **Verify Application Configuration**
|
||||
|
||||
```bash
|
||||
curl http://app:9464/metrics
|
||||
```
|
||||
|
||||
3. **Check Trace Data**
|
||||
- Access Jaeger UI
|
||||
- Search for recent traces
|
||||
- Verify span attributes
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Advanced Features
|
||||
|
||||
- Custom dashboards for avatar metrics
|
||||
- Advanced sampling strategies
|
||||
- Log correlation with traces
|
||||
- Performance profiling integration
|
||||
|
||||
### Scalability Improvements
|
||||
|
||||
- Horizontal collector scaling
|
||||
- Load balancing for collectors
|
||||
- Multi-region deployment
|
||||
- Edge collection points
|
||||
|
||||
### Integration Enhancements
|
||||
|
||||
- Additional exporter backends
|
||||
- Custom processors
|
||||
- Advanced filtering
|
||||
- Data transformation
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
### Infrastructure Costs
|
||||
|
||||
- Additional compute resources for collectors
|
||||
- Storage costs for trace data
|
||||
- Network bandwidth costs
|
||||
|
||||
### Operational Costs
|
||||
|
||||
- Monitoring and maintenance
|
||||
- Configuration management
|
||||
- Troubleshooting and support
|
||||
|
||||
### Optimization Strategies
|
||||
|
||||
- Implement efficient sampling
|
||||
- Use appropriate retention policies
|
||||
- Optimize batch processing
|
||||
- Monitor resource usage
|
||||
|
||||
## Conclusion
|
||||
|
||||
The OpenTelemetry integration for ivatar provides comprehensive observability while leveraging the existing monitoring infrastructure. The phased deployment approach ensures minimal disruption to production services while providing valuable insights into avatar generation performance and user behavior.
|
||||
|
||||
Key success factors:
|
||||
|
||||
- Gradual rollout with monitoring
|
||||
- Performance impact assessment
|
||||
- Proper resource planning
|
||||
- Security considerations
|
||||
- Ongoing optimization
|
||||
83
README.md
83
README.md
@@ -10,6 +10,89 @@
|
||||
- [Coverage HTML report](http://oliver.git.linux-kernel.at/ivatar)
|
||||
- [Code documentation (autogenerated, pycco)](http://oliver.git.linux-kernel.at/ivatar/pycco/)
|
||||
|
||||
# Environment Variables
|
||||
|
||||
## OpenTelemetry Configuration
|
||||
|
||||
OpenTelemetry instrumentation is always enabled in ivatar. The following environment variables control the behavior:
|
||||
|
||||
### Core Configuration
|
||||
|
||||
- `OTEL_SERVICE_NAME`: Service name for OpenTelemetry (default: "ivatar")
|
||||
- `OTEL_ENVIRONMENT`: Deployment environment (default: "production")
|
||||
- `OTEL_EXPORT_ENABLED`: Enable/disable data export (default: "false")
|
||||
- Set to "true" to enable sending telemetry data to external collectors
|
||||
- Set to "false" to disable export (instrumentation still active)
|
||||
|
||||
### Export Configuration
|
||||
|
||||
- `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP endpoint for traces and metrics export
|
||||
- Example: "http://localhost:4317" (gRPC) or "http://localhost:4318" (HTTP)
|
||||
- `OTEL_PROMETHEUS_ENDPOINT`: Prometheus metrics endpoint (default: "0.0.0.0:9464")
|
||||
|
||||
## Example Configurations
|
||||
|
||||
### Development (Export Disabled)
|
||||
|
||||
```bash
|
||||
export OTEL_EXPORT_ENABLED=false
|
||||
export OTEL_SERVICE_NAME=ivatar-dev
|
||||
export OTEL_ENVIRONMENT=development
|
||||
```
|
||||
|
||||
### Production (Export Enabled)
|
||||
|
||||
```bash
|
||||
export OTEL_EXPORT_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar
|
||||
export OTEL_ENVIRONMENT=production
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
|
||||
```
|
||||
|
||||
# Testing
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Local Development (Recommended)
|
||||
|
||||
For local development, use the provided script to skip Bluesky tests that require external API credentials:
|
||||
|
||||
```bash
|
||||
./scripts/run_tests_local.sh
|
||||
```
|
||||
|
||||
This runs all tests except those marked with `@pytest.mark.bluesky`.
|
||||
|
||||
### All Tests
|
||||
|
||||
To run all tests including Bluesky tests (requires Bluesky API credentials):
|
||||
|
||||
```bash
|
||||
python3 manage.py test -v3
|
||||
```
|
||||
|
||||
### Specific Test Categories
|
||||
|
||||
```bash
|
||||
# Run only Bluesky tests
|
||||
python3 manage.py test ivatar.ivataraccount.test_views_bluesky -v3
|
||||
|
||||
# Run only file upload security tests
|
||||
python3 manage.py test ivatar.test_file_security -v3
|
||||
|
||||
# Run only upload tests
|
||||
python3 manage.py test ivatar.ivataraccount.test_views -v3
|
||||
```
|
||||
|
||||
## Test Markers
|
||||
|
||||
Tests are categorized using pytest markers:
|
||||
|
||||
- `@pytest.mark.bluesky`: Tests requiring Bluesky API credentials
|
||||
- `@pytest.mark.slow`: Long-running tests
|
||||
- `@pytest.mark.integration`: Integration tests
|
||||
- `@pytest.mark.unit`: Unit tests
|
||||
|
||||
# Authors and contributors
|
||||
|
||||
Lead developer/Owner: Oliver Falk (aka ofalk or falko) - https://git.linux-kernel.at/oliver
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
https://django-debug-toolbar.readthedocs.io/en/latest/installation.html
|
||||
https://stackoverflow.com/questions/6548947/how-can-django-debug-toolbar-be-set-to-work-for-just-some-users/6549317#6549317
|
||||
@@ -1,49 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
import os
|
||||
import django
|
||||
import timeit
|
||||
|
||||
os.environ.setdefault(
|
||||
"DJANGO_SETTINGS_MODULE", "ivatar.settings"
|
||||
) # pylint: disable=wrong-import-position
|
||||
django.setup() # pylint: disable=wrong-import-position
|
||||
|
||||
from ivatar.ivataraccount.models import ConfirmedEmail, APIKey
|
||||
from simplecrypt import decrypt
|
||||
from binascii import unhexlify
|
||||
|
||||
digest = None
|
||||
digest_sha256 = None
|
||||
|
||||
|
||||
def get_digest_sha256():
|
||||
digest_sha256 = ConfirmedEmail.objects.first().encrypted_digest_sha256(
|
||||
secret_key=APIKey.objects.first()
|
||||
)
|
||||
return digest_sha256
|
||||
|
||||
|
||||
def get_digest():
|
||||
digest = ConfirmedEmail.objects.first().encrypted_digest(
|
||||
secret_key=APIKey.objects.first()
|
||||
)
|
||||
return digest
|
||||
|
||||
|
||||
def decrypt_digest():
|
||||
return decrypt(APIKey.objects.first().secret_key, unhexlify(digest))
|
||||
|
||||
|
||||
def decrypt_digest_256():
|
||||
return decrypt(APIKey.objects.first().secret_key, unhexlify(digest_sha256))
|
||||
|
||||
|
||||
digest = get_digest()
|
||||
digest_sha256 = get_digest_sha256()
|
||||
|
||||
print("Encrypt digest: %s" % timeit.timeit(get_digest, number=1))
|
||||
print("Encrypt digest_sha256: %s" % timeit.timeit(get_digest_sha256, number=1))
|
||||
print("Decrypt digest: %s" % timeit.timeit(decrypt_digest, number=1))
|
||||
print("Decrypt digest_sha256: %s" % timeit.timeit(decrypt_digest_256, number=1))
|
||||
@@ -1,7 +0,0 @@
|
||||
DATABASES['default'] = {
|
||||
'ENGINE': 'django.db.backends.mysql',
|
||||
'NAME': 'libravatar',
|
||||
'USER': 'libravatar',
|
||||
'PASSWORD': 'libravatar',
|
||||
'HOST': 'localhost',
|
||||
}
|
||||
30
config.py
30
config.py
@@ -34,6 +34,12 @@ MIDDLEWARE.extend(
|
||||
"ivatar.middleware.CustomLocaleMiddleware",
|
||||
]
|
||||
)
|
||||
|
||||
# Add OpenTelemetry middleware (always enabled now)
|
||||
MIDDLEWARE.insert(0, "ivatar.opentelemetry_middleware.OpenTelemetryMiddleware")
|
||||
|
||||
# Add OpenTelemetry middleware only if feature flag is enabled
|
||||
# Note: This will be checked at runtime, not at import time
|
||||
MIDDLEWARE.insert(
|
||||
0,
|
||||
"ivatar.middleware.MultipleProxyMiddleware",
|
||||
@@ -166,9 +172,10 @@ if "POSTGRES_DB" in os.environ:
|
||||
"USER": os.environ["POSTGRES_USER"],
|
||||
"PASSWORD": os.environ["POSTGRES_PASSWORD"],
|
||||
"HOST": os.environ["POSTGRES_HOST"],
|
||||
"TEST": {
|
||||
"NAME": os.environ["POSTGRES_DB"],
|
||||
},
|
||||
# Let Django use its default test database naming
|
||||
# "TEST": {
|
||||
# "NAME": os.environ["POSTGRES_DB"],
|
||||
# },
|
||||
}
|
||||
|
||||
SESSION_SERIALIZER = "django.contrib.sessions.serializers.JSONSerializer"
|
||||
@@ -296,9 +303,26 @@ TRUSTED_DEFAULT_URLS = list(map(map_legacy_config, TRUSTED_DEFAULT_URLS))
|
||||
BLUESKY_IDENTIFIER = os.environ.get("BLUESKY_IDENTIFIER", None)
|
||||
BLUESKY_APP_PASSWORD = os.environ.get("BLUESKY_APP_PASSWORD", None)
|
||||
|
||||
# File upload security settings
|
||||
FILE_UPLOAD_MAX_MEMORY_SIZE = 5 * 1024 * 1024 # 5MB
|
||||
DATA_UPLOAD_MAX_MEMORY_SIZE = 5 * 1024 * 1024 # 5MB
|
||||
FILE_UPLOAD_PERMISSIONS = 0o644
|
||||
|
||||
# Enhanced file upload security
|
||||
ENABLE_FILE_SECURITY_VALIDATION = True
|
||||
ENABLE_EXIF_SANITIZATION = True
|
||||
ENABLE_MALICIOUS_CONTENT_SCAN = True
|
||||
|
||||
# Logging configuration - can be overridden in local config
|
||||
# Example: LOGS_DIR = "/var/log/ivatar" # For production deployments
|
||||
|
||||
# OpenTelemetry feature flag - can be disabled for F/LOSS deployments
|
||||
ENABLE_OPENTELEMETRY = os.environ.get("ENABLE_OPENTELEMETRY", "false").lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
)
|
||||
|
||||
# This MUST BE THE LAST!
|
||||
if os.path.isfile(os.path.join(BASE_DIR, "config_local.py")):
|
||||
from config_local import * # noqa # flake8: noqa # NOQA # pragma: no cover
|
||||
|
||||
@@ -12,6 +12,11 @@ import os
|
||||
# Override logs directory for development with custom location
|
||||
# LOGS_DIR = os.path.join(os.path.expanduser("~"), "ivatar_logs")
|
||||
|
||||
# File upload security settings
|
||||
# ENABLE_FILE_SECURITY_VALIDATION = True
|
||||
# ENABLE_EXIF_SANITIZATION = True
|
||||
# ENABLE_MALICIOUS_CONTENT_SCAN = True
|
||||
|
||||
# Example production overrides:
|
||||
# DEBUG = False
|
||||
# SECRET_KEY = "your-production-secret-key-here"
|
||||
@@ -39,3 +44,7 @@ import os
|
||||
|
||||
# Example: Override logs directory for production
|
||||
# LOGS_DIR = "/var/log/ivatar"
|
||||
|
||||
# Bluesky integration credentials
|
||||
# BLUESKY_IDENTIFIER = "your-bluesky-handle"
|
||||
# BLUESKY_APP_PASSWORD = "your-app-password"
|
||||
|
||||
3
config_local_test.py
Normal file
3
config_local_test.py
Normal file
@@ -0,0 +1,3 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Test configuration to verify LOGS_DIR override
|
||||
LOGS_DIR = "/tmp/ivatar_test_logs"
|
||||
1
cropperjs.zip
Normal file
1
cropperjs.zip
Normal file
@@ -0,0 +1 @@
|
||||
Not Found
|
||||
337
ivatar/file_security.py
Normal file
337
ivatar/file_security.py
Normal file
@@ -0,0 +1,337 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
File upload security utilities for ivatar
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import magic
|
||||
import os
|
||||
from io import BytesIO
|
||||
from typing import Dict, Tuple
|
||||
|
||||
from PIL import Image
|
||||
|
||||
# Initialize logger
|
||||
logger = logging.getLogger("ivatar.security")
|
||||
|
||||
# Security constants
|
||||
ALLOWED_MIME_TYPES = [
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/gif",
|
||||
"image/webp",
|
||||
"image/bmp",
|
||||
"image/tiff",
|
||||
]
|
||||
|
||||
ALLOWED_EXTENSIONS = [".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".tiff"]
|
||||
|
||||
# Magic byte signatures for image formats
|
||||
IMAGE_SIGNATURES = {
|
||||
b"\xff\xd8\xff": "image/jpeg",
|
||||
b"\x89PNG\r\n\x1a\n": "image/png",
|
||||
b"GIF87a": "image/gif",
|
||||
b"GIF89a": "image/gif",
|
||||
b"RIFF": "image/webp", # WebP starts with RIFF
|
||||
b"BM": "image/bmp",
|
||||
b"II*\x00": "image/tiff", # Little-endian TIFF
|
||||
b"MM\x00*": "image/tiff", # Big-endian TIFF
|
||||
}
|
||||
|
||||
# Maximum file size for different operations (in bytes)
|
||||
MAX_FILE_SIZE_BASIC = 5 * 1024 * 1024 # 5MB for basic validation
|
||||
MAX_FILE_SIZE_SCAN = 10 * 1024 * 1024 # 10MB for virus scanning
|
||||
MAX_FILE_SIZE_PROCESS = 50 * 1024 * 1024 # 50MB for processing
|
||||
|
||||
|
||||
class FileUploadSecurityError(Exception):
|
||||
"""Custom exception for file upload security issues"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class FileValidator:
|
||||
"""Comprehensive file validation for uploads"""
|
||||
|
||||
def __init__(self, file_data: bytes, filename: str):
|
||||
self.file_data = file_data
|
||||
self.filename = filename
|
||||
self.file_size = len(file_data)
|
||||
self.file_hash = hashlib.sha256(file_data).hexdigest()
|
||||
|
||||
def validate_basic(self) -> Dict[str, any]:
|
||||
"""
|
||||
Perform basic file validation
|
||||
Returns validation results dictionary
|
||||
"""
|
||||
results = {
|
||||
"valid": True,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"file_info": {
|
||||
"size": self.file_size,
|
||||
"hash": self.file_hash,
|
||||
"filename": self.filename,
|
||||
},
|
||||
}
|
||||
|
||||
# Check file size
|
||||
if self.file_size > MAX_FILE_SIZE_BASIC:
|
||||
results["valid"] = False
|
||||
results["errors"].append(f"File too large: {self.file_size} bytes")
|
||||
|
||||
# Check filename
|
||||
if not self.filename or len(self.filename) > 255:
|
||||
results["valid"] = False
|
||||
results["errors"].append("Invalid filename")
|
||||
|
||||
# Check file extension
|
||||
ext = os.path.splitext(self.filename)[1].lower()
|
||||
if ext not in ALLOWED_EXTENSIONS:
|
||||
results["valid"] = False
|
||||
results["errors"].append(f"File extension not allowed: {ext}")
|
||||
|
||||
return results
|
||||
|
||||
def validate_magic_bytes(self) -> Dict[str, any]:
|
||||
"""
|
||||
Validate file using magic bytes (file signatures)
|
||||
"""
|
||||
results = {"valid": True, "detected_type": None, "errors": []}
|
||||
|
||||
# Check magic bytes
|
||||
detected_type = None
|
||||
for signature, mime_type in IMAGE_SIGNATURES.items():
|
||||
if self.file_data.startswith(signature):
|
||||
detected_type = mime_type
|
||||
break
|
||||
|
||||
# Special handling for WebP (RIFF + WEBP)
|
||||
if self.file_data.startswith(b"RIFF") and b"WEBP" in self.file_data[:12]:
|
||||
detected_type = "image/webp"
|
||||
|
||||
if not detected_type:
|
||||
results["valid"] = False
|
||||
results["errors"].append(
|
||||
"File signature does not match any supported image format"
|
||||
)
|
||||
else:
|
||||
results["detected_type"] = detected_type
|
||||
|
||||
return results
|
||||
|
||||
def validate_mime_type(self) -> Dict[str, any]:
|
||||
"""
|
||||
Validate MIME type using python-magic
|
||||
"""
|
||||
results = {"valid": True, "detected_mime": None, "errors": []}
|
||||
|
||||
try:
|
||||
# Use python-magic to detect MIME type
|
||||
detected_mime = magic.from_buffer(self.file_data, mime=True)
|
||||
results["detected_mime"] = detected_mime
|
||||
|
||||
if detected_mime not in ALLOWED_MIME_TYPES:
|
||||
results["valid"] = False
|
||||
results["errors"].append(f"MIME type not allowed: {detected_mime}")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"MIME type detection failed: {e}")
|
||||
results["warnings"].append("Could not detect MIME type")
|
||||
|
||||
return results
|
||||
|
||||
def validate_pil_image(self) -> Dict[str, any]:
|
||||
"""
|
||||
Validate using PIL to ensure it's a valid image
|
||||
"""
|
||||
results = {"valid": True, "image_info": {}, "errors": []}
|
||||
|
||||
try:
|
||||
# Open image with PIL
|
||||
image = Image.open(BytesIO(self.file_data))
|
||||
|
||||
# Get image information
|
||||
results["image_info"] = {
|
||||
"format": image.format,
|
||||
"mode": image.mode,
|
||||
"size": image.size,
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"has_transparency": image.mode in ("RGBA", "LA", "P"),
|
||||
}
|
||||
|
||||
# Verify image can be loaded
|
||||
image.load()
|
||||
|
||||
# Check for suspicious characteristics
|
||||
if image.width > 10000 or image.height > 10000:
|
||||
results["warnings"].append("Image dimensions are very large")
|
||||
|
||||
if image.width < 1 or image.height < 1:
|
||||
results["valid"] = False
|
||||
results["errors"].append("Invalid image dimensions")
|
||||
|
||||
except Exception as e:
|
||||
results["valid"] = False
|
||||
results["errors"].append(f"Invalid image format: {str(e)}")
|
||||
|
||||
return results
|
||||
|
||||
def sanitize_exif_data(self) -> bytes:
|
||||
"""
|
||||
Remove EXIF data from image to prevent metadata leaks
|
||||
"""
|
||||
try:
|
||||
image = Image.open(BytesIO(self.file_data))
|
||||
|
||||
# Create new image without EXIF data
|
||||
if image.mode in ("RGBA", "LA"):
|
||||
# Preserve transparency
|
||||
new_image = Image.new("RGBA", image.size, (255, 255, 255, 0))
|
||||
new_image.paste(image, mask=image.split()[-1])
|
||||
else:
|
||||
new_image = Image.new("RGB", image.size, (255, 255, 255))
|
||||
new_image.paste(image)
|
||||
|
||||
# Save without EXIF data
|
||||
output = BytesIO()
|
||||
new_image.save(output, format=image.format or "JPEG", quality=95)
|
||||
return output.getvalue()
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"EXIF sanitization failed: {e}")
|
||||
return self.file_data # Return original if sanitization fails
|
||||
|
||||
def scan_for_malicious_content(self) -> Dict[str, any]:
|
||||
"""
|
||||
Scan for potentially malicious content patterns
|
||||
"""
|
||||
results = {"suspicious": False, "threats": [], "warnings": []}
|
||||
|
||||
# Check for embedded scripts or executable content
|
||||
suspicious_patterns = [
|
||||
b"<script",
|
||||
b"javascript:",
|
||||
b"vbscript:",
|
||||
b"data:text/html",
|
||||
b"<?php",
|
||||
b"<%",
|
||||
b"#!/bin/",
|
||||
b"MZ", # PE executable header
|
||||
b"\x7fELF", # ELF executable header
|
||||
]
|
||||
|
||||
for pattern in suspicious_patterns:
|
||||
if pattern in self.file_data:
|
||||
results["suspicious"] = True
|
||||
results["threats"].append(f"Suspicious pattern detected: {pattern}")
|
||||
|
||||
# Check for polyglot files (valid in multiple formats)
|
||||
if self.file_data.startswith(b"GIF89a") and b"<script" in self.file_data:
|
||||
results["suspicious"] = True
|
||||
results["threats"].append("Potential polyglot attack detected")
|
||||
|
||||
return results
|
||||
|
||||
def comprehensive_validation(self) -> Dict[str, any]:
|
||||
"""
|
||||
Perform comprehensive file validation
|
||||
"""
|
||||
results = {
|
||||
"valid": True,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"file_info": {},
|
||||
"security_score": 100,
|
||||
}
|
||||
|
||||
# Basic validation
|
||||
basic_results = self.validate_basic()
|
||||
if not basic_results["valid"]:
|
||||
results["valid"] = False
|
||||
results["errors"].extend(basic_results["errors"])
|
||||
results["security_score"] -= 30
|
||||
|
||||
results["file_info"].update(basic_results["file_info"])
|
||||
results["warnings"].extend(basic_results["warnings"])
|
||||
|
||||
# Magic bytes validation
|
||||
magic_results = self.validate_magic_bytes()
|
||||
if not magic_results["valid"]:
|
||||
results["valid"] = False
|
||||
results["errors"].extend(magic_results["errors"])
|
||||
results["security_score"] -= 10 # Reduced from 25 - basic format issue, not security threat
|
||||
|
||||
results["file_info"]["detected_type"] = magic_results["detected_type"]
|
||||
|
||||
# MIME type validation
|
||||
mime_results = self.validate_mime_type()
|
||||
if not mime_results["valid"]:
|
||||
results["valid"] = False
|
||||
results["errors"].extend(mime_results["errors"])
|
||||
results["security_score"] -= 10 # Reduced from 20 - basic format issue, not security threat
|
||||
|
||||
results["file_info"]["detected_mime"] = mime_results["detected_mime"]
|
||||
results["warnings"].extend(mime_results.get("warnings", []))
|
||||
|
||||
# PIL image validation
|
||||
pil_results = self.validate_pil_image()
|
||||
if not pil_results["valid"]:
|
||||
results["valid"] = False
|
||||
results["errors"].extend(pil_results["errors"])
|
||||
results["security_score"] -= 10 # Reduced from 15 - basic format issue, not security threat
|
||||
|
||||
results["file_info"]["image_info"] = pil_results["image_info"]
|
||||
results["warnings"].extend(pil_results.get("warnings", []))
|
||||
|
||||
# Security scan
|
||||
security_results = self.scan_for_malicious_content()
|
||||
if security_results["suspicious"]:
|
||||
results["valid"] = False
|
||||
results["errors"].extend(security_results["threats"])
|
||||
results["security_score"] -= 50
|
||||
|
||||
results["warnings"].extend(security_results.get("warnings", []))
|
||||
|
||||
# Log security events
|
||||
if not results["valid"]:
|
||||
logger.warning(f"File upload validation failed: {results['errors']}")
|
||||
elif results["security_score"] < 80:
|
||||
logger.info(
|
||||
f"File upload with low security score: {results['security_score']}"
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def validate_uploaded_file(
|
||||
file_data: bytes, filename: str
|
||||
) -> Tuple[bool, Dict[str, any], bytes]:
|
||||
"""
|
||||
Main function to validate uploaded files
|
||||
|
||||
Returns:
|
||||
(is_valid, validation_results, sanitized_data)
|
||||
"""
|
||||
validator = FileValidator(file_data, filename)
|
||||
|
||||
# Perform comprehensive validation
|
||||
results = validator.comprehensive_validation()
|
||||
|
||||
if not results["valid"]:
|
||||
return False, results, file_data
|
||||
|
||||
# Sanitize EXIF data
|
||||
sanitized_data = validator.sanitize_exif_data()
|
||||
|
||||
return True, results, sanitized_data
|
||||
|
||||
|
||||
def get_file_security_report(file_data: bytes, filename: str) -> Dict[str, any]:
|
||||
"""
|
||||
Generate a security report for a file without modifying it
|
||||
"""
|
||||
validator = FileValidator(file_data, filename)
|
||||
return validator.comprehensive_validation()
|
||||
@@ -6,15 +6,22 @@ from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from django import forms
|
||||
from django.utils.translation import gettext_lazy as _
|
||||
from django.core.exceptions import ValidationError
|
||||
|
||||
from ipware import get_client_ip
|
||||
|
||||
from ivatar import settings
|
||||
from ivatar.settings import MIN_LENGTH_EMAIL, MAX_LENGTH_EMAIL
|
||||
from ivatar.settings import MIN_LENGTH_URL, MAX_LENGTH_URL
|
||||
from ivatar.settings import ENABLE_FILE_SECURITY_VALIDATION
|
||||
from ivatar.file_security import validate_uploaded_file, FileUploadSecurityError
|
||||
from .models import UnconfirmedEmail, ConfirmedEmail, Photo
|
||||
from .models import UnconfirmedOpenId, ConfirmedOpenId
|
||||
from .models import UserPreference
|
||||
import logging
|
||||
|
||||
# Initialize logger
|
||||
logger = logging.getLogger("ivatar.ivataraccount.forms")
|
||||
|
||||
|
||||
MAX_NUM_UNCONFIRMED_EMAILS_DEFAULT = 5
|
||||
@@ -81,7 +88,7 @@ class AddEmailForm(forms.Form):
|
||||
|
||||
class UploadPhotoForm(forms.Form):
|
||||
"""
|
||||
Form handling photo upload
|
||||
Form handling photo upload with enhanced security validation
|
||||
"""
|
||||
|
||||
photo = forms.FileField(
|
||||
@@ -107,16 +114,102 @@ class UploadPhotoForm(forms.Form):
|
||||
},
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def save(request, data):
|
||||
def clean_photo(self):
|
||||
"""
|
||||
Save the model and assign it to the current user
|
||||
Enhanced photo validation with security checks
|
||||
"""
|
||||
photo = self.cleaned_data.get("photo")
|
||||
|
||||
if not photo:
|
||||
raise ValidationError(_("No file provided"))
|
||||
|
||||
# Read file data
|
||||
try:
|
||||
# Handle different file types
|
||||
if hasattr(photo, 'read'):
|
||||
file_data = photo.read()
|
||||
elif hasattr(photo, 'file'):
|
||||
file_data = photo.file.read()
|
||||
else:
|
||||
file_data = bytes(photo)
|
||||
filename = photo.name
|
||||
except Exception as e:
|
||||
logger.error(f"Error reading uploaded file: {e}")
|
||||
raise ValidationError(_("Error reading uploaded file"))
|
||||
|
||||
# Perform comprehensive security validation (if enabled)
|
||||
if ENABLE_FILE_SECURITY_VALIDATION:
|
||||
try:
|
||||
is_valid, validation_results, sanitized_data = validate_uploaded_file(
|
||||
file_data, filename
|
||||
)
|
||||
|
||||
if not is_valid:
|
||||
# Log security violation
|
||||
logger.warning(
|
||||
f"File upload security violation: {validation_results['errors']}"
|
||||
)
|
||||
|
||||
# Only reject truly malicious files at the form level
|
||||
# Allow basic format issues to pass through to Photo.save() for original error handling
|
||||
if validation_results.get("security_score", 100) < 30:
|
||||
raise ValidationError(
|
||||
_("File appears to be malicious and cannot be uploaded")
|
||||
)
|
||||
else:
|
||||
# For format issues, don't raise ValidationError - let Photo.save() handle it
|
||||
# This preserves the original error handling behavior
|
||||
logger.info(f"File format issue detected, allowing Photo.save() to handle: {validation_results['errors']}")
|
||||
# Store the validation results for potential use, but don't reject the form
|
||||
self.validation_results = validation_results
|
||||
self.file_data = file_data
|
||||
else:
|
||||
# Store sanitized data for later use
|
||||
self.sanitized_data = sanitized_data
|
||||
self.validation_results = validation_results
|
||||
# Store original file data for fallback
|
||||
self.file_data = file_data
|
||||
|
||||
# Log successful validation
|
||||
logger.info(
|
||||
f"File upload validated successfully: {filename}, security_score: {validation_results.get('security_score', 100)}"
|
||||
)
|
||||
|
||||
except FileUploadSecurityError as e:
|
||||
logger.error(f"File upload security error: {e}")
|
||||
raise ValidationError(_("File security validation failed"))
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error during file validation: {e}")
|
||||
raise ValidationError(_("File validation failed"))
|
||||
else:
|
||||
# Security validation disabled (e.g., in tests)
|
||||
logger.debug(f"File upload security validation disabled for: {filename}")
|
||||
self.file_data = file_data
|
||||
|
||||
return photo
|
||||
|
||||
def save(self, request, data):
|
||||
"""
|
||||
Save the model and assign it to the current user with enhanced security
|
||||
"""
|
||||
# Link this file to the user's profile
|
||||
photo = Photo()
|
||||
photo.user = request.user
|
||||
photo.ip_address = get_client_ip(request)[0]
|
||||
photo.data = data.read()
|
||||
|
||||
# Use sanitized data if available, otherwise use stored file data
|
||||
if hasattr(self, "sanitized_data"):
|
||||
photo.data = self.sanitized_data
|
||||
elif hasattr(self, "file_data"):
|
||||
photo.data = self.file_data
|
||||
else:
|
||||
# Fallback: try to read from the file object
|
||||
try:
|
||||
photo.data = data.read()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read file data: {e}")
|
||||
photo.data = b""
|
||||
|
||||
photo.save()
|
||||
return photo if photo.pk else None
|
||||
|
||||
|
||||
130
ivatar/ivataraccount/migrations/0021_add_performance_indexes.py
Normal file
130
ivatar/ivataraccount/migrations/0021_add_performance_indexes.py
Normal file
@@ -0,0 +1,130 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Generated manually for performance optimization
|
||||
|
||||
from typing import Any, List, Tuple, Optional
|
||||
from django.db import migrations, connection
|
||||
|
||||
|
||||
def create_indexes(apps: Any, schema_editor: Any) -> None:
|
||||
"""
|
||||
Create performance indexes for both PostgreSQL and MySQL compatibility.
|
||||
Uses CONCURRENTLY for PostgreSQL production, regular CREATE INDEX for tests/transactions.
|
||||
"""
|
||||
db_engine = connection.vendor
|
||||
|
||||
indexes: List[Tuple[str, str, str, Optional[str]]] = [
|
||||
# ConfirmedEmail indexes
|
||||
("idx_cemail_digest", "ivataraccount_confirmedemail", "digest", None),
|
||||
(
|
||||
"idx_cemail_digest_sha256",
|
||||
"ivataraccount_confirmedemail",
|
||||
"digest_sha256",
|
||||
None,
|
||||
),
|
||||
(
|
||||
"idx_cemail_access_count",
|
||||
"ivataraccount_confirmedemail",
|
||||
"access_count",
|
||||
None,
|
||||
),
|
||||
(
|
||||
"idx_cemail_bluesky_handle",
|
||||
"ivataraccount_confirmedemail",
|
||||
"bluesky_handle",
|
||||
"WHERE bluesky_handle IS NOT NULL",
|
||||
),
|
||||
# Photo indexes
|
||||
("idx_photo_format", "ivataraccount_photo", "format", None),
|
||||
("idx_photo_access_count", "ivataraccount_photo", "access_count", None),
|
||||
# Composite indexes
|
||||
(
|
||||
"idx_cemail_user_access",
|
||||
"ivataraccount_confirmedemail",
|
||||
"user_id, access_count",
|
||||
None,
|
||||
),
|
||||
(
|
||||
"idx_cemail_photo_access",
|
||||
"ivataraccount_confirmedemail",
|
||||
"photo_id, access_count",
|
||||
None,
|
||||
),
|
||||
("idx_photo_user_format", "ivataraccount_photo", "user_id, format", None),
|
||||
]
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# Check if we're in a transaction (test environment)
|
||||
try:
|
||||
cursor.execute("SELECT 1")
|
||||
in_transaction = connection.in_atomic_block
|
||||
except Exception:
|
||||
in_transaction = True
|
||||
|
||||
for index_name, table_name, columns, where_clause in indexes:
|
||||
try:
|
||||
if db_engine == "postgresql":
|
||||
# Use CONCURRENTLY only if not in a transaction (production)
|
||||
# Use regular CREATE INDEX if in a transaction (tests)
|
||||
if in_transaction:
|
||||
# In transaction (test environment) - use regular CREATE INDEX
|
||||
if where_clause:
|
||||
sql = f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name}({columns}) {where_clause};"
|
||||
else:
|
||||
sql = f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name}({columns});"
|
||||
else:
|
||||
# Not in transaction (production) - use CONCURRENTLY
|
||||
if where_clause:
|
||||
sql = f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {index_name} ON {table_name}({columns}) {where_clause};"
|
||||
else:
|
||||
sql = f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {index_name} ON {table_name}({columns});"
|
||||
else:
|
||||
# MySQL and other databases - skip partial indexes
|
||||
if where_clause:
|
||||
print(
|
||||
f"Skipping partial index {index_name} for {db_engine} (not supported)"
|
||||
)
|
||||
continue
|
||||
sql = f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name}({columns});"
|
||||
|
||||
cursor.execute(sql)
|
||||
print(f"Created index: {index_name}")
|
||||
|
||||
except Exception as e:
|
||||
# Index might already exist or other error - log and continue
|
||||
print(f"Index {index_name} creation skipped: {e}")
|
||||
|
||||
|
||||
def drop_indexes(apps: Any, schema_editor: Any) -> None:
|
||||
"""
|
||||
Drop the performance indexes.
|
||||
"""
|
||||
indexes: List[str] = [
|
||||
"idx_cemail_digest",
|
||||
"idx_cemail_digest_sha256",
|
||||
"idx_cemail_access_count",
|
||||
"idx_cemail_bluesky_handle",
|
||||
"idx_photo_format",
|
||||
"idx_photo_access_count",
|
||||
"idx_cemail_user_access",
|
||||
"idx_cemail_photo_access",
|
||||
"idx_photo_user_format",
|
||||
]
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
for index_name in indexes:
|
||||
try:
|
||||
cursor.execute(f"DROP INDEX IF EXISTS {index_name};")
|
||||
print(f"Dropped index: {index_name}")
|
||||
except Exception as e:
|
||||
print(f"Index {index_name} drop skipped: {e}")
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("ivataraccount", "0020_confirmedopenid_bluesky_handle"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RunPython(create_indexes, drop_indexes),
|
||||
]
|
||||
@@ -139,6 +139,11 @@ class Photo(BaseAccountModel):
|
||||
|
||||
verbose_name = _("photo")
|
||||
verbose_name_plural = _("photos")
|
||||
indexes = [
|
||||
models.Index(fields=["format"], name="idx_photo_format"),
|
||||
models.Index(fields=["access_count"], name="idx_photo_access_count"),
|
||||
models.Index(fields=["user_id", "format"], name="idx_photo_user_format"),
|
||||
]
|
||||
|
||||
def import_image(self, service_name, email_address):
|
||||
"""
|
||||
@@ -336,6 +341,20 @@ class ConfirmedEmail(BaseAccountModel):
|
||||
|
||||
verbose_name = _("confirmed email")
|
||||
verbose_name_plural = _("confirmed emails")
|
||||
indexes = [
|
||||
models.Index(fields=["digest"], name="idx_cemail_digest"),
|
||||
models.Index(fields=["digest_sha256"], name="idx_cemail_digest_sha256"),
|
||||
models.Index(fields=["access_count"], name="idx_cemail_access_count"),
|
||||
models.Index(fields=["bluesky_handle"], name="idx_cemail_bluesky_handle"),
|
||||
models.Index(
|
||||
fields=["user_id", "access_count"],
|
||||
name="idx_cemail_user_access",
|
||||
),
|
||||
models.Index(
|
||||
fields=["photo_id", "access_count"],
|
||||
name="idx_cemail_photo_access",
|
||||
),
|
||||
]
|
||||
|
||||
def set_photo(self, photo):
|
||||
"""
|
||||
@@ -379,9 +398,28 @@ class ConfirmedEmail(BaseAccountModel):
|
||||
)
|
||||
|
||||
cache_key = f"views.decorators.cache.cache_page.{quote(str(cache_url))}"
|
||||
if cache.has_key(cache_key):
|
||||
cache.delete(cache_key)
|
||||
logger.debug("Successfully cleaned up cached page: %s" % cache_key)
|
||||
try:
|
||||
if cache.has_key(cache_key):
|
||||
cache.delete(cache_key)
|
||||
logger.debug("Successfully cleaned up cached page: %s" % cache_key)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to clean up cached page %s: %s" % (cache_key, exc)
|
||||
)
|
||||
|
||||
# Invalidate Bluesky avatar URL cache if bluesky_handle changed
|
||||
if hasattr(self, "bluesky_handle") and self.bluesky_handle:
|
||||
try:
|
||||
cache.delete(self.bluesky_handle)
|
||||
logger.debug(
|
||||
"Successfully cleaned up Bluesky avatar cache for handle: %s"
|
||||
% self.bluesky_handle
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to clean up Bluesky avatar cache for handle %s: %s"
|
||||
% (self.bluesky_handle, exc)
|
||||
)
|
||||
|
||||
return super().save(force_insert, force_update, using, update_fields)
|
||||
|
||||
@@ -551,6 +589,36 @@ class ConfirmedOpenId(BaseAccountModel):
|
||||
openid_variations(lowercase_url)[3].encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
# Invalidate page caches and Bluesky avatar cache
|
||||
if self.pk:
|
||||
# Invalidate assign_photo_openid page cache
|
||||
cache_url = reverse_lazy(
|
||||
"assign_photo_openid", kwargs={"openid_id": int(self.pk)}
|
||||
)
|
||||
cache_key = f"views.decorators.cache.cache_page.{quote(str(cache_url))}"
|
||||
try:
|
||||
if cache.has_key(cache_key):
|
||||
cache.delete(cache_key)
|
||||
logger.debug("Successfully cleaned up cached page: %s" % cache_key)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to clean up cached page %s: %s" % (cache_key, exc)
|
||||
)
|
||||
|
||||
# Invalidate Bluesky avatar URL cache if bluesky_handle exists
|
||||
if hasattr(self, "bluesky_handle") and self.bluesky_handle:
|
||||
try:
|
||||
cache.delete(self.bluesky_handle)
|
||||
logger.debug(
|
||||
"Successfully cleaned up Bluesky avatar cache for handle: %s"
|
||||
% self.bluesky_handle
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to clean up Bluesky avatar cache for handle %s: %s"
|
||||
% (self.bluesky_handle, exc)
|
||||
)
|
||||
|
||||
return super().save(force_insert, force_update, using, update_fields)
|
||||
|
||||
def __str__(self):
|
||||
|
||||
@@ -573,16 +573,25 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
self.login()
|
||||
url = reverse("upload_photo")
|
||||
# rb => Read binary
|
||||
with open(TEST_IMAGE_FILE, "rb") as photo:
|
||||
response = self.client.post(
|
||||
url,
|
||||
{
|
||||
"photo": photo,
|
||||
"not_porn": True,
|
||||
"can_distribute": True,
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
with open(TEST_IMAGE_FILE, "rb") as photo_file:
|
||||
photo_data = photo_file.read()
|
||||
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
uploaded_file = SimpleUploadedFile(
|
||||
"deadbeef.png",
|
||||
photo_data,
|
||||
content_type="image/png"
|
||||
)
|
||||
|
||||
response = self.client.post(
|
||||
url,
|
||||
{
|
||||
"photo": uploaded_file,
|
||||
"not_porn": True,
|
||||
"can_distribute": True,
|
||||
},
|
||||
follow=True,
|
||||
)
|
||||
if not test_only_one:
|
||||
return response
|
||||
self.assertEqual(
|
||||
|
||||
@@ -8,6 +8,7 @@ import contextlib
|
||||
# pylint: disable=too-many-lines
|
||||
import os
|
||||
import django
|
||||
import pytest
|
||||
from django.test import TestCase
|
||||
from django.test import Client
|
||||
|
||||
@@ -22,7 +23,7 @@ django.setup()
|
||||
# pylint: disable=wrong-import-position
|
||||
from ivatar import settings
|
||||
from ivatar.ivataraccount.models import ConfirmedOpenId, ConfirmedEmail
|
||||
from ivatar.utils import random_string
|
||||
from ivatar.utils import random_string, Bluesky
|
||||
|
||||
from libravatar import libravatar_url
|
||||
|
||||
@@ -62,6 +63,16 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
)
|
||||
settings.EMAIL_BACKEND = "django.core.mail.backends.dummy.EmailBackend"
|
||||
|
||||
# Clear any existing Bluesky session to ensure clean test state
|
||||
Bluesky.clear_shared_session()
|
||||
|
||||
def tearDown(self):
|
||||
"""
|
||||
Clean up after tests
|
||||
"""
|
||||
# Clear Bluesky session to avoid affecting other tests
|
||||
Bluesky.clear_shared_session()
|
||||
|
||||
def create_confirmed_openid(self):
|
||||
"""
|
||||
Create a confirmed openid
|
||||
@@ -83,6 +94,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
|
||||
# The following tests need to be moved over to the model tests
|
||||
# and real web UI tests added
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_handle_for_mail_via_model_handle_does_not_exist(self):
|
||||
"""
|
||||
Add Bluesky handle to a confirmed mail address
|
||||
@@ -99,6 +111,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Setting Bluesky handle that doesn't exist works?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_handle_for_mail_via_model_handle_exists(self):
|
||||
"""
|
||||
Add Bluesky handle to a confirmed mail address
|
||||
@@ -113,6 +126,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Setting Bluesky handle doesn't work?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_handle_for_openid_via_model_handle_does_not_exist(self):
|
||||
"""
|
||||
Add Bluesky handle to a confirmed openid address
|
||||
@@ -129,6 +143,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Setting Bluesky handle that doesn't exist works?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_handle_for_openid_via_model_handle_exists(self):
|
||||
"""
|
||||
Add Bluesky handle to a confirmed openid address
|
||||
@@ -143,6 +158,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Setting Bluesky handle doesn't work?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_fetch_mail(self):
|
||||
"""
|
||||
Check if we can successfully fetch a Bluesky avatar via email
|
||||
@@ -158,6 +174,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
self.assertEqual(response.status_code, 302)
|
||||
self.assertEqual(response["Location"], f"/blueskyproxy/{confirmed.digest}")
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_bluesky_fetch_openid(self):
|
||||
"""
|
||||
Check if we can successfully fetch a Bluesky avatar via OpenID
|
||||
@@ -173,6 +190,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
self.assertEqual(response.status_code, 302)
|
||||
self.assertEqual(response["Location"], f"/blueskyproxy/{confirmed.digest}")
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_assign_bluesky_handle_to_openid(self):
|
||||
"""
|
||||
Assign a Bluesky handle to an OpenID
|
||||
@@ -185,6 +203,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Adding Bluesky handle to OpenID fails?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_assign_bluesky_handle_to_email(self):
|
||||
"""
|
||||
Assign a Bluesky handle to an email
|
||||
@@ -215,6 +234,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
"Setting Bluesky handle doesn't work?",
|
||||
)
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_assign_photo_to_mail_removes_bluesky_handle(self):
|
||||
"""
|
||||
Assign a Photo to a mail, removes Bluesky handle
|
||||
@@ -223,6 +243,7 @@ class Tester(TestCase): # pylint: disable=too-many-public-methods
|
||||
confirmed = self.create_confirmed_email()
|
||||
self._assign_bluesky_handle(confirmed, "assign_photo_email")
|
||||
|
||||
@pytest.mark.bluesky
|
||||
def test_assign_photo_to_openid_removes_bluesky_handle(self):
|
||||
"""
|
||||
Assign a Photo to a OpenID, removes Bluesky handle
|
||||
|
||||
@@ -617,7 +617,7 @@ class DeletePhotoView(SuccessMessageMixin, View):
|
||||
@method_decorator(login_required, name="dispatch")
|
||||
class UploadPhotoView(SuccessMessageMixin, FormView):
|
||||
"""
|
||||
View class responsible for photo upload
|
||||
View class responsible for photo upload with enhanced security
|
||||
"""
|
||||
|
||||
model = Photo
|
||||
@@ -627,26 +627,46 @@ class UploadPhotoView(SuccessMessageMixin, FormView):
|
||||
success_url = reverse_lazy("profile")
|
||||
|
||||
def post(self, request, *args, **kwargs):
|
||||
# Check maximum number of photos
|
||||
num_photos = request.user.photo_set.count()
|
||||
if num_photos >= MAX_NUM_PHOTOS:
|
||||
messages.error(
|
||||
request, _("Maximum number of photos (%i) reached" % MAX_NUM_PHOTOS)
|
||||
)
|
||||
return HttpResponseRedirect(reverse_lazy("profile"))
|
||||
|
||||
return super().post(request, *args, **kwargs)
|
||||
|
||||
def form_valid(self, form):
|
||||
photo_data = self.request.FILES["photo"]
|
||||
|
||||
# Additional size check (redundant but good for security)
|
||||
if photo_data.size > MAX_PHOTO_SIZE:
|
||||
messages.error(self.request, _("Image too big"))
|
||||
return HttpResponseRedirect(reverse_lazy("profile"))
|
||||
|
||||
# Enhanced security logging
|
||||
security_logger.info(
|
||||
f"Photo upload attempt by user {self.request.user.id} "
|
||||
f"from IP {get_client_ip(self.request)[0]}, "
|
||||
f"file size: {photo_data.size} bytes"
|
||||
)
|
||||
|
||||
photo = form.save(self.request, photo_data)
|
||||
|
||||
if not photo:
|
||||
security_logger.warning(
|
||||
f"Photo upload failed for user {self.request.user.id} - invalid format"
|
||||
)
|
||||
messages.error(self.request, _("Invalid Format"))
|
||||
return HttpResponseRedirect(reverse_lazy("profile"))
|
||||
|
||||
# Log successful upload
|
||||
security_logger.info(
|
||||
f"Photo uploaded successfully by user {self.request.user.id}, "
|
||||
f"photo ID: {photo.pk}"
|
||||
)
|
||||
|
||||
# Override success URL -> Redirect to crop page.
|
||||
self.success_url = reverse_lazy("crop_photo", args=[photo.pk])
|
||||
return super().form_valid(form)
|
||||
|
||||
224
ivatar/opentelemetry_config.py
Normal file
224
ivatar/opentelemetry_config.py
Normal file
@@ -0,0 +1,224 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
OpenTelemetry configuration for ivatar project.
|
||||
|
||||
This module provides OpenTelemetry setup and configuration for the ivatar
|
||||
Django application, including tracing, metrics, and logging integration.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
|
||||
from opentelemetry import trace, metrics
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.prometheus import PrometheusMetricReader
|
||||
from opentelemetry.instrumentation.django import DjangoInstrumentor
|
||||
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
|
||||
from opentelemetry.instrumentation.pymysql import PyMySQLInstrumentor
|
||||
from opentelemetry.instrumentation.requests import RequestsInstrumentor
|
||||
from opentelemetry.instrumentation.urllib3 import URLLib3Instrumentor
|
||||
|
||||
# Note: Memcached instrumentation not available in OpenTelemetry Python
|
||||
|
||||
logger = logging.getLogger("ivatar")
|
||||
|
||||
|
||||
class OpenTelemetryConfig:
|
||||
"""
|
||||
OpenTelemetry configuration manager for ivatar.
|
||||
|
||||
Handles setup of tracing, metrics, and instrumentation for the Django application.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.enabled = True # Always enable OpenTelemetry instrumentation
|
||||
self.export_enabled = self._is_export_enabled()
|
||||
self.service_name = self._get_service_name()
|
||||
self.environment = self._get_environment()
|
||||
self.resource = self._create_resource()
|
||||
|
||||
def _is_export_enabled(self) -> bool:
|
||||
"""Check if OpenTelemetry data export is enabled via environment variable."""
|
||||
return os.environ.get("OTEL_EXPORT_ENABLED", "false").lower() in (
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
)
|
||||
|
||||
def _get_service_name(self) -> str:
|
||||
"""Get service name from environment or default."""
|
||||
return os.environ.get("OTEL_SERVICE_NAME", "ivatar")
|
||||
|
||||
def _get_environment(self) -> str:
|
||||
"""Get environment name (production, development, etc.)."""
|
||||
return os.environ.get("OTEL_ENVIRONMENT", "development")
|
||||
|
||||
def _create_resource(self) -> Resource:
|
||||
"""Create OpenTelemetry resource with service information."""
|
||||
return Resource.create(
|
||||
{
|
||||
"service.name": self.service_name,
|
||||
"service.version": os.environ.get("IVATAR_VERSION", "1.8.0"),
|
||||
"service.namespace": "libravatar",
|
||||
"deployment.environment": self.environment,
|
||||
"service.instance.id": os.environ.get("HOSTNAME", "unknown"),
|
||||
}
|
||||
)
|
||||
|
||||
def setup_tracing(self) -> None:
|
||||
"""Set up OpenTelemetry tracing."""
|
||||
try:
|
||||
# Set up tracer provider
|
||||
trace.set_tracer_provider(TracerProvider(resource=self.resource))
|
||||
tracer_provider = trace.get_tracer_provider()
|
||||
|
||||
# Configure OTLP exporter if export is enabled and endpoint is provided
|
||||
if self.export_enabled:
|
||||
otlp_endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
|
||||
if otlp_endpoint:
|
||||
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
|
||||
span_processor = BatchSpanProcessor(otlp_exporter)
|
||||
tracer_provider.add_span_processor(span_processor)
|
||||
logger.info(
|
||||
f"OpenTelemetry tracing configured with OTLP endpoint: {otlp_endpoint}"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"OpenTelemetry tracing configured without OTLP endpoint"
|
||||
)
|
||||
else:
|
||||
logger.info("OpenTelemetry tracing configured (export disabled)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup OpenTelemetry tracing: {e}")
|
||||
self.enabled = False
|
||||
|
||||
def setup_metrics(self) -> None:
|
||||
"""Set up OpenTelemetry metrics."""
|
||||
try:
|
||||
# Configure metric readers
|
||||
metric_readers = []
|
||||
|
||||
# Always configure Prometheus exporter for metrics (for local development)
|
||||
prometheus_endpoint = os.environ.get(
|
||||
"OTEL_PROMETHEUS_ENDPOINT", "0.0.0.0:9464"
|
||||
)
|
||||
prometheus_reader = PrometheusMetricReader()
|
||||
metric_readers.append(prometheus_reader)
|
||||
|
||||
# Configure OTLP exporter if export is enabled and endpoint is provided
|
||||
if self.export_enabled:
|
||||
otlp_endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
|
||||
if otlp_endpoint:
|
||||
otlp_exporter = OTLPMetricExporter(endpoint=otlp_endpoint)
|
||||
metric_reader = PeriodicExportingMetricReader(otlp_exporter)
|
||||
metric_readers.append(metric_reader)
|
||||
logger.info(
|
||||
f"OpenTelemetry metrics configured with OTLP endpoint: {otlp_endpoint}"
|
||||
)
|
||||
|
||||
# Set up meter provider with readers
|
||||
meter_provider = MeterProvider(
|
||||
resource=self.resource, metric_readers=metric_readers
|
||||
)
|
||||
metrics.set_meter_provider(meter_provider)
|
||||
|
||||
logger.info(
|
||||
f"OpenTelemetry metrics configured with Prometheus endpoint: {prometheus_endpoint}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup OpenTelemetry metrics: {e}")
|
||||
self.enabled = False
|
||||
|
||||
def setup_instrumentation(self) -> None:
|
||||
"""Set up OpenTelemetry instrumentation for various libraries."""
|
||||
try:
|
||||
# Django instrumentation
|
||||
DjangoInstrumentor().instrument()
|
||||
logger.info("Django instrumentation enabled")
|
||||
|
||||
# Database instrumentation
|
||||
Psycopg2Instrumentor().instrument()
|
||||
PyMySQLInstrumentor().instrument()
|
||||
logger.info("Database instrumentation enabled")
|
||||
|
||||
# HTTP client instrumentation
|
||||
RequestsInstrumentor().instrument()
|
||||
URLLib3Instrumentor().instrument()
|
||||
logger.info("HTTP client instrumentation enabled")
|
||||
|
||||
# Note: Memcached instrumentation not available in OpenTelemetry Python
|
||||
# Cache operations will be traced through Django instrumentation
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup OpenTelemetry instrumentation: {e}")
|
||||
self.enabled = False
|
||||
|
||||
def get_tracer(self, name: str) -> trace.Tracer:
|
||||
"""Get a tracer instance."""
|
||||
return trace.get_tracer(name)
|
||||
|
||||
def get_meter(self, name: str) -> metrics.Meter:
|
||||
"""Get a meter instance."""
|
||||
return metrics.get_meter(name)
|
||||
|
||||
|
||||
# Global OpenTelemetry configuration instance (lazy-loaded)
|
||||
_ot_config = None
|
||||
|
||||
|
||||
def get_ot_config():
|
||||
"""Get the global OpenTelemetry configuration instance."""
|
||||
global _ot_config
|
||||
if _ot_config is None:
|
||||
_ot_config = OpenTelemetryConfig()
|
||||
return _ot_config
|
||||
|
||||
|
||||
def setup_opentelemetry() -> None:
|
||||
"""
|
||||
Set up OpenTelemetry for the ivatar application.
|
||||
|
||||
This function should be called during Django application startup.
|
||||
"""
|
||||
logger.info("Setting up OpenTelemetry...")
|
||||
|
||||
ot_config = get_ot_config()
|
||||
ot_config.setup_tracing()
|
||||
ot_config.setup_metrics()
|
||||
ot_config.setup_instrumentation()
|
||||
|
||||
if ot_config.enabled:
|
||||
if ot_config.export_enabled:
|
||||
logger.info("OpenTelemetry setup completed successfully (export enabled)")
|
||||
else:
|
||||
logger.info("OpenTelemetry setup completed successfully (export disabled)")
|
||||
else:
|
||||
logger.info("OpenTelemetry setup failed")
|
||||
|
||||
|
||||
def get_tracer(name: str) -> trace.Tracer:
|
||||
"""Get a tracer instance for the given name."""
|
||||
return get_ot_config().get_tracer(name)
|
||||
|
||||
|
||||
def get_meter(name: str) -> metrics.Meter:
|
||||
"""Get a meter instance for the given name."""
|
||||
return get_ot_config().get_meter(name)
|
||||
|
||||
|
||||
def is_enabled() -> bool:
|
||||
"""Check if OpenTelemetry is enabled (always True now)."""
|
||||
return True
|
||||
|
||||
|
||||
def is_export_enabled() -> bool:
|
||||
"""Check if OpenTelemetry data export is enabled."""
|
||||
return get_ot_config().export_enabled
|
||||
419
ivatar/opentelemetry_middleware.py
Normal file
419
ivatar/opentelemetry_middleware.py
Normal file
@@ -0,0 +1,419 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
OpenTelemetry middleware and custom instrumentation for ivatar.
|
||||
|
||||
This module provides custom OpenTelemetry instrumentation for avatar-specific
|
||||
operations, including metrics and tracing for avatar generation, file uploads,
|
||||
and authentication flows.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from functools import wraps
|
||||
|
||||
from django.http import HttpRequest, HttpResponse
|
||||
from django.utils.deprecation import MiddlewareMixin
|
||||
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.trace import Status, StatusCode
|
||||
|
||||
from ivatar.opentelemetry_config import get_tracer, get_meter, is_enabled
|
||||
|
||||
logger = logging.getLogger("ivatar")
|
||||
|
||||
|
||||
class OpenTelemetryMiddleware(MiddlewareMixin):
|
||||
"""
|
||||
Custom OpenTelemetry middleware for ivatar-specific metrics and tracing.
|
||||
|
||||
This middleware adds custom attributes and metrics to OpenTelemetry spans
|
||||
for avatar-related operations.
|
||||
"""
|
||||
|
||||
def __init__(self, get_response):
|
||||
self.get_response = get_response
|
||||
# Don't get metrics instance here - get it lazily in __call__
|
||||
|
||||
def __call__(self, request):
|
||||
# Get metrics instance lazily
|
||||
if not hasattr(self, "metrics"):
|
||||
self.metrics = get_avatar_metrics()
|
||||
|
||||
# Process request to start tracing
|
||||
self.process_request(request)
|
||||
|
||||
response = self.get_response(request)
|
||||
|
||||
# Process response to complete tracing
|
||||
self.process_response(request, response)
|
||||
|
||||
return response
|
||||
|
||||
def process_request(self, request: HttpRequest) -> None:
|
||||
"""Process incoming request and start tracing."""
|
||||
# Start span for the request
|
||||
span_name = f"{request.method} {request.path}"
|
||||
span = get_tracer("ivatar.middleware").start_span(span_name)
|
||||
|
||||
# Add request attributes
|
||||
span.set_attributes(
|
||||
{
|
||||
"http.method": request.method,
|
||||
"http.url": request.build_absolute_uri(),
|
||||
"http.user_agent": request.META.get("HTTP_USER_AGENT", ""),
|
||||
"http.remote_addr": self._get_client_ip(request),
|
||||
"ivatar.path": request.path,
|
||||
}
|
||||
)
|
||||
|
||||
# Check if this is an avatar request
|
||||
if self._is_avatar_request(request):
|
||||
span.set_attribute("ivatar.request_type", "avatar")
|
||||
self._add_avatar_attributes(span, request)
|
||||
|
||||
# Store span in request for later use
|
||||
request._ot_span = span
|
||||
|
||||
# Record request start time
|
||||
request._ot_start_time = time.time()
|
||||
|
||||
def process_response(
|
||||
self, request: HttpRequest, response: HttpResponse
|
||||
) -> HttpResponse:
|
||||
"""Process response and complete tracing."""
|
||||
span = getattr(request, "_ot_span", None)
|
||||
if not span:
|
||||
return response
|
||||
|
||||
try:
|
||||
# Calculate request duration
|
||||
start_time = getattr(request, "_ot_start_time", time.time())
|
||||
duration = time.time() - start_time
|
||||
|
||||
# Add response attributes
|
||||
span.set_attributes(
|
||||
{
|
||||
"http.status_code": response.status_code,
|
||||
"http.response_size": len(response.content)
|
||||
if hasattr(response, "content")
|
||||
else 0,
|
||||
"http.request.duration": duration,
|
||||
}
|
||||
)
|
||||
|
||||
# Set span status based on response
|
||||
if response.status_code >= 400:
|
||||
span.set_status(
|
||||
Status(StatusCode.ERROR, f"HTTP {response.status_code}")
|
||||
)
|
||||
else:
|
||||
span.set_status(Status(StatusCode.OK))
|
||||
|
||||
# Record metrics
|
||||
# Note: HTTP request metrics are handled by Django instrumentation
|
||||
# We only record avatar-specific metrics here
|
||||
|
||||
# Record avatar-specific metrics
|
||||
if self._is_avatar_request(request):
|
||||
# Record avatar request metric using the new metrics system
|
||||
self.metrics.record_avatar_request(
|
||||
size=self._get_avatar_size(request),
|
||||
format_type=self._get_avatar_format(request),
|
||||
)
|
||||
|
||||
finally:
|
||||
span.end()
|
||||
|
||||
return response
|
||||
|
||||
def _is_avatar_request(self, request: HttpRequest) -> bool:
|
||||
"""Check if this is an avatar request."""
|
||||
return request.path.startswith("/avatar/") or request.path.startswith("/avatar")
|
||||
|
||||
def _add_avatar_attributes(self, span: trace.Span, request: HttpRequest) -> None:
|
||||
"""Add avatar-specific attributes to span."""
|
||||
try:
|
||||
# Extract avatar parameters
|
||||
size = self._get_avatar_size(request)
|
||||
format_type = self._get_avatar_format(request)
|
||||
email = self._get_avatar_email(request)
|
||||
|
||||
span.set_attributes(
|
||||
{
|
||||
"ivatar.avatar_size": size,
|
||||
"ivatar.avatar_format": format_type,
|
||||
"ivatar.avatar_email": email,
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to add avatar attributes: {e}")
|
||||
|
||||
def _get_avatar_size(self, request: HttpRequest) -> str:
|
||||
"""Extract avatar size from request."""
|
||||
size = request.GET.get("s", "80")
|
||||
return str(size)
|
||||
|
||||
def _get_avatar_format(self, request: HttpRequest) -> str:
|
||||
"""Extract avatar format from request."""
|
||||
format_type = request.GET.get("d", "png")
|
||||
return str(format_type)
|
||||
|
||||
def _get_avatar_email(self, request: HttpRequest) -> str:
|
||||
"""Extract email from avatar request path."""
|
||||
try:
|
||||
# Extract email from path like /avatar/user@example.com
|
||||
path_parts = request.path.strip("/").split("/")
|
||||
if len(path_parts) >= 2 and path_parts[0] == "avatar":
|
||||
return path_parts[1]
|
||||
except Exception:
|
||||
pass
|
||||
return "unknown"
|
||||
|
||||
def _get_client_ip(self, request: HttpRequest) -> str:
|
||||
"""Get client IP address from request."""
|
||||
x_forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR")
|
||||
if x_forwarded_for:
|
||||
return x_forwarded_for.split(",")[0].strip()
|
||||
return request.META.get("REMOTE_ADDR", "unknown")
|
||||
|
||||
|
||||
def trace_avatar_operation(operation_name: str):
|
||||
"""
|
||||
Decorator to trace avatar operations.
|
||||
|
||||
Args:
|
||||
operation_name: Name of the operation being traced
|
||||
"""
|
||||
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
if not is_enabled():
|
||||
return func(*args, **kwargs)
|
||||
|
||||
tracer = get_tracer("ivatar.avatar")
|
||||
with tracer.start_as_current_span(f"avatar.{operation_name}") as span:
|
||||
try:
|
||||
result = func(*args, **kwargs)
|
||||
span.set_status(Status(StatusCode.OK))
|
||||
return result
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.set_attribute("error.message", str(e))
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def trace_file_upload(operation_name: str):
|
||||
"""
|
||||
Decorator to trace file upload operations.
|
||||
|
||||
Args:
|
||||
operation_name: Name of the file upload operation being traced
|
||||
"""
|
||||
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
tracer = get_tracer("ivatar.file_upload")
|
||||
with tracer.start_as_current_span(f"file_upload.{operation_name}") as span:
|
||||
try:
|
||||
# Add file information if available
|
||||
if args and hasattr(args[0], "FILES"):
|
||||
files = args[0].FILES
|
||||
if files:
|
||||
file_info = list(files.values())[0]
|
||||
span.set_attributes(
|
||||
{
|
||||
"file.name": file_info.name,
|
||||
"file.size": file_info.size,
|
||||
"file.content_type": file_info.content_type,
|
||||
}
|
||||
)
|
||||
|
||||
result = func(*args, **kwargs)
|
||||
span.set_status(Status(StatusCode.OK))
|
||||
return result
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.set_attribute("error.message", str(e))
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def trace_authentication(operation_name: str):
|
||||
"""
|
||||
Decorator to trace authentication operations.
|
||||
|
||||
Args:
|
||||
operation_name: Name of the authentication operation being traced
|
||||
"""
|
||||
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
tracer = get_tracer("ivatar.auth")
|
||||
with tracer.start_as_current_span(f"auth.{operation_name}") as span:
|
||||
try:
|
||||
result = func(*args, **kwargs)
|
||||
span.set_status(Status(StatusCode.OK))
|
||||
return result
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.set_attribute("error.message", str(e))
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
class AvatarMetrics:
|
||||
"""
|
||||
Custom metrics for avatar operations.
|
||||
|
||||
This class provides methods to record custom metrics for avatar-specific
|
||||
operations like generation, caching, and external service calls.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.meter = get_meter("ivatar.avatar")
|
||||
|
||||
# Create custom metrics
|
||||
self.avatar_generated = self.meter.create_counter(
|
||||
name="ivatar_avatars_generated_total",
|
||||
description="Total number of avatars generated",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.avatar_requests = self.meter.create_counter(
|
||||
name="ivatar_avatar_requests_total",
|
||||
description="Total number of avatar image requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.avatar_cache_hits = self.meter.create_counter(
|
||||
name="ivatar_avatar_cache_hits_total",
|
||||
description="Total number of avatar cache hits",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.avatar_cache_misses = self.meter.create_counter(
|
||||
name="ivatar_avatar_cache_misses_total",
|
||||
description="Total number of avatar cache misses",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.external_avatar_requests = self.meter.create_counter(
|
||||
name="ivatar_external_avatar_requests_total",
|
||||
description="Total number of external avatar requests",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.file_uploads = self.meter.create_counter(
|
||||
name="ivatar_file_uploads_total",
|
||||
description="Total number of file uploads",
|
||||
unit="1",
|
||||
)
|
||||
|
||||
self.file_upload_size = self.meter.create_histogram(
|
||||
name="ivatar_file_upload_size_bytes",
|
||||
description="File upload size in bytes",
|
||||
unit="bytes",
|
||||
)
|
||||
|
||||
def record_avatar_request(self, size: str, format_type: str):
|
||||
"""Record avatar request."""
|
||||
self.avatar_requests.add(
|
||||
1,
|
||||
{
|
||||
"size": size,
|
||||
"format": format_type,
|
||||
},
|
||||
)
|
||||
|
||||
def record_avatar_generated(
|
||||
self, size: str, format_type: str, source: str = "generated"
|
||||
):
|
||||
"""Record avatar generation."""
|
||||
self.avatar_generated.add(
|
||||
1,
|
||||
{
|
||||
"size": size,
|
||||
"format": format_type,
|
||||
"source": source,
|
||||
},
|
||||
)
|
||||
|
||||
def record_cache_hit(self, size: str, format_type: str):
|
||||
"""Record cache hit."""
|
||||
self.avatar_cache_hits.add(
|
||||
1,
|
||||
{
|
||||
"size": size,
|
||||
"format": format_type,
|
||||
},
|
||||
)
|
||||
|
||||
def record_cache_miss(self, size: str, format_type: str):
|
||||
"""Record cache miss."""
|
||||
self.avatar_cache_misses.add(
|
||||
1,
|
||||
{
|
||||
"size": size,
|
||||
"format": format_type,
|
||||
},
|
||||
)
|
||||
|
||||
def record_external_request(self, service: str, status_code: int):
|
||||
"""Record external avatar service request."""
|
||||
self.external_avatar_requests.add(
|
||||
1,
|
||||
{
|
||||
"service": service,
|
||||
"status_code": str(status_code),
|
||||
},
|
||||
)
|
||||
|
||||
def record_file_upload(self, file_size: int, content_type: str, success: bool):
|
||||
"""Record file upload."""
|
||||
self.file_uploads.add(
|
||||
1,
|
||||
{
|
||||
"content_type": content_type,
|
||||
"success": str(success),
|
||||
},
|
||||
)
|
||||
|
||||
self.file_upload_size.record(
|
||||
file_size,
|
||||
{
|
||||
"content_type": content_type,
|
||||
"success": str(success),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# Global metrics instance (lazy-loaded)
|
||||
_avatar_metrics = None
|
||||
|
||||
|
||||
def get_avatar_metrics():
|
||||
"""Get the global avatar metrics instance."""
|
||||
global _avatar_metrics
|
||||
if _avatar_metrics is None:
|
||||
_avatar_metrics = AvatarMetrics()
|
||||
return _avatar_metrics
|
||||
|
||||
|
||||
def reset_avatar_metrics():
|
||||
"""Reset the global avatar metrics instance (for testing)."""
|
||||
global _avatar_metrics
|
||||
_avatar_metrics = None
|
||||
@@ -16,8 +16,38 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
# Logging directory - can be overridden in local config
|
||||
LOGS_DIR = os.path.join(BASE_DIR, "logs")
|
||||
|
||||
# Ensure logs directory exists
|
||||
os.makedirs(LOGS_DIR, exist_ok=True)
|
||||
|
||||
def _test_logs_directory_writeability(logs_dir):
|
||||
"""
|
||||
Test if a logs directory is actually writable by attempting to create and write a test file
|
||||
"""
|
||||
try:
|
||||
# Ensure directory exists
|
||||
os.makedirs(logs_dir, exist_ok=True)
|
||||
|
||||
# Test if we can actually write to the directory
|
||||
test_file = os.path.join(logs_dir, ".write_test")
|
||||
with open(test_file, "w") as f:
|
||||
f.write("test")
|
||||
|
||||
# Clean up test file
|
||||
os.remove(test_file)
|
||||
return True
|
||||
except (OSError, PermissionError):
|
||||
return False
|
||||
|
||||
|
||||
# Ensure logs directory exists and is writable - worst case, fall back to /tmp
|
||||
if not _test_logs_directory_writeability(LOGS_DIR):
|
||||
LOGS_DIR = "/tmp/libravatar-logs"
|
||||
if not _test_logs_directory_writeability(LOGS_DIR):
|
||||
# If even /tmp fails, use a user-specific temp directory
|
||||
import tempfile
|
||||
|
||||
LOGS_DIR = os.path.join(tempfile.gettempdir(), f"libravatar-logs-{os.getuid()}")
|
||||
_test_logs_directory_writeability(LOGS_DIR) # This should always succeed
|
||||
|
||||
logger.warning(f"Failed to write to logs directory, falling back to {LOGS_DIR}")
|
||||
|
||||
# SECURITY WARNING: keep the secret key used in production secret!
|
||||
SECRET_KEY = "=v(+-^t#ahv^a&&e)uf36g8algj$d1@6ou^w(r0@%)#8mlc*zk"
|
||||
@@ -73,7 +103,7 @@ LOGGING = {
|
||||
"loggers": {
|
||||
"ivatar": {
|
||||
"handlers": ["file", "console"],
|
||||
"level": "INFO",
|
||||
"level": "INFO", # Restore normal logging level
|
||||
"propagate": True,
|
||||
},
|
||||
"ivatar.security": {
|
||||
@@ -279,3 +309,18 @@ STATIC_ROOT = os.path.join(BASE_DIR, "static")
|
||||
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
|
||||
|
||||
from config import * # pylint: disable=wildcard-import,wrong-import-position,unused-wildcard-import # noqa
|
||||
|
||||
# OpenTelemetry setup - must be after config import
|
||||
# Only setup if feature flag is enabled
|
||||
try:
|
||||
if getattr(globals(), "ENABLE_OPENTELEMETRY", False):
|
||||
from ivatar.opentelemetry_config import setup_opentelemetry
|
||||
|
||||
setup_opentelemetry()
|
||||
|
||||
# Add OpenTelemetry middleware if enabled
|
||||
MIDDLEWARE.append("ivatar.opentelemetry_middleware.OpenTelemetryMiddleware")
|
||||
except (ImportError, NameError):
|
||||
# OpenTelemetry packages not installed or configuration failed
|
||||
# ENABLE_OPENTELEMETRY not defined (shouldn't happen but be safe)
|
||||
pass
|
||||
|
||||
275
ivatar/test_file_security.py
Normal file
275
ivatar/test_file_security.py
Normal file
@@ -0,0 +1,275 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Tests for file upload security enhancements
|
||||
"""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from django.test import TestCase, override_settings
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
from django.contrib.auth.models import User
|
||||
|
||||
from ivatar.file_security import (
|
||||
FileValidator,
|
||||
validate_uploaded_file,
|
||||
get_file_security_report,
|
||||
)
|
||||
from ivatar.ivataraccount.forms import UploadPhotoForm
|
||||
|
||||
|
||||
class FileSecurityTestCase(TestCase):
|
||||
"""Test cases for file upload security"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test data"""
|
||||
self.user = User.objects.create_user(
|
||||
username="testuser", email="test@example.com", password="testpass123"
|
||||
)
|
||||
|
||||
# Create test image data
|
||||
self.valid_jpeg_data = b"\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x01\x00H\x00H\x00\x00\xff\xdb\x00C\x00\x08\x06\x06\x07\x06\x05\x08\x07\x07\x07\t\t\x08\n\x0c\x14\r\x0c\x0b\x0b\x0c\x19\x12\x13\x0f\x14\x1d\x1a\x1f\x1e\x1d\x1a\x1c\x1c $.' \",#\x1c\x1c(7),01444\x1f'9=82<.342\xff\xc0\x00\x11\x08\x00\x01\x00\x01\x01\x01\x11\x00\x02\x11\x01\x03\x11\x01\xff\xc4\x00\x1f\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\xff\xc4\x00\xb5\x10\x00\x02\x01\x03\x03\x02\x04\x03\x05\x05\x04\x04\x00\x00\x01}\x01\x02\x03\x00\x04\x11\x05\x12!1A\x06\x13Qa\x07\"q\x142\x81\x91\xa1\x08#B\xb1\xc1\x15R\xd1\xf0$3br\x82\t\n\x16\x17\x18\x19\x1a%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\x83\x84\x85\x86\x87\x88\x89\x8a\x92\x93\x94\x95\x96\x97\x98\x99\x9a\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xff\xda\x00\x0c\x03\x01\x00\x02\x11\x03\x11\x00\x3f\x00\xf9\xff\xd9"
|
||||
|
||||
self.malicious_data = b'GIF89a<script>alert("xss")</script>'
|
||||
self.large_data = b"x" * (10 * 1024 * 1024) # 10MB
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up after tests"""
|
||||
pass
|
||||
|
||||
def test_valid_jpeg_validation(self):
|
||||
"""Test validation of valid JPEG file"""
|
||||
validator = FileValidator(self.valid_jpeg_data, "test.jpg")
|
||||
|
||||
# Mock PIL validation to avoid issues with test data
|
||||
with patch.object(validator, "validate_pil_image") as mock_pil:
|
||||
mock_pil.return_value = {
|
||||
"valid": True,
|
||||
"image_info": {
|
||||
"format": "JPEG",
|
||||
"mode": "RGB",
|
||||
"size": (100, 100),
|
||||
"width": 100,
|
||||
"height": 100,
|
||||
"has_transparency": False,
|
||||
},
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
}
|
||||
|
||||
results = validator.comprehensive_validation()
|
||||
|
||||
self.assertTrue(results["valid"])
|
||||
self.assertEqual(results["file_info"]["detected_type"], "image/jpeg")
|
||||
self.assertGreaterEqual(results["security_score"], 80)
|
||||
|
||||
def test_magic_bytes_validation(self):
|
||||
"""Test magic bytes validation"""
|
||||
validator = FileValidator(self.valid_jpeg_data, "test.jpg")
|
||||
results = validator.validate_magic_bytes()
|
||||
|
||||
self.assertTrue(results["valid"])
|
||||
self.assertEqual(results["detected_type"], "image/jpeg")
|
||||
|
||||
def test_malicious_content_detection(self):
|
||||
"""Test detection of malicious content"""
|
||||
validator = FileValidator(self.malicious_data, "malicious.gif")
|
||||
results = validator.scan_for_malicious_content()
|
||||
|
||||
self.assertTrue(results["suspicious"])
|
||||
self.assertGreater(len(results["threats"]), 0)
|
||||
|
||||
def test_file_size_validation(self):
|
||||
"""Test file size validation"""
|
||||
validator = FileValidator(self.large_data, "large.jpg")
|
||||
results = validator.validate_basic()
|
||||
|
||||
self.assertFalse(results["valid"])
|
||||
self.assertIn("File too large", results["errors"][0])
|
||||
|
||||
def test_invalid_extension_validation(self):
|
||||
"""Test invalid file extension validation"""
|
||||
validator = FileValidator(self.valid_jpeg_data, "test.exe")
|
||||
results = validator.validate_basic()
|
||||
|
||||
self.assertFalse(results["valid"])
|
||||
self.assertIn("File extension not allowed", results["errors"][0])
|
||||
|
||||
def test_exif_sanitization(self):
|
||||
"""Test EXIF data sanitization"""
|
||||
validator = FileValidator(self.valid_jpeg_data, "test.jpg")
|
||||
sanitized_data = validator.sanitize_exif_data()
|
||||
|
||||
# Should return data (may be same or sanitized)
|
||||
self.assertIsInstance(sanitized_data, bytes)
|
||||
self.assertGreater(len(sanitized_data), 0)
|
||||
|
||||
def test_comprehensive_validation_function(self):
|
||||
"""Test the main validation function"""
|
||||
# Mock PIL validation to avoid issues with test data
|
||||
with patch("ivatar.file_security.FileValidator.validate_pil_image") as mock_pil:
|
||||
mock_pil.return_value = {
|
||||
"valid": True,
|
||||
"image_info": {"format": "JPEG", "size": (100, 100)},
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
}
|
||||
|
||||
is_valid, results, sanitized_data = validate_uploaded_file(
|
||||
self.valid_jpeg_data, "test.jpg"
|
||||
)
|
||||
|
||||
self.assertTrue(is_valid)
|
||||
self.assertIsInstance(results, dict)
|
||||
self.assertIsInstance(sanitized_data, bytes)
|
||||
|
||||
def test_security_report_generation(self):
|
||||
"""Test security report generation"""
|
||||
# Mock PIL validation to avoid issues with test data
|
||||
with patch("ivatar.file_security.FileValidator.validate_pil_image") as mock_pil:
|
||||
mock_pil.return_value = {
|
||||
"valid": True,
|
||||
"image_info": {"format": "JPEG", "size": (100, 100)},
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
}
|
||||
|
||||
report = get_file_security_report(self.valid_jpeg_data, "test.jpg")
|
||||
|
||||
self.assertIn("valid", report)
|
||||
self.assertIn("security_score", report)
|
||||
self.assertIn("file_info", report)
|
||||
|
||||
@patch("ivatar.file_security.magic.from_buffer")
|
||||
def test_mime_type_validation(self, mock_magic):
|
||||
"""Test MIME type validation with mocked magic"""
|
||||
mock_magic.return_value = "image/jpeg"
|
||||
|
||||
validator = FileValidator(self.valid_jpeg_data, "test.jpg")
|
||||
results = validator.validate_mime_type()
|
||||
|
||||
self.assertTrue(results["valid"])
|
||||
self.assertEqual(results["detected_mime"], "image/jpeg")
|
||||
|
||||
def test_polyglot_attack_detection(self):
|
||||
"""Test detection of polyglot attacks"""
|
||||
polyglot_data = b'GIF89a<script>alert("xss")</script>'
|
||||
validator = FileValidator(polyglot_data, "polyglot.gif")
|
||||
results = validator.scan_for_malicious_content()
|
||||
|
||||
self.assertTrue(results["suspicious"])
|
||||
# Check for either polyglot attack or suspicious script pattern
|
||||
threats_text = " ".join(results["threats"]).lower()
|
||||
self.assertTrue(
|
||||
"polyglot attack" in threats_text or "suspicious pattern" in threats_text,
|
||||
f"Expected polyglot attack or suspicious pattern, got: {results['threats']}",
|
||||
)
|
||||
|
||||
|
||||
class UploadPhotoFormSecurityTestCase(TestCase):
|
||||
"""Test cases for UploadPhotoForm security enhancements"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test data"""
|
||||
self.user = User.objects.create_user(
|
||||
username="testuser", email="test@example.com", password="testpass123"
|
||||
)
|
||||
|
||||
def test_form_validation_with_valid_file(self):
|
||||
"""Test form validation with valid file"""
|
||||
valid_jpeg_data = b"\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x01\x00H\x00H\x00\x00\xff\xdb\x00C\x00\x08\x06\x06\x07\x06\x05\x08\x07\x07\x07\t\t\x08\n\x0c\x14\r\x0c\x0b\x0b\x0c\x19\x12\x13\x0f\x14\x1d\x1a\x1f\x1e\x1d\x1a\x1c\x1c $.' \",#\x1c\x1c(7),01444\x1f'9=82<.342\xff\xc0\x00\x11\x08\x00\x01\x00\x01\x01\x01\x11\x00\x02\x11\x01\x03\x11\x01\xff\xc4\x00\x1f\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\xff\xc4\x00\xb5\x10\x00\x02\x01\x03\x03\x02\x04\x03\x05\x05\x04\x04\x00\x00\x01}\x01\x02\x03\x00\x04\x11\x05\x12!1A\x06\x13Qa\x07\"q\x142\x81\x91\xa1\x08#B\xb1\xc1\x15R\xd1\xf0$3br\x82\t\n\x16\x17\x18\x19\x1a%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\x83\x84\x85\x86\x87\x88\x89\x8a\x92\x93\x94\x95\x96\x97\x98\x99\x9a\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xff\xda\x00\x0c\x03\x01\x00\x02\x11\x03\x11\x00\x3f\x00\xf9\xff\xd9"
|
||||
|
||||
uploaded_file = SimpleUploadedFile(
|
||||
"test.jpg", valid_jpeg_data, content_type="image/jpeg"
|
||||
)
|
||||
|
||||
form_data = {"photo": uploaded_file, "not_porn": True, "can_distribute": True}
|
||||
|
||||
form = UploadPhotoForm(data=form_data, files={"photo": uploaded_file})
|
||||
|
||||
# Mock the validation to avoid PIL issues in tests
|
||||
with patch(
|
||||
"ivatar.ivataraccount.forms.validate_uploaded_file"
|
||||
) as mock_validate:
|
||||
mock_validate.return_value = (
|
||||
True,
|
||||
{"security_score": 95, "errors": [], "warnings": []},
|
||||
valid_jpeg_data,
|
||||
)
|
||||
|
||||
self.assertTrue(form.is_valid())
|
||||
|
||||
def test_form_validation_with_malicious_file(self):
|
||||
"""Test form validation with malicious file"""
|
||||
malicious_data = b'GIF89a<script>alert("xss")</script>'
|
||||
|
||||
uploaded_file = SimpleUploadedFile(
|
||||
"malicious.gif", malicious_data, content_type="image/gif"
|
||||
)
|
||||
|
||||
form_data = {"photo": uploaded_file, "not_porn": True, "can_distribute": True}
|
||||
|
||||
form = UploadPhotoForm(data=form_data, files={"photo": uploaded_file})
|
||||
|
||||
# Mock the validation to return malicious file detection
|
||||
with patch(
|
||||
"ivatar.ivataraccount.forms.validate_uploaded_file"
|
||||
) as mock_validate:
|
||||
mock_validate.return_value = (
|
||||
False,
|
||||
{
|
||||
"security_score": 20,
|
||||
"errors": ["Malicious content detected"],
|
||||
"warnings": [],
|
||||
},
|
||||
malicious_data,
|
||||
)
|
||||
|
||||
self.assertFalse(form.is_valid())
|
||||
# Check for any error message indicating validation failure
|
||||
error_text = str(form.errors["photo"]).lower()
|
||||
self.assertTrue(
|
||||
"malicious" in error_text or "validation failed" in error_text,
|
||||
f"Expected malicious or validation failed message, got: {form.errors['photo']}",
|
||||
)
|
||||
|
||||
|
||||
class UploadPhotoViewSecurityTestCase(TestCase):
|
||||
"""Test cases for UploadPhotoView security enhancements"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test data"""
|
||||
self.user = User.objects.create_user(
|
||||
username="testuser", email="test@example.com", password="testpass123"
|
||||
)
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up after tests"""
|
||||
pass
|
||||
|
||||
|
||||
@override_settings(
|
||||
ENABLE_FILE_SECURITY_VALIDATION=True,
|
||||
ENABLE_EXIF_SANITIZATION=True,
|
||||
ENABLE_MALICIOUS_CONTENT_SCAN=True,
|
||||
ENABLE_RATE_LIMITING=True,
|
||||
)
|
||||
class FileSecurityIntegrationTestCase(TestCase):
|
||||
"""Integration tests for file upload security"""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test data"""
|
||||
self.user = User.objects.create_user(
|
||||
username="testuser", email="test@example.com", password="testpass123"
|
||||
)
|
||||
|
||||
def test_end_to_end_security_validation(self):
|
||||
"""Test end-to-end security validation"""
|
||||
# This would test the complete flow from upload to storage
|
||||
# with all security checks enabled
|
||||
pass
|
||||
|
||||
def test_security_logging(self):
|
||||
"""Test that security events are properly logged"""
|
||||
# This would test that security events are logged
|
||||
# when malicious files are uploaded
|
||||
pass
|
||||
439
ivatar/test_opentelemetry.py
Normal file
439
ivatar/test_opentelemetry.py
Normal file
@@ -0,0 +1,439 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Tests for OpenTelemetry integration in ivatar.
|
||||
|
||||
This module contains comprehensive tests for OpenTelemetry functionality,
|
||||
including configuration, middleware, metrics, and tracing.
|
||||
"""
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import patch, MagicMock
|
||||
from django.test import TestCase, RequestFactory
|
||||
from django.http import HttpResponse
|
||||
|
||||
from ivatar.opentelemetry_config import (
|
||||
OpenTelemetryConfig,
|
||||
is_enabled,
|
||||
)
|
||||
from ivatar.opentelemetry_middleware import (
|
||||
OpenTelemetryMiddleware,
|
||||
trace_avatar_operation,
|
||||
trace_file_upload,
|
||||
trace_authentication,
|
||||
AvatarMetrics,
|
||||
get_avatar_metrics,
|
||||
reset_avatar_metrics,
|
||||
)
|
||||
|
||||
|
||||
class OpenTelemetryConfigTest(TestCase):
|
||||
"""Test OpenTelemetry configuration."""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test environment."""
|
||||
self.original_env = os.environ.copy()
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up test environment."""
|
||||
os.environ.clear()
|
||||
os.environ.update(self.original_env)
|
||||
|
||||
def test_config_always_enabled(self):
|
||||
"""Test that OpenTelemetry instrumentation is always enabled."""
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertTrue(config.enabled)
|
||||
|
||||
def test_config_enabled_with_env_var(self):
|
||||
"""Test that OpenTelemetry can be enabled with environment variable."""
|
||||
os.environ["OTEL_ENABLED"] = "true"
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertTrue(config.enabled)
|
||||
|
||||
def test_service_name_default(self):
|
||||
"""Test default service name."""
|
||||
# Clear environment variables to test default behavior
|
||||
original_env = os.environ.copy()
|
||||
os.environ.pop("OTEL_SERVICE_NAME", None)
|
||||
|
||||
try:
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertEqual(config.service_name, "ivatar")
|
||||
finally:
|
||||
os.environ.clear()
|
||||
os.environ.update(original_env)
|
||||
|
||||
def test_service_name_custom(self):
|
||||
"""Test custom service name."""
|
||||
os.environ["OTEL_SERVICE_NAME"] = "custom-service"
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertEqual(config.service_name, "custom-service")
|
||||
|
||||
def test_environment_default(self):
|
||||
"""Test default environment."""
|
||||
# Clear environment variables to test default behavior
|
||||
original_env = os.environ.copy()
|
||||
os.environ.pop("OTEL_ENVIRONMENT", None)
|
||||
|
||||
try:
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertEqual(config.environment, "development")
|
||||
finally:
|
||||
os.environ.clear()
|
||||
os.environ.update(original_env)
|
||||
|
||||
def test_environment_custom(self):
|
||||
"""Test custom environment."""
|
||||
os.environ["OTEL_ENVIRONMENT"] = "production"
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertEqual(config.environment, "production")
|
||||
|
||||
def test_resource_creation(self):
|
||||
"""Test resource creation with service information."""
|
||||
os.environ["OTEL_SERVICE_NAME"] = "test-service"
|
||||
os.environ["OTEL_ENVIRONMENT"] = "test"
|
||||
os.environ["IVATAR_VERSION"] = "1.0.0"
|
||||
os.environ["HOSTNAME"] = "test-host"
|
||||
|
||||
config = OpenTelemetryConfig()
|
||||
resource = config.resource
|
||||
|
||||
self.assertEqual(resource.attributes["service.name"], "test-service")
|
||||
self.assertEqual(resource.attributes["service.version"], "1.0.0")
|
||||
self.assertEqual(resource.attributes["deployment.environment"], "test")
|
||||
self.assertEqual(resource.attributes["service.instance.id"], "test-host")
|
||||
|
||||
@patch("ivatar.opentelemetry_config.OTLPSpanExporter")
|
||||
@patch("ivatar.opentelemetry_config.BatchSpanProcessor")
|
||||
@patch("ivatar.opentelemetry_config.trace")
|
||||
def test_setup_tracing_with_otlp(self, mock_trace, mock_processor, mock_exporter):
|
||||
"""Test tracing setup with OTLP endpoint."""
|
||||
os.environ["OTEL_EXPORT_ENABLED"] = "true"
|
||||
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4317"
|
||||
|
||||
config = OpenTelemetryConfig()
|
||||
config.setup_tracing()
|
||||
|
||||
mock_exporter.assert_called_once_with(endpoint="http://localhost:4317")
|
||||
mock_processor.assert_called_once()
|
||||
mock_trace.get_tracer_provider().add_span_processor.assert_called_once()
|
||||
|
||||
@patch("ivatar.opentelemetry_config.PrometheusMetricReader")
|
||||
@patch("ivatar.opentelemetry_config.PeriodicExportingMetricReader")
|
||||
@patch("ivatar.opentelemetry_config.OTLPMetricExporter")
|
||||
@patch("ivatar.opentelemetry_config.metrics")
|
||||
def test_setup_metrics_with_prometheus_and_otlp(
|
||||
self,
|
||||
mock_metrics,
|
||||
mock_otlp_exporter,
|
||||
mock_periodic_reader,
|
||||
mock_prometheus_reader,
|
||||
):
|
||||
"""Test metrics setup with Prometheus and OTLP."""
|
||||
os.environ["OTEL_EXPORT_ENABLED"] = "true"
|
||||
os.environ["OTEL_PROMETHEUS_ENDPOINT"] = "0.0.0.0:9464"
|
||||
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4317"
|
||||
|
||||
config = OpenTelemetryConfig()
|
||||
config.setup_metrics()
|
||||
|
||||
mock_prometheus_reader.assert_called_once()
|
||||
mock_otlp_exporter.assert_called_once_with(endpoint="http://localhost:4317")
|
||||
mock_periodic_reader.assert_called_once()
|
||||
mock_metrics.set_meter_provider.assert_called_once()
|
||||
|
||||
@patch("ivatar.opentelemetry_config.DjangoInstrumentor")
|
||||
@patch("ivatar.opentelemetry_config.Psycopg2Instrumentor")
|
||||
@patch("ivatar.opentelemetry_config.PyMySQLInstrumentor")
|
||||
@patch("ivatar.opentelemetry_config.RequestsInstrumentor")
|
||||
@patch("ivatar.opentelemetry_config.URLLib3Instrumentor")
|
||||
def test_setup_instrumentation(
|
||||
self,
|
||||
mock_urllib3,
|
||||
mock_requests,
|
||||
mock_pymysql,
|
||||
mock_psycopg2,
|
||||
mock_django,
|
||||
):
|
||||
"""Test instrumentation setup."""
|
||||
os.environ["OTEL_ENABLED"] = "true"
|
||||
|
||||
config = OpenTelemetryConfig()
|
||||
config.setup_instrumentation()
|
||||
|
||||
mock_django().instrument.assert_called_once()
|
||||
mock_psycopg2().instrument.assert_called_once()
|
||||
mock_pymysql().instrument.assert_called_once()
|
||||
mock_requests().instrument.assert_called_once()
|
||||
mock_urllib3().instrument.assert_called_once()
|
||||
|
||||
|
||||
class OpenTelemetryMiddlewareTest(TestCase):
|
||||
"""Test OpenTelemetry middleware."""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test environment."""
|
||||
self.factory = RequestFactory()
|
||||
reset_avatar_metrics() # Reset global metrics instance
|
||||
self.middleware = OpenTelemetryMiddleware(lambda r: HttpResponse("test"))
|
||||
|
||||
@patch("ivatar.opentelemetry_middleware.get_tracer")
|
||||
def test_middleware_enabled(self, mock_get_tracer):
|
||||
"""Test middleware when OpenTelemetry is enabled."""
|
||||
mock_tracer = MagicMock()
|
||||
mock_span = MagicMock()
|
||||
mock_tracer.start_span.return_value = mock_span
|
||||
mock_get_tracer.return_value = mock_tracer
|
||||
|
||||
request = self.factory.get("/avatar/test@example.com")
|
||||
response = self.middleware(request)
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertTrue(hasattr(request, "_ot_span"))
|
||||
mock_tracer.start_span.assert_called_once()
|
||||
mock_span.set_attributes.assert_called()
|
||||
mock_span.end.assert_called_once()
|
||||
|
||||
@patch("ivatar.opentelemetry_middleware.get_tracer")
|
||||
def test_avatar_request_attributes(self, mock_get_tracer):
|
||||
"""Test that avatar requests get proper attributes."""
|
||||
mock_tracer = MagicMock()
|
||||
mock_span = MagicMock()
|
||||
mock_tracer.start_span.return_value = mock_span
|
||||
mock_get_tracer.return_value = mock_tracer
|
||||
|
||||
request = self.factory.get("/avatar/test@example.com?s=128&d=png")
|
||||
# Reset metrics to ensure we get a fresh instance
|
||||
reset_avatar_metrics()
|
||||
self.middleware.process_request(request)
|
||||
|
||||
# Check that avatar-specific attributes were set
|
||||
calls = mock_span.set_attributes.call_args_list
|
||||
avatar_attrs = any(
|
||||
call[0][0].get("ivatar.request_type") == "avatar" for call in calls
|
||||
)
|
||||
# Also check for individual set_attribute calls
|
||||
set_attribute_calls = mock_span.set_attribute.call_args_list
|
||||
individual_avatar_attrs = any(
|
||||
call[0][0] == "ivatar.request_type" and call[0][1] == "avatar"
|
||||
for call in set_attribute_calls
|
||||
)
|
||||
self.assertTrue(avatar_attrs or individual_avatar_attrs)
|
||||
|
||||
def test_is_avatar_request(self):
|
||||
"""Test avatar request detection."""
|
||||
avatar_request = self.factory.get("/avatar/test@example.com")
|
||||
non_avatar_request = self.factory.get("/stats/")
|
||||
|
||||
self.assertTrue(self.middleware._is_avatar_request(avatar_request))
|
||||
self.assertFalse(self.middleware._is_avatar_request(non_avatar_request))
|
||||
|
||||
def test_get_avatar_size(self):
|
||||
"""Test avatar size extraction."""
|
||||
request = self.factory.get("/avatar/test@example.com?s=256")
|
||||
size = self.middleware._get_avatar_size(request)
|
||||
self.assertEqual(size, "256")
|
||||
|
||||
def test_get_avatar_format(self):
|
||||
"""Test avatar format extraction."""
|
||||
request = self.factory.get("/avatar/test@example.com?d=jpg")
|
||||
format_type = self.middleware._get_avatar_format(request)
|
||||
self.assertEqual(format_type, "jpg")
|
||||
|
||||
def test_get_avatar_email(self):
|
||||
"""Test email extraction from avatar request."""
|
||||
request = self.factory.get("/avatar/test@example.com")
|
||||
email = self.middleware._get_avatar_email(request)
|
||||
self.assertEqual(email, "test@example.com")
|
||||
|
||||
|
||||
class AvatarMetricsTest(TestCase):
|
||||
"""Test AvatarMetrics class."""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test environment."""
|
||||
self.metrics = AvatarMetrics()
|
||||
|
||||
@patch("ivatar.opentelemetry_middleware.get_meter")
|
||||
def test_metrics_enabled(self, mock_get_meter):
|
||||
"""Test metrics when OpenTelemetry is enabled."""
|
||||
mock_meter = MagicMock()
|
||||
mock_counter = MagicMock()
|
||||
mock_histogram = MagicMock()
|
||||
|
||||
mock_meter.create_counter.return_value = mock_counter
|
||||
mock_meter.create_histogram.return_value = mock_histogram
|
||||
mock_get_meter.return_value = mock_meter
|
||||
|
||||
avatar_metrics = AvatarMetrics()
|
||||
|
||||
# Test avatar generation recording
|
||||
avatar_metrics.record_avatar_generated("128", "png", "generated")
|
||||
mock_counter.add.assert_called_with(
|
||||
1, {"size": "128", "format": "png", "source": "generated"}
|
||||
)
|
||||
|
||||
# Test cache hit recording
|
||||
avatar_metrics.record_cache_hit("128", "png")
|
||||
mock_counter.add.assert_called_with(1, {"size": "128", "format": "png"})
|
||||
|
||||
# Test file upload recording
|
||||
avatar_metrics.record_file_upload(1024, "image/png", True)
|
||||
mock_histogram.record.assert_called_with(
|
||||
1024, {"content_type": "image/png", "success": "True"}
|
||||
)
|
||||
|
||||
|
||||
class TracingDecoratorsTest(TestCase):
|
||||
"""Test tracing decorators."""
|
||||
|
||||
@patch("ivatar.opentelemetry_middleware.get_tracer")
|
||||
def test_trace_avatar_operation(self, mock_get_tracer):
|
||||
"""Test trace_avatar_operation decorator."""
|
||||
mock_tracer = MagicMock()
|
||||
mock_span = MagicMock()
|
||||
mock_tracer.start_as_current_span.return_value.__enter__.return_value = (
|
||||
mock_span
|
||||
)
|
||||
mock_get_tracer.return_value = mock_tracer
|
||||
|
||||
@trace_avatar_operation("test_operation")
|
||||
def test_function():
|
||||
return "success"
|
||||
|
||||
result = test_function()
|
||||
|
||||
self.assertEqual(result, "success")
|
||||
mock_tracer.start_as_current_span.assert_called_once_with(
|
||||
"avatar.test_operation"
|
||||
)
|
||||
mock_span.set_status.assert_called_once()
|
||||
|
||||
@patch("ivatar.opentelemetry_middleware.get_tracer")
|
||||
def test_trace_avatar_operation_exception(self, mock_get_tracer):
|
||||
"""Test trace_avatar_operation decorator with exception."""
|
||||
mock_tracer = MagicMock()
|
||||
mock_span = MagicMock()
|
||||
mock_tracer.start_as_current_span.return_value.__enter__.return_value = (
|
||||
mock_span
|
||||
)
|
||||
mock_get_tracer.return_value = mock_tracer
|
||||
|
||||
@trace_avatar_operation("test_operation")
|
||||
def test_function():
|
||||
raise ValueError("test error")
|
||||
|
||||
with self.assertRaises(ValueError):
|
||||
test_function()
|
||||
|
||||
mock_span.set_status.assert_called_once()
|
||||
mock_span.set_attribute.assert_called_with("error.message", "test error")
|
||||
|
||||
def test_trace_file_upload(self):
|
||||
"""Test trace_file_upload decorator."""
|
||||
|
||||
@trace_file_upload("test_upload")
|
||||
def test_function():
|
||||
return "success"
|
||||
|
||||
result = test_function()
|
||||
self.assertEqual(result, "success")
|
||||
|
||||
def test_trace_authentication(self):
|
||||
"""Test trace_authentication decorator."""
|
||||
|
||||
@trace_authentication("test_auth")
|
||||
def test_function():
|
||||
return "success"
|
||||
|
||||
result = test_function()
|
||||
self.assertEqual(result, "success")
|
||||
|
||||
|
||||
class IntegrationTest(TestCase):
|
||||
"""Integration tests for OpenTelemetry."""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test environment."""
|
||||
self.original_env = os.environ.copy()
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up test environment."""
|
||||
os.environ.clear()
|
||||
os.environ.update(self.original_env)
|
||||
|
||||
@patch("ivatar.opentelemetry_config.setup_opentelemetry")
|
||||
def test_setup_opentelemetry_called(self, mock_setup):
|
||||
"""Test that setup_opentelemetry is called during Django startup."""
|
||||
# This would be called during Django settings import
|
||||
from ivatar.opentelemetry_config import setup_opentelemetry as setup_func
|
||||
|
||||
setup_func()
|
||||
mock_setup.assert_called_once()
|
||||
|
||||
def test_is_enabled_function(self):
|
||||
"""Test is_enabled function."""
|
||||
# OpenTelemetry is now always enabled
|
||||
self.assertTrue(is_enabled())
|
||||
|
||||
# Test enabled with environment variable
|
||||
os.environ["OTEL_ENABLED"] = "true"
|
||||
config = OpenTelemetryConfig()
|
||||
self.assertTrue(config.enabled)
|
||||
|
||||
|
||||
class OpenTelemetryDisabledTest(TestCase):
|
||||
"""Test OpenTelemetry behavior when disabled (no-op mode)."""
|
||||
|
||||
def setUp(self):
|
||||
"""Set up test environment."""
|
||||
self.original_env = os.environ.copy()
|
||||
# Ensure OpenTelemetry is disabled
|
||||
os.environ.pop("ENABLE_OPENTELEMETRY", None)
|
||||
os.environ.pop("OTEL_ENABLED", None)
|
||||
|
||||
def tearDown(self):
|
||||
"""Clean up test environment."""
|
||||
os.environ.clear()
|
||||
os.environ.update(self.original_env)
|
||||
|
||||
def test_opentelemetry_always_enabled(self):
|
||||
"""Test that OpenTelemetry instrumentation is always enabled."""
|
||||
# OpenTelemetry instrumentation is now always enabled
|
||||
self.assertTrue(is_enabled())
|
||||
|
||||
def test_decorators_work(self):
|
||||
"""Test that decorators work when OpenTelemetry is enabled."""
|
||||
|
||||
@trace_avatar_operation("test_operation")
|
||||
def test_function():
|
||||
return "success"
|
||||
|
||||
result = test_function()
|
||||
self.assertEqual(result, "success")
|
||||
|
||||
def test_metrics_work(self):
|
||||
"""Test that metrics work when OpenTelemetry is enabled."""
|
||||
avatar_metrics = get_avatar_metrics()
|
||||
|
||||
# These should not raise exceptions
|
||||
avatar_metrics.record_avatar_generated("80", "png", "uploaded")
|
||||
avatar_metrics.record_cache_hit("80", "png")
|
||||
avatar_metrics.record_cache_miss("80", "png")
|
||||
avatar_metrics.record_external_request("gravatar", 200)
|
||||
avatar_metrics.record_file_upload(1024, "image/png", True)
|
||||
|
||||
def test_middleware_enabled(self):
|
||||
"""Test that middleware works when OpenTelemetry is enabled."""
|
||||
factory = RequestFactory()
|
||||
middleware = OpenTelemetryMiddleware(lambda r: HttpResponse("test"))
|
||||
|
||||
request = factory.get("/avatar/test@example.com")
|
||||
response = middleware(request)
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertEqual(response.content.decode(), "test")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -10,7 +10,7 @@ from django.conf.urls.static import static
|
||||
from django.views.generic import TemplateView, RedirectView
|
||||
from ivatar import settings
|
||||
from .views import AvatarImageView, StatsView
|
||||
from .views import GravatarProxyView, BlueskyProxyView
|
||||
from .views import GravatarProxyView, BlueskyProxyView, DeploymentVersionView
|
||||
|
||||
urlpatterns = [ # pylint: disable=invalid-name
|
||||
path("admin/", admin.site.urls),
|
||||
@@ -69,6 +69,11 @@ urlpatterns = [ # pylint: disable=invalid-name
|
||||
),
|
||||
path("talk_to_us/", RedirectView.as_view(url="/contact"), name="talk_to_us"),
|
||||
path("stats/", StatsView.as_view(), name="stats"),
|
||||
path(
|
||||
"deployment/version/",
|
||||
DeploymentVersionView.as_view(),
|
||||
name="deployment_version",
|
||||
),
|
||||
]
|
||||
|
||||
MAINTENANCE = False
|
||||
|
||||
@@ -36,13 +36,15 @@ def urlopen(url, timeout=URL_TIMEOUT):
|
||||
|
||||
class Bluesky:
|
||||
"""
|
||||
Handle Bluesky client access
|
||||
Handle Bluesky client access with persistent session management
|
||||
"""
|
||||
|
||||
identifier = ""
|
||||
app_password = ""
|
||||
service = "https://bsky.social"
|
||||
session = None
|
||||
_shared_session = None # Class-level shared session
|
||||
_session_expires_at = None # Track session expiration
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -54,10 +56,29 @@ class Bluesky:
|
||||
self.app_password = app_password
|
||||
self.service = service
|
||||
|
||||
def _is_session_valid(self) -> bool:
|
||||
"""
|
||||
Check if the current session is still valid
|
||||
"""
|
||||
if not self._shared_session or not self._session_expires_at:
|
||||
return False
|
||||
|
||||
import time
|
||||
|
||||
# Add 5 minute buffer before actual expiration
|
||||
return time.time() < (self._session_expires_at - 300)
|
||||
|
||||
def login(self):
|
||||
"""
|
||||
Login to Bluesky
|
||||
Login to Bluesky with session persistence
|
||||
"""
|
||||
# Use shared session if available and valid
|
||||
if self._is_session_valid():
|
||||
self.session = self._shared_session
|
||||
logger.debug("Reusing existing Bluesky session")
|
||||
return
|
||||
|
||||
logger.debug("Creating new Bluesky session")
|
||||
auth_response = requests.post(
|
||||
f"{self.service}/xrpc/com.atproto.server.createSession",
|
||||
json={"identifier": self.identifier, "password": self.app_password},
|
||||
@@ -65,6 +86,29 @@ class Bluesky:
|
||||
auth_response.raise_for_status()
|
||||
self.session = auth_response.json()
|
||||
|
||||
# Store session data for reuse
|
||||
self._shared_session = self.session
|
||||
import time
|
||||
|
||||
# Sessions typically expire in 24 hours, but we'll refresh every 12 hours
|
||||
self._session_expires_at = time.time() + (12 * 60 * 60)
|
||||
|
||||
logger.debug(
|
||||
"Created new Bluesky session, expires at: %s",
|
||||
time.strftime(
|
||||
"%Y-%m-%d %H:%M:%S", time.localtime(self._session_expires_at)
|
||||
),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def clear_shared_session(cls):
|
||||
"""
|
||||
Clear the shared session (useful for testing)
|
||||
"""
|
||||
cls._shared_session = None
|
||||
cls._session_expires_at = None
|
||||
logger.debug("Cleared shared Bluesky session")
|
||||
|
||||
def normalize_handle(self, handle: str) -> str:
|
||||
"""
|
||||
Return the normalized handle for given handle
|
||||
@@ -79,11 +123,10 @@ class Bluesky:
|
||||
handle = handle[:-1]
|
||||
return handle
|
||||
|
||||
def get_profile(self, handle: str) -> str:
|
||||
if not self.session:
|
||||
self.login()
|
||||
profile_response = None
|
||||
|
||||
def _make_profile_request(self, handle: str):
|
||||
"""
|
||||
Make a profile request to Bluesky API with automatic retry on session expiration
|
||||
"""
|
||||
try:
|
||||
profile_response = requests.get(
|
||||
f"{self.service}/xrpc/app.bsky.actor.getProfile",
|
||||
@@ -91,11 +134,32 @@ class Bluesky:
|
||||
params={"actor": handle},
|
||||
)
|
||||
profile_response.raise_for_status()
|
||||
return profile_response.json()
|
||||
except requests.exceptions.HTTPError as exc:
|
||||
if exc.response.status_code == 401:
|
||||
# Session expired, try to login again
|
||||
logger.warning("Bluesky session expired, re-authenticating")
|
||||
self.clear_shared_session()
|
||||
self.login()
|
||||
# Retry the request
|
||||
profile_response = requests.get(
|
||||
f"{self.service}/xrpc/app.bsky.actor.getProfile",
|
||||
headers={"Authorization": f'Bearer {self.session["accessJwt"]}'},
|
||||
params={"actor": handle},
|
||||
)
|
||||
profile_response.raise_for_status()
|
||||
return profile_response.json()
|
||||
else:
|
||||
logger.warning(f"Bluesky profile fetch failed with HTTP error: {exc}")
|
||||
return None
|
||||
except Exception as exc:
|
||||
logger.warning(f"Bluesky profile fetch failed with HTTP error: {exc}")
|
||||
logger.warning(f"Bluesky profile fetch failed with error: {exc}")
|
||||
return None
|
||||
|
||||
return profile_response.json()
|
||||
def get_profile(self, handle: str) -> str:
|
||||
if not self.session or not self._is_session_valid():
|
||||
self.login()
|
||||
return self._make_profile_request(handle)
|
||||
|
||||
def get_avatar(self, handle: str):
|
||||
"""
|
||||
|
||||
208
ivatar/views.py
208
ivatar/views.py
@@ -8,6 +8,7 @@ from io import BytesIO
|
||||
from os import path
|
||||
import hashlib
|
||||
import logging
|
||||
import threading
|
||||
from ivatar.utils import urlopen, Bluesky
|
||||
from urllib.error import HTTPError, URLError
|
||||
from ssl import SSLError
|
||||
@@ -39,6 +40,65 @@ from .ivataraccount.models import Photo
|
||||
from .ivataraccount.models import pil_format, file_format
|
||||
from .utils import is_trusted_url, mm_ng, resize_animated_gif
|
||||
|
||||
# Import OpenTelemetry only if feature flag is enabled
|
||||
try:
|
||||
from django.conf import settings
|
||||
|
||||
if getattr(settings, "ENABLE_OPENTELEMETRY", False):
|
||||
from .opentelemetry_middleware import trace_avatar_operation, get_avatar_metrics
|
||||
|
||||
avatar_metrics = get_avatar_metrics()
|
||||
else:
|
||||
# Create no-op decorators and metrics when OpenTelemetry is disabled
|
||||
def trace_avatar_operation(operation_name):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
class NoOpMetrics:
|
||||
def record_avatar_generated(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_cache_hit(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_cache_miss(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_external_request(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_file_upload(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
avatar_metrics = NoOpMetrics()
|
||||
except ImportError:
|
||||
# Django not available or settings not loaded
|
||||
def trace_avatar_operation(operation_name):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
class NoOpMetrics:
|
||||
def record_avatar_generated(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_cache_hit(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_cache_miss(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_external_request(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def record_file_upload(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
avatar_metrics = NoOpMetrics()
|
||||
|
||||
# Initialize loggers
|
||||
logger = logging.getLogger("ivatar")
|
||||
security_logger = logging.getLogger("ivatar.security")
|
||||
@@ -121,6 +181,8 @@ class AvatarImageView(TemplateView):
|
||||
# Check the cache first
|
||||
if CACHE_RESPONSE:
|
||||
if centry := caches["filesystem"].get(uri):
|
||||
# Record cache hit
|
||||
avatar_metrics.record_cache_hit(size=str(size), format_type=imgformat)
|
||||
# For DEBUG purpose only
|
||||
# print('Cached entry for %s' % uri)
|
||||
return HttpResponse(
|
||||
@@ -130,6 +192,9 @@ class AvatarImageView(TemplateView):
|
||||
reason=centry["reason"],
|
||||
charset=centry["charset"],
|
||||
)
|
||||
else:
|
||||
# Record cache miss
|
||||
avatar_metrics.record_cache_miss(size=str(size), format_type=imgformat)
|
||||
|
||||
# In case no digest at all is provided, return to home page
|
||||
if "digest" not in kwargs:
|
||||
@@ -297,6 +362,14 @@ class AvatarImageView(TemplateView):
|
||||
obj.save()
|
||||
if imgformat == "jpg":
|
||||
imgformat = "jpeg"
|
||||
|
||||
# Record avatar generation metrics
|
||||
avatar_metrics.record_avatar_generated(
|
||||
size=str(size),
|
||||
format_type=imgformat,
|
||||
source="uploaded" if obj else "generated",
|
||||
)
|
||||
|
||||
response = CachingHttpResponse(uri, data, content_type=f"image/{imgformat}")
|
||||
response["Cache-Control"] = "max-age=%i" % CACHE_IMAGES_MAX_AGE
|
||||
# Remove Vary header for images since language doesn't matter
|
||||
@@ -323,6 +396,7 @@ class AvatarImageView(TemplateView):
|
||||
response["Vary"] = ""
|
||||
return response
|
||||
|
||||
@trace_avatar_operation("generate_png")
|
||||
def _return_cached_png(self, arg0, data, uri):
|
||||
arg0.save(data, "PNG", quality=JPEG_QUALITY)
|
||||
return self._return_cached_response(data, uri)
|
||||
@@ -335,6 +409,7 @@ class GravatarProxyView(View):
|
||||
|
||||
# TODO: Do cache images!! Memcached?
|
||||
|
||||
@trace_avatar_operation("gravatar_proxy")
|
||||
def get(
|
||||
self, request, *args, **kwargs
|
||||
): # pylint: disable=too-many-branches,too-many-statements,too-many-locals,no-self-use,unused-argument,too-many-return-statements
|
||||
@@ -768,3 +843,136 @@ class StatsView(TemplateView, JsonResponse):
|
||||
}
|
||||
|
||||
return JsonResponse(retval)
|
||||
|
||||
|
||||
# Thread-safe version cache
|
||||
_version_cache = None
|
||||
_version_cache_lock = threading.Lock()
|
||||
|
||||
|
||||
def _get_git_info_from_files():
|
||||
"""
|
||||
Safely extract git information from .git files without subprocess calls
|
||||
"""
|
||||
try:
|
||||
# Get the project root directory
|
||||
project_root = path.dirname(path.dirname(path.abspath(__file__)))
|
||||
git_dir = path.join(project_root, ".git")
|
||||
|
||||
if not path.exists(git_dir):
|
||||
return None
|
||||
|
||||
# Read HEAD to get current branch/commit
|
||||
head_file = path.join(git_dir, "HEAD")
|
||||
if not path.exists(head_file):
|
||||
return None
|
||||
|
||||
with open(head_file, "r") as f:
|
||||
head_content = f.read().strip()
|
||||
|
||||
# Parse HEAD content
|
||||
if head_content.startswith("ref: "):
|
||||
# We're on a branch
|
||||
branch_ref = head_content[5:] # Remove 'ref: '
|
||||
branch_name = path.basename(branch_ref)
|
||||
|
||||
# Read the commit hash from the ref
|
||||
ref_file = path.join(git_dir, branch_ref)
|
||||
if path.exists(ref_file):
|
||||
with open(ref_file, "r") as f:
|
||||
commit_hash = f.read().strip()
|
||||
else:
|
||||
return None
|
||||
else:
|
||||
# Detached HEAD state
|
||||
commit_hash = head_content
|
||||
branch_name = "detached"
|
||||
|
||||
# Try to get commit date from git log file (if available)
|
||||
commit_date = None
|
||||
log_file = path.join(git_dir, "logs", "HEAD")
|
||||
if path.exists(log_file):
|
||||
try:
|
||||
with open(log_file, "r") as f:
|
||||
# Read last line to get most recent commit info
|
||||
lines = f.readlines()
|
||||
if lines:
|
||||
last_line = lines[-1].strip()
|
||||
# Git log format: <old_hash> <new_hash> <author> <timestamp> <timezone> <message>
|
||||
parts = last_line.split("\t")
|
||||
if len(parts) >= 2:
|
||||
# Extract timestamp and convert to readable date
|
||||
timestamp_part = parts[0].split()[-2] # Get timestamp
|
||||
if timestamp_part.isdigit():
|
||||
import datetime
|
||||
|
||||
timestamp = int(timestamp_part)
|
||||
commit_date = datetime.datetime.fromtimestamp(
|
||||
timestamp
|
||||
).strftime("%Y-%m-%d %H:%M:%S %z")
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
# Fallback: try to get date from commit object if available
|
||||
if not commit_date and len(commit_hash) == 40:
|
||||
try:
|
||||
commit_dir = path.join(git_dir, "objects", commit_hash[:2])
|
||||
commit_file = path.join(commit_dir, commit_hash[2:])
|
||||
if path.exists(commit_file):
|
||||
# This would require decompressing the git object, which is complex
|
||||
# For now, we'll use a placeholder
|
||||
commit_date = "unknown"
|
||||
except Exception:
|
||||
commit_date = "unknown"
|
||||
|
||||
return {
|
||||
"commit_hash": commit_hash,
|
||||
"short_hash": commit_hash[:7] if len(commit_hash) >= 7 else commit_hash,
|
||||
"branch": branch_name,
|
||||
"commit_date": commit_date or "unknown",
|
||||
"deployment_status": "active",
|
||||
"version": f"{branch_name}-{commit_hash[:7] if len(commit_hash) >= 7 else commit_hash}",
|
||||
}
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning(f"Failed to read git info from files: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
def _get_cached_version_info():
|
||||
"""
|
||||
Get cached version information, loading it if not available
|
||||
"""
|
||||
global _version_cache
|
||||
|
||||
with _version_cache_lock:
|
||||
if _version_cache is None:
|
||||
# Get version info from git files
|
||||
_version_cache = _get_git_info_from_files()
|
||||
|
||||
# If that fails, return error
|
||||
if _version_cache is None:
|
||||
_version_cache = {
|
||||
"error": "Unable to determine version - .git directory not found",
|
||||
"deployment_status": "unknown",
|
||||
}
|
||||
|
||||
return _version_cache
|
||||
|
||||
|
||||
class DeploymentVersionView(View):
|
||||
"""
|
||||
View to return deployment version information for CI/CD verification
|
||||
Uses cached version info to prevent DDoS attacks and improve performance
|
||||
"""
|
||||
|
||||
def get(self, request, *args, **kwargs):
|
||||
"""
|
||||
Return cached deployment version information
|
||||
"""
|
||||
version_info = _get_cached_version_info()
|
||||
|
||||
if "error" in version_info:
|
||||
return JsonResponse(version_info, status=500)
|
||||
|
||||
return JsonResponse(version_info)
|
||||
|
||||
@@ -23,6 +23,16 @@ git+https://github.com/ofalk/identicon.git
|
||||
git+https://github.com/ofalk/monsterid.git
|
||||
git+https://github.com/ofalk/Robohash.git@devel
|
||||
notsetuptools
|
||||
# OpenTelemetry dependencies (optional - can be disabled via feature flag)
|
||||
opentelemetry-api>=1.20.0
|
||||
opentelemetry-exporter-otlp>=1.20.0
|
||||
opentelemetry-exporter-prometheus>=0.54b0
|
||||
opentelemetry-instrumentation-django>=0.42b0
|
||||
opentelemetry-instrumentation-psycopg2>=0.42b0
|
||||
opentelemetry-instrumentation-pymysql>=0.42b0
|
||||
opentelemetry-instrumentation-requests>=0.42b0
|
||||
opentelemetry-instrumentation-urllib3>=0.42b0
|
||||
opentelemetry-sdk>=1.20.0
|
||||
Pillow
|
||||
pip
|
||||
psycopg2-binary
|
||||
@@ -32,8 +42,10 @@ pyLibravatar
|
||||
pylint
|
||||
pymemcache
|
||||
PyMySQL
|
||||
pytest
|
||||
python-coveralls
|
||||
python-language-server
|
||||
python-magic>=0.4.27
|
||||
pytz
|
||||
rope
|
||||
setuptools
|
||||
|
||||
448
scripts/check_deployment.py
Executable file
448
scripts/check_deployment.py
Executable file
@@ -0,0 +1,448 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Libravatar Deployment Verification Script
|
||||
|
||||
This script verifies that Libravatar deployments are working correctly by:
|
||||
- Checking version endpoint
|
||||
- Testing avatar functionality with various sizes
|
||||
- Verifying stats endpoint
|
||||
- Testing redirect behavior
|
||||
|
||||
Usage:
|
||||
python3 check_deployment.py --dev # Test dev deployment
|
||||
python3 check_deployment.py --prod # Test production deployment
|
||||
python3 check_deployment.py --endpoint <url> # Test custom endpoint
|
||||
python3 check_deployment.py --dev --prod # Test both deployments
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import ssl
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Dict, Optional, Tuple
|
||||
from urllib.parse import urljoin
|
||||
from urllib.request import urlopen, Request
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
try:
|
||||
from PIL import Image
|
||||
|
||||
PIL_AVAILABLE = True
|
||||
except ImportError:
|
||||
PIL_AVAILABLE = False
|
||||
|
||||
# Configuration
|
||||
DEV_URL = "https://dev.libravatar.org"
|
||||
PROD_URL = "https://libravatar.org"
|
||||
MAX_RETRIES = 5
|
||||
RETRY_DELAY = 10
|
||||
|
||||
# ANSI color codes
|
||||
|
||||
|
||||
class Colors:
|
||||
RED = "\033[0;31m"
|
||||
GREEN = "\033[0;32m"
|
||||
YELLOW = "\033[1;33m"
|
||||
BLUE = "\033[0;34m"
|
||||
NC = "\033[0m" # No Color
|
||||
|
||||
|
||||
def colored_print(message: str, color: str = Colors.NC) -> None:
|
||||
"""Print a colored message."""
|
||||
print(f"{color}{message}{Colors.NC}")
|
||||
|
||||
|
||||
def make_request(
|
||||
url: str,
|
||||
method: str = "GET",
|
||||
headers: Optional[Dict[str, str]] = None,
|
||||
follow_redirects: bool = True,
|
||||
binary: bool = False,
|
||||
) -> Tuple[bool, Optional[bytes], Optional[Dict[str, str]]]:
|
||||
"""
|
||||
Make an HTTP request and return success status, content, and headers.
|
||||
|
||||
Args:
|
||||
url: URL to request
|
||||
method: HTTP method
|
||||
headers: Additional headers
|
||||
follow_redirects: Whether to follow redirects automatically
|
||||
|
||||
Returns:
|
||||
Tuple of (success, content, headers)
|
||||
"""
|
||||
req = Request(url, headers=headers or {})
|
||||
req.get_method = lambda: method
|
||||
|
||||
# Create SSL context that handles certificate verification issues
|
||||
ssl_context = ssl.create_default_context()
|
||||
|
||||
# Try with SSL verification first
|
||||
try:
|
||||
opener = urlopen
|
||||
if not follow_redirects:
|
||||
# Create a custom opener that doesn't follow redirects
|
||||
import urllib.request
|
||||
|
||||
class NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
return None
|
||||
|
||||
opener = urllib.request.build_opener(NoRedirectHandler)
|
||||
|
||||
if follow_redirects:
|
||||
with opener(req, context=ssl_context) as response:
|
||||
content = response.read()
|
||||
if not binary:
|
||||
try:
|
||||
content = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
# If content is not text (e.g., binary image), return empty string
|
||||
content = ""
|
||||
headers = dict(response.headers)
|
||||
return True, content, headers
|
||||
else:
|
||||
response = opener.open(req)
|
||||
content = response.read()
|
||||
if not binary:
|
||||
try:
|
||||
content = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
content = ""
|
||||
headers = dict(response.headers)
|
||||
return True, content, headers
|
||||
except URLError as url_err:
|
||||
# Check if this is an SSL error wrapped in URLError
|
||||
if isinstance(url_err.reason, ssl.SSLError):
|
||||
# If SSL fails, try with unverified context (less secure but works for testing)
|
||||
ssl_context_unverified = ssl.create_default_context()
|
||||
ssl_context_unverified.check_hostname = False
|
||||
ssl_context_unverified.verify_mode = ssl.CERT_NONE
|
||||
|
||||
try:
|
||||
if follow_redirects:
|
||||
with urlopen(req, context=ssl_context_unverified) as response:
|
||||
content = response.read()
|
||||
if not binary:
|
||||
try:
|
||||
content = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
content = ""
|
||||
headers = dict(response.headers)
|
||||
return True, content, headers
|
||||
else:
|
||||
import urllib.request
|
||||
|
||||
class NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
return None
|
||||
|
||||
opener = urllib.request.build_opener(NoRedirectHandler)
|
||||
response = opener.open(req)
|
||||
content = response.read()
|
||||
if not binary:
|
||||
try:
|
||||
content = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
content = ""
|
||||
headers = dict(response.headers)
|
||||
return True, content, headers
|
||||
except Exception:
|
||||
return False, None, None
|
||||
else:
|
||||
return False, None, None
|
||||
except HTTPError:
|
||||
return False, None, None
|
||||
|
||||
|
||||
def check_version_endpoint(base_url: str) -> Tuple[bool, Optional[Dict]]:
|
||||
"""Check the version endpoint and return deployment info."""
|
||||
version_url = urljoin(base_url, "/deployment/version/")
|
||||
success, content, _ = make_request(version_url)
|
||||
|
||||
if not success or not content:
|
||||
return False, None
|
||||
|
||||
try:
|
||||
version_info = json.loads(content)
|
||||
return True, version_info
|
||||
except json.JSONDecodeError:
|
||||
return False, None
|
||||
|
||||
|
||||
def test_avatar_redirect(base_url: str) -> bool:
|
||||
"""Test that invalid avatar requests redirect to default image."""
|
||||
avatar_url = urljoin(base_url, "/avatar/test@example.com")
|
||||
|
||||
# Use a simple approach: check if the final URL after redirect contains deadbeef.png
|
||||
try:
|
||||
req = Request(avatar_url)
|
||||
ssl_context = ssl.create_default_context()
|
||||
ssl_context.check_hostname = False
|
||||
ssl_context.verify_mode = ssl.CERT_NONE
|
||||
|
||||
with urlopen(req, context=ssl_context) as response:
|
||||
final_url = response.geturl()
|
||||
return "deadbeef.png" in final_url
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def test_avatar_sizing(base_url: str) -> bool:
|
||||
"""Test avatar endpoint with random sizes."""
|
||||
# Use a known test hash for consistent testing
|
||||
test_hash = "63a75a80e6b1f4adfdb04c1ca02e596c"
|
||||
|
||||
# Generate random sizes between 50-250
|
||||
sizes = [random.randint(50, 250) for _ in range(2)]
|
||||
|
||||
for size in sizes:
|
||||
avatar_url = urljoin(base_url, f"/avatar/{test_hash}?s={size}")
|
||||
|
||||
# Download image to temporary file
|
||||
success, content, _ = make_request(avatar_url, binary=True)
|
||||
if not success or not content:
|
||||
colored_print(f"❌ Avatar endpoint failed for size {size}", Colors.RED)
|
||||
return False
|
||||
|
||||
# Check image dimensions
|
||||
if PIL_AVAILABLE:
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".jpg") as temp_file:
|
||||
temp_file.write(content)
|
||||
temp_file.flush()
|
||||
|
||||
with Image.open(temp_file.name) as img:
|
||||
width, height = img.size
|
||||
if width == size and height == size:
|
||||
colored_print(
|
||||
f"✅ Avatar size {size}x{size} verified", Colors.GREEN
|
||||
)
|
||||
else:
|
||||
colored_print(
|
||||
f"❌ Avatar wrong size: expected {size}x{size}, got {width}x{height}",
|
||||
Colors.RED,
|
||||
)
|
||||
return False
|
||||
except Exception as e:
|
||||
colored_print(f"❌ Error checking image dimensions: {e}", Colors.RED)
|
||||
return False
|
||||
else:
|
||||
# Fallback: just check if we got some content
|
||||
if len(content) > 100: # Assume valid image if we got substantial content
|
||||
colored_print(
|
||||
f"✅ Avatar size {size} downloaded (PIL not available for verification)",
|
||||
Colors.YELLOW,
|
||||
)
|
||||
else:
|
||||
colored_print(
|
||||
f"❌ Avatar endpoint returned insufficient content for size {size}",
|
||||
Colors.RED,
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def test_stats_endpoint(base_url: str) -> bool:
|
||||
"""Test that the stats endpoint is accessible."""
|
||||
stats_url = urljoin(base_url, "/stats/")
|
||||
success, _, _ = make_request(stats_url)
|
||||
return success
|
||||
|
||||
|
||||
def test_deployment(
|
||||
base_url: str,
|
||||
name: str,
|
||||
max_retries: int = MAX_RETRIES,
|
||||
retry_delay: int = RETRY_DELAY,
|
||||
) -> bool:
|
||||
"""
|
||||
Test a deployment with retry logic.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of the deployment
|
||||
name: Human-readable name for the deployment
|
||||
max_retries: Maximum number of retry attempts
|
||||
|
||||
Returns:
|
||||
True if all tests pass, False otherwise
|
||||
"""
|
||||
colored_print(f"Testing {name} deployment at {base_url}", Colors.YELLOW)
|
||||
|
||||
for attempt in range(1, max_retries + 1):
|
||||
colored_print(
|
||||
f"Attempt {attempt}/{max_retries}: Checking {name} deployment...",
|
||||
Colors.BLUE,
|
||||
)
|
||||
|
||||
# Check if site is responding
|
||||
success, version_info = check_version_endpoint(base_url)
|
||||
if success and version_info:
|
||||
colored_print(
|
||||
f"{name} site is responding, checking version...", Colors.GREEN
|
||||
)
|
||||
|
||||
# Display version information
|
||||
commit_hash = version_info.get("commit_hash", "Unknown")
|
||||
branch = version_info.get("branch", "Unknown")
|
||||
version = version_info.get("version", "Unknown")
|
||||
|
||||
colored_print(f"Deployed commit: {commit_hash}", Colors.BLUE)
|
||||
colored_print(f"Deployed branch: {branch}", Colors.BLUE)
|
||||
colored_print(f"Deployed version: {version}", Colors.BLUE)
|
||||
|
||||
# Run functionality tests
|
||||
colored_print("Running basic functionality tests...", Colors.YELLOW)
|
||||
|
||||
# Test avatar redirect
|
||||
if test_avatar_redirect(base_url):
|
||||
colored_print("✅ Invalid avatar redirects correctly", Colors.GREEN)
|
||||
else:
|
||||
colored_print("❌ Invalid avatar redirect failed", Colors.RED)
|
||||
return False
|
||||
|
||||
# Test avatar sizing
|
||||
if test_avatar_sizing(base_url):
|
||||
pass # Success messages are printed within the function
|
||||
else:
|
||||
return False
|
||||
|
||||
# Test stats endpoint
|
||||
if test_stats_endpoint(base_url):
|
||||
colored_print("✅ Stats endpoint working", Colors.GREEN)
|
||||
else:
|
||||
colored_print("❌ Stats endpoint failed", Colors.RED)
|
||||
return False
|
||||
|
||||
colored_print(
|
||||
f"🎉 {name} deployment verification completed successfully!",
|
||||
Colors.GREEN,
|
||||
)
|
||||
return True
|
||||
else:
|
||||
colored_print(f"{name} site not responding yet...", Colors.YELLOW)
|
||||
|
||||
if attempt < max_retries:
|
||||
colored_print(
|
||||
f"Waiting {retry_delay} seconds before next attempt...", Colors.BLUE
|
||||
)
|
||||
time.sleep(retry_delay)
|
||||
|
||||
colored_print(
|
||||
f"❌ FAILED: {name} deployment verification timed out after {max_retries} attempts",
|
||||
Colors.RED,
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function with command-line argument parsing."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Libravatar Deployment Verification Script",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python3 check_deployment.py --dev # Test dev deployment
|
||||
python3 check_deployment.py --prod # Test production deployment
|
||||
python3 check_deployment.py --endpoint <url> # Test custom endpoint
|
||||
python3 check_deployment.py --dev --prod # Test both deployments
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--dev",
|
||||
action="store_true",
|
||||
help="Test dev deployment (https://dev.libravatar.org)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prod",
|
||||
action="store_true",
|
||||
help="Test production deployment (https://libravatar.org)",
|
||||
)
|
||||
parser.add_argument("--endpoint", type=str, help="Test custom endpoint URL")
|
||||
parser.add_argument(
|
||||
"--max-retries",
|
||||
type=int,
|
||||
default=MAX_RETRIES,
|
||||
help=f"Maximum number of retry attempts (default: {MAX_RETRIES})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--retry-delay",
|
||||
type=int,
|
||||
default=RETRY_DELAY,
|
||||
help=f"Delay between retry attempts in seconds (default: {RETRY_DELAY})",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate arguments
|
||||
if not any([args.dev, args.prod, args.endpoint]):
|
||||
parser.error("At least one of --dev, --prod, or --endpoint must be specified")
|
||||
|
||||
# Update configuration if custom values provided
|
||||
max_retries = args.max_retries
|
||||
retry_delay = args.retry_delay
|
||||
|
||||
colored_print("Libravatar Deployment Verification Script", Colors.BLUE)
|
||||
colored_print("=" * 50, Colors.BLUE)
|
||||
|
||||
# Check dependencies
|
||||
if not PIL_AVAILABLE:
|
||||
colored_print(
|
||||
"⚠️ Warning: PIL/Pillow not available. Image dimension verification will be limited.",
|
||||
Colors.YELLOW,
|
||||
)
|
||||
colored_print(" Install with: pip install Pillow", Colors.YELLOW)
|
||||
|
||||
results = []
|
||||
|
||||
# Test dev deployment
|
||||
if args.dev:
|
||||
colored_print("", Colors.NC)
|
||||
dev_result = test_deployment(DEV_URL, "Dev", max_retries, retry_delay)
|
||||
results.append(("Dev", dev_result))
|
||||
|
||||
# Test production deployment
|
||||
if args.prod:
|
||||
colored_print("", Colors.NC)
|
||||
prod_result = test_deployment(PROD_URL, "Production", max_retries, retry_delay)
|
||||
results.append(("Production", prod_result))
|
||||
|
||||
# Test custom endpoint
|
||||
if args.endpoint:
|
||||
colored_print("", Colors.NC)
|
||||
custom_result = test_deployment(
|
||||
args.endpoint, "Custom", max_retries, retry_delay
|
||||
)
|
||||
results.append(("Custom", custom_result))
|
||||
|
||||
# Summary
|
||||
colored_print("", Colors.NC)
|
||||
colored_print("=" * 50, Colors.BLUE)
|
||||
colored_print("Deployment Verification Summary:", Colors.BLUE)
|
||||
colored_print("=" * 50, Colors.BLUE)
|
||||
|
||||
all_passed = True
|
||||
for name, result in results:
|
||||
if result:
|
||||
colored_print(f"✅ {name} deployment: PASSED", Colors.GREEN)
|
||||
else:
|
||||
colored_print(f"❌ {name} deployment: FAILED", Colors.RED)
|
||||
all_passed = False
|
||||
|
||||
if all_passed:
|
||||
colored_print("🎉 All deployment verifications passed!", Colors.GREEN)
|
||||
sys.exit(0)
|
||||
else:
|
||||
colored_print("❌ Some deployment verifications failed!", Colors.RED)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
28
scripts/run_tests_local.sh
Executable file
28
scripts/run_tests_local.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/bin/bash
|
||||
# Run tests locally, skipping Bluesky tests that require external API credentials
|
||||
# OpenTelemetry instrumentation is always enabled, but export is disabled for local testing
|
||||
|
||||
echo "Running tests locally (skipping Bluesky tests, OpenTelemetry export disabled)..."
|
||||
echo "============================================================================="
|
||||
|
||||
# OpenTelemetry instrumentation is always enabled, but disable export for local testing
|
||||
export OTEL_EXPORT_ENABLED=false
|
||||
export OTEL_SERVICE_NAME=ivatar-local
|
||||
export OTEL_ENVIRONMENT=development
|
||||
|
||||
# Run Django tests excluding Bluesky tests (OpenTelemetry tests are included)
|
||||
python3 manage.py test \
|
||||
--exclude-tag=bluesky \
|
||||
-v2
|
||||
|
||||
echo ""
|
||||
echo "To run all tests including Bluesky (requires API credentials):"
|
||||
echo "python3 manage.py test -v3"
|
||||
echo ""
|
||||
echo "To run only Bluesky tests:"
|
||||
echo "python3 manage.py test ivatar.ivataraccount.test_views_bluesky -v3"
|
||||
echo ""
|
||||
echo "To run tests with OpenTelemetry export enabled:"
|
||||
echo "OTEL_EXPORT_ENABLED=true python3 manage.py test -v2"
|
||||
echo ""
|
||||
echo "Note: OpenTelemetry instrumentation is always enabled. Only export is controlled by OTEL_EXPORT_ENABLED."
|
||||
19
scripts/run_tests_no_ot.sh
Executable file
19
scripts/run_tests_no_ot.sh
Executable file
@@ -0,0 +1,19 @@
|
||||
#!/bin/bash
|
||||
# Run tests with OpenTelemetry instrumentation enabled but export disabled
|
||||
# This is the default test mode for most users
|
||||
|
||||
set -e
|
||||
|
||||
echo "Running tests with OpenTelemetry instrumentation (export disabled)..."
|
||||
echo "===================================================================="
|
||||
|
||||
# OpenTelemetry instrumentation is always enabled, but disable export for testing
|
||||
export OTEL_EXPORT_ENABLED=false
|
||||
export OTEL_SERVICE_NAME=ivatar-test
|
||||
export OTEL_ENVIRONMENT=test
|
||||
|
||||
# Run Django tests (Django will auto-discover all tests)
|
||||
python3 manage.py test -v3
|
||||
|
||||
echo ""
|
||||
echo "Tests completed successfully (OpenTelemetry instrumentation enabled, export disabled)"
|
||||
50
scripts/run_tests_with_coverage.py
Executable file
50
scripts/run_tests_with_coverage.py
Executable file
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Run tests with OpenTelemetry instrumentation and export enabled, plus coverage measurement.
|
||||
This script is designed to be used with 'coverage run' command.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import django
|
||||
from django.conf import settings
|
||||
from django.test.utils import get_runner
|
||||
|
||||
|
||||
def main():
|
||||
# Enable OpenTelemetry instrumentation and export
|
||||
os.environ["OTEL_EXPORT_ENABLED"] = "true"
|
||||
os.environ["OTEL_SERVICE_NAME"] = "ivatar-test"
|
||||
os.environ["OTEL_ENVIRONMENT"] = "test"
|
||||
|
||||
print("Running tests with OpenTelemetry instrumentation and export enabled...")
|
||||
print("====================================================================")
|
||||
|
||||
# Add current directory to Python path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
# Setup Django
|
||||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "ivatar.settings")
|
||||
django.setup()
|
||||
|
||||
# Get Django test runner
|
||||
TestRunner = get_runner(settings)
|
||||
test_runner = TestRunner()
|
||||
|
||||
# Run tests
|
||||
failures = test_runner.run_tests([])
|
||||
|
||||
if failures:
|
||||
print(f"Tests failed with {failures} failures")
|
||||
return 1
|
||||
else:
|
||||
print("")
|
||||
print(
|
||||
"Tests completed successfully (OpenTelemetry instrumentation and export enabled)"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
19
scripts/run_tests_with_ot.sh
Executable file
19
scripts/run_tests_with_ot.sh
Executable file
@@ -0,0 +1,19 @@
|
||||
#!/bin/bash
|
||||
# Run tests with OpenTelemetry instrumentation and export enabled
|
||||
# This is used in CI to test OpenTelemetry functionality
|
||||
|
||||
set -e
|
||||
|
||||
echo "Running tests with OpenTelemetry instrumentation and export enabled..."
|
||||
echo "===================================================================="
|
||||
|
||||
# Enable OpenTelemetry instrumentation and export
|
||||
export OTEL_EXPORT_ENABLED=true
|
||||
export OTEL_SERVICE_NAME=ivatar-test
|
||||
export OTEL_ENVIRONMENT=test
|
||||
|
||||
# Run Django tests (Django will auto-discover all tests)
|
||||
python3 manage.py test -v3
|
||||
|
||||
echo ""
|
||||
echo "Tests completed successfully (OpenTelemetry instrumentation and export enabled)"
|
||||
Reference in New Issue
Block a user