Commit Graph

1148 Commits

Author SHA1 Message Date
Oliver Falk
d228674d4b Fix test with DjangoInstrumentor 2025-10-18 11:08:40 +02:00
Oliver Falk
c3c4a23619 Enable OTEL again, but only in one place 2025-10-18 11:00:49 +02:00
Oliver Falk
54426c84bc Merge branch 'master' into devel 2025-10-17 16:54:54 +02:00
Oliver Falk
a10a73c56e Do not add the OTEL middleware currently 2025-10-17 16:54:04 +02:00
Oliver Falk
f699a66236 Temporarily disable Django instrumentation to test Host header issue
- Comment out DjangoInstrumentor().instrument() to test if it's causing duplicate Host headers
- Remove unused DjangoInstrumentor import
- Keep other instrumentation (database, HTTP client) enabled
- This is a temporary test to isolate the Host header duplication issue
2025-10-17 15:42:53 +02:00
Oliver Falk
ac9c8fcbc3 Merge devel into master - resolve OpenTelemetry conflict
- Resolve merge conflict in opentelemetry_config.py
- Keep improved OSError handling for 'Address in use' error
- Include latest OpenTelemetry multiple initialization fix
2025-10-17 14:57:48 +02:00
Oliver Falk
ad4e5068b9 Fix OpenTelemetry multiple initialization issue
- Add _ot_initialized flag to prevent multiple setup calls
- Make setup_opentelemetry() idempotent
- Handle 'Address in use' error gracefully for Prometheus server
- Prevent OpenTelemetry setup failures due to multiple initialization
2025-10-17 14:56:15 +02:00
Oliver Falk
684cdca4e4 Merge branch 'devel' into 'master'
Enhance the version endpoint and fix OTEL deployment

See merge request oliver/ivatar!267
2025-10-17 14:49:10 +02:00
Oliver Falk
6db3450b20 Enhance the version endpoint and fix OTEL deployment 2025-10-17 14:49:10 +02:00
Oliver Falk
8e61f02702 Fix OpenTelemetry initialization - remove ENABLE_OPENTELEMETRY dependency
- Always initialize OpenTelemetry in Django settings (instrumentation always enabled)
- Remove ENABLE_OPENTELEMETRY feature flag from config.py
- Simplify views.py OpenTelemetry imports to always use real implementation
- Export control now handled by OTEL_EXPORT_ENABLED environment variable only

This ensures OpenTelemetry is properly initialized during Django startup
and the Prometheus metrics server starts correctly.
2025-10-17 14:43:05 +02:00
Oliver Falk
c17b078913 Add prometheus-client dependency for OpenTelemetry metrics server 2025-10-17 14:32:34 +02:00
Oliver Falk
ba4c587e5c Simplify version endpoint caching to indefinite cache
Since containers restart on content changes, remove TTL-based cache expiration
and cache version information indefinitely. This provides maximum performance
benefit while maintaining correctness.

- Remove timestamp-based cache expiration logic
- Cache version info indefinitely until container restart
- Simplify caching function by removing TTL complexity
- Maintain optimized git log file reading
2025-10-17 14:14:14 +02:00
Oliver Falk
8a1ccb1e0f Fix OpenTelemetry Prometheus metrics server startup
- Add _start_prometheus_server method to start HTTP server
- Register PrometheusMetricReader collector with prometheus_client REGISTRY
- Parse endpoint to extract host and port for HTTP server
- This fixes the issue where metrics endpoint was not accessible
2025-10-17 14:07:27 +02:00
Oliver Falk
e79398bc33 Fix performance issues in /deployment/version/ endpoint
- Add proper cache expiration with 5-minute TTL
- Optimize git log file reading to avoid loading entire file
- Read only last chunk (1024 bytes) instead of all lines
- Add shorter TTL (30s) for error cases to allow retry
- Improve error handling with UnicodeDecodeError support
- Maintain backward compatibility and security (no subprocess calls)

This fixes the 30-second response time issue by implementing efficient
caching and optimized file I/O operations.
2025-10-17 14:03:45 +02:00
Oliver Falk
25e9e489c3 Merge branch 'devel' into 'master'
Pull latest fixed from devel

See merge request oliver/ivatar!266
2025-10-17 12:56:16 +02:00
Oliver Falk
d3ffef3407 Pull latest fixed from devel 2025-10-17 12:56:15 +02:00
Oliver Falk
e8951014f9 Resolve merge conflict in test_views.py - choose devel version formatting 2025-10-17 12:42:47 +02:00
Oliver Falk
a6751d79d4 Merge branch 'master' into devel 2025-10-17 12:38:37 +02:00
Oliver Falk
1411420c65 Add comprehensive OpenID error handling tests for error.html template coverage
- Add OpenIDErrorHandlingTestCase with 8 test methods
- Test OpenID discovery failures, confirmation failures, and cancellations
- Test template inheritance (openid/failure.html extends error.html)
- Test direct error.html template rendering with authenticated/anonymous users
- Use unittest.mock to simulate OpenID failures without external dependencies
- Verify error messages are properly displayed to users
- Ensure comprehensive coverage of error.html template through OpenID scenarios

All tests pass and maintain existing test coverage.
2025-10-17 11:25:26 +02:00
Oliver Falk
2966a85083 Merge branch 'devel' into 'master'
File upload security (iteration 1), security enhancements and OpenTelemetry (OTEL) implementation (sending data disabled by default)

See merge request oliver/ivatar!265
2025-10-17 11:16:49 +02:00
Oliver Falk
780dc18fa4 File upload security (iteration 1), security enhancements and OpenTelemetry (OTEL) implementation (sending data disabled by default) 2025-10-17 11:16:48 +02:00
Oliver Falk
dcdbc6b608 Update test scripts and documentation for simplified OpenTelemetry approach
- Update all test scripts to use OTEL_EXPORT_ENABLED instead of legacy flags
- Remove references to deprecated ENABLE_OPENTELEMETRY and OTEL_ENABLED
- Simplify run_tests_local.sh to use --exclude-tag=bluesky
- Update documentation to reflect instrumentation always enabled
- Remove legacy configuration section from README.md

All scripts now use the new approach where:
- OpenTelemetry instrumentation is always enabled
- Only data export is controlled by OTEL_EXPORT_ENABLED flag
- Cleaner configuration with single export control flag
2025-10-17 11:00:04 +02:00
Oliver Falk
2eb38445d7 Fix OpenTelemetry tests to use OTEL_EXPORT_ENABLED
- Update test_setup_tracing_with_otlp to use OTEL_EXPORT_ENABLED instead of OTEL_ENABLED
- Update test_setup_metrics_with_prometheus_and_otlp to use OTEL_EXPORT_ENABLED instead of OTEL_ENABLED
- These tests now correctly test the export-enabled behavior by setting the export flag
2025-10-17 10:38:07 +02:00
Oliver Falk
4d86a11728 Simplify OpenTelemetry approach - always enable instrumentation
- Always enable OpenTelemetry instrumentation, use OTEL_EXPORT_ENABLED for data export control
- Remove conditional checks from middleware, metrics, and decorators
- Simplify CI configuration to use single test job instead of parallel jobs
- Update tests to remove conditional logic and mocking of is_enabled()
- Add comprehensive environment variable documentation to README
- Update config.py to always add OpenTelemetry middleware
- Replace ENABLE_OPENTELEMETRY/OTEL_ENABLED with OTEL_EXPORT_ENABLED

This approach is much simpler and eliminates the complexity of conditional
OpenTelemetry loading while still allowing control over data export.
2025-10-17 09:54:59 +02:00
Oliver Falk
b4e10e3ec5 Increase verbosity 2025-10-16 20:18:12 +02:00
Oliver Falk
69044a12e6 Remove redundant semgrep job from GitLab CI
- The standalone semgrep job was scanning /tmp/app instead of project files
- It produced no useful output and wasted CI resources
- semgrep-sast from SAST template already provides comprehensive security scanning
- This eliminates redundancy and reduces pipeline time by ~31 seconds
2025-10-16 20:10:42 +02:00
Oliver Falk
c926869e6d Convert deployment testing from shell script to Python
- Replace scripts/test_deployment.sh with scripts/check_deployment.py
- Add command-line parameters: --dev, --prod, --endpoint, --max-retries, --retry-delay
- Improve maintainability with Python instead of shell script
- Add proper SSL certificate handling with fallback to unverified SSL
- Add binary content support for image downloads
- Add comprehensive error handling and colored output
- Add type hints and better documentation

- Update GitLab CI deployment verification jobs to use new Python script
- Replace ~140 lines of inline shell script with simple Python calls
- Change CI images from alpine:latest to python:3.11-alpine
- Add Pillow dependency for image processing in CI
- Maintain same retry logic and timing as before

- Remove obsolete test runner scripts that were deleted earlier
- All deployment tests now use consistent Python-based approach
2025-10-16 20:05:59 +02:00
Oliver Falk
b95bf287bf Fix Python path issue in coverage script
- Add current directory to Python path before calling django.setup()
- This fixes ModuleNotFoundError: No module named 'ivatar'
- The script now properly finds the ivatar module when running tests
- Coverage should now work correctly with Django test runner
2025-10-16 19:59:25 +02:00
Oliver Falk
e6596b925a Fix coverage measurement in CI
- Replace subprocess call with direct Django test runner invocation
- This allows coverage tool to properly track test execution
- Use django.setup() and get_runner() to run tests directly
- Coverage should now show proper test coverage instead of 1%
2025-10-16 19:54:53 +02:00
Oliver Falk
9767ebf110 Skip OpenTelemetry disabled test in CI environment
- Skip test_opentelemetry_disabled_by_default when OTEL_ENABLED=true in CI
- This test is specifically about testing disabled behavior, which can't be properly tested
  when OpenTelemetry is enabled by CI configuration
- Use skipTest() to gracefully skip the test instead of failing
- This ensures the test passes in both OTel-enabled and OTel-disabled CI jobs
2025-10-16 19:50:11 +02:00
Oliver Falk
530dbdbb6c Fix OpenTelemetry disabled test global config reset
- Fix the global config reset by using module-level access instead of global statement
- Use ivatar.opentelemetry_config._ot_config = None to properly reset the singleton
- This ensures the test can properly test disabled behavior when environment is cleared
2025-10-16 19:44:38 +02:00
Oliver Falk
a3a2220d15 Fix OpenTelemetry disabled test for CI environment
- Fix test_opentelemetry_disabled_by_default to handle CI environment correctly
- When OTEL_ENABLED=true in CI, test that OpenTelemetry is actually enabled
- When testing default disabled behavior, reset global config singleton
- This ensures the test works in both OTel-enabled and OTel-disabled CI jobs
2025-10-16 19:36:51 +02:00
Oliver Falk
32c854a545 Remove all remaining pytest markers from OpenTelemetry tests
- Remove @pytest.mark.opentelemetry from IntegrationTest class (line 407)
- Remove @pytest.mark.no_opentelemetry from OpenTelemetryDisabledTest class (line 456)
- All pytest references have now been completely removed
- Tests will now work with Django's test runner in CI
2025-10-16 19:30:35 +02:00
Oliver Falk
1601e86ad8 Remove remaining pytest marker from AvatarMetricsTest
- Remove @pytest.mark.opentelemetry decorator that was causing ImportError
- All pytest markers now removed from OpenTelemetry tests
- Tests now compatible with Django test runner
2025-10-16 19:22:02 +02:00
Oliver Falk
a2affffdd1 Fix OpenTelemetry tests for CI environment
- Update tests that expect OpenTelemetry to be disabled by default
- Handle case where CI environment has OpenTelemetry enabled
- Remove pytest markers since we're using Django test runner
- Tests now work correctly in both OpenTelemetry-enabled and disabled environments
- Fixes 3 failing tests in CI pipeline
2025-10-16 19:18:16 +02:00
Oliver Falk
f0182c7d51 Fix Django test database naming conflict
- Remove explicit TEST.NAME configuration that was causing conflicts
- Let Django use its default test database naming convention
- Prevents 'database already exists' errors in CI
- Django will now create test databases with names like 'test_django_db_with_otel'
2025-10-16 19:10:37 +02:00
Oliver Falk
ff0543a0bb Fix CI database naming conflict for parallel jobs
- Use different database names for each parallel job:
  - test_without_opentelemetry: django_db_no_otel
  - test_with_opentelemetry_and_coverage: django_db_with_otel
- Prevents database conflicts when jobs run in parallel
- Each job gets its own isolated PostgreSQL database
2025-10-16 18:31:23 +02:00
Oliver Falk
eeeb8a4f3a Simplify test scripts and move run_tests_local.sh to scripts/
- Move run_tests_local.sh to scripts/ directory for consistency
- Remove explicit test module listing from all test scripts
- Let Django auto-discover all tests instead of maintaining explicit lists
- Update README.md to reference new script location
- Simplify scripts/run_tests_with_coverage.py to use auto-discovery
- Reduce maintenance burden by eliminating duplicate test module lists
2025-10-16 18:00:17 +02:00
Oliver Falk
6f205ccad9 Refactor test scripts into scripts/ directory and improve CI
- Move run_tests_no_ot.sh and run_tests_with_ot.sh to scripts/ directory
- Create scripts/run_tests_with_coverage.py for coverage measurement
- Update CI to use scripts from scripts/ directory
- Eliminate code duplication between shell scripts and CI configuration
- Use Python script with coverage run for proper coverage measurement
2025-10-16 17:58:33 +02:00
Oliver Falk
b5186f2081 Fix CI to run Django tests directly instead of shell scripts
- Replace shell script calls with direct Django test commands
- Include specific test modules for both OpenTelemetry enabled/disabled scenarios
- Fix coverage run command to work with Django test suite
2025-10-16 17:57:28 +02:00
Oliver Falk
37e24df9dc Remove pytest.ini as we now use Django test suite 2025-10-16 17:54:32 +02:00
Oliver Falk
19facb4bec Split CI testing into parallel jobs for OpenTelemetry
- test_without_opentelemetry: Run baseline tests without OpenTelemetry
- test_with_opentelemetry_and_coverage: Run comprehensive tests with OpenTelemetry enabled and measure coverage
- Both jobs run in parallel for faster CI execution
- Coverage is measured only on OpenTelemetry-enabled run to capture additional code paths
- Updated pages job dependency to use the new coverage job
2025-10-16 17:35:52 +02:00
Oliver Falk
a19fd6ffa2 Fix test scripts to use Django test suite instead of pytest
- Replace pytest with python3 manage.py test in both scripts
- Remove pytest.ini configuration file
- Maintain consistency with existing testing approach
- Include all test modules explicitly for better control
2025-10-16 17:28:21 +02:00
Oliver Falk
a98ab6bb4a Fix test scripts to use Django test suite instead of pytest
- Replace pytest with python3 manage.py test in both scripts
- Remove pytest.ini configuration file
- Maintain consistency with existing testing approach
- Include all test modules explicitly for better control
2025-10-16 17:27:21 +02:00
Oliver Falk
919f2ddf73 Merge branch 'opentelemetry' into 'devel'
Reduce version requirement for OT, for older Python installations (eg. 3.8)

See merge request oliver/ivatar!264
2025-10-16 15:39:28 +02:00
Oliver Falk
b747854ae5 Merge branch 'devel' into 'opentelemetry'
# Conflicts:
#   requirements.txt
2025-10-16 15:34:10 +02:00
Oliver Falk
847fda66f8 Fix OpenTelemetry package versions for Python 3.8 compatibility
- Change opentelemetry-exporter-prometheus from >=0.59b0 to >=0.54b0
- Reorder packages for better organization
- Fixes compatibility issue with older Python/Django versions on dev instance
2025-10-16 15:27:54 +02:00
Oliver Falk
cde3ce10bd Merge branch 'opentelemetry' into 'devel'
Add OpenTelemetry integration

See merge request oliver/ivatar!263
2025-10-16 15:22:55 +02:00
Oliver Falk
a9021fc0f7 Add OpenTelemetry integration 2025-10-16 15:22:54 +02:00
Oliver Falk
5ff79cf7ae Fix OpenTelemetry middleware and tests
- Add missing avatar_requests counter to AvatarMetrics class
- Fix middleware to get metrics instance lazily in __call__ method
- Add reset_avatar_metrics() function for testing
- Fix test_avatar_request_attributes to check both set_attributes and set_attribute calls
- Add http.request.duration span attribute to fix flake8 unused variable warning
- All 29 OpenTelemetry tests now passing
- All 117 non-OpenTelemetry tests still passing
2025-10-16 15:02:59 +02:00