Merged
Conversation
- Add onMetrics callback option to StartOptions - Track rolling averages for total, segmentation, and GPU time - Report metrics for both worker and main pipelines - Display metrics in demo UI with utilization and tier info
- Fix background source handling for canvas2d pipeline using optional chaining - Add metrics tracking for canvas2d rendering time - Reduce blurDownsampleScale from 0.5 to 0.25 for tiers A and B - Fix mask ImageData normalization to properly handle alpha channel
- Add computeBlurRadius() function for dynamic blur radius calculation - Blur radius now scales with blurStrength (0-1) instead of fixed per tier - Supports tier scaling and downsampling compensation - Clamps to MAX_SHADER_RADIUS (16) - Increase blur shader MAX_RADIUS from 8 to 16 pixels - Update gaussianBlurH.frag and gaussianBlurV.frag - Allows for stronger blur effects - Improve blur quality for tier A - Increase blurDownsampleScale from 0.25 to 0.5 (better quality) - Increase segmentationHeight from 144 to 256 (better mask quality) - Enhance quality controller stability - Increase hysteresis from 30 to 60 frames - Add downgrade confirmation (10 consecutive over-budget frames) - Add warm-up period (90 samples) before allowing downgrades - Implement maxTier cap to prevent re-upgrade to tier A after issues - Adjust thresholds: upgrade 50% (was 60%), downgrade 100% (was 85%) - Add debug logging in non-production mode - Make segmenter initialization non-blocking in worker - Prevent concurrent initialization attempts - Use promise pattern to avoid blocking frame processing - Add segmenterInitPromise state tracking - Ensure blur consistency across pipelines - Canvas2DPipeline now uses computeBlurRadius() like WebGL - Both pipelines use same calculation logic - Update demo defaults (virtual mode, auto quality) Fixes weak blur visibility issue where blur was barely noticeable even at maximum settings. Blur now scales smoothly with blurStrength parameter.
- Implement debt-based downgrade system - Accumulates over-budget debt instead of requiring consecutive frames - Debt recovery when performance improves - More stable and predictable tier transitions - Add EWMA (Exponentially Weighted Moving Average) smoothing - Smoother performance measurements - Reduces noise in tier decision making - Tracks separate EWMA for total, segmentation, and GPU time - Add severe downgrade detection - Fast downgrade when severely over budget (2x threshold) - Separate confirmation and warmup requirements - Prevents severe performance degradation - Implement upgrade failure tracking - Tracks failed upgrade attempts - Caps upgrades after repeated failures - Prevents oscillation between tiers - Refactor to ring buffer for sample storage - More efficient O(1) sample insertion - Maintains running totals for fast averaging - Better memory usage patterns - Extract modules for better organization - quality/tuning.ts: All tuning constants in one place - quality/samples.ts: Performance sample types - Better separation of concerns - Enhance metrics system - MetricsWindow interface with ring buffer - createMetricsWindow and resetMetricsWindow helpers - More efficient sample management - Improve quality resolution - Better handling of passthrough mode - resolveQualityTierForEffectMode for mode-aware resolution - effectModeToProcessingMode helper function - Enhance pipeline initialization - Conditional segmenter initialization (skip for tier D/passthrough) - Better error handling and state management - Use new metrics window API - Add comprehensive documentation - JSDoc comments throughout quality modules - Better parameter descriptions - Clearer function documentation - Update tests for new behavior - Tests for debt-based system - Tests for EWMA smoothing - Tests for severe downgrade detection This refactoring significantly improves the stability and predictability of the quality controller while maintaining backward compatibility.
- Document current status as production-ready code in real-world testing phase - Explain limited release to Edge and Internal builds only - Highlight areas requiring validation: capability detection, quality controller, performance - Note known areas for future improvement: segmentation and overlay quality
- Move BackgroundEffects directory from src/script/repositories/media/ to apps/webapp/src/script/repositories/media/ - Update component imports and references - Update webpack configuration - Remove old BackgroundEffects location
- Remove VideoBackgroundBlur.ts (replaced by BackgroundEffects system) - Add mediapipe model to .gitignore - Update config files for BackgroundEffects - Update GroupVideoGrid and VideoControls components - Enhance CallingRepository and Participant for background effects - Update BackgroundEffects quality definitions and capability policy
- Add comprehensive pipeline implementations (Canvas2D, MainWebGL, Passthrough, WorkerWebGL) - Enhance BackgroundEffectsController with additional functionality - Update pipeline types and interfaces - Add background images assets - Remove selfie_segmenter.tflite model file - Update i18n strings for background effects - Improve context loss handling in worker - Update README with pipeline documentation
- Tier B: increase blurDownsampleScale (0.25 -> 0.5), bilateralRadius (4 -> 5), bilateralSpatialSigma (3.0 -> 3.5) - Tier C: increase segmentationCadence (2 -> 3), decrease maskRefineScale (0.5 -> 0.4) - Update README to reflect quality tier adjustments
- Add comprehensive WebGL renderer implementation (BackgroundRenderer, RenderPasses, ShaderPrograms, WebGLResources) - Enhance mask post-processing and segmentation capabilities - Improve worker frame processing and context loss handling - Add shared utilities for mask and timestamp management - Enhance FrameSource with additional functionality - Update Canvas2D and WorkerWebGL pipelines - Remove old background image assets - Update i18n strings and README documentation
…uperhigh quality model
…ove effect handling to global scope
Background effects are no longer attached to the self participant.
Effect management is now handled globally and the BackgroundEffectsController
is implemented as a singleton to centralize lifecycle and state handling.
… persisted feature toggle
…er the app restarts
zskhan
requested review from
V-Gira,
arjita-mitra,
e-maad,
nkaramarko,
otto-the-bot,
screendriver,
thisisamir98 and
zskhan
as code owners
thisisamir98
reviewed
Apr 1, 2026
thisisamir98
reviewed
Apr 1, 2026
thisisamir98
reviewed
Apr 1, 2026
...ebapp/src/script/repositories/media/backgroundEffects/effects/backgroundEffectsController.ts
Show resolved
Hide resolved
thisisamir98
reviewed
Apr 1, 2026
.../src/script/repositories/media/backgroundEffects/effects/backgroundEffectsController.test.ts
Show resolved
Hide resolved
thisisamir98
reviewed
Apr 1, 2026
.../src/script/repositories/media/backgroundEffects/effects/backgroundEffectsController.test.ts
Show resolved
Hide resolved
thisisamir98
reviewed
Apr 1, 2026
thisisamir98
approved these changes
Apr 1, 2026
thisisamir98
reviewed
Apr 1, 2026
apps/webapp/src/script/repositories/media/backgroundEffects/effects/frameSource.ts
Show resolved
Hide resolved
thisisamir98
reviewed
Apr 1, 2026
apps/webapp/src/script/repositories/media/backgroundEffects/effects/videoSource.test.ts
Show resolved
Hide resolved
|
zskhan
approved these changes
Apr 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Security Checklist (required)
Accessibility (required)
Standards Acknowledgement (required)
Screenshots or demo (if the user interface changed)
Notes for reviewers