fix: shader compilation error on Firefox with Nvidia driver

[lsdch]

Closes #9869

Background

As explained in #9869, a shader compilation error occurs on examples involving GPU aggregation (e.g. Hexagon layer) when running Firefox with open drivers for Nvidia. This does not occur when running Chromium on the same machine.

LLMs suggest the following explanation:

Firefox is translating WebGL2 (GLSL ES 3.00) → desktop GLSL 1.50. Chromium does this differently (via ANGLE), which is why Chromium works.

After investigating the issue, @chrisgervang pointed out that the error might stem from the usage of uintBitsToFloat which is not supported in GLSL 1.50. The initial attempt to fix it by replacing NaN generation using uintBitsToFloat to use 0.0 / 0.0 was not successful.

Turns out that luma.gl automatically injects a fallback for NaN in the shader, replacing it with uintBitsToFloat(0xFFFFFFFF) in the transpiled code. Or at least that's the interpretation I got from the LLM (upon which I heavily relied, since I am not familiar with GLSL).

The suggested fix for this issue is to avoid defining NaN values entirely in the shader to represent empty values. Instead, a sentinel value is defined to signal empty values.

Change List

• replace NaN definition in WebGLAggregationTransform shaders to use a large negative constant to represent empty values

Before / after screenshots :

---

Note

Medium Risk
Touches GPU aggregation shaders and changes how empty bins are represented/filtered, which could affect min/max domain calculations in edge cases (e.g., extremely negative values) across different drivers.

Overview
Fixes a Firefox/Nvidia shader compilation issue by removing NaN generation/usage from the GPU aggregation domain shaders in webgl-aggregation-transform.ts.

Empty bins are now encoded with a large negative EMPTY_BIN sentinel in the vertex shader and filtered in the fragment shader via a threshold check (with epsilon) instead of isnan(...) + discard, keeping the domain reduction path functional on drivers that choke on NaN-related bit ops.

^{Written by Cursor Bugbot for commit bfcd415. This will update automatically on new commits. Configure here.}

[chrisgervang]

Nice digging. This seems harmless, but I would like to also get input from the other maintainers closer to luma.gl. It may be the case that this fix should be pushed down further if luma is falling NaN back to a value unsupported by Firefox

[Pessimistress]

Yes the NaN value has caused a few cross-platform issues. This approach seems reasonable to me and hopefully it sticks around!

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

[cursor]

Sentinel value passes Number.isFinite check, corrupting scale computations

Medium Severity

The sentinel value EMPTY_BIN = -3e38 is a finite number, so Number.isFinite(-3e38) returns true. The scale utility functions in scale-utils.ts rely on Number.isFinite() to filter out empty bin values — applyScaleQuantile uses .filter(Number.isFinite) and applyScaleOrdinal checks Number.isFinite(x). Previously, NaN was correctly excluded by these filters. Now, -3e38 values from empty bins pass through and contaminate percentile threshold calculations and ordinal category mappings, skewing results when quantile or ordinal scale types are used.

Additional Locations (1)

• modules/aggregation-layers/src/common/aggregator/gpu-aggregator/webgl-aggregation-transform.ts#L162-L163

 

[cursor]

Epsilon 1e-5 is meaningless at magnitude 3e38

Low Severity

The comparison value3.x <= EMPTY_BIN + 1e-5 adds an epsilon of 1e-5 to -3e38, but at that magnitude, float32 precision is on the order of ~1e31. The addition (-3e38) + 1e-5 evaluates to exactly -3e38 in floating-point arithmetic, so the epsilon provides zero additional tolerance despite the comment claiming it accounts for "floating point precision issues." The check still works for exact sentinel matches, but the claimed safety margin is illusory.