[#2739] feat(server): Trigger flush when there are too many blocks in shuffle buffer

[xianjingfeng]

What changes were proposed in this pull request?

Trigger flush when there are too many blocks in shuffle buffer.

Why are the changes needed?

Fix: #2739
Prevent memory overflow

Does this PR introduce any user-facing change?

set rss.server.buffer.blockCount.capacity

How was this patch tested?

UT

[github-actions]

Test Results

 3 185 files  + 33   3 185 suites  +33   6h 51m 39s ⏱️ + 1m 38s
 1 247 tests +  1   1 244 ✅ ±  0   1 💤 ±0  1 ❌ ±0  1 🔥 +1 
15 804 runs  +193  15 787 ✅ +193  15 💤 ±0  1 ❌  - 1  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit f3f4b42. ± Comparison against base commit b80940d.

♻️ This comment has been updated with latest results.

[zuston]

Could you provide more context about this mechanism? My understanding is that we only need to enforce a limit on total memory usage. Even if there are many blocks, as long as each block is small, it shouldn’t pose a problem.

[xianjingfeng]

Could you provide more context about this mechanism? My understanding is that we only need to enforce a limit on total memory usage. Even if there are many blocks, as long as each block is small, it shouldn’t pose a problem.

If there are many small blocks in memory, the number of ShufflePartitionedBlock instances will be large, and the heap memory will not be able to hold them all.

[zuston]

Could you provide more context about this mechanism? My understanding is that we only need to enforce a limit on total memory usage. Even if there are many blocks, as long as each block is small, it shouldn’t pose a problem.

If there are many small blocks in memory, the number of ShufflePartitionedBlock instances will be large, and the heap memory will not be able to hold them all.

Got it.