Test bench

[Antonov548]

No description provided.

[Antonov548]

/bench

[Antonov548]

/bench

[github-actions]

Benchmarks

Triggered by /bench - Run: https://github.com/cybersemics/treecrdt/actions/runs/21714258505
Base: main (442df42) - Head: feat/test-bench (14debf9)
Compared entries: 67 - Improved: 24 - Regressed: 43

Impl	Storage	Workload	Base ops/s	Head ops/s	Δ ops/s	Base p50 ms	Head p50 ms	Δ p50	Status
core-inmem-crdt	memory	insert-move-100	120933.31	120574.68	-0.30%	1.65	1.66	+0.30%	✅
core-inmem-crdt	memory	insert-move-1000	14337.05	14192.51	-1.01%	139.50	140.92	+1.02%	✅
core-inmem-crdt	memory	insert-move-10000	951.27	954.75	+0.37%	21024.53	20947.88	-0.36%	✅
sqlite-node	file	insert-move-100	21857.73	21817.01	-0.19%	9.15	9.17	+0.19%	✅
sqlite-node	file	insert-move-1000	25454.13	25531.32	+0.30%	78.57	78.34	-0.30%	✅
sqlite-node	file	insert-move-10000	22970.56	22864.67	-0.46%	870.68	874.71	+0.46%	✅
sqlite-node	file	sync-all-100	3367.77	3354.44	-0.40%	29.69	29.81	+0.40%	✅
sqlite-node	file	sync-all-1000	13003.34	12943.46	-0.46%	76.90	77.26	+0.46%	✅
sqlite-node	file	sync-all-10000	12850.63	12578.05	-2.12%	778.17	795.04	+2.17%	✅
sqlite-node	file	sync-children-100	1848.68	1878.72	+1.63%	27.05	26.61	-1.60%	✅
sqlite-node	file	sync-children-1000	9736.06	9676.79	-0.61%	51.36	51.67	+0.61%	✅
sqlite-node	file	sync-children-10000	12778.12	13011.14	+1.82%	391.29	384.29	-1.79%	✅
sqlite-node	file	sync-one-missing-100	42.94	43.84	+2.09%	23.29	22.81	-2.05%	✅
sqlite-node	file	sync-one-missing-1000	36.56	37.11	+1.50%	27.35	26.95	-1.47%	✅
sqlite-node	file	sync-one-missing-10000	13.91	14.03	+0.85%	71.91	71.30	-0.84%	✅
sqlite-node	file	sync-root-children-fanout10-1110	84.26	84.47	+0.25%	23.74	23.68	-0.25%	✅
sqlite-node	memory	insert-move-100	27796.81	27855.42	+0.21%	7.20	7.18	-0.21%	✅
sqlite-node	memory	insert-move-1000	26515.69	26488.05	-0.10%	75.43	75.51	+0.10%	✅
sqlite-node	memory	insert-move-10000	23958.62	23837.80	-0.50%	834.77	839.00	+0.51%	✅
sqlite-node	memory	sync-all-100	3645.58	3227.61	-11.47%	27.43	30.98	+12.95%	🟠
sqlite-node	memory	sync-all-1000	14009.96	13805.42	-1.46%	71.38	72.44	+1.48%	✅
sqlite-node	memory	sync-all-10000	14497.46	13957.71	-3.72%	689.78	716.45	+3.87%	✅
sqlite-node	memory	sync-children-100	2156.93	1844.96	-14.46%	23.18	27.10	+16.91%	🟠
sqlite-node	memory	sync-children-1000	11482.15	10871.27	-5.32%	43.55	45.99	+5.62%	🟡
sqlite-node	memory	sync-children-10000	15405.30	14520.43	-5.74%	324.56	344.34	+6.09%	🟡
sqlite-node	memory	sync-one-missing-100	45.56	46.28	+1.58%	21.95	21.61	-1.55%	✅
sqlite-node	memory	sync-one-missing-1000	38.76	39.19	+1.11%	25.80	25.52	-1.10%	✅
sqlite-node	memory	sync-one-missing-10000	14.13	14.26	+0.91%	70.79	70.14	-0.91%	✅
sqlite-node	memory	sync-root-children-fanout10-1110	89.79	79.52	-11.44%	22.27	25.15	+12.92%	🟠
wa-sqlite	browser-memory	insert-chain-100	3305.79	3169.57	-4.12%	30.25	31.55	+4.30%	✅
wa-sqlite	browser-memory	insert-chain-1000	427.04	417.78	-2.17%	2341.70	2393.60	+2.22%	✅
wa-sqlite	browser-memory	insert-move-100	7490.64	6688.96	-10.70%	26.70	29.90	+11.99%	🟠
wa-sqlite	browser-memory	insert-move-1000	9708.74	8818.34	-9.17%	206.00	226.80	+10.10%	🟠
wa-sqlite	browser-memory	replay-log-100	3246.75	3144.65	-3.14%	30.80	31.80	+3.25%	✅
wa-sqlite	browser-memory	replay-log-1000	427.22	418.43	-2.06%	2340.70	2389.90	+2.10%	✅
wa-sqlite	browser-memory	sync-all-100	2265.01	2107.48	-6.95%	44.15	47.45	+7.47%	🟡
wa-sqlite	browser-memory	sync-all-1000	11806.38	11627.91	-1.51%	84.70	86.00	+1.53%	✅
wa-sqlite	browser-memory	sync-all-10000	15410.70	15020.65	-2.53%	648.90	665.75	+2.60%	✅
wa-sqlite	browser-memory	sync-children-100	1440.92	1414.43	-1.84%	34.70	35.35	+1.87%	✅
wa-sqlite	browser-memory	sync-children-1000	8510.64	8361.20	-1.76%	58.75	59.80	+1.79%	✅
wa-sqlite	browser-memory	sync-children-10000	14983.52	14463.41	-3.47%	333.70	345.70	+3.60%	✅
wa-sqlite	browser-memory	sync-one-missing-100	31.10	30.96	-0.46%	32.15	32.30	+0.47%	✅
wa-sqlite	browser-memory	sync-one-missing-1000	27.55	26.99	-2.02%	36.30	37.05	+2.07%	✅
wa-sqlite	browser-memory	sync-one-missing-10000	11.66	11.49	-1.38%	85.80	87.00	+1.40%	✅
wa-sqlite	browser-memory	sync-root-children-fanout10-1110	58.48	57.80	-1.16%	34.20	34.60	+1.17%	✅
wa-sqlite	browser-opfs-coop-sync	insert-chain-100	1523.23	1583.53	+3.96%	65.65	63.15	-3.81%	✅
wa-sqlite	browser-opfs-coop-sync	insert-chain-1000	409.53	408.85	-0.17%	2441.80	2445.90	+0.17%	✅
wa-sqlite	browser-opfs-coop-sync	insert-move-100	3257.33	3327.79	+2.16%	61.40	60.10	-2.12%	✅
wa-sqlite	browser-opfs-coop-sync	insert-move-1000	5464.48	5929.44	+8.51%	366.00	337.30	-7.84%	🟡
wa-sqlite	browser-opfs-coop-sync	replay-log-100	1603.85	1592.36	-0.72%	62.35	62.80	+0.72%	✅
wa-sqlite	browser-opfs-coop-sync	replay-log-1000	409.94	410.91	+0.24%	2439.40	2433.60	-0.24%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-100	1289.49	1302.93	+1.04%	77.55	76.75	-1.03%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-1000	2573.01	2582.98	+0.39%	388.65	387.15	-0.39%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-10000	2851.32	2853.56	+0.08%	3507.15	3504.40	-0.08%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-100	904.16	901.71	-0.27%	55.30	55.45	+0.27%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-1000	2172.97	2160.29	-0.58%	230.10	231.45	+0.59%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-10000	2655.69	2647.04	-0.33%	1882.75	1888.90	+0.33%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-100	28.61	28.09	-1.83%	34.95	35.60	+1.86%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-1000	23.75	22.73	-4.32%	42.10	44.00	+4.51%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-10000	4.62	4.53	-1.99%	216.50	220.90	+2.03%	✅
wa-sqlite	browser-opfs-coop-sync	sync-root-children-fanout10-1110	32.49	30.26	-6.88%	61.55	66.10	+7.39%	🟡
wasm	memory	insert-chain-100	68254.03	68578.69	+0.48%	1.47	1.46	-0.47%	✅
wasm	memory	insert-chain-1000	31955.75	29999.80	-6.12%	31.29	33.33	+6.52%	🟡
wasm	memory	insert-chain-10000	4126.43	4127.64	+0.03%	2423.40	2422.69	-0.03%	✅
wasm	memory	insert-move-100	31496.57	47625.42	+51.21%	6.35	4.20	-33.87%	🔴
wasm	memory	insert-move-1000	31278.91	31393.62	+0.37%	63.94	63.71	-0.37%	✅
wasm	memory	insert-move-10000	4213.66	4225.20	+0.27%	4746.47	4733.51	-0.27%	✅

Status: ✅ <5% change | 🟡 5–10% | 🟠 10–30% | 🔴 >30%

Artifacts:

• base summary: bench-base/benchmarks/summary.json
• head summary: bench-head/benchmarks/summary.json

[marcus-pousette]

nice!

[marcus-pousette]

"

wa-sqlite	browser-memory	insert-move-10	632.91	2247.19	+255.06%

"

the deltas seem very high even though we did not do any chanegs here, I wonder if we are running the bench for long enough time for some cases.

[marcus-pousette]

I want to expect smaller difference between main and this branch somehow

[Antonov548]

I want to expect smaller difference between main and this branch somehow

I also was surprised by results. I was trying to figure out from where this difference comes from. But so far I can't explain why it happens.

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

/bench

[Antonov548]

reference:

Impl	Storage	Workload	Base ops/s	Head ops/s	Δ ops/s	Base p50 ms	Head p50 ms	Δ p50	Status
core-inmem-crdt	memory	insert-move-100	119931.97	120969.16	+0.86%	1.67	1.65	-0.86%	✅
core-inmem-crdt	memory	insert-move-1000	14253.56	14092.57	-1.13%	140.32	141.92	+1.14%	✅
core-inmem-crdt	memory	insert-move-10000	843.07	858.18	+1.79%	23722.94	23305.19	-1.76%	✅
sqlite-node	file	insert-move-100	21255.47	21114.17	-0.66%	9.41	9.47	+0.67%	✅
sqlite-node	file	insert-move-1000	24509.73	25282.32	+3.15%	81.60	79.11	-3.06%	✅
sqlite-node	file	insert-move-10000	22461.11	22616.83	+0.69%	890.43	884.30	-0.69%	✅
sqlite-node	file	sync-all-100	3252.52	3217.09	-1.09%	30.75	31.08	+1.10%	✅
sqlite-node	file	sync-all-1000	12672.45	12465.64	-1.63%	78.91	80.22	+1.66%	✅
sqlite-node	file	sync-all-10000	12640.14	12381.23	-2.05%	791.13	807.67	+2.09%	✅
sqlite-node	file	sync-children-100	1735.78	1722.85	-0.74%	28.81	29.02	+0.75%	✅
sqlite-node	file	sync-children-1000	9417.58	9209.82	-2.21%	53.09	54.29	+2.26%	✅
sqlite-node	file	sync-children-10000	13178.23	12150.86	-7.80%	379.41	411.49	+8.46%	🟡
sqlite-node	file	sync-one-missing-100	41.98	41.21	-1.85%	23.82	24.27	+1.88%	✅
sqlite-node	file	sync-one-missing-1000	36.52	36.01	-1.39%	27.38	27.77	+1.41%	✅
sqlite-node	file	sync-one-missing-10000	13.91	13.43	-3.47%	71.87	74.45	+3.59%	✅
sqlite-node	file	sync-root-children-fanout10-1110	83.57	81.45	-2.53%	23.93	24.56	+2.60%	✅
sqlite-node	memory	insert-move-100	27911.32	27306.27	-2.17%	7.17	7.32	+2.22%	✅
sqlite-node	memory	insert-move-1000	26318.90	26448.58	+0.49%	75.99	75.62	-0.49%	✅
sqlite-node	memory	insert-move-10000	23276.84	23416.61	+0.60%	859.22	854.09	-0.60%	✅
sqlite-node	memory	sync-all-100	3638.12	3548.28	-2.47%	27.49	28.18	+2.53%	✅
sqlite-node	memory	sync-all-1000	14501.73	14675.90	+1.20%	68.96	68.14	-1.19%	✅
sqlite-node	memory	sync-all-10000	14175.82	13991.14	-1.30%	705.43	714.74	+1.32%	✅
sqlite-node	memory	sync-children-100	2126.24	2076.51	-2.34%	23.52	24.08	+2.39%	✅
sqlite-node	memory	sync-children-1000	11337.83	11060.06	-2.45%	44.10	45.21	+2.51%	✅
sqlite-node	memory	sync-children-10000	15177.87	14824.34	-2.33%	329.43	337.28	+2.38%	✅
sqlite-node	memory	sync-one-missing-100	45.59	44.71	-1.93%	21.94	22.37	+1.97%	✅
sqlite-node	memory	sync-one-missing-1000	38.58	36.50	-5.38%	25.92	27.40	+5.69%	🟡
sqlite-node	memory	sync-one-missing-10000	14.12	13.92	-1.37%	70.84	71.83	+1.39%	✅
sqlite-node	memory	sync-root-children-fanout10-1110	91.71	88.18	-3.85%	21.81	22.68	+4.00%	✅
wa-sqlite	browser-memory	insert-chain-100	3262.64	3412.97	+4.61%	30.65	29.30	-4.40%	✅
wa-sqlite	browser-memory	insert-chain-1000	430.89	429.68	-0.28%	2320.80	2327.30	+0.28%	✅
wa-sqlite	browser-memory	insert-move-100	6756.76	7220.22	+6.86%	29.60	27.70	-6.42%	🟡
wa-sqlite	browser-memory	insert-move-1000	9174.31	9420.63	+2.68%	218.00	212.30	-2.61%	✅
wa-sqlite	browser-memory	replay-log-100	3311.26	3407.16	+2.90%	30.20	29.35	-2.81%	✅
wa-sqlite	browser-memory	replay-log-1000	432.00	429.00	-0.69%	2314.80	2331.00	+0.70%	✅
wa-sqlite	browser-memory	sync-all-100	2298.85	2143.62	-6.75%	43.50	46.65	+7.24%	🟡
wa-sqlite	browser-memory	sync-all-1000	11834.32	11267.61	-4.79%	84.50	88.75	+5.03%	🟡
wa-sqlite	browser-memory	sync-all-10000	14833.49	14892.03	+0.39%	674.15	671.50	-0.39%	✅
wa-sqlite	browser-memory	sync-children-100	1375.52	1381.22	+0.41%	36.35	36.20	-0.41%	✅
wa-sqlite	browser-memory	sync-children-1000	8396.31	8163.27	-2.78%	59.55	61.25	+2.85%	✅
wa-sqlite	browser-memory	sync-children-10000	14526.44	14257.20	-1.85%	344.20	350.70	+1.89%	✅
wa-sqlite	browser-memory	sync-one-missing-100	30.49	29.76	-2.38%	32.80	33.60	+2.44%	✅
wa-sqlite	browser-memory	sync-one-missing-1000	27.17	26.01	-4.29%	36.80	38.45	+4.48%	✅
wa-sqlite	browser-memory	sync-one-missing-10000	11.41	11.47	+0.57%	87.65	87.15	-0.57%	✅
wa-sqlite	browser-memory	sync-root-children-fanout10-1110	56.26	51.48	-8.49%	35.55	38.85	+9.28%	🟡
wa-sqlite	browser-opfs-coop-sync	insert-chain-100	1471.67	1573.56	+6.92%	67.95	63.55	-6.48%	🟡
wa-sqlite	browser-opfs-coop-sync	insert-chain-1000	405.28	415.02	+2.40%	2467.40	2409.50	-2.35%	✅
wa-sqlite	browser-opfs-coop-sync	insert-move-100	3129.89	3189.79	+1.91%	63.90	62.70	-1.88%	✅
wa-sqlite	browser-opfs-coop-sync	insert-move-1000	5246.59	5676.98	+8.20%	381.20	352.30	-7.58%	🟡
wa-sqlite	browser-opfs-coop-sync	replay-log-100	1564.95	1566.17	+0.08%	63.90	63.85	-0.08%	✅
wa-sqlite	browser-opfs-coop-sync	replay-log-1000	407.48	418.62	+2.73%	2454.10	2388.80	-2.66%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-100	1290.32	1286.17	-0.32%	77.50	77.75	+0.32%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-1000	2527.81	2540.01	+0.48%	395.60	393.70	-0.48%	✅
wa-sqlite	browser-opfs-coop-sync	sync-all-10000	2805.25	2786.41	-0.67%	3564.75	3588.85	+0.68%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-100	887.31	881.83	-0.62%	56.35	56.70	+0.62%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-1000	2142.70	2103.93	-1.81%	233.35	237.65	+1.84%	✅
wa-sqlite	browser-opfs-coop-sync	sync-children-10000	2603.76	2600.64	-0.12%	1920.30	1922.60	+0.12%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-100	27.51	27.51	+0.00%	36.35	36.35	-0.00%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-1000	22.75	23.26	+2.21%	43.95	43.00	-2.16%	✅
wa-sqlite	browser-opfs-coop-sync	sync-one-missing-10000	4.39	4.42	+0.71%	227.90	226.30	-0.70%	✅
wa-sqlite	browser-opfs-coop-sync	sync-root-children-fanout10-1110	30.82	30.86	+0.15%	64.90	64.80	-0.15%	✅
wasm	memory	insert-chain-100	66719.93	66953.09	+0.35%	1.50	1.49	-0.35%	✅
wasm	memory	insert-chain-1000	32483.96	30869.53	-4.97%	30.78	32.39	+5.23%	🟡
wasm	memory	insert-chain-10000	4078.03	4171.67	+2.30%	2452.17	2397.12	-2.24%	✅
wasm	memory	insert-move-100	38870.75	30706.63	-21.00%	5.15	6.51	+26.59%	🟠
wasm	memory	insert-move-1000	30845.21	30518.12	-1.06%	64.84	65.53	+1.07%	✅
wasm	memory	insert-move-10000	4164.37	4190.38	+0.62%	4802.65	4772.84	-0.62%	✅

Status: ✅ <5% change | 🟡 5–10% | 🟠 10–30% | 🔴 >30%

[Antonov548]

/bench