Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Tree CRDT workspace targeting SQLite/wa-sqlite + WASM bindings with a shared Typ
- `packages/treecrdt-sqlite-node`: TreeCRDT bundled for Node.js use
- `packages/treecrdt-wa-sqlite`: TreeCRDT bunlded for browser use
- `packages/treecrdt-benchmark`: Benchmark utilities
- `packages/discovery`: bootstrap contract for resolving docs to attachment plans
- `packages/sync/protocol`: sync protocol/runtime core
- `packages/sync/material/sqlite`: SQLite-backed sync adapters and proof-material stores
- `packages/sync/material/postgres`: Postgres-backed sync proof-material stores
Expand Down
3 changes: 3 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ flowchart TD
%% TypeScript packages (pnpm workspace)
subgraph TS["TypeScript packages (pnpm workspace)"]
iface["@treecrdt/interface"]
discovery["@treecrdt/discovery"]
discovery_server["@treecrdt/discovery-server-node"]
sync_core["@treecrdt/sync"]
sync_sqlite["@treecrdt/sync-sqlite"]
sync_postgres["@treecrdt/sync-postgres"]
Expand All @@ -49,6 +51,7 @@ flowchart TD

%% Runtime dependencies
sync_core --> iface
discovery_server --> discovery
sync_core --> riblt_pkg
sync_sqlite --> sync_core
sync_sqlite --> iface
Expand Down
31 changes: 31 additions & 0 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ pnpm benchmark:sync:direct
pnpm benchmark:sync:local
pnpm benchmark:sync:prime
pnpm benchmark:sync:remote
pnpm benchmark:sync:bootstrap
pnpm benchmark:web
pnpm benchmark:wasm
pnpm benchmark:postgres
Expand All @@ -36,6 +37,7 @@ pnpm benchmark:postgres
- First view on a new device, with payloads: `benchmark:sync:*` with `sync-balanced-children-payloads-cold-start`
- Re-sync the same subtree on a restarted client that already has that scope locally: `benchmark:sync:*` with `sync-balanced-children-resync` or `sync-balanced-children-payloads-resync`
- Single end-to-end time-to-first-visible-page number: `benchmark:sync:*` with the same balanced workloads plus `--first-view`
- One-time bootstrap/discovery tax before opening the regional websocket: `benchmark:sync:bootstrap`
- Local render cost after the data is already present: `benchmark:sqlite-node:note-paths -- --benches=read-children-payloads`
- Local mutation cost inside a large existing tree: `benchmark:sqlite-node:note-paths -- --benches=insert-into-large-tree`
- Protocol/storage baselines and worst-case stress: `sync-one-missing`, `sync-all`, `sync-children*`, `sync-root-children-fanout10`
Expand Down Expand Up @@ -136,6 +138,8 @@ pnpm benchmark:sync:remote -- \
--server-fixture-cache=reuse
```

For remote targets, `prime` now records the exact fixture doc ID locally under `tmp/sqlite-node-sync-bench/server-fixtures/`. That means a fresh endpoint can be primed once with `--server-fixture-cache=rebuild`, and later `--server-fixture-cache=reuse` runs on the same machine can reopen that exact remote fixture doc instead of relying on historical deterministic fixture residue.

By default, the local sync target runs the Postgres sync server in a spawned child process so local and remote measurements are closer to each other. When you add `--profile-backend`, the local target intentionally switches to the in-process server so per-backend timings are visible inside the benchmark process.

Local server benchmarks now seed the Postgres backend directly before the timer starts. That keeps the measured path honest, because the actual sync to the client still goes through the real websocket server, while avoiding huge protocol-seed setup costs that are not part of the benchmark question.
Expand Down Expand Up @@ -213,6 +217,33 @@ pnpm benchmark:sync:remote -- \
--max-ops-per-batch=500
```

### Bootstrap / Resolve Bench

Use `benchmark:sync:bootstrap` when you want to isolate the one-time discovery
layer from the steady-state sync path.

The benchmark target can be a standalone bootstrap server such as
`@treecrdt/discovery-server-node`, not just a colocated sync-server route.

It measures:

- `resolveSamplesMs`: `GET /resolve-doc?docId=...`
- `connectSamplesMs`: first websocket open after resolve
- `totalSamplesMs`: resolve + first websocket open
- `cachedConnectSamplesMs`: direct websocket reconnect using the already resolved attachment

```sh
TREECRDT_DISCOVERY_URL=https://bootstrap-host \
pnpm benchmark:sync:bootstrap -- \
--iterations=5
```

This is the benchmark to use when you want to answer:

- how expensive the bootstrap lookup is on cold open
- how much faster cached reconnects are
- whether discovery is staying off the steady-state hot path

### Backend Call Profiling

Add `--profile-backend` when you want per-backend timings for:
Expand Down
23 changes: 22 additions & 1 deletion examples/playground/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
A small, self-contained demo that exercises the `@treecrdt/wa-sqlite` adapter inside a Vite + React + Tailwind UI. It runs the TreeCRDT SQLite extension in wa-sqlite and lets you insert, move, and delete nodes in an expandable tree while watching the underlying operation log.

## Features

- Insert children under any node, reorder siblings (up/down), move nodes back to the root, or delete them (root is protected).
- Collapsible tree with per-node controls and a composer form to target any parent.
- Live CRDT operation log with lamport/counter metadata.
Expand All @@ -11,6 +12,7 @@ A small, self-contained demo that exercises the `@treecrdt/wa-sqlite` adapter in
- Optional auth/ACL demo (COSE_Sign1 + CWT subtree capabilities) with invite links, per-op signatures, and a pending-op inspector.

## Running locally

```bash
pnpm install --filter @treecrdt/playground
pnpm -C examples/playground dev
Expand All @@ -27,16 +29,34 @@ pnpm build
pnpm sync-server:postgres:db:start
# Start the TreeCRDT sync server on ws://localhost:8787 using that Postgres DB.
pnpm sync-server:postgres:local
# Start the standalone bootstrap server on http://localhost:8788.
pnpm discovery-server:local
# Start the playground UI.
pnpm -C examples/playground dev
```

Then in the playground:

- Open the `Connections` panel
- Paste `ws://localhost:8787` into `Remote sync server`
- Paste `http://localhost:8788` into `Remote sync / bootstrap`
- Leave mode as `Hybrid`, or switch to `Remote server` if you want to disable local tab sync

## Bootstrap endpoint

If you want to test against a bootstrap endpoint instead of entering the
websocket sync server directly:

- Open the `Connections` panel
- Paste the HTTPS bootstrap URL you want to test
- Use `Hybrid` for browser-local tabs plus remote sync, or `Remote server` for remote-only behavior

The playground will call `/resolve-doc` once, cache the returned websocket
attachment, and then connect directly to the resolved `wss://.../sync`
endpoint.

If you want to skip bootstrap entirely, you can still paste a direct websocket
endpoint such as `ws://localhost:8787`.

`pnpm sync-server:postgres:db:start` starts a disposable local Postgres at:

```bash
Expand Down Expand Up @@ -83,6 +103,7 @@ pnpm --filter @treecrdt/wa-sqlite-vendor rebuild
The example does not depend on the npm `wa-sqlite` package; it consumes the repo's git submodule build directly via the copy step above.

## Building / deploying to GitHub Pages

```bash
pnpm -C examples/playground build # outputs to dist/
pnpm -C examples/playground deploy # pushes dist/ via gh-pages
Expand Down
3 changes: 2 additions & 1 deletion examples/playground/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,14 @@
"typecheck": "tsc -p tsconfig.json --noEmit",
"build": "vite build",
"preview": "vite preview",
"test:e2e": "pnpm -C ../../packages/sync/protocol run build && pnpm -C ../../packages/sync/server/core run build && pnpm -C ../../packages/treecrdt-auth run build && pnpm -C ../../packages/treecrdt-wa-sqlite-vendor run build && pnpm -C ../../packages/treecrdt-wa-sqlite run build:ts && playwright test",
"test:e2e": "pnpm -C ../../packages/discovery run build && pnpm -C ../../packages/sync/protocol run build && pnpm -C ../../packages/sync/server/core run build && pnpm -C ../../packages/treecrdt-auth run build && pnpm -C ../../packages/treecrdt-wa-sqlite-vendor run build && pnpm -C ../../packages/treecrdt-wa-sqlite run build:ts && playwright test",
"deploy": "pnpm run build && gh-pages -d dist"
},
"dependencies": {
"@tanstack/react-virtual": "^3.13.13",
"@treecrdt/auth": "workspace:*",
"@treecrdt/crypto": "workspace:*",
"@treecrdt/discovery": "workspace:*",
"@treecrdt/interface": "workspace:*",
"@treecrdt/sync": "workspace:*",
"@treecrdt/sync-sqlite": "workspace:*",
Expand Down
119 changes: 68 additions & 51 deletions examples/playground/src/playground/components/PeersPanel.tsx
Original file line number Diff line number Diff line change
@@ -1,43 +1,43 @@
import React from "react";
import { MdCheckCircle, MdCloudOff, MdCloudQueue, MdErrorOutline, MdSync } from "react-icons/md";
import React from 'react';
import { MdCheckCircle, MdCloudOff, MdCloudQueue, MdErrorOutline, MdSync } from 'react-icons/md';

import type { PeerInfo, RemoteSyncStatus, SyncTransportMode } from "../types";
import type { PeerInfo, RemoteSyncStatus, SyncTransportMode } from '../types';

function formatPeerId(id: string): string {
if (id.startsWith("remote:")) return `remote(${id.slice("remote:".length)})`;
if (id.startsWith('remote:')) return `remote(${id.slice('remote:'.length)})`;
return id.length > 18 ? `${id.slice(0, 8)}…${id.slice(-6)}` : id;
}

function transportModeButtonClass(active: boolean): string {
return active
? "border-accent bg-accent/15 text-white"
: "border-slate-700 bg-slate-900/70 text-slate-300 hover:border-slate-500 hover:text-white";
? 'border-accent bg-accent/15 text-white'
: 'border-slate-700 bg-slate-900/70 text-slate-300 hover:border-slate-500 hover:text-white';
}

function remoteStatusTone(status: RemoteSyncStatus): string {
switch (status.state) {
case "connected":
return "border-emerald-500/40 bg-emerald-500/10 text-emerald-100";
case "connecting":
return "border-sky-500/40 bg-sky-500/10 text-sky-100";
case "disabled":
return "border-slate-700 bg-slate-900/70 text-slate-400";
case "missing_url":
return "border-amber-500/30 bg-amber-500/10 text-amber-100";
case "invalid":
case "error":
return "border-rose-500/40 bg-rose-500/10 text-rose-100";
case 'connected':
return 'border-emerald-500/40 bg-emerald-500/10 text-emerald-100';
case 'connecting':
return 'border-sky-500/40 bg-sky-500/10 text-sky-100';
case 'disabled':
return 'border-slate-700 bg-slate-900/70 text-slate-400';
case 'missing_url':
return 'border-amber-500/30 bg-amber-500/10 text-amber-100';
case 'invalid':
case 'error':
return 'border-rose-500/40 bg-rose-500/10 text-rose-100';
}
}

function RemoteStatusIcon({ status }: { status: RemoteSyncStatus }) {
if (status.state === "connected") {
if (status.state === 'connected') {
return <MdCheckCircle className="text-[14px]" />;
}
if (status.state === "connecting") {
if (status.state === 'connecting') {
return <MdSync className="text-[14px]" />;
}
if (status.state === "disabled") {
if (status.state === 'disabled') {
return <MdCloudOff className="text-[14px]" />;
}
return <MdErrorOutline className="text-[14px]" />;
Expand All @@ -62,7 +62,7 @@ export function PeersPanel({
remoteSyncStatus: RemoteSyncStatus;
peers: PeerInfo[];
}) {
const requiresRemoteUrl = syncTransportMode !== "local";
const requiresRemoteUrl = syncTransportMode !== 'local';
const hasRemoteUrl = syncServerUrl.trim().length > 0;

return (
Expand All @@ -72,73 +72,86 @@ export function PeersPanel({
>
<div className="flex flex-wrap items-start justify-between gap-3">
<div>
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">Connections</div>
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">
Connections
</div>
<div className="mt-1 max-w-xl text-[11px] text-slate-400">
Choose how this tab syncs. Local tabs use `BroadcastChannel`. Remote server uses a websocket sync endpoint.
Choose how this tab syncs. Local tabs use `BroadcastChannel`. Remote transport can use
either a direct websocket sync endpoint or a separate HTTP bootstrap endpoint.
</div>
</div>
<button
className={`flex h-8 items-center gap-2 rounded-lg border px-3 text-[11px] font-semibold transition ${
online
? "border-slate-700 bg-slate-800/70 text-slate-200 hover:border-accent hover:text-white"
: "border-amber-500/60 bg-amber-500/10 text-amber-100 hover:border-amber-400"
? 'border-slate-700 bg-slate-800/70 text-slate-200 hover:border-accent hover:text-white'
: 'border-amber-500/60 bg-amber-500/10 text-amber-100 hover:border-amber-400'
}`}
onClick={() => setOnline((v) => !v)}
type="button"
title={online ? "Pause sync activity" : "Resume sync activity"}
title={online ? 'Pause sync activity' : 'Resume sync activity'}
>
{online ? <MdCloudQueue className="text-[16px]" /> : <MdCloudOff className="text-[16px]" />}
<span>{online ? "Sync enabled" : "Sync paused"}</span>
{online ? (
<MdCloudQueue className="text-[16px]" />
) : (
<MdCloudOff className="text-[16px]" />
)}
<span>{online ? 'Sync enabled' : 'Sync paused'}</span>
</button>
</div>

<div className="mt-3 rounded-lg border border-slate-800/70 bg-slate-950/30 p-2">
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">Transport</div>
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">
Transport
</div>
<div className="mt-2 flex flex-wrap gap-2">
<button
type="button"
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === "local")}`}
onClick={() => setSyncTransportMode("local")}
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === 'local')}`}
onClick={() => setSyncTransportMode('local')}
>
Local tabs
</button>
<button
type="button"
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === "remote")}`}
onClick={() => setSyncTransportMode("remote")}
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === 'remote')}`}
onClick={() => setSyncTransportMode('remote')}
>
Remote server
</button>
<button
type="button"
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === "hybrid")}`}
onClick={() => setSyncTransportMode("hybrid")}
className={`rounded-md border px-3 py-1.5 text-[11px] font-semibold transition ${transportModeButtonClass(syncTransportMode === 'hybrid')}`}
onClick={() => setSyncTransportMode('hybrid')}
>
Hybrid
</button>
</div>
<div className="mt-2 text-[11px] text-slate-500">
{syncTransportMode === "local" && "Only same-origin tabs in this browser will sync."}
{syncTransportMode === "remote" && "Only the configured websocket sync server will be used."}
{syncTransportMode === "hybrid" && "Use both same-origin tabs and the configured websocket sync server."}
{syncTransportMode === 'local' && 'Only same-origin tabs in this browser will sync.'}
{syncTransportMode === 'remote' &&
'Only the configured remote websocket or bootstrap endpoint will be used.'}
{syncTransportMode === 'hybrid' &&
'Use both same-origin tabs and the configured remote websocket or bootstrap endpoint.'}
</div>
</div>

<div className="mt-3 rounded-lg border border-slate-800/70 bg-slate-950/30 p-2">
<div className="flex items-center justify-between gap-2">
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">Remote sync server</div>
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">
Remote sync / bootstrap
</div>
<div
className={`inline-flex items-center gap-1.5 rounded-full border px-2 py-1 text-[10px] font-semibold ${remoteStatusTone(remoteSyncStatus)}`}
title={remoteSyncStatus.detail}
>
<RemoteStatusIcon status={remoteSyncStatus} />
<span>
{remoteSyncStatus.state === "connected" && "Connected"}
{remoteSyncStatus.state === "connecting" && "Connecting"}
{remoteSyncStatus.state === "disabled" && "Inactive"}
{remoteSyncStatus.state === "missing_url" && "Missing URL"}
{remoteSyncStatus.state === "invalid" && "Invalid URL"}
{remoteSyncStatus.state === "error" && "Unreachable"}
{remoteSyncStatus.state === 'connected' && 'Connected'}
{remoteSyncStatus.state === 'connecting' && 'Connecting'}
{remoteSyncStatus.state === 'disabled' && 'Inactive'}
{remoteSyncStatus.state === 'missing_url' && 'Missing URL'}
{remoteSyncStatus.state === 'invalid' && 'Invalid URL'}
{remoteSyncStatus.state === 'error' && 'Unreachable'}
</span>
</div>
</div>
Expand All @@ -149,17 +162,17 @@ export function PeersPanel({
onChange={(event) => {
const next = event.target.value;
setSyncServerUrl(next);
if (syncTransportMode === "local" && next.trim().length > 0) {
setSyncTransportMode("hybrid");
if (syncTransportMode === 'local' && next.trim().length > 0) {
setSyncTransportMode('hybrid');
}
}}
placeholder="ws://localhost:8787 or ws://localhost:8787/sync"
placeholder="https://bootstrap-host or ws://localhost:8787"
spellCheck={false}
/>
<button
type="button"
className="rounded-md border border-slate-700 px-2 py-1.5 text-[11px] font-semibold text-slate-300 transition hover:border-slate-500 hover:text-white disabled:opacity-50"
onClick={() => setSyncServerUrl("")}
onClick={() => setSyncServerUrl('')}
disabled={syncServerUrl.trim().length === 0}
title="Clear remote sync server URL"
>
Expand All @@ -174,14 +187,18 @@ export function PeersPanel({

<div className="mt-3 rounded-lg border border-slate-800/70 bg-slate-950/30 p-2">
<div className="flex items-center justify-between gap-2">
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">Connected peers</div>
<div className="text-[10px] font-semibold uppercase tracking-wide text-slate-400">
Connected peers
</div>
<div className="text-[10px] text-slate-500">{peers.length}</div>
</div>
<div className="mt-2 max-h-32 overflow-auto pr-1">
{peers.map((p) => (
<div key={p.id} className="flex items-center justify-between gap-2 py-1">
<span className="font-mono text-slate-200">{formatPeerId(p.id)}</span>
<span className="text-[10px] text-slate-500">{Math.max(0, Date.now() - p.lastSeen)}ms</span>
<span className="text-[10px] text-slate-500">
{Math.max(0, Date.now() - p.lastSeen)}ms
</span>
</div>
))}
{peers.length === 0 && (
Expand Down
Loading
Loading