feat: Add Rust FFI client for integration testing

.claude/skills/rust-ffi-compat.skill

@@ -0,0 +1,137 @@
+---
+name: rust-ffi-compat
+description: Run Go integration tests against the Rust FFI implementation of DefraDB and generate compatibility reports. Use when checking Rust compatibility status, running FFI tests, generating test reports, or tracking progress on Rust implementation. Triggers on "rust compatibility", "ffi tests", "rust test status", "compatibility report", or "run rust tests".
+---
+
+# Rust FFI Compatibility Testing
+
+Run Go DefraDB integration tests against the Rust FFI implementation to verify compatibility.
+
+## Architecture: Single Go Repo + Multiple Rust Worktrees
+
+**ONE Go repo** (`defradb-rust-compat` on `jack/ffi-rust-compat` branch) serves all Rust worktrees. Each Rust worktree focuses on a specific test package for tight iteration loops.
+
+| Rust Worktree | Test Package | GOCACHE |
+|---------------|--------------|---------|
+| `defradb.rs-query-simple` | query/simple | `/tmp/gocache-query-simple` |
+| `defradb.rs-one-to-many` | query/one_to_many | `/tmp/gocache-one-to-many` |
+| `defradb.rs-index` | index | `/tmp/gocache-index` |
+| `defradb.rs-collection-version` | collection_version | `/tmp/gocache-collection-version` |
+| `defradb.rs-explain` | explain | `/tmp/gocache-explain` |
+
+### Why This Design
+
+- **No library copying**: `CGO_LDFLAGS` points directly to each Rust worktree's `target/release/`
+- **No Go duplication**: Single Go repo, tests run against any Rust worktree
+- **Cache isolation**: Unique `GOCACHE` per Rust worktree prevents stale CGO objects
+- **Parallel-safe**: Multiple Claude sessions can work on different Rust worktrees simultaneously
+
+## Prerequisites
+
+1. Build Rust FFI library from the Rust worktree:
+   ```bash
+   cd /path/to/defradb.rs-<package>
+   cargo build --release -p ffi
+   ```
+
+2. Ensure `defra.h` is in the Go repo's `tests/clients/rustffi/` directory (committed to repo, rarely changes)
+
+## Running Tests
+
+### ⚠️ CRITICAL: Use inline env vars with `go test -C`
+
+**DO NOT use `cd`, `export`, or wrapper scripts.** Claude Code's sandbox blocks these patterns:
+- `cd /path && go test` → "permission denied"
+- `export VAR=value` → "not valid in this context"
+- `bash script.sh` → subprocess blocked
+
+**ALWAYS use this pattern:** All env vars inline, then `go test -C <dir>`:
+
+```bash
+CGO_ENABLED=1 CGO_LDFLAGS="-L/path/to/defradb.rs-<package>/target/release -lffi -ldl -lpthread -lm" DEFRA_CLIENT_RUST_FFI=true GOCACHE=/tmp/gocache-<package> go test -C /path/to/defradb-rust-compat ./tests/integration/<package>/... -v -count=1 -timeout 120s
+```
+
+The `-C` flag changes directory internally (equivalent to `cd` but works in the sandbox).
+
+### Example: Testing query/simple against defradb.rs-query-simple
+
+```bash
+CGO_ENABLED=1 CGO_LDFLAGS="-L/Users/johnzampolin/go/src/github.com/sourcenetwork/defradb.rs-query-simple/target/release -lffi -ldl -lpthread -lm" DEFRA_CLIENT_RUST_FFI=true GOCACHE=/tmp/gocache-query-simple go test -C /Users/johnzampolin/go/src/github.com/sourcenetwork/defradb-rust-compat ./tests/integration/query/simple/... -v -count=1 -timeout 120s
+```
+
+### Verify you're testing the right client
+
+Check the test output for `client=rust-ffi`:
+```
+INF tests.integration TestQuerySimple database=badger-in-memory client=rust-ffi ...
+```
+
+If it shows a different client (e.g., `client=go`), the env vars are wrong or missing.
+
+### After rebuilding Rust
+
+Clear the GOCACHE to ensure fresh CGO compilation:
+```bash
+rm -rf /tmp/gocache-<package>
+```
+
+Go's build cache doesn't detect changes to `libffi.a` (known bug #25419), so this step is required after Rust rebuilds.
+
+## Test Packages
+
+| Package | Description | Current Status |
+|---------|-------------|----------------|
+| `query/simple` | Basic queries, filters, ordering, limits | 74% (322/433) |
+| `query/one_to_many` | One-to-many relation queries | 26% (23/87) |
+| `index` | Secondary index queries | 29% (23/79) |
+| `collection_version` | Schema versioning | 20% (75/372) |
+| `explain` | Query plan explain output | 0% (0/249) |
+| `query/inline_array` | Array field queries | **100%** (172/172) |
+| `query/json` | JSON field queries | **100%** (67/67) |
+| `mutation/delete` | Document deletion | 88% (30/34) |
+| `mutation/update` | Document updates | 82% (70/85) |
+| `mutation/create` | Document creation | 72% (48/67) |
+| `query/one_to_one` | One-to-one relation queries | 80% (33/41) |
+| `query/commits` | Commit history queries | 86% (75/87) |
+| `collection` | Collection management | 0% (0/16) |
+
+## Full Report Generation
+
+Run all packages in parallel with shared GOCACHE for fastest execution:
+
+```bash
+# Use the main defradb.rs repo (or any worktree with a complete build)
+RS=/Users/johnzampolin/go/src/github.com/sourcenetwork/defradb.rs
+GO=/Users/johnzampolin/go/src/github.com/sourcenetwork/defradb-rust-compat
+
+CGO_ENABLED=1 \
+CGO_LDFLAGS="-L$RS/target/release -lffi -ldl -lpthread -lm" \
+DEFRA_CLIENT_RUST_FFI=true \
+GOCACHE=/tmp/gocache-full-report \
+go test -C $GO ./tests/integration/<pkg>/... -v -count=1 -timeout 120s
+```
+
+Launch 13 background Bash tasks (one per package) to run simultaneously.
+
+## Troubleshooting
+
+| Problem | Fix |
+|---------|-----|
+| `permission denied` or `not valid in this context` | Don't use `cd`, `export`, or scripts. Use inline env vars with `go test -C` |
+| `client=go` in output (not `rust-ffi`) | Missing `DEFRA_CLIENT_RUST_FFI=true` env var |
+| Undefined symbols at link time | Ensure `CGO_LDFLAGS` points to correct `target/release/` with fresh `libffi.a` |
+| Stale results after Rust rebuild | Delete the GOCACHE: `rm -rf /tmp/gocache-<package>` |
+| Tests stuck / hanging | Check `DEFRA_CLIENT_RUST_FFI=true` is set |
+| Header not found | Ensure `defra.h` exists in Go repo's `tests/clients/rustffi/` |
+| Wrong test results | Verify `CGO_LDFLAGS` points to the intended Rust worktree |
+
+## Creating a New Rust Worktree
+
+```bash
+cd /Users/johnzampolin/go/src/github.com/sourcenetwork/defradb.rs
+git worktree add ../defradb.rs-<package> -b ffi/<package>
+cd ../defradb.rs-<package>
+cargo build --release -p ffi
+```
+
+Then test with the canonical invocation above.

.gitignore

@@ -70,3 +70,13 @@ crash.log
 
 # Ignore .env files containing sensitive information.
 .env
+
+# Rust FFI test reports
+reports/
+
+# Rust FFI test history (see make test:ffi-history)
+ffi-test-history/
+
+# Rust FFI library files (built from defradb.rs)
+tests/clients/rustffi/*.a
+tests/clients/rustffi/*.dylib

Makefile

@@ -474,3 +474,232 @@ API_LEVEL ?= 21
 .PHONY: build-c-shared-android
 build-c-shared-android:
 	@tools/scripts/build-c-shared-android.sh $(ANDROID_NDK) $(API_LEVEL) "$(BUILD_FLAGS)"
+
+# ============================================================================
+# Rust FFI Testing
+# ============================================================================
+#
+# Run Go integration tests against the Rust FFI implementation.
+#
+# Usage:
+#   make test:ffi RUST_LIB=/path/to/defradb.rs FFI_PKG=query/simple
+#   make test:ffi RUST_LIB=/path/to/defradb.rs FFI_PKG="query/simple mutation/create"
+#   make test:ffi-report      # View last test report
+#   make test:ffi-save        # Save current reports to history
+#   make test:ffi-history     # View test history
+#
+# Environment:
+#   RUST_LIB    - Path to Rust defradb.rs repo (required)
+#   FFI_PKG     - Test package(s), space-separated (default: query/simple)
+#   FFI_TIMEOUT - Test timeout (default: 300s)
+#
+# History:
+#   Test results are saved to ffi-test-history/ (gitignored) with metadata
+#   about which Rust branch/commit produced the results. Use test:ffi-save
+#   after a test run to save results, or test:ffi-history to view past runs.
+#
+# Parallel Worktree Support:
+#   Each RUST_LIB path gets its own GOCACHE, so multiple Claude sessions can
+#   run tests against different Rust worktrees simultaneously without cache
+#   collisions. Cache dir is derived from RUST_LIB basename + package name.
+
+FFI_PKG ?= query/simple
+FFI_TIMEOUT ?= 300s
+FFI_REPORT_DIR ?= /tmp/ffi-test-reports
+FFI_HISTORY_DIR ?= $(CURDIR)/ffi-test-history
+
+.PHONY: test\:ffi
+test\:ffi:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi RUST_LIB=/path/to/defradb.rs FFI_PKG=query/simple)
+endif
+	@echo "=== Generating defra.h from $(RUST_LIB) ==="
+	@cd $(RUST_LIB) && cbindgen --config crates/ffi/cbindgen.toml --crate ffi --output $(CURDIR)/tests/clients/rustffi/defra.h 2>&1
+	@echo "=== Copying Rust FFI library ==="
+	@cp -f $(RUST_LIB)/target/release/libffi.dylib $(CURDIR)/tests/clients/rustffi/libdefra_ffi.dylib 2>/dev/null && \
+		install_name_tool -id @rpath/libdefra_ffi.dylib $(CURDIR)/tests/clients/rustffi/libdefra_ffi.dylib 2>/dev/null || \
+		cp -f $(RUST_LIB)/target/release/libffi.so $(CURDIR)/tests/clients/rustffi/libdefra_ffi.so 2>/dev/null || \
+		echo "WARNING: Could not copy Rust FFI library (run: cargo build --release -p ffi)"
+	@mkdir -p $(FFI_REPORT_DIR)
+	@rust_lib_name=$$(basename $(RUST_LIB)); \
+	rust_branch=$$(cd $(RUST_LIB) && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown"); \
+	rust_commit=$$(cd $(RUST_LIB) && git rev-parse --short HEAD 2>/dev/null || echo "unknown"); \
+	for pkg in $(FFI_PKG); do \
+		pkg_safe=$$(echo $$pkg | tr '/' '-'); \
+		report_file=$(FFI_REPORT_DIR)/$$pkg_safe.log; \
+		meta_file=$(FFI_REPORT_DIR)/$$pkg_safe.meta; \
+		cache_dir=/tmp/gocache-$$rust_lib_name-$$pkg_safe; \
+		echo ""; \
+		echo "=== Testing $$pkg ==="; \
+		echo "Clearing GOCACHE: $$cache_dir"; \
+		rm -rf $$cache_dir; \
+		echo "Running tests..."; \
+		CGO_ENABLED=1 \
+		DEFRA_CLIENT_RUST_FFI=true \
+		$(if $(DEFRA_MUTATION_TYPE),DEFRA_MUTATION_TYPE=$(DEFRA_MUTATION_TYPE)) \
+		$(if $(DEFRA_BADGER_FILE),DEFRA_BADGER_FILE=$(DEFRA_BADGER_FILE)) \
+		GOCACHE=$$cache_dir \
+		go test ./tests/integration/$$pkg/... -v -count=1 -timeout $(FFI_TIMEOUT) 2>&1 | tee $$report_file; \
+		echo "{\"timestamp\":\"$$(date +%Y-%m-%d\ %H:%M:%S)\",\"rust_lib\":\"$$rust_lib_name\",\"rust_branch\":\"$$rust_branch\",\"rust_commit\":\"$$rust_commit\"}" > $$meta_file; \
+		echo ""; \
+		echo "=== Results for $$pkg ==="; \
+		passed=$$(grep -c "^--- PASS:" $$report_file 2>/dev/null || true); \
+		failed=$$(grep -c "^--- FAIL:" $$report_file 2>/dev/null || true); \
+		passed=$${passed:-0}; \
+		failed=$${failed:-0}; \
+		total=$$((passed + failed)); \
+		echo "  Passed: $$passed"; \
+		echo "  Failed: $$failed"; \
+		echo "  Total:  $$total"; \
+		if [ "$$failed" -gt 0 ]; then \
+			echo ""; \
+			echo "Failed tests:"; \
+			grep "^--- FAIL:" $$report_file | sed 's/--- FAIL: /  - /' | sed 's/ (.*//' ; \
+		fi; \
+		echo ""; \
+		echo "Full output: $$report_file"; \
+	done
+
+# All test packages organized by priority (see issue #18)
+# P0 Foundation: query/simple, mutation/*
+# P1 Core Query: inline_array, commits, json
+# P2 Relations: one_to_*, many_to_many
+# P3 Features: explain, acp, index, versions, etc.
+FFI_PKG_P0 := query/simple mutation/create mutation/update mutation/delete mutation/mix mutation/special mutation/upsert
+FFI_PKG_P1 := query/inline_array query/commits query/json
+FFI_PKG_P2 := query/one_to_one query/one_to_many query/one_to_many_multiple query/one_to_many_to_many query/one_to_many_to_one query/one_to_one_multiple query/one_to_one_to_many query/one_to_one_to_one query/one_to_two_many query/many_to_many
+FFI_PKG_P3 := index explain collection collection_version view acp backup encryption net node signature subscription searchable_encryption issues
+
+FFI_PKG_ALL := $(FFI_PKG_P0) $(FFI_PKG_P1) $(FFI_PKG_P2) $(FFI_PKG_P3)
+
+.PHONY: test\:ffi-all
+test\:ffi-all:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-all RUST_LIB=/path/to/defradb.rs)
+endif
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG_ALL)"
+
+# Run FFI tests with both default (collection-save) and GQL mutation types.
+# This catches tests that only run with GQLRequestMutationType.
+.PHONY: test\:ffi-full
+test\:ffi-full:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-full RUST_LIB=/path/to/defradb.rs FFI_PKG=acp)
+endif
+	@echo "=== Pass 1: default mutation type ==="
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG)"
+	@echo ""
+	@echo "=== Pass 2: GQL mutation type ==="
+	@DEFRA_MUTATION_TYPE=gql $(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG)"
+
+.PHONY: test\:ffi-p0
+test\:ffi-p0:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-p0 RUST_LIB=/path/to/defradb.rs)
+endif
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG_P0)"
+
+.PHONY: test\:ffi-p1
+test\:ffi-p1:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-p1 RUST_LIB=/path/to/defradb.rs)
+endif
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG_P1)"
+
+.PHONY: test\:ffi-p2
+test\:ffi-p2:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-p2 RUST_LIB=/path/to/defradb.rs)
+endif
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG_P2)"
+
+.PHONY: test\:ffi-p3
+test\:ffi-p3:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-p3 RUST_LIB=/path/to/defradb.rs)
+endif
+	@$(MAKE) test:ffi RUST_LIB=$(RUST_LIB) FFI_PKG="$(FFI_PKG_P3)"
+
+.PHONY: test\:ffi-report
+test\:ffi-report:
+	@echo "=== FFI Test Reports ==="
+	@echo ""
+	@total_passed=0; total_failed=0; \
+	for f in $(FFI_REPORT_DIR)/*.log; do \
+		if [ -f "$$f" ]; then \
+			name=$$(basename $$f .log); \
+			passed=$$(grep -c "^--- PASS:" $$f 2>/dev/null; true); \
+			failed=$$(grep -c "^--- FAIL:" $$f 2>/dev/null; true); \
+			passed=$${passed:-0}; \
+			failed=$${failed:-0}; \
+			total=$$((passed + failed)); \
+			total_passed=$$((total_passed + passed)); \
+			total_failed=$$((total_failed + failed)); \
+			meta=""; \
+			meta_file="$(FFI_REPORT_DIR)/$$name.meta"; \
+			if [ -f "$$meta_file" ]; then \
+				branch=$$(grep -o '"rust_branch":"[^"]*"' $$meta_file | cut -d'"' -f4); \
+				commit=$$(grep -o '"rust_commit":"[^"]*"' $$meta_file | cut -d'"' -f4); \
+				ts=$$(grep -o '"timestamp":"[^"]*"' $$meta_file | cut -d'"' -f4); \
+				meta="$$branch@$$commit $$ts"; \
+			fi; \
+			if [ "$$total" -gt 0 ]; then \
+				pct=$$((passed * 100 / total)); \
+				printf "%-40s %3d/%3d (%3d%%)  %s\n" "$$name" $$passed $$total $$pct "$$meta"; \
+			fi; \
+		fi; \
+	done; \
+	echo ""; \
+	grand_total=$$((total_passed + total_failed)); \
+	if [ "$$grand_total" -gt 0 ]; then \
+		grand_pct=$$((total_passed * 100 / grand_total)); \
+		printf "%-40s %3d/%3d (%3d%%)\n" "TOTAL" $$total_passed $$grand_total $$grand_pct; \
+	fi
+
+.PHONY: test\:ffi-save
+test\:ffi-save:
+ifndef RUST_LIB
+	$(error RUST_LIB is required. Usage: make test:ffi-save RUST_LIB=/path/to/defradb.rs)
+endif
+	@mkdir -p $(FFI_HISTORY_DIR)
+	@rust_branch=$$(cd $(RUST_LIB) && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown"); \
+	rust_commit=$$(cd $(RUST_LIB) && git rev-parse --short HEAD 2>/dev/null || echo "unknown"); \
+	rust_lib_name=$$(basename $(RUST_LIB)); \
+	timestamp=$$(date +%Y%m%d-%H%M%S); \
+	history_dir=$(FFI_HISTORY_DIR)/$$timestamp-$$rust_lib_name-$$rust_branch-$$rust_commit; \
+	mkdir -p $$history_dir; \
+	echo "Saving FFI test results to $$history_dir"; \
+	cp $(FFI_REPORT_DIR)/*.log $$history_dir/ 2>/dev/null || true; \
+	echo "{ \"timestamp\": \"$$timestamp\", \"rust_lib\": \"$(RUST_LIB)\", \"rust_branch\": \"$$rust_branch\", \"rust_commit\": \"$$rust_commit\" }" > $$history_dir/metadata.json; \
+	total_passed=0; total_failed=0; \
+	for f in $$history_dir/*.log; do \
+		if [ -f "$$f" ]; then \
+			passed=$$(grep -c "^--- PASS:" $$f 2>/dev/null; true); \
+			failed=$$(grep -c "^--- FAIL:" $$f 2>/dev/null; true); \
+			total_passed=$$((total_passed + passed)); \
+			total_failed=$$((total_failed + failed)); \
+		fi; \
+	done; \
+	grand_total=$$((total_passed + total_failed)); \
+	echo "{ \"passed\": $$total_passed, \"failed\": $$total_failed, \"total\": $$grand_total }" >> $$history_dir/metadata.json; \
+	echo "Saved: $$total_passed passed, $$total_failed failed ($$grand_total total)"
+
+.PHONY: test\:ffi-history
+test\:ffi-history:
+	@echo "=== FFI Test History ==="
+	@echo ""
+	@if [ ! -d $(FFI_HISTORY_DIR) ]; then \
+		echo "No history found. Run 'make test:ffi-save RUST_LIB=...' after tests."; \
+		exit 0; \
+	fi; \
+	for d in $$(ls -1d $(FFI_HISTORY_DIR)/*/ 2>/dev/null | sort -r | head -20); do \
+		if [ -f "$$d/metadata.json" ]; then \
+			name=$$(basename $$d); \
+			passed=$$(grep -o '"passed": [0-9]*' $$d/metadata.json | grep -o '[0-9]*' || echo 0); \
+			total=$$(grep -o '"total": [0-9]*' $$d/metadata.json | grep -o '[0-9]*' || echo 0); \
+			if [ "$$total" -gt 0 ]; then \
+				pct=$$((passed * 100 / total)); \
+				printf "%-60s %4d/%4d (%3d%%)\n" "$$name" $$passed $$total $$pct; \
+			fi; \
+		fi; \
+	done