crate publishing workflow

.github/PUBLISH_CRATES.md

@@ -0,0 +1,140 @@
+# Release Process
+
+Publishing DiskANN crates to [crates.io](https://crates.io).
+
+## Overview
+
+All workspace crates are published together with synchronized version numbers using `cargo publish --workspace`, which automatically resolves dependency order and waits for each crate to be indexed before publishing its dependents. The Rust toolchain version is read from [`rust-toolchain.toml`](../../rust-toolchain.toml).
+
+Releases follow a pull-request workflow: bump the version on a branch, open a PR, let the dry-run check pass, merge, then tag the release via the GitHub UI.
+
+## Prerequisites
+
+1. **CRATES_IO_TOKEN Secret**: A crates.io API token configured as a GitHub repository secret named `CRATES_IO_TOKEN` with publish permissions for all DiskANN crates.
+   - Create a token: [crates.io/settings/tokens](https://crates.io/settings/tokens)
+   - Add the secret: Repository Settings → Secrets and variables → Actions → New repository secret
+
+2. **Maintainer Access**: Write access to the repository and owner/maintainer of all crates on crates.io.
+
+## Dry-Run Testing
+
+A `cargo publish --workspace --dry-run` runs **automatically** as a pull-request check whenever `Cargo.toml` is changed. You can also trigger a dry-run manually:
+
+### Manual: GitHub Actions
+
+1. Navigate to: `https://github.com/microsoft/DiskANN/actions/workflows/publish.yml`
+2. Click **Run workflow**, select your branch, keep **dry-run = true**
+3. Watch the workflow — look for successful `cargo publish --workspace --dry-run`
+
+### Manual: Local
+
+```bash
+cargo publish --locked --workspace --dry-run
+```
+
+### What Dry-Run Tests
+
+- Crate metadata and packaging validation
+- Dependency resolution and publish ordering
+- Build verification
+
+### What It Does NOT Test
+
+- Actual publishing, registry token auth, upload reliability
+
+## Release Steps
+
+1. **Create a release branch** from `main`:
+
+   ```bash
+   git checkout main && git pull
+   git checkout -b release-X.Y.Z
+   ```
+
+2. **Update version** in root `Cargo.toml`:
+
+   - Set `workspace.package.version`:
+
+     ```toml
+     [workspace.package]
+     version = "X.Y.Z"
+     ```
+
+   - Update **all internal crate entries** under `[workspace.dependencies]` to match:
+
+     ```toml
+     diskann-wide = { path = "diskann-wide", version = "X.Y.Z" }
+     diskann-vector = { path = "diskann-vector", version = "X.Y.Z" }
+     # ... etc
+     ```
+
+   Member crates inherit `workspace.package.version` via `version.workspace = true`,
+   but `[workspace.dependencies]` versions must be set explicitly (they're baked into
+   published manifests for crates.io consumers). 
+
+3. **Update CHANGELOG** (if applicable).
+
+4. **Push and open a pull request** to `main`:
+
+   ```bash
+   git commit -am "Bump version to X.Y.Z"
+   git push origin release-X.Y.Z
+   ```
+
+   Open a PR on GitHub. The **Publish to crates.io / Dry-run publish test** check runs automatically.
+
+5. **Wait for checks** — the dry-run and CI must both pass before merge.
+
+6. **Merge the PR** into `main` and delete the release branch (GitHub offers this on the merged PR page).
+
+7. **Create a release** via the GitHub UI:
+   - Go to **Releases → Draft a new release**
+   - Create a new tag `vX.Y.Z` targeting `main`
+   - Add release notes describing changes
+   - Click **Publish release**
+
+   Pushing the tag triggers the real publish workflow.
+
+8. **Verify** the published crates — confirm the new version appears in the output:
+
+   ```bash
+   cargo search diskann --limit 20
+   ```
+
+## Recovering from a Partial Publish
+
+If the publish workflow fails midway through, some crates will have been published
+and others will not. `cargo publish --workspace` will refuse to re-publish crates
+whose version already exists on crates.io, so you have two options:
+
+### Option 1: Retry with `--exclude` (same version)
+
+If the failure was transient (network issue, registry timeout), exclude the
+already-published crates and re-run:
+
+```bash
+cargo publish --locked --workspace \
+  --exclude already-published-crate-1 \
+  --exclude already-published-crate-2
+```
+
+You can trigger this via **workflow_dispatch** with `dry_run = false`, or run it
+locally with a valid `CARGO_REGISTRY_TOKEN`.
+
+### Option 2: Bump version and re-release
+
+If the failure was caused by a packaging or metadata error in a specific crate,
+fix the issue, bump the version, and go through the full release process again.
+Already-published crates at the old version will remain on crates.io — this is
+harmless since consumers pin to specific versions.
+
+## Pre-release Checklist
+
+The release author should verify the following on the version-bump PR before merging:
+
+- [ ] `workspace.package.version` and all `[workspace.dependencies]` versions are updated in `Cargo.toml`
+- [ ] CHANGELOG includes an entry for the new version summarising notable changes
+- [ ] Breaking changes are called out in the CHANGELOG and in the PR description
+- [ ] All CI checks pass
+- [ ] **Dry-run publish check passes**
+- [ ] Release branch deleted after merge

.github/workflows/publish.yml

@@ -0,0 +1,132 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT license.
+
+# Publishes all workspace crates to crates.io.
+#
+# Triggers:
+#   - push tag v{major}.{minor}.{patch} → real publish
+#   - pull_request touching Cargo.toml   → automatic dry-run (pre-merge check)
+#   - workflow_dispatch                  → manual dry-run or real publish
+#
+# Requires CRATES_IO_TOKEN secret. Rust toolchain version is read from rust-toolchain.toml.
+
+name: Publish to crates.io
+
+on:
+  push:
+    tags:
+      - 'v[0-9]+.[0-9]+.[0-9]+'
+  pull_request:
+    branches: ["main"]
+    paths:
+      - 'Cargo.toml'
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: 'Run in dry-run mode (test without actually publishing)'
+        required: false
+        default: 'true'
+        type: choice
+        options:
+          - 'true'
+          - 'false'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: false
+
+env:
+  RUST_BACKTRACE: 1
+
+defaults:
+  run:
+    shell: bash
+
+permissions:
+  contents: read
+
+jobs:
+  publish:
+    name: >-
+      ${{
+        (github.event_name == 'pull_request' || github.event.inputs.dry_run == 'true')
+          && 'Dry-run publish test'
+          || 'Publish crates to crates.io'
+      }}
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          lfs: true
+          fetch-depth: 0
+
+      - name: Read Rust version from rust-toolchain.toml
+        id: rust-version
+        run: |
+          RUST_VERSION=$(sed -n 's/^channel = "\(.*\)"/\1/p' rust-toolchain.toml)
+          echo "channel=$RUST_VERSION" >> "$GITHUB_OUTPUT"
+
+      - name: Install Rust ${{ steps.rust-version.outputs.channel }}
+        uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: ${{ steps.rust-version.outputs.channel }}
+
+      - uses: Swatinem/rust-cache@v2
+
+      - name: Prevent publish from non-main branch
+        if: >-
+          github.event_name == 'workflow_dispatch'
+          && github.event.inputs.dry_run != 'true'
+          && github.ref != 'refs/heads/main'
+        run: |
+          echo "::error::Live publishing is only allowed from main. Use dry-run for other branches."
+          exit 1
+
+      - name: Verify version matches tag
+        if: github.event_name == 'push'
+        run: |
+          TAG_VERSION="${GITHUB_REF#refs/tags/v}"
+          CARGO_VERSION=$(grep -A 5 '^\[workspace\.package\]' Cargo.toml | grep 'version = ' | head -n1 | sed 's/.*"\(.*\)".*/\1/')
+          if [ "$TAG_VERSION" != "$CARGO_VERSION" ]; then
+            echo "::error::Tag version ($TAG_VERSION) does not match Cargo.toml version ($CARGO_VERSION)"
+            exit 1
+          fi
+
+      - name: Verify tag is on main branch
+        if: github.event_name == 'push'
+        run: |
+          if ! git branch --contains "$GITHUB_SHA" | grep -qE '\bmain$'; then
+            echo "::error::Tag must point to a commit on main"
+            exit 1
+          fi
+
+      - name: Verify all crates use workspace version
+        run: |
+          bad_crates=()
+          for manifest in $(cargo metadata --no-deps --format-version 1 | jq -r '.packages[].manifest_path'); do
+            dir=$(dirname "$manifest")
+            name=$(basename "$dir")
+            if [ "$manifest" != "$(pwd)/Cargo.toml" ] && ! grep -qE 'version\s*=\s*\{\s*workspace\s*=\s*true\s*\}|version\.workspace\s*=\s*true' "$manifest"; then
+              bad_crates+=("$name")
+            fi
+          done
+          if [ ${#bad_crates[@]} -gt 0 ]; then
+            echo "::error::The following crates do not use version.workspace = true: ${bad_crates[*]}"
+            exit 1
+          fi
+
+      - name: Publish workspace crates
+        env:
+          CARGO_REGISTRY_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}
+          DRY_RUN: ${{ github.event.inputs.dry_run || 'false' }}
+        run: |
+          DRY_RUN_FLAG=""
+          if [ "${{ github.event_name }}" = "pull_request" ] || [ "$DRY_RUN" = "true" ]; then
+            DRY_RUN_FLAG="--dry-run"
+            echo "🧪 DRY-RUN MODE"
+          else
+            echo "📦 LIVE MODE - Publishing to crates.io"
+          fi
+          cargo publish --locked --workspace $DRY_RUN_FLAG