refactor: carve out ARF & Storage into first-class Code Blue domains

[felixschmetz]

Summary

Extracts ARF (raw entity capture/replay) and Storage (file/object backend) out of the
monolithic platform/ package into self-contained domain modules under domains/,
establishing the "Code Blue" domain architecture pattern for infrastructure-level concerns.

What changed

• domains/arf/ — new domain owning the full ARF lifecycle: capture, replay, reader,
  manifest management. Protocols (ArfServiceProtocol, ArfReaderProtocol), concrete
  service, types, fakes, and co-located unit tests.
• domains/storage/ — new domain owning all persistent storage: backend protocol
  (StorageBackend), four backend implementations (filesystem, S3, Azure Blob, GCS),
  SyncFileManager, factory, paths, exceptions, fakes, and co-located unit tests.
• Container DI wiring — StorageBackend, SyncFileManager, and ArfService are now
  first-class Container slots, injected via Inject() across the codebase.
• Deleted platform/storage/ and platform/sync/arf/ (all code migrated, not duplicated).
• All 25+ sources updated to import from domains.storage instead of platform.storage.

Why

The platform/ package had grown into a grab-bag mixing orchestration logic with
infrastructure concerns. Pulling storage and ARF into domains/ gives each module:

• Explicit protocol boundaries (no hidden coupling)
• Fakes for unit testing without I/O
• Co-located tests that travel with the code
• A reproducible pattern for future domain extractions

Migration shape

• platform/storage/ → domains/storage/
• platform/storage/arf_reader → domains/arf/reader
• platform/sync/arf/ → domains/arf/

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[felixschmetz]

Will be taken care of in the next PR: #1614

[felixschmetz]

Addressed in next PR: #1614

[felixschmetz]

temporary - let's defer restructuring the admin stuff

[felixschmetz]

Hmm, are we ok with this in the short term?

[cubic-dev-ai]

5 issues found across 82 files

Note: This PR contains a large number of files. cubic only reviews up to 75 files per PR, so some files may not have been reviewed.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/airweave/domains/arf/service.py">

<violation number="1" location="backend/airweave/domains/arf/service.py:47">
P2: Reuse `StoragePaths` here instead of duplicating the ARF path helpers and `_safe_filename` logic.</violation>
</file>

<file name="backend/airweave/domains/storage/fakes/backend.py">

<violation number="1" location="backend/airweave/domains/storage/fakes/backend.py:70">
P1: Include `write_file` entries in `list_files`; otherwise binary fixtures disappear from `list_files` and `count_files`.</violation>
</file>

<file name="backend/list_syncs_for_migration.py">

<violation number="1" location="backend/list_syncs_for_migration.py:108">
P2: This now creates a new storage backend for every sync. In large migration runs that means repeated backend/client initialization and, for cloud backends, leaked async clients because the new instances are never closed.</violation>
</file>

<file name="backend/airweave/platform/sync/actions/entity/builder.py">

<violation number="1" location="backend/airweave/platform/sync/actions/entity/builder.py:120">
P1: Custom agent: **Check for Cursor Rules Drift**

Update the ARF/sync Cursor rules for this file-structure and API-pattern change. This builder now injects `ArfServiceProtocol` into `ArfHandler`, but `.cursor/rules/arf.mdc` and `.cursor/rules/sync-architecture.mdc` still document deleted `platform/storage/*` / `platform/sync/raw_data.py` paths and obsolete `RawDataHandler` wiring.</violation>
</file>

<file name="backend/airweave/domains/arf/fakes/service.py">

<violation number="1" location="backend/airweave/domains/arf/fakes/service.py:11">
P1: Custom agent: **Explicit Protocol Implementation**

Explicit Protocol Implementation: this fake service implements `ArfServiceProtocol` but does not inherit it. Declare `FakeArfService(ArfServiceProtocol)` so the ARF write-path contract stays explicit and statically enforced.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P1: Include write_file entries in list_files; otherwise binary fixtures disappear from list_files and count_files.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/airweave/domains/storage/fakes/backend.py, line 70:

<comment>Include `write_file` entries in `list_files`; otherwise binary fixtures disappear from `list_files` and `count_files`.</comment>

<file context>
@@ -0,0 +1,86 @@
+        return len(json_keys) > 0 or len(file_keys) > 0
+
+    async def list_files(self, prefix: str = "") -> List[str]:
+        return [k for k in self._json_store if k.startswith(prefix)]
+
+    async def list_dirs(self, prefix: str = "") -> List[str]:
</file context>

[cubic-dev-ai]

P1: Custom agent: Check for Cursor Rules Drift

Update the ARF/sync Cursor rules for this file-structure and API-pattern change. This builder now injects ArfServiceProtocol into ArfHandler, but .cursor/rules/arf.mdc and .cursor/rules/sync-architecture.mdc still document deleted platform/storage/* / platform/sync/raw_data.py paths and obsolete RawDataHandler wiring.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/airweave/platform/sync/actions/entity/builder.py, line 120:

<comment>Update the ARF/sync Cursor rules for this file-structure and API-pattern change. This builder now injects `ArfServiceProtocol` into `ArfHandler`, but `.cursor/rules/arf.mdc` and `.cursor/rules/sync-architecture.mdc` still document deleted `platform/storage/*` / `platform/sync/raw_data.py` paths and obsolete `RawDataHandler` wiring.</comment>

<file context>
@@ -106,16 +102,24 @@ def _add_destination_handler(
+                logger.warning("Skipping ArfHandler (no arf_service provided)")
+            return
+
+        handlers.append(ArfHandler(arf_service=arf_service))
+        if logger:
+            logger.debug("Added ArfHandler")
</file context>

[cubic-dev-ai]

P1: Custom agent: Explicit Protocol Implementation

Explicit Protocol Implementation: this fake service implements ArfServiceProtocol but does not inherit it. Declare FakeArfService(ArfServiceProtocol) so the ARF write-path contract stays explicit and statically enforced.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/airweave/domains/arf/fakes/service.py, line 11:

<comment>Explicit Protocol Implementation: this fake service implements `ArfServiceProtocol` but does not inherit it. Declare `FakeArfService(ArfServiceProtocol)` so the ARF write-path contract stays explicit and statically enforced.</comment>

<file context>
@@ -0,0 +1,132 @@
+from airweave.platform.entities._base import BaseEntity
+
+
+class FakeArfService:
+    """In-memory fake for ArfServiceProtocol.
+
</file context>

[cubic-dev-ai]

P2: Reuse StoragePaths here instead of duplicating the ARF path helpers and _safe_filename logic.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/airweave/domains/arf/service.py, line 47:

<comment>Reuse `StoragePaths` here instead of duplicating the ARF path helpers and `_safe_filename` logic.</comment>

<file context>
@@ -0,0 +1,330 @@
+    # =========================================================================
+
+    @staticmethod
+    def _sync_path(sync_id: str) -> str:
+        return f"raw/{sync_id}"
+
</file context>

[cubic-dev-ai]

P2: This now creates a new storage backend for every sync. In large migration runs that means repeated backend/client initialization and, for cloud backends, leaked async clients because the new instances are never closed.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/list_syncs_for_migration.py, line 108:

<comment>This now creates a new storage backend for every sync. In large migration runs that means repeated backend/client initialization and, for cloud backends, leaked async clients because the new instances are never closed.</comment>

<file context>
@@ -102,10 +102,10 @@ async def get_postgres_entity_count(conn, sync_id: UUID) -> int:
+        from airweave.domains.storage.factory import get_storage_backend
 
-        arf_service = ArfService(storage=storage_backend)
+        arf_service = ArfService(storage=get_storage_backend())
         count = await arf_service.get_entity_count(str(sync_id))
         return count
</file context>