Remove opendal dependency, preserve exporter-side presigned downloads

[ambient-code]

Related to #218.

The opendal dependency should be removed from the codebase. However, the exporter-side download functionality (presigned URL equivalent) must be preserved.

The goal is to replace the opendal-based storage layer with a lighter approach that still allows the exporter to download artifacts via presigned URLs (or an equivalent mechanism), without pulling in the full opendal dependency tree.

Requirements

• Remove the opendal dependency
• Maintain exporter-side download capability using presigned URLs or equivalent
• Ensure client-side progress reporting continues to work (see no progress at the client side during presigned download #218)

[ambient-code]

Analysis of opendal removal scope

I've done a thorough investigation of the codebase to assess the feasibility of removing the opendal dependency. Here's my detailed findings:

Current opendal usage

Core package: jumpstarter-driver-opendal (6 source files) wraps the opendal library and provides:

• Opendal driver class: Full storage backend abstraction using AsyncOperator — file CRUD, directory operations, hashing, presigned URLs, capability detection, cleanup tracking
• OpendalClient: Client-side counterpart with CLI integration
• OpendalAdapter: Critical bridge component that handles presigned URL generation and file streaming between client and exporter
• AsyncFileStream: anyio-compatible wrapper for opendal.AsyncFile
• FlasherInterface / StorageMuxInterface / StorageMuxFlasherInterface: Abstract interfaces used by many driver packages
• FlasherClient / StorageMuxClient / StorageMuxFlasherClient: Client implementations
• Common models: Capability, Metadata, PresignedRequest, EntryMode

13 packages depend on jumpstarter-driver-opendal:

• jumpstarter-driver-dutlink (uses StorageMuxFlasherInterface)
• jumpstarter-driver-sdwire (uses StorageMuxFlasherInterface)
• jumpstarter-driver-esp32 (uses FlasherInterface, FlasherClient)
• jumpstarter-driver-flashers (uses FlasherClient, OpendalClient, operator_for_path, Operator directly)
• jumpstarter-driver-http (uses Opendal driver directly for storage backend, Operator in client)
• jumpstarter-driver-iscsi (uses Opendal driver, Operator in client)
• jumpstarter-driver-pi-pico (uses FlasherInterface, FlasherClient)
• jumpstarter-driver-probe-rs (uses OpendalAdapter, Operator)
• jumpstarter-driver-qemu (uses FlasherInterface, Operator in tests)
• jumpstarter-driver-ridesx (uses Opendal, FlasherClient, operator_for_path, Operator)
• jumpstarter-driver-tftp (uses Opendal and AsyncOperator directly in TFTP server)
• jumpstarter-all (meta-package)

~17 e2e config files reference jumpstarter_driver_opendal.driver.MockStorageMux

Direct opendal imports (not through driver-opendal wrapper):

• jumpstarter-driver-tftp/server.py: Uses AsyncOperator directly for file serving
• jumpstarter-driver-flashers/client.py: Uses opendal.Operator and opendal.Metadata directly
• jumpstarter-driver-http/client.py: Uses opendal.Operator directly
• jumpstarter-driver-iscsi/client.py: Uses opendal.Operator directly
• jumpstarter-driver-ridesx/client.py: Uses opendal.Operator directly
• jumpstarter-driver-probe-rs/client.py: Uses opendal.Operator directly

Key challenges

1. opendal is used as a generic storage abstraction on BOTH sides: The Operator class is used on the client side for local filesystem access and HTTP downloads, not just as a pass-through to the remote driver. Functions like operator_for_path() create local Operator("fs", root="/") or Operator("http", ...) instances.

2. The OpendalAdapter is the critical integration point: It handles presigned URL detection (operator.capability().presign_read), presigned URL generation, and streaming fallback. This is the component mentioned in the issue title.

3. TFTP server directly uses AsyncOperator: The TFTP server implementation reads files using await self.server.operator.open(filepath, "rb") and await self.server.operator.stat(filename).

4. The driver package provides abstract interfaces: FlasherInterface, StorageMuxInterface, etc. are imported by many packages. These need to be preserved or relocated.

Recommendation

This change is too large and risky for a single PR. I recommend breaking it into phases:

Phase 1: Create a lightweight storage abstraction (new package or within driver-opendal)

• Create simple FileOperator / AsyncFileOperator classes that wrap Python's built-in file I/O (aiofiles, pathlib, os) for local filesystem operations
• Create an HTTP client wrapper using httpx or aiohttp for HTTP operations
• Keep the same API surface as the current Operator usage patterns

Phase 2: Replace client-side opendal usage

• Replace Operator("fs", root="/") with the new filesystem abstraction
• Replace Operator("http", ...) with the new HTTP client wrapper
• Update OpendalAdapter to use the new abstractions while preserving presigned URL support
• Update operator_for_path() function

Phase 3: Replace driver-side opendal usage

• Replace AsyncOperator in the Opendal driver class
• Update TFTP server to use native async file I/O
• Update AsyncFileStream to wrap native file objects

Phase 4: Remove opendal dependency

• Remove opendal from all pyproject.toml files
• Rename package if desired (the package name jumpstarter-driver-opendal itself references opendal)
• Update e2e configs and documentation

Presigned URL preservation

The current presigned URL flow:

1. Client calls operator.capability().presign_read to check if the backend supports it
2. If yes, calls operator.to_async_operator().presign_read(path, expire_second=60) to get a presigned URL
3. Returns a PresignedRequestResource which the exporter-side _resource_from_presigned() method (in jumpstarter/driver/base.py) downloads using aiohttp

The exporter-side download via aiohttp is already independent of opendal — it's in the core jumpstarter package. The change needed is on the client side: replacing opendal's presigned URL generation. For S3-compatible backends, this would require boto3 or a lightweight S3 signing implementation. For other backends, it depends on what storage backends need to be supported.

[mangelajo]

1. I don't like the current operator patterns, it over complicates things. So I think we should not try to replicate what opendal does and simplify. I.e. , When a pre-signed content must be downloaded at the other side, just pass that request to download the item, etc.

2. I agree that it's a very big change, we should probably start with one single driver, for example, the qemu driver, which is well isolated and testable.

3. I agree we should offer common structures, and driver interfaces for uploading/reading/etc files in drivers (like in HTTP/TFTP storage), we need to maintain those interfaces for backwards compatibility. But let's focus first on the qemu driver as a first step.

Based on those ideas, can you make a plan?

[ambient-code]

Revised plan: QEMU driver first, simplified approach

Based on @mangelajo's feedback, here is a focused plan starting with the QEMU driver as the pilot.

How the QEMU driver uses opendal today

The QEMU driver (jumpstarter-driver-qemu) has a narrow dependency on jumpstarter-driver-opendal:

1. Driver side (driver.py): Imports only FlasherInterface from jumpstarter_driver_opendal.driver. QemuFlasher extends FlasherInterface and implements flash() / dump() using self.resource(source) — which is a method from the core jumpstarter Driver base class, not from opendal. The actual flash logic uses anyio.streams.file.FileWriteStream / FileReadStream to write to local disk images. No Operator, AsyncOperator, or opendal API is called directly in the driver.

2. Client side (client.py): No opendal imports at all. QemuClient extends CompositeClient and does not reference FlasherClient, OpendalAdapter, or Operator. The flasher child is accessed via qemu.flasher which resolves to whatever client class FlasherInterface.client() returns — currently jumpstarter_driver_opendal.client.FlasherClient.

3. Tests (driver_test.py): Uses opendal.Operator("http", ...) directly to construct an HTTP operator for downloading Fedora images via presigned-URL-like flow. This is the only direct opendal import in the test.

4. pyproject.toml: Lists jumpstarter-driver-opendal as a dependency.

What the QEMU driver actually needs from opendal

• FlasherInterface: An abstract base class with flash() and dump() abstract methods, plus a client() classmethod returning "jumpstarter_driver_opendal.client.FlasherClient".
• FlasherClient: Client-side implementation that uses OpendalAdapter to handle file transfer (local fs via Operator("fs") or HTTP via Operator("http")) and presigned URL generation.
• OpendalAdapter: The bridge that checks operator.capability().presign_read, generates presigned URLs if supported, or falls back to streaming file content.

Proposed plan for the QEMU driver

Step 1: Create a lightweight flasher abstraction in a new or existing package

Create common flasher interfaces and a simple client that does not depend on opendal. The key insight is that FlasherInterface itself has zero opendal imports — it is pure ABC. The complexity is in FlasherClient and OpendalAdapter.

For the QEMU driver specifically, we only need:

• Local filesystem reads (for flashing a local .qcow2 file)
• HTTP downloads (for fetching images from URLs like the Fedora mirror)
• Presigned URL pass-through (the exporter side already handles presigned downloads via aiohttp in jumpstarter/driver/base.py._resource_from_presigned() — this is opendal-free)

Step 2: Replace FlasherClient with a simplified implementation

Instead of routing through opendal's Operator abstraction, the new client would:

def _flash_single(self, image, *, target, compression):
    path = Path(image)
    if str(image).startswith(("http://", "https://")):
        # For HTTP sources: pass a PresignedRequestResource directly
        # The exporter already knows how to download from presigned URLs
        resource = PresignedRequestResource(
            url=str(image), method="GET", headers={}
        ).model_dump(mode="json")
        return self.call("flash", resource, target)
    else:
        # For local files: stream via client resource (existing mechanism)
        async with FileReadStream.from_path(path) as stream:
            async with self.resource_async(stream, content_encoding=compression) as handle:
                return self.call("flash", handle, target)

This eliminates the Operator / OpendalAdapter layer entirely. The exporter-side self.resource() in Driver.resource() already handles both ClientStreamResource and PresignedRequestResource natively.

Step 3: Move FlasherInterface (and StorageMuxInterface, etc.) out of jumpstarter-driver-opendal

These ABCs should live in a common location — either in the core jumpstarter package or in a new lightweight jumpstarter-driver-common package. This preserves backward compatibility for all 13 packages that import these interfaces.

Step 4: Update QEMU driver

• Change from jumpstarter_driver_opendal.driver import FlasherInterface to import from the new location
• Remove jumpstarter-driver-opendal from pyproject.toml dependencies
• Update test to use httpx or PresignedRequestResource directly instead of Operator("http", ...)

What this preserves

• Exporter-side presigned downloads: Fully preserved. The _resource_from_presigned() method in jumpstarter/driver/base.py uses aiohttp directly and has no opendal dependency.
• Progress reporting: ProgressStream wrapping in _resource_from_presigned() continues to work.
• Backward compatibility: Other drivers that still need full opendal functionality continue to use jumpstarter-driver-opendal unchanged.

What changes for other drivers (future phases)

The pattern established with QEMU can be replicated for other drivers. The main categories:

• Simple flasher drivers (esp32, pi-pico, probe-rs): Same pattern as QEMU — replace FlasherClient with simplified version
• Storage drivers (http, iscsi, tftp): These use opendal more deeply (e.g., TFTP server uses AsyncOperator for file serving). These need more careful replacement with native async file I/O
• Full storage abstraction drivers (flashers, ridesx): Use Operator on client side for local fs + HTTP. Replace with native pathlib + httpx

Estimated scope for QEMU-only change

• ~4 files modified in jumpstarter-driver-qemu
• ~1-2 new files for relocated FlasherInterface (or modification to core jumpstarter package)
• No changes to core jumpstarter package's resource handling
• No changes to other driver packages

[ambient-code]

Implementation Plan for QEMU Driver

Starting implementation now. Here are the specific changes:

1. Move FlasherInterface to core jumpstarter package

• Create jumpstarter/driver/flasher.py containing FlasherInterface (pure ABC, no opendal imports)
• Export it from jumpstarter/driver/__init__.py
• Add backward-compatible re-export in jumpstarter_driver_opendal/driver.py

2. Create simplified FlasherClient without opendal

• Create jumpstarter/client/flasher.py containing FlasherClient and FlasherClientInterface
• For local files: stream via existing resource_async mechanism using anyio.open_file
• For HTTP URLs: pass PresignedRequestResource directly (exporter already handles presigned downloads via aiohttp)
• No opendal dependency needed — uses pure anyio/aiohttp for file streaming
• Add backward-compatible re-export in jumpstarter_driver_opendal/client.py

3. Update FlasherInterface.client() to point to new location

• FlasherInterface.client() will return "jumpstarter.client.flasher.FlasherClient" instead of "jumpstarter_driver_opendal.client.FlasherClient"

4. Update QEMU driver

• Import FlasherInterface from jumpstarter.driver instead of jumpstarter_driver_opendal.driver
• Remove jumpstarter-driver-opendal from pyproject.toml dependencies
• Update test to not use opendal.Operator — use local file paths and HTTP URLs directly

5. Test backward compatibility

• The opendal package's FlasherInterface will re-export from the new core location
• Existing drivers (esp32, pi-pico, etc.) that import from opendal will continue to work unchanged