v1.18.4

• Fix server-side bug that would prevent forkableHub from correctly updating metrics when receiving partial or out-of-order blocks

v1.18.3

Server

• Add optional 'secret key' authentication between tier1 and tier2 services.
• Fixed a case where a nil pointer exception could happen on storage error(s).
• Substreams client library will no longer forcefully strip authentication from plaintext connections.
• Substreams tier1 will now retry failed tier2 jobs that stream data directly if they have not produced any data yet.
• Substreams worker jobs will always retry tier2 jobs that return with codes.Unavailable: no healthy upstream, considering this error as load-balancer version of codes.ResourceExhausted.
• Substreams will reject requests with stores when expected total memory usage from stores alone are above 45% (down from prev. 75%)
  You can set SUBSTREAMS_TOTAL_STORE_SIZE_LIMIT_PERCENT=75 to go back to previous behavior or SUBSTREAMS_ENFORCE_TOTAL_STORE_SIZE_LIMIT=false to disable the feature.

v1.18.2

Sink

• Fix InferOutputModuleFromPackage sentinel value causing error when used with manifests containing a networks: section. The sentinel value was being passed to the manifest reader before resolution, causing a "could not find module @!##InferOutputModuleFromSpkg##!@ in graph" error.

v1.18.1

Server-side fixes

• Fix V4 buffer flushing on end-of-file: the buffer is now properly flushed when the last block in a segment is reached
• Fix error handling for InvalidArgument and other validation errors: they were shown as Unknown
• Fix partial blocks output in V4: prevent NewPartial and UndoPartial cursor steps from being exposed to clients (normalized to New/Undo respectively)
• Fix partial block data to be correctly wrapped in BlockScopedDatas on v4.
• Last partial blocks are now accepted interchangeably with new blocks and vice-versa, allowing faster full blocks for requests that do not ask for partial blocks.

v1.18.0

CLI

• Improved substreams auth command to better guide new users by showing the registration link alongside the authentication link. The command now also accepts API keys directly and automatically exchanges them for JWT tokens.

• Added support for buf.build commit references in descriptor set module versions. In addition to semantic versions (e.g., v1.2.3), you can now use buf.build commit hashes (32 lowercase hex characters, e.g., @f3ab5976b9ba4f9bac28b26271fca7d7). Commit references are also cached since they are immutable.

• Fixed protogen hash caching behavior when descriptor sets don't have a pinned version. Previously, when using descriptor sets without a version (resolving to "latest"), the .last_generated_hash file would cache incorrectly and skip regeneration even when the remote content had changed. Now:

  • Descriptor sets without a deterministic version (semver or commit ref) will always trigger regeneration
  • A warning is emitted listing which descriptor sets need pinned versions
  • The hash file is removed when non-deterministic descriptor sets are present to prevent stale caches

Server

• Improved 'partial blocks': support new pbbstream's "LastPartial" field, fix 'undo' scenarios for stores

Performance (RPC V4)

This release introduces significant performance optimizations to the Substreams gRPC communication layer. See RPC Protocol Reference for details.

• RPC V4 protocol with BlockScopedDatas batching: The new V4 protocol batches multiple BlockScopedData messages into a single BlockScopedDatas response, reducing gRPC round-trips and message framing overhead during backfill.

• S2 compression (new default): S2 compression replaces gzip as the default compression algorithm. S2 provides ~3-5x faster compression/decompression than gzip with comparable compression ratios. The client automatically negotiates compression with the server.

• VTProtobuf fast serialization: Both client and server now use vtprotobuf for protobuf marshaling/unmarshaling, providing ~2-3x faster serialization with reduced memory allocations.

• Server-side message buffering: Configurable via OutputBufferSize flag (default: 100 blocks) or MESSAGE_BUFFER_MAX_DATA_SIZE environment variable (default: 10MB).

• Automatic protocol fallback: Clients gracefully fall back V4 → V3 → V2 when connecting to older servers.

• Improved Connect/gRPC protocol selection: Server now efficiently routes requests to the appropriate handler based on content-type, improving performance by ~15% for pure gRPC clients (previously all requests went through Connect RPC layer).

Sink

• Updated sink library to leverage RPC V4 protocol with BlockScopedDatas batching, improving throughput by reducing per-message processing overhead.

v1.17.11

Server

• Fix issue where a retry on dstore while writing a fullKV would corrupt the file, making it unreadable. Fix prevents this and also now deletes affected files when they are detected.
• Fix bug in event loop where loop.NewQuitMsg() (which returns *QuitMsg pointer) was not being handled, causing quit messages from error paths to be silently ignored and requests to hang indefinitely.
• Fix issue where transient HTTP/2 stream errors (e.g., INTERNAL_ERROR) from dstore were being treated as fatal errors instead of being retried. These transient network errors are now detected and retried with exponential backoff. Added comprehensive diagnostic logging including working duration tracking and ERROR-level alerts when walker is stuck for more than 5 minutes.

Sink

• Fix progress_running_jobs not resetting its counter when reaching 0 jobs running on the server.

v1.17.10

Server

• Added bucketed prometheus metrics head_block_relative_time_sum{app=substreams_output} that shows latency between outputing live blocks and their blocktime.
• Fixed underflow in 'FailedPrecondition desc = request needs to process a total of x blocks' error when running from 'substreams run' with a start-block in the future.

CLI

• Fix parsing of params with modules derived with use: now allows reusing the same modules with different params
• Fix substreams pack for a substreams.yaml where no modules are defined, but a SinkConfig points to an imported module (force modules inclusion)
• The substreams init command will now list generators in the server's specified order (instead of randomly display them).
• The substreams init has now proper error handling when a file cannot be uploaded to the server.
• added substreams sink protojson (migrated from https://github.com/streamingfast/substreams-sink-files)

v1.17.9

v1.17.9

Server

• Fix issue where "live backfiller" would not create segments after reconnecting with a cursor starting from a previous quicksave, causing delays in future reconnection
• Prevent "panic" when log messages are too large: instead, they will be truncated with a 'some logs were truncated' message.
• Raise max individual log message size from 128k to 512k
• Raise max log message size for a full block from 128k to 5MiB
• Reduce log level from Warn to Debug when we fail to get or set the store size (for backends that don't support it)

CLI

• Added --bytes-encoding flag to run and gui, accepted values: ['', 'hex', 'base58', 'base64', 'string'] (default: '' still auto-detects from network)

Partial blocks (experimental)

• Removed PartialsData message and brought back this data inside the good old BlockScopedData
• added the following fields to BlockScopedData:
  • bool is_partial to indicate if this block is a partial block. The following two fields are only present when is_partial==true
  • optional bool is_last_partial to indicate if this is the last partial of a given block (with correct block hash)
  • optional uint32 partial_index to indicate the index of this partial block within the full block
• renamed --partial_blocks_only flag to partial_blocks on substreams Blocks request
• removed --include_partial_blocks flag from substreams Blocks request

v1.17.8-joinedpartials

Changes to experimental partial blocks:

• a single flag: --partial-blocks
• removed 'PartialBlockData' message, now everything is in the original BlockScopedData
• added params to BlockScopedData:

• bool is_partial
• optional bool is_last_partial
• optional uint32 partial_index

Client will receive a mix of 'full' and 'partial' blocks in a stream, which they can manage as they wish (in some cases, existing implementation will "just work" by adding the 'partial-blocks' parameter to the substreams request, customers can also decide to add more partial-aware logic to their handlers.

v1.17.8

• Added experimental support for partial blocks (ex: Base's Flash Blocks) -- only supported on https://base-mainnet-flash.streamingfast.io endpoint
• See more details in the documentation

CLI

• commands run and sink webhook now support these flags:
  • --include-partial-blocks: sends every block as partial(s) but also as real block
  • --partial-blocks-only: only sends partials (for every block, every bit of data should be there)

Sink library

• To use the new partial blocks in a sink that uses github.com/streamingfast/substreams/sink, simply:
  • Define your sink flags with sink.FlagIncludePartialBlocks and/or sink.FlagPartialBlocksOnly under FlagIncludeOptional()
  • Implement the function HandlePartialBlockData(...) and pass it to NewSinkerFullHandlersWithPartial(...) when creating the sinker.

Server

• Server accepts new include_partial_blocks and partial_blocks_only boolean params in the request body.
• Response, when requested with above params, now include new message PartialBlockData, containing the usual "map module output", the clock and the index of that partial.
• Server accepts environment variable SUBSTREAMS_BIGGEST_PARTIAL_BLOCK_INDEX (default 10) -- it will emit a partial with this index when it gets the final part of a block.