feat: dspy - Enable DSPy with dynamic context (e.g. from RAG)

[tomasruizt]

Related Issues

• Related to DSPy Integration #1635

Context

The original DSPy + Haystack cookbook notebook demonstrates a RAG flow where a retriever dynamically looks up the context for each question. With the recently merged DSPySignatureChatGenerator (#2831), this exact use case is not yet possible inside a Haystack Pipeline:

• run() declares only messages and generation_kwargs as input sockets.
• Signature fields like context live behind **kwargs, so Pipeline.connect() cannot bind anything to them.

The net effect: today the chat generator only works with static, hardcoded context (as in test_run_with_input_mapping and test_live_run_with_multi_field_signature). A retriever cannot dynamically supply question-dependent context.

Proposed Changes

Add a pipeline_inputs: list[str] parameter to DSPySignatureChatGenerator. When set, the listed signature input fields are registered as real Haystack input sockets via component.set_input_type(self, name, str, ""). This unlocks the pipeline input/output discovery mechanism, so a retriever can be wired directly to the generator and the context is resolved at run time, per question. See the new unit test test_rag_pipeline_question_dependent_context for a concrete usage example.

The change should be backwards compatible: when pipeline_inputs is None, the existing behavior is unchanged. Let me know if this PR makes sense in your opinion @anakin87 @julian-risch

How did you test it?

• Added a unit test test_rag_pipeline_question_dependent_context that wires InMemoryBM25Retriever to the DSPySignatureChatGenerator inside a Pipeline and asserts the retrieved context reaching the (mocked) DSPy module is question-dependent.
• hatch run test:unit — 39 passed
• hatch run test:types — no issues
• hatch run fmt — clean

Checklist

• I have read the contributors guidelines and the code of conduct
• I added unit tests and updated the docstrings
• I've used a conventional commit type for the PR title

AI disclosure: Claude Code was used.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

Coverage report (dspy)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/dspy/src/haystack_integrations/components/generators/dspy/chat
chat_generator.py
Project Total

This report was generated by python-coverage-comment-action

[anakin87]

I'll take a look in the next few days...

[anakin87]

@julian-risch I'd also appreciate your opinion