Dump collection fails if runtime exceeds BMCWeb timeout (attachment lost)

[dkamat27]

Description

When using phosphor-debug-collector, if the dump collection takes longer than the BMCWeb dump collection task timeout, the request fails and the attachment is lost.

Example:

• Dump involving ipmi sel list took ~84 minutes.
• BMCWeb’s dump collection task times out at 45 minutes.
• Result: the dump completes in the backend but the request fails and no attachment is returned.

---

Impact

• Long or slow dump collections cannot be retrieved.
• Users experience failed dump retrieval even though the dump finishes successfully.
• This results in loss of important debug data for customer systems.

---

Expected Behavior

• Dump collection should succeed and the attachment should be retrievable, regardless of runtime.

---

Open Question

Should this be addressed by:

1. Increasing the BMCWeb timeout, or
2. Or any other design?

[ojayanth]

After reviewing the behavior and architecture and based on your inputs it appears that the core functionality of dump creation is working correctly in phosphor-debug-collector. The dump is eventually created and visible in the system once completed. However, the failure occurs due to a timeout in BMCWeb’s task tracking, which currently defaults to 5 minutes. This timeout is shorter than the actual dump collection time (e.g., 84 minutes in your example), leading to a failed HTTP response and no attachment being returned to the client.

Clarifications:

• The dump is not lost. It is still created and can be retrieved manually or via Redfish once completed.
• The issue lies in task timeout handling in bmcweb, not in the dump collection logic itself.
• Therefore, this issue would be more appropriately tracked under the openbmc/bmcweb repository.

Suggestions:

• Verify whether the dump entry is created and downloadable after completion.
• Evaluate whether the 84-minute duration is expected for the specific dump type (e.g., IPMI SEL list). If not, consider optimizing the backend collection logic.

[dkamat27]

Yes, the dump itself hasn’t failed, but the problem is that once the BMCWeb task times out, we don’t know when the dump will actually complete. The event log entries may eventually show the new dump, but that requires constant polling and doesn’t tell us the exact completion time.

So I wanted to ask for suggestions — do you think it makes sense to tie the timeout to a completion signal from phosphor-debug-collector, instead of a fixed value in BMCWeb? For example, in the case of ipmi sel list, we are still investigating why it took so long (~84 mins), but the same issue could happen with other long-running commands. How can we ensure the dump collection gets a proper timeout or async handling in such scenarios?

[mdmillerii]

The Redfish way is to create a task for things that are expected to take longer than an HTTP request response timeout.

In OpenBMC and bmcweb tasks used to be tied to code update. While that may no longer be true I am not aware of any additional usage.

[ojayanth]

@dkamat27 the dump itself is not failing, but the core issue is that BMCWeb’s task tracking mechanism times out before the dump completes, leading to a misleading failure response to the client.

From a debug data collection design perspective, user-initiated dumps are intended to capture a snapshot of the system state at or near the time of issue detection. The assumption is that the system is still responsive and that the data can be collected within a short, bounded time window — typically under a few minutes. Taking significantly longer (e.g., 84 minutes) undermines this design goal, as:

• The system state may have changed significantly by the time the dump completes.
• The user may no longer be waiting for the result.
• The collected data may no longer be relevant for root cause analysis.

[dkamat27]

Hi @ojayanth just one more thought what if instead of relying only on the overall BMCWeb timeout, we added a per-script timeout inside phosphor-debug-collector dreport so each script would exit after a defined duration? That way we avoid a single long-running command blocking the whole dump. Do you think this would be a good design approach, or could it cause unintended issues?

[rohitpai]

From a customer perspective we need to make sure the dump never fails. Meaning we need to provide the dump within specified timeout in bmcweb. Dump is only way to triage issue in field so if this stops working then we are lost. There is no ssh for further triaging.

So resiliency goal for dump must be - it will always work within the timeout period.

On healthy systems the dump will work always but we need to think about systems that have failed due to various conditions. These error conditions can cause time delay in the plugin scripts. This brings the issue of when faults happens we cant make sure the scripts complete within the expected timeout.
Hence I feel we need to have mechanism to time check on each plugin script and abort the script its taking longer duration.
It's good to have partial data collected from these faulty system then no data collected at all because the dump did not work.

So lets align on the need of this before we find out what is the best way to accomplish this.
@ojayanth - please share your thoughts.

[ojayanth]

A key design point for the dump plugin is to capture data quickly to avoid timeouts. In most cases, data collection relies on reading files or executing busctl commands, which typically complete within the default D-Bus timeout window.

I fully agree that if any existing plugin is known to require longer execution time, we should start by implementing plugin-specific timeout handling within that plugin. This approach avoids sweeping changes to the core system and helps maintain modularity.

Please feel free to share patches — happy to review and collaborate!

[dkamat27]

Hi @ojayanth what are your opinion's on putting the timeout on each scripts? what i think on this approach is that the timeout may let the loss of data in some positive case which can be collected otherwise also for different platforms the timeout which may become harder the maintains as the platforms grows?

[ojayanth]

I checked the current plugin list under tools/dreport.d/plugins.d, Not seeing "ipmi sel list" . can you point me the plugin , which is causing the failure.

Regarding the timeout behavior: since each plugin either performs a file copy or invokes a D-Bus command, both of which inherently have their own timeout mechanisms, the messaging timeout is indeed plugin-specific. If a plugin requires a custom timeout adjustment, it should be handled within that plugin's logic.

Feel free to push a commit that introduces the necessary timeout changes for the plugin in question.

[edtanous]

I found this in a random search and wanted to clear some things up. The bmcweb 5 minute timeout is a timeout BETWEEN progress updates. So long as the service in question is sending progress updates (which it should be doing every 30 seconds at the latest) bmcweb will not kill the task. If we're seeing failures here, it means that dump collection is stalling or not updating the progress API, which might make sense if there's no progress interface from the bash. That needs corrected on the phosphor-debug-collector side.

[ojayanth]

I found this in a random search and wanted to clear some things up. The bmcweb 5 minute timeout is a timeout BETWEEN progress updates. So long as the service in question is sending progress updates (which it should be doing every 30 seconds at the latest) bmcweb will not kill the task. If we're seeing failures here, it means that dump collection is stalling or not updating the progress API, which might make sense if there's no progress interface from the bash. That needs corrected on the phosphor-debug-collector side.

Agree with the concern raised—currently, user-initiated BMC dumps via phosphor-debug-collector lack robust progress and timeout handling, especially when backend scripts like dreport hang or take too long.
To address this, Need look hybrid solution:

• Recommended Enhancements:

• Progress Updates: As recommended Emit D-Bus progress signals periodically (e.g., every 30 seconds) during dump creation to reflect activity.

• Configurable Timeouts: Add timeout logic in the C++ layer with platform-configurable values to detect stalled executions and update status to Failed.

• Selective Script-Level Timeout (Optional): For known long-running or unreliable scripts, allow optional wrapping with shell-level timeout, but avoid enforcing this globally to reduce integration effort.

[cvam0000]

I want to propose a design for this issue :

Three-Layer Timeout Protection System

LAYER 1: Dynamic Per-Plugin Timeout
Detects and handles individual hung or slow-running plugins before they consume all available time. Implements dynamic timeouts that decrease as collection progresses: 30 minutes for early plugins, 25 minutes after 5 minutes elapsed, and 20 minutes after 10 minutes elapsed.

LAYER 2: Fallback Collection
When any plugin times out, immediately switches to collecting only 10 critical diagnostic plugins: bmcstate, fw_version, os_version, checkstop, gard, pels, systemd, eventlog, proc_info, and network_info. These run WITHOUT per-plugin timeouts, only limited by the Layer 3 global timeout. eg : journal , dmesg , disk usage , all states etc( essential for debug most of the issues)

LAYER 3: Graceful Shutdown with Data Preservation
When the 45-minute overall timeout is reached, sends SIGTERM to allow dreport to package collected data, waits 30 seconds for graceful completion, then sends SIGKILL only if still running. This transforms timeout from data-loss to data-preservation.

Benefits
Zero data loss on timeout
No dumps are ever discarded.
Guaranteed minimum data from critical fallback plugins.