Bonsai freezes and cannot stop workflow (or Bonsai) normally

[ZegCricket]

Hello! I hope you are doing well.

My main Bonsai project consists on a sound lateralization task. It's nothing new, but the idea is to port the current setups to these ones centered around Harp and Bonsai. With that said, the project has already grown quite big, since it uses a lot of different hardware, it has a lot of constraints and configurations the user can pick from and it has a small GUI to display real-time data and interact with the task. The project is not finished by any means, but it's already possible to use in experiments, and it's in fact being used.

One issue that has been happening for a few months now is that the Bonsai workflow freezes and to stop it, I have to close the Bonsai window and kill the process. I've already tried a lot of stuff and honestly I don't know where to look at (or what to look at), so I'll just describe everything that I've tried and everything that I know about the setup, so I apologize in advance for this being a super long discussion.

The first thing I should mention is that this is not a consistent issue. It may happen 3 times in a setup in one day and then the next day it may not happen at all, with the exact same computer model (and remaining hardware) it may happen more in a setup than in another. The fact that it doesn't happen everytime makes it hard to find a cause and effect relationship.

The second thing is that the issue is not explicit. What I mean by that is that there's no error message, the camera keeps recording and I can still give reward to the animals (I could probably do a bunch of other things, but this is a thing I can do with the GUI). However, it's like the main task logic is not working anymore (and the fact that I indeed have to kill the process).

I've already tried to improve my Bonsai code, since I started the project a year ago and since then I think I improved my Bonsai knowledge. I tried to take the unnecessary logic out of the state machine, simplifying some of the logic, using some tricks I've learned along the way.

The majority of the reports I received related to this issue coincide with the end of the state machine and everytime I came across this issue it happened at the end of state machine. The best I could do regarding understanding what was happening was to see the final state send an event that is propagated to the last node, but for some reason the Repeat node doesn't trigger the resubscription process. I know that because when that happens the first node of the workflow (Timer with the properties at 0 - so it sends an OnComplete after sending an event) doesn't send the event it would send if the resubscription process happened. To make sure that all states are terminated, every state has a Take 1 in the end, so in principle the stream leading to the Repeat is more than terminated.

I'm using Bonsai 2.8.2 and regarding hardware, I use a PointGrey/FLIR camera and the following Harp boards:

• Harp Behavior
• Harp SoundCard
• Harp SyringePump (2 per setup)

I've also set Windows configurations to use Best Performance and to never sleep/hibernate.

However, when I need to develop something, I usually don't use the camera or the SyringePumps and it still may happen, so I doubt it's an issue related to any of these pieces of hardware. If it's helpful, you can find the documentation for the project here.

Thank you in advance for your help and let me know if there's something else I can explain/provide so that issue gets solved!

[bruno-f-cruz]

First of all nice project organization! Awesome to see people using Bonsai.Sgen and pydantic for experiments :)!

So I can help you, is there a minimal workflow that reproduces the issue?

[bruno-f-cruz]

Hey. I took a quick look but the workflow is indeed already quite complex. I didnt see anything super obvious. However, one thing that you may want to refactor to make this easier to debug, is to scope your subjects only to the place you need them. Right now, the whole state of the workflow is "global" and understanding a local subject requires you to transverse the whole workflow. Remember subjects have scope that you can control using nested operators like Defer, SelectMany, CreateObservable, etc...

Here's where I would start:

1. Is it the case that the bug occurs when people use the UI? From what I see in your workflow, this is the only thing that may "inject" an unpredictable event in the state machine are the UI elements.
2. Look at the logging. Can you tell what is the last event logged? When bonsai crashes, can you still see data coming in from all devices (e.g. Harp).
3. Create a memory dump of the crash, and read it offline to see where the deadlock is coming from. (https://learn.microsoft.com/en-us/visualstudio/debugger/using-dump-files?view=vs-2022). These files are sometimes hard to parse, but most of the deadlock conditions I have found in Bonsai were solved this way.

[ZegCricket]

First of all, sorry for taking so long to reply!

At some point, my goal was to take the majority of the task's logic outside of the state machine. That's how I got to the current organization of nested GroupWorkflows which requires me to use these global subjects to interact between the state machine and the task's logic. How could I explore a sort of middle ground?

1. I don't think so. The issue already happened before I created the UI and it's supposed to be used more for monitoring than for control.
2. I'm not sure, but I'll keep an eye on it on the next times it happens!
3. Thank you for the suggestion, I'll try to make it work.

Thank you once again for trying to help me! I'll try your suggestions in the next few days and hopefully update you on how it's going :)

[ZegCricket]

Hello again! I was able to see the issue a few more times and tried to pay attention to the Harp events. I used a minimal setup that only uses the Harp Behavior and the Harp SoundCard. I don't get any events from the SoundCard, but the last event from the Behavior is from address 32 (DigitalInputState) and corresponded to me poking in one of the lateral pokes (to give an answer). I haven't received any other events after that (which would be expected given the AnalogData register collects data at 1 kHz).

I'm also trying to use the memory dumps, but I'm still figuring it out.

[banchan86]

Hey @ZegCricket, I'm not sure if you've managed to make progress on this. One other idea I had was that in Bonsai 2.9, you can use the new Watch feature to try and figure out where the error is coming from. I know the workflow is still on an old bonsai version, but maybe you can test bits of it out in Bonsai 2.9 instead of migrating the whole thing over.

[ZegCricket]

Hey! I think I was able to make it not happen again (yet), but I don't want to risk saying it's fixed. I will try to explain what I think was happening and what I changed, so that someone can try to explain me in detail what was happening under the hood in order for me to avoid to make a similar mistake in the future (I only have an intuition, but don't have a formal explaination for it), so I apologize for possibly making this comment too long.

I mentioned that when this issue happened, what seemed to be happening was that the main workflow was able to reach the end of it, but for some reason the Repeat node would not trigger. I was straight up wrong about this. After some time trying to debug the workflow I was able to find that the origin of the problem was in the Decision > LNPTime SelectMany. This node basically checks whether the animal stayed enough time in one of the lateral pokes in order for the answer to be considered valid (otherwise, the trial will be considered an abort). The code inside this node was way too complicated for its purpose, as you can see below:

I won't get into the details of this workflow, but I will mention 2 things:

• This node should be executed in a matter of milliseconds;
• The Timestamp subject (found at the left of the workflow) corresponds to the Harp timestamp given by the AnalogData register of the Harp Behavior, which receives events at 1kHz. This way, I can timestamp everything (hardware and software events) with the Harp timestamp.

With these 2 things in mind, what I think was happening is that, since for every execution of this node, the LNPStart subject only sends a single event, the values received from the Timestamp subject will accumulate since it preceeds a Zip node. For some reason, I think this causes the code related to the Harp Behavior to close and when the workflow gets to a point where it needs a value from the Timestamp subject, the workflow will stop its execution.

So, I refactored this node in a way I can get rid of this Timestamp node and that, at the same time, leverages the Harp hardware.

In hardware, I short circuited the DO with the DIO of a PortBreakout (which is used to connect the poke to the Harp Behavior), so that when the animal answers, the DO triggers a pulse (which will be detected in the DIO). This way, the code inside the new LNPTime node just checks whether the animal stops poking before the pulse is over. If the animal leaves the poke before the pulse ends, the trial is aborted, otherwise the answer will be considered valid. I do something similar during fixation time.

After this change, the issue stop happening. My intuition tells me it was related to the Timestamp subject + Zip combo. Does it makes sense? If yes, can someone give me a more formal explaination?

Feel free to ask me any questions you have.

Thank you for your help so far, @bruno-f-cruz and @banchan86! :D

[adaptive-control]

Personally, I have exactly the same type of struggles with a complex setup making use of two Behavior boards, in which I am also implementing a condition checking how long an animal is poking before updating the state of the task.
@ZegCricket are you still happy with your solution? Do you consider the issue fixed? Because my workflow is quite different, but I have also this issue where I suspect a problem coming from the Behavior board nodes.
Romain

[bruno-f-cruz]

Have you tried to debug the program using the suggestion above?

Create a memory dump of the crash, and read it offline to see where the deadlock is coming from. (https://learn.microsoft.com/en-us/visualstudio/debugger/using-dump-files?view=vs-2022). These files are sometimes hard to parse, but most of the deadlock conditions I have found in Bonsai were solved this way.

I would be curious to look at this. In your case, how often does the workflow crash?

[ZegCricket]

Hi! The solution helped a little bit, but it's far from being the end of the story. At the moment, we have 10 setups running 3 sessions of 2 hours each per day (sometimes 4 sessions), 5-6 days a week. Additionally, we have 2 more setups that run 1-2 sessions a day. I can say that since the beginning of 2026, we have experienced around 6-7 freezes in total, which I would say is pretty tolerable given the amount of sessions we have.

With that said, what I think that was causing the majority of the freezes was when I was trying to timestamp software events with the Harp timestamp (via the WithLatestTimestamp node - with basically is a WithLatestFrom with the Harp timestamp as the second input), specially when these happen at the beginning of a new state (i.e. SelectMany). The reason for this would be that if the event that I would like to be timestamped arrived at the WithLatestTimestamp before the Timestamp emitted a new event, the workflow would not advance. It may not explain every freeze, but I was able to caught as a source of freezes. Why does this causes Bonsai to freeze completely and not just the workflow? I have no clue. A workaround is to add a SubscribeWhen at the beginning of the SelectMany with the Timestamp subject as the second input, this way we make sure we have Timestamp events. I don't know if this applies to your workflow at all, but it was just to update on my situation.

Now, these freezes are becoming way more frequent than one would expect. In the past month, I heard of 2 other independent groups that are experiencing this issue, which makes me wonder whether it's us as users that are doing something dramatically wrong, but if that's the case, doesn't this make the already steep learning curve of Bonsai even steeper?

@bruno-f-cruz, thank you for the suggestion! I remember that I tried to follow it back when I started this discussion, but the truth is that I didn't know what to look for or how to interpret the dump files or even if I was generating the right thing in the first place. Could you elaborate a bit on how this could help debugging the deadlock?

Lastly, @adaptive-control, to answer your question, I don't consider the issue completely fixed and I'm not sure if I ever will, but for me that's not the point at this point. I see it more as a matter of attenuating the issue as much as possible. Nonetheless, I have spent so much time debugging this issue that maybe I can help you, in case you want to share your workflow. :D

[glopesdev]

Why does this causes Bonsai to freeze completely and not just the workflow? I have no clue.

@ZegCricket Well done researching and sharing this diagnosis. I can see how this particular pattern is very difficult to debug.

Unfortunately, all individual operators are doing exactly what they should.

The behaviour of WithLatestFrom is to ensure only the first source sequence raises notifications. If that sequence has a single event, and that event is delayed relative to the second, indeed the sequence will complete immediately, as there was never any state available for pairing.

The Repeat node will resubscribe the entire sequence when it completes successfully. Together with the above behaviour, this will cause an infinite loop in the state machine as you described.

I can see two possible recommendations emerging here:

1. If you know the source sequence has only one event and you don't mind the timestamp might be assigned in the future relative to the event occurrence, then you can use CombineLatest > Take(1) instead. CombineLatest will guarantee that a pair is generated as soon as notifications have been emitted by both sequences.

2. To avoid "silent" failures, you can use First instead of Take(1) at the exit of your states. This will ensure errors are raised if states complete with an empty sequence.

My only reservation with using First is that it unsubscribes from the source later than Take(1), and this has itself created infinite loop issues in the past.

Perhaps with the upcoming trace feature it will be easier to debug this. Unfortunately it is impossible in general to detect occurrence of infinite loops in programs, so there is no way to comprehensively detect these situations.

[ZegCricket]

@glopesdev, thank you for the clarification!

As I mentioned before, despite this being a big source of these infinite loops, it seems that it's not the only one. I'm currently trying to help a group which is struggling with these freezes as well. I was already able to move the WithLatestTimestamp nodes outside of the state machine, but the freezes keep happening. This setup uses a Harp LoadCells (so I'm using the LoadCells events as the timestamp source) and when a freeze happens you basically stop seeing new LoadCells events as if there was some problem with the connection with the device (although the workflow doesn't panic from this). It's not an issue with the firmware either, since I've already tried to run the workflow with the device in debug mode and when the freeze happens the device keeps working as it's supposed to. Additionally, when in a freeze, it's usually possible to interact with other parts of the workflow (example: delivering water from a GUI button).

With that said, I would like to ask if there are other known freeze sources, specially Harp-related freezes.

Thanks in advance!