The timeout bug — why three user positions hung 33 hours

A subscriber asked a simple question: "why are these positions on Binance still open after two days?". The strategy spec defined a maximum hold time well below that. The positions had been open 33 hours. The honest post-mortem on the bug, the fix, and what it cost us in trust — without exposing how the underlying model works, because that part is not for publication.

What we will and won't say

The signal model behind the trades — the firm's proprietary in-house quant + AI research — is not described here. Inputs, scoring, and the maximum hold the spec defines are deliberately not disclosed. What is on the table for this post: the execution-layer code path, the bug, and the fix. That part is reproducible engineering, not edge.

The discovery

Three long positions on a subscriber's account, opened in the same morning window. The strategy spec defines a maximum hold time, after which the position is force-closed at market regardless of P&L (because the underlying thesis decays past that point — by design, that horizon stays internal).

The customer noticed the positions still showing on Binance well past the intended close. They opened the conversation by saying "why are these still open?". That's the exact moment a quant firm earns or loses a customer. If we'd been the kind of shop that hand-waves and blames Binance, we would have been a different shop. We dug in.

The mechanism

Execution flow when a signal fires:

1. Detector emits a signal with coin, score, side, suggested entry
2. Executor fans out to (a) firm account, (b) all Pro subscribers with that strategy enabled
3. For each fanout target, place market entry + emergency SL on Binance
4. Register the position with positionMonitor for the strategy-specific timeout

Step 4 is where the bug lived. The executor handled firm and user trades on different code paths — firm trades got a positionMonitor registration; user trades just marked status=filled. No timeout registration for user accounts. Binance held the SL order, but if price never reached SL or any TP, the position would sit there indefinitely.

How it survived to production

The original execution path was firm-only. When we added per-user execution two months ago, we forked the path but didn't propagate the timeout-registration logic. The unit tests we wrote for per-user execution covered "did the order place correctly" and "did the SL register on Binance" — but not "does the spec'd timeout fire", because a meaningful test of that takes hours of wall-clock time to run end-to-end, and we leaned on firm-side production as the safety net.

Firm-side production never tripped the bug because firm positions had always closed via SL or TP within their window, by chance. The first time conditions lined up to keep a position alive past the spec'd timeout was on a user account — no registration, on a session where price stayed in a range that didn't touch SL or TP.

The fix

We added a per-strategy hold map in userPositionGuard.js with a per-tick force-close check. The actual hold values are part of the strategy spec and stay private. The mechanism is straightforward: every 60 seconds the guard scans open user trades, compares age against the strategy's spec'd maximum, and force-closes via reduceOnly market order when the threshold is crossed.

Deployed within 20 minutes of the customer report. The next tick caught all three stuck positions and closed them at market. Net P&L across the three was close to flat — the dollar cost of the bug was symbolic. The trust cost was the real one.

The real cost

A subscriber paying for a system that defines a maximum hold deserves a system that honors that hold, not one that drifts to 33 hours past it. We told the customer exactly what happened, deployed the fix while they were still on the line, and showed them the closed records with computed PnL the moment they appeared.

The customer kept their subscription. That's the right outcome only because we caught it within hours of the report. If the same bug had hit during a market crash and a position rode through a -25% drawdown because no timeout fired, the conversation would have been very different.

What we changed beyond the fix

• Test parity between firm and user execution paths. Any feature added to firm execution must have an equivalent test on user execution. No more relying on "firm production catches everything."
• External-close PnL computation. While debugging, we found that when Binance closed a position externally (SL fill, manual close), we were marking status=closed but not computing pnlPct/pnlUsd. Fixed in the same commit. Now every closed trade has full P&L data regardless of who triggered the close.

Why we publish this

The firm-vs-user-execution-parity bug is exactly the kind of thing other crypto-bot operators would hide. Most platforms run on closed-source code, opaque P&L attribution, and "trust us, the system works as advertised". When something goes wrong, customers eat it.

We'd rather publish the bug, the fix, and the lesson. The model itself stays in the vault — that part is the firm's IP and the reason the service exists at all. But the engineering scaffolding around it should be inspectable, and when it breaks, the post-mortem should be public. That's the standard we hold ourselves to.