Testing a trading bot against a fake exchange

You cannot test a trading bot against the real exchange for the paths that matter. The real exchange does not time out when you ask it to. It does not fill half an order on demand. It does not report a duplicate client id so you can see what your code does next.

So we build a small fake exchange, in memory, that does exactly those things when told.

The idea#

The fake exchange holds orders in a dictionary and follows the same rules the real one documents. On top of that it has a script: a list of faults to inject, consumed one per call. A test names the fault it wants, and the failure path runs the same way every time.

class FakeExchange:
    """In-memory exchange for tests. Failures are scripted, so every run hits the same rare path."""
    def __init__(self):
        self.orders, self.script = {}, []        # script: faults to inject, consumed in order

    def inject(self, *faults):                   # e.g. "timeout_after_accept", "reject", "partial_fill"
        self.script.extend(faults)

    def create_order(self, symbol, side, qty, client_order_id, **_):
        fault = self.script.pop(0) if self.script else None
        if client_order_id in self.orders:
            raise DuplicateClientId(client_order_id)
        if fault == "reject":
            raise OrderRejected("insufficient margin")
        order = {"id": len(self.orders) + 1, "client_order_id": client_order_id, "symbol": symbol,
                 "side": side, "qty": qty, "filled": qty / 2 if fault == "partial_fill" else qty}
        self.orders[client_order_id] = order
        if fault == "timeout_after_accept":      # the order exists, but the caller never hears back
            raise SendTimeout(client_order_id)
        return order

    def find_order(self, symbol, client_order_id):
        return self.orders.get(client_order_id)

A test that would have caught a classic bug#

The nastiest failure is not a reject. It is the timeout after the exchange accepted the order. The bot never heard back, so it sends again, and now there are two orders for one decision.

def test_timeout_after_accept_does_not_double_send():
    ex = FakeExchange()
    ex.inject("timeout_after_accept")
    order = send_once(ex, {"symbol": "BTCUSDT", "side": "buy", "qty": 1})
    assert len(ex.orders) == 1                      # one decision, one order
    assert order["client_order_id"] in ex.orders

send_once is in the post about unknown order states.

The fake has to follow the real rules#

A lenient fake hides real bugs. If the real exchange only answers a limited time window for some query, the fake must refuse the same queries, or your tests will pass on code that fails in production. Every time we learn a rule of the real exchange, it goes into the fake first and into a test second.

Break your own safety code#

Tests that pass prove little about safety code. So we also break it on purpose: flip a comparison in the risk gate, remove a stop, skip a check, and run the suite. Every one of those breaks must make a test fail. A break that survives means a safety path nobody is testing. This is called mutation testing, and for the code that decides what happens to money, it is worth the afternoon.