Calling C++ from Python, and where the time actually goes

The matching engine from the last post settles an order in about 0.4 microseconds. In that time light travels roughly the length of a tennis court. The C++ is not the problem.

And yet the live dashboard, the thing in your browser showing the order book update, feels a beat slow. So where does the time go? The honest answer surprised me, and it is a trap worth knowing about any time you bolt a fast language onto a friendly one.

Two languages that do not speak the same way

The engine is C++, chosen because it is fast. The web server is Python, chosen because writing a server in Python is pleasant and writing one in C++ is not. That split is common and sensible. The catch is that C++ and Python store data in completely different shapes, so they cannot call each other directly. They need a translator.

That translator is a binding: a small layer of glue code that lets Python call a C++ function as if it were a normal Python function. The tool that generates the glue here is pybind11. You describe your C++ functions once, and Python gets to call them.

The thing nobody warns you about is that the translation is not free. Every value that crosses the border has to be unpacked from its C++ shape and repacked into a Python shape, and that repacking can cost more than the actual work.

The work is a sliver

Here is the full journey, from an order arriving to the new book showing up as pixels on a screen. The bar widths are on a stretched scale so the tiny stages are still visible:

The C++ match is the thinnest bar on the chart. Everything to its right, repacking the result for Python, hopping through the async server, serializing to send over the network, and finally the browser drawing a frame, is larger by a wide margin. The browser alone paints at about 60 frames a second, so a single frame is roughly 16 milliseconds. That is forty thousand times longer than the match it is displaying.

The lesson lands hard the first time: optimizing the C++ further would be polishing the smallest bar on the chart.

What the border actually costs

The expensive step in the middle is the repacking. Each time the engine matches an order, it produces a little pile of events ("a trade happened," "an order was accepted"), and every one of those has to become a Python dictionary so Python can read it:

py::dict event_to_dict(const Event& e) {
    py::dict d;                 // allocate a fresh Python object...
    d["type"]     = type_name(e.type);
    d["price"]    = e.price;    // ...and copy each field across the border
    d["quantity"] = e.quantity;
    return d;
}

Allocating a Python dictionary and filling it in is real work, and it happens for every event on every order. Multiply by the order rate and this, not the matching, is the dominant cost at the boundary. If this needed to go faster, the fix is not faster matching; it is crossing the border less often, by handing back many results in one batch instead of one dictionary at a time.

A couple of smaller choices in the same file are worth seeing, because they trade raw speed for a nicer Python experience. The engine accepts the order side as a plain string:

// "BUY" / "SELL" as strings, converted inside. A string compare per call,
// but the caller writes engine.submit_order("BUY", ...) and it just reads.
engine.def("submit_order", [](MatchingEngine& self, const std::string& side, ...) { ... });

Comparing strings is slower than comparing a number, but the Python caller gets to write "BUY" instead of importing an enum, and at the rates here that trade is free in practice. The other nicety: when there is no best bid yet, the C++ returns an "empty" value that pybind11 turns into Python's None, so the caller writes the natural if engine.best_bid is None.

One more term, since it comes up the moment you mix C++ and Python threads. Python has a rule called the GIL, the global interpreter lock, that lets only one piece of Python run at a time. It can be a real bottleneck under heavy threading. Here the engine is called ten times a second, so the lock is never the thing standing in the way.

The takeaway

When you glue a fast language to a friendly one, the glue is usually the bottleneck, not either language. The only way to know that is to measure the whole path from input to pixel, not the one function you assume is hot. I assumed the C++ would dominate. The chart said otherwise, and the chart was right.