One line, .cpu(), inside a forward hook can stall your whole GPU.
To record activations you attach a hook to a layer. The obvious version copies the tensor to the CPU right away:
python
def hook(module, inp, out):
data = out.detach().cpu().numpy() # forces the GPU to wait
.cpu() and .item() make the CPU wait for the GPU. Do that on many layers for every token and the GPU spends much of its time idle.
GlassBox’s approach:
- Register hooks only on parent modules, not on single neurons.
- Select the needed columns on the GPU with
index_select.
- Copy them to pinned host memory on a separate CUDA stream, without blocking.
- A background writer saves the data to disk.
What we measured. The repo’s Colab benchmark compared plain PyTorch, a naive .cpu() hook, and GlassBox in eager mode:
| Test |
GlassBox overhead |
| Qwen2.5-0.5B, batch 1 |
about +27% |
| Qwen2.5-0.5B, batch 4 |
about +11% |
| Qwen2.5-3B (T4 GPU), batch 1 to 4 |
about +32% to +35% |
In these eager-mode runs, GlassBox did not beat the naive hook. Python hook dispatch dominates on small models. We are not claiming otherwise.
The architecture aims to remove GPU pipeline stalls, and the 2.0 release adds a CUDA Graph-safe wiretap and a lock-free ring buffer for serving engines. A verified sub-1% figure for that path still needs independent benchmarks. The repo includes benchmark.sh so you can measure your own hardware and submit the result.
Takeaway: Always benchmark instrumentation on your own setup. Publish the numbers even when they are not flattering.
Thanks for publishing the unflattering numbers alongside the design — that honesty makes the post more useful than most performance write-ups.
The +27% to +35% result on small models makes sense to me. On a 0.5B model at batch 1, the forward pass itself is so short that Python hook dispatch, the extra kernel launches for index_select, and the bookkeeping around the ring buffer can easily cost more than the stall you are trying to remove. The stall only becomes the bottleneck once the model is large enough, or the sequence long enough, that the GPU actually has meaningful work to overlap with the copy.
Two things I would want to see in the next benchmark run, if you are collecting submissions: first, separate the timing into GPU time and wall-clock time, ideally with torch.cuda.Event around the forward pass and a profiler trace, so readers can tell whether the overhead is a real GPU stall or just CPU-side dispatch. Second, vary the number of hooked layers and the recorded width. Recording every layer at full width is the worst case for this kind of tool, and in practice most people only need a handful of layers or a subset of neurons, so a curve showing overhead against coverage would help people decide when it is worth turning on.
The CUDA Graph point is interesting too. A lot of naive hooks break graphs precisely because of the .cpu() or .item() call, so even if the eager-mode numbers are not better yet, a graph-safe path that keeps everything on device could matter more for serving, where the same graph is replayed many times. I would be curious whether the 2.0 wiretap keeps that advantage when the model is run under torch.compile, since that is where most people will actually hit the graph constraints.
For anyone reading this and deciding whether to add hooks to their own model, my rule of thumb would be: hook as few layers as you can, copy as little as you can, never synchronize inside the forward pass, and measure with the profiler rather than end-to-end latency alone, because end-to-end latency hides whether you are paying in GPU idle time or in Python overhead. Those are different problems with different fixes.