Every night, ground-based gamma-ray telescopes like the Large-Sized Telescopes (LSTs) of the Cherenkov Telescope Array Observatory (CTAO) watch the sky for the faint, brief (~5 ns) flashes of light produced when a gamma ray strikes the atmosphere. For every genuine cosmic event, the camera also picks up a large amount of background light from stars, distant galaxies, and the atmosphere itself. Deciding, in real time, which flashes are worth keeping and which are just noise is one of the hardest engineering problems in the whole telescope.

A new digital camera, the Advanced Camera, is being proposed as a future upgrade for the LSTs. Unlike the current design, which briefly stores each pixel’s signal in analog memory before deciding whether to keep it, this camera would continuously stream every pixel’s data, at a rate of one billion samples per second, directly to a dedicated processing board. That board, the Central Trigger Processor (CTPb), would have only microseconds to sift through this flood of information and decide what to keep. In this article, I present the design of the CTPb prototype, developed by the High Energy Physics group at Universidad Complutense de Madrid, together with two dedicated testbenches that validate its key underlying technologies. This work forms part of my industrial PhD project, and was awarded the First Prize Student Paper Award at the 25th IEEE Real Time Conference.

Splitting the Work: the CTPb Architecture

The Advanced Camera’s focal plane is made up of 7987 individual sensors, grouped in sevens into 1141 small clusters, which are in turn grouped in sevens again into 163 larger units, processed together in independent Front-End Boards (FEBs). Each FEB digitizes the signals from its group of sensors and sends the resulting data as a continuous, high-speed optical stream to the Central Trigger Processor board (CTPb), a single board hosting four processing chips (FPGAs), like 163 separate puzzle pieces being dumped on a table.

Just as a single puzzle piece rarely tells you much on its own, it is only by putting enough neighboring pieces together that the full picture starts to emerge. The chip receiving these streams does not judge each piece in isolation either: it always looks up its immediate neighbors and puts them back together into the full local picture before deciding whether a real event just happened.

In the CTPb, three of the board’s four processing chips (FPGAs) each take on this task for roughly one third of the 163 FEBs, reconstructing local neighborhoods to make a first-level decision (as discussed later in this article, one promising approach is to feed these reconstructed neighborhoods into a neural network, trained to recognize the pattern of a genuine shower and reject background noise). A fourth chip then combines the three partial decisions and adds precise timing information. Because a single telescope can still be fooled by a random coincidence of background light, this fourth chip also compares notes with neighboring telescopes, checking not only whether they recorded the same event at the same instant, but also whether it landed in the same patch of sky. Together, these checks form an additional layer of confirmation known as the stereo-topological trigger.

Because building and testing the full four-chip board directly would be a costly and risky first step, involving highly specialized components and tens of thousands of euros in hardware, we designed a smaller prototype covering one third of the camera, built around two processing chips instead of four. This prototype lets us validate the same architecture and the same optical links at a smaller, more manageable scale before committing to the final design.

Putting the Pieces to the Test

Before committing to building the full prototype board, we validated its two key building blocks separately: the optical links that carry data from the camera to the CTPb and the neural network we mentioned above, which each FPGA receiving camera data will run.

For the optical links, we needed to choose a communication protocol: the set of rules that governs exactly how data is packaged and interpreted at both ends of the fiber, in our 10 Gbps links. We built a dedicated test board and compared two candidates: JESD204C, a protocol originally designed to connect analog-to-digital converters to FPGAs and already used elsewhere in the Advanced Camera’s front-end electronics, and Aurora, a lighter, simpler alternative more commonly used to distribute control commands in large physics experiments. We sent data through fiber cables of varying lengths, well beyond the distances expected in the final installation. Both protocols showed an excellent, virtually error-free performance across all tested configurations, but only one of them, JESD204C, guaranteed that every piece of data always takes exactly the same amount of time to arrive, with no exceptions. This predictability, known as deterministic latency, is essential for a real-time trigger system, where timing must be known precisely down to the nanosecond.

For the neural network, specifically a convolutional neural network, the same type of model widely used for image recognition, we implemented it on a smaller evaluation board, built around the same family of FPGA (Kintex UltraScale) as the prototype, to confirm that the inference ran correctly and to measure the hardware resources it required. The network itself had been trained beforehand on realistic, simulated gamma-ray showers and background-only events, and only afterwards converted from the high-precision numbers used during training to the lower-precision, fixed-point numbers the FPGA operates on. The results were encouraging: the network reached a decision on each incoming event in under two microseconds, and could start working on the next one roughly every 525 nanoseconds, so the chip is never idle waiting for one decision to finish before starting another. To put these numbers in perspective, we compared this implementation against the same neural network running on a regular computer processor and on a graphics card (GPU). The FPGA reached a decision around 46 times faster than the processor and roughly seven times faster than the GPU, while consuming about 167 and 120 times less energy per decision, respectively, and, unlike the GPU, with the same fixed, predictable timing every single time, exactly the kind of reliability a real-time trigger system cannot do without.

What’s Next

These two testbenches gave us the confidence to move forward with the physical design of the 1/3-camera prototype itself. The next steps are the completion of its electrical schematics and printed circuit board layout, followed by manufacturing and, eventually, integration and testing with real camera data.

Regarding the neural network, the version validated here is only one of several variants we have explored. In the final board, each first-stage chip will only have to handle a third of the camera instead of the whole thing, automatically tripling its throughput, and a more capable FPGA leaves enough spare resources to process several sectors at once instead of one at a time, a form of parallel processing that further multiplies throughput on top of that. Combined with quantization techniques that free up further resources without hurting accuracy, our preliminary studies show that some of these model variants could go considerably faster this way. The higher the rate, the more information the network can draw on to make its decision. The next hardware milestone is implementing this parallel processing for the most promising of these variants.

If successful, the CTPb will give the proposed Advanced Camera for the CTAO Large-Sized Telescopes the ability to make smarter, faster decisions about which fleeting flashes of light are worth keeping, bringing real-time, AI-assisted cosmic-ray detection a step closer to reality.

María Molina-Delicado is a doctoral researcher at Universidad Complutense de Madrid, working on real-time trigger systems for the CTAO Large-Sized Telescopes and ML algorithms implementation on FPGAs. She can be reached at [email protected].