Can a neural network increase LiDAR resolution?
Yes — a network can infer a denser scan from a sparse one. Here a 16-channel 3D LiDAR was upscaled to 64-channel resolution, running in real time on an FPGA.
AUTOMOTIVE VISION
Turning a 16-channel LiDAR into a 64-channel one, in real time, on an FPGA.
High-channel-count LiDAR is expensive; low-channel-count LiDAR is affordable but sparse. A neural network can bridge the gap — inferring the denser scan from the sparse one — but only if it runs in the vehicle, in real time, inside a power budget a car can carry, which rules out doing it on a datacentre GPU.
CONTEXT
Where this started.
The task was to take a 16-channel 3D LiDAR and produce the resolution of a 64-channel unit, as a neural-network IP core on a Xilinx Zynq UltraScale+ device, processing frames as they arrive rather than in a batch after the fact.
Two implementation routes were taken deliberately: one from a high-level Python model through HLS tools, and one using Xilinx DPU parametrisable deep-learning cores. The comparison is the point — the same network, two paths to fabric, with different area, effort and performance trade-offs.
CHALLENGES
4 problems, named.
Stated before they had answers.
01
Inference has to be real time, in the car
The upscaling only has value if it keeps up with the sensor inside the vehicle's power envelope — not offline on a GPU.
02
A sparse-to-dense network in fixed hardware
Upscaling 16 channels to 64 is a learned mapping that had to be realised as an IP core, not a script.
03
Two routes to fabric, honestly compared
HLS from a Python model versus Xilinx DPU cores give different area and performance; both were built to make the trade-off visible.
04
Moving point clouds through the fabric
3D LiDAR data had to be pre-processed into a form the network consumes and buffered without stalling the pipeline.
ARCHITECTURE
How the system was built.
A sparse 16-channel scan arrives over Ethernet and is pre-processed from a 3D point cloud into a 2D range image. The neural-network IP core — built two ways, through HLS from a Python model and using Xilinx DPU cores — upscales it, with ping-pong buffering keeping the pipeline moving, and the 64-channel result leaves over 1 Gbps Ethernet at 10–15 frames per second.
CONTRIBUTION
What Faststream did.
The specific scope, rather than a capability list.
- Neural-network IP core — upscaling a 16-channel 3D LiDAR to 64-channel resolution on Zynq UltraScale+.
- Real-time datapath — point-cloud pre-processing, 3D-to-2D range conversion and ping-pong buffering at 10–15 fps.
- HLS implementation — the network realised from a high-level Python model through HLS tools.
- DPU implementation — the same network on Xilinx DPU parametrisable deep-learning cores.
- Implementation comparison — area, effort and performance trade-offs made explicit across the two routes.
- Host interface — upscaled scans delivered over 1 Gbps Ethernet for downstream processing.
WHAT WAS HARD
The parts that consumed the schedule.
Written out because a reader facing the same programme gets more from this than from a summary of what went well.
01
Real time inside a vehicle budget
The network had to keep up with the sensor on an in-vehicle device, which shaped every quantisation and buffering choice.
02
Two routes, deliberately
Building both the HLS and DPU implementations was more work, but it is the only way to compare area and performance honestly rather than argue about them.
03
Point clouds are not images
Pre-processing 3D LiDAR into a representation the network consumes — and back — is where much of the real engineering sat.
04
Keeping the pipeline from stalling
Ping-pong buffering was needed so acquisition, inference and output overlap rather than serialise.
OUTCOME
What resulted.
A real-time LiDAR upscaler on FPGA
16-to-64-channel super-resolution running at 10–15 fps on Zynq UltraScale+.
Two validated implementation routes
The same network through HLS and on Xilinx DPU cores, with the trade-offs measured, not asserted.
In-vehicle, not offline
Inference in the vehicle's power budget rather than on a datacentre GPU.
A reusable inference datapath
Pre-processing, buffering and NN integration reusable for other sparse-to-dense sensing tasks.
Confidentiality
Customer projects are presented at property, capability, outcome and integration level. Customer names, internal architecture and confidential deliverables are not disclosed. Where a figure would identify a customer or a design, it is omitted rather than approximated. More detail is available under a non-disclosure agreement, within the limits each customer has agreed.
CAPABILITY USED
What this was built from.
Every item links to its own page.
PLATFORM
Faststream Vision
Machine vision and edge inference.
CAPABILITYAI Silicon & Acceleration
Accelerators, quantisation, RTL.
CAPABILITYSensor Fusion & Perception
Camera, LiDAR, radar, fused.
CAPABILITYEdge AI & Embedded ML
Models inside the power budget.
SOLUTIONImage Processing & FPGA Vision
The wider body of imaging work.
SERVICEFPGA Design
Architecture, HDL, timing closure.
WHERE THIS APPLIES
Industries this serves.
COMMON QUESTIONS
Questions this programme gets asked.
Why run it on an FPGA instead of a GPU?
Because the inference has to happen in the vehicle, in real time, within a power budget a car can carry — which a datacentre GPU cannot meet.
What is the difference between the HLS and DPU routes?
HLS builds the network from a high-level model into custom RTL; the DPU uses Xilinx's parametrisable deep-learning cores. They trade area, effort and performance differently, so both were built and compared.
What frame rate was achieved?
The design processed frames in real time at 10–15 fps, with the upscaled output sent to a host over 1 Gbps Ethernet.
KEEP READING
Related work.
BUILD WITH FASTSTREAM
Bring us the difficult part.
Tell us the sensor, the target device and the frame rate. Real-time inference on FPGA fabric, inside a real power budget, is core Faststream work.