In the previous days, I tried to consume SPAN traffic from an HP Switch that provides the traffic with its protocol, the HP ERM (HP Encapsulated Remote Mirror) Remote SPAN. After a while, I concluded, that it didn’t work like other remote SPAN protocols that facilitate GRE (ERSPAN) connections and PCAP-over-IP to send the mirrored traffic.
So I was gasping to find a solution, cause it was mandatory to make it work, for my project.
As I was drifting on the internet for a solution, I found an issue (https://github.com/zeek/zeek/issues/1968) on the GitHub page of Zeek. One guy had the same problem, and a solution was proposed. The HP ERM Protocol uses a 12-byte header after the UDP Header. So skipping the 12 bytes, you can get the encapsulated traffic.
export {
redef PacketAnalyzer::SKIP::skip_bytes = 12;
redef PacketAnalyzer::SKIP::default_analyzer = PacketAnalyzer::ANALYZER_ETHERNET;
}
event zeek_init()
{
PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_UDP, 7932, PacketAnalyzer::ANALYZER_SKIP);
}
Without hesitation, I loaded the script in the Zeek instance and deployed the new promising configuration. It worked. Now Zeek was able to understand what was inside the encapsulated packet. But after a while, the real problem has emerged. The decapsulation process was taking place inside the Zeek engine, so all the traffic was handled by one Zeek worker. This was happening because the AF-Packet was only seeing the same tuple of source and destination IPs and ports (HP Switch IP [Source] and Zeek Capturing Interface IP [Destination] and the same ports) all the time.
Hence the chosen Zeek worker was always the same if we have in mind how AF-Packet dispatches the traffic. The packets of a specific flow must be processed by the same worker every time, so no traffic gaps exist. Another solution was needed. The goal here was for the decapsulation process to happen as close as possible to the NIC.
So after a brief conversation on Zeek’s Slack channel, I came up with the idea of the XDP (https://en.wikipedia.org/wiki/Express_Data_Path). With a lot of help from Arne Welzel (Lead Software engineer at Corelight), because it was the first time I was writing something in XDP, we wrote an XDP script, which removes the first 12 bytes (54 bytes from the start of the frame) on the fly, before the packet enters the Linux kernel. So AF-Packet can read the real tuple of source and destination IPs and ports and dispatch it the right way.
The XDP code is hosted at https://github.com/chrisanag1985/xdp-erm with instructions on how to compile it and load it to your Linux system. Just remember this works independently from Zeek and you have to find a persistent way to load it at the system’s reboots.
