EuroBSDCon 2019, Lillehammer: NUMA Optimizations in the FreeBSD Network Stack - Drew Gallatin

will discuss optimizations to keep network connections and their resources local to NUMA domains. These changes include: Allocating NUMA local memory to back files sent via sendfile(9). Allocating NUMA local memory for Kernel TLS crypto buffers. Directing connections to TCP Pacers and kTLS workers bound to the local domain. Directing incoming connections to Nginx workers bound to the local domain via modifications to SO_REUSEPORT_LB listen sockets. I will present data from real Netflix servers showing an improvement of almost 2x on AMD EPYC (85Gbs - 165Gbs), and 1.3x on Intel Xeon (140Gb/s - 180Gbs). I will present data from the Xeon system showing a 50% reduction in cross-domain traffic. Drew Gallatin Drew started working on FreeBSD at Duke in the 90s, and was one of the people behind the FreeBSD/alpha port. He worked on zero-copy TCP optimizations for FreeBSD and was sending data at over 1Gb/s before gigabit Ethernet was generally available. He spent a decade at Myricom, optimizing their drivers. After a brief hiatus at Google, he landed at Netflix, where he works on optimizing the FreeBSD kernel and network stack for content delivery. He worked on the optimizations to serve unencrypted Netflix traffic at 100Gb/s, and then on more optimizations to send encrypted traffic at 100Gb/s.