The Piltdown Man remains were supposedly discovered in Sussex, England in the early 1900s. The discoverer claimed it was the remains of a half-million-year-old missing link between ape and human.
It took 40 years to be thoroughly debunked but it was, beyond any doubt, a hoax.
When I first heard about CXL (Compute Express Link) a couple years ago, I was skeptical. I often have a rather cynical view of new technology, having seen a lot debut with no reasonable purpose. Does it add any usable customer value? Or is it just some vendors’ efforts to foist high-margin hardware on budget-flush buyers? FCoE is a prime example of the latter.
I wasn’t sure high-end computing needed a new device interconnect. It smelled like a hoax.
I attended Tech Field Day 27 as a delegate in Milpitas earlier this year. The first day was chock full of CXL discussion, starting with a CXL roundtable discussion in the late morning. This was followed by a presentation from Memverge (with segments from Astera Labs and XConn—possibly one of the most poorly conceived names in the tech industry). We wrapped up with a CXL update from the former head of the CXL Consortium, Siamak Tavallaei.
This is not going to be a CXL tutorial. The link to the consortium above is going to be far better at that. A summary is that CXL is a nascent memory interconnect that attaches via PCIexpress (PCIe). It allows memory to be installed internally via a PCIe slot, or by connection outside the box through a CXL adapter.
The current version of CXL with released products is CXL 1.1. This works with PCIe 5.0, available on some of the latest generation of Intel and AMD processors. The CXL wikipedia entry is almost certainly kept up to date and provides compatibility info for the various versions.
The variants in the graphic show some of the ways CXL can be applied.
CXL version 2.0 is specified at this point, with hardware support underway from several companies. A primary new feature is the ability to have a single switching layer between the processor and the memory. Another main feature is the ability to divvy up a memory device to various different processors.
The nice thing about CXL-connected memory is that the CXL connection can just look like a laggy NUMA bridge to a processor. Instead of the memory being connected to another processor, the memory is on the potentially long and winding road of a PCIe connection. But at least it ends up looking like a memory access, not an off-host I/O request.
I’ve spent a long time in the computing high-tech industry. Over that time, I’ve watched the continued modularization and commoditization of compute resources. From stand-alone computers, the industry went to networks. Internal storage got broken out to external, and then shared external, storage.
A parallel advancement was going from single processor to multiprocessor systems. This pair of co-incidental technology progressions crapped out the child, virtualization, which brought with it continuing evolution.
(image source: Greg Perry, Toronto Star)
Virtualization gave rise to virtual storage and virtual networking, meaning that in addition to slicing and dicing with hardware, you could now do (some of) the same with software. Virtualization also found a natural home in converged and hyper-converged systems, meaning you could divvy up resources and configure hardware in more variable ways. This was eventually called the Software-Defined Data Center (SDDC).
SDDC evolved (and devolved in many cases) into the idea of hardware composable infrastructure. The thinking with composable infrastructure is that you could use hardware building blocks to build any system you want. It could be a single-purpose system, but more likely it would be the underpinning of virtualized systems.
What I realized at the Tech Field Day discussions was that CXL supplied the next step in composable infrastructure. Until this concept came up, the ratio and relationship between processor and memory were always fixed. There was no way to turn the knob to adjust that ratio. CXL provides the ability to add or subtract physical memory for a processor or set of processors, much the way virtualization provisions between virtual memory and virtual processors. It puts these two components into the mix of flexible resources. CXL is a missing link in the evolution of composable infrastructure. It allows memory and processors to be scaled independently.
So I might have been the Encino Man in seeing the value of CXL. CXL probably isn’t for every configuration, especially anything other than top, high-end compute systems. But it’s no hoax. It ain’t the Piltdown Man.
I’ll end with the comment that CXL version 3.0, with multi-tier switching and peer-to-peer communications, has lots of challenges in front of it. Attempting to treat a network as a bus is fraught with difficulties. In the end, I foresee CXL leveling out at a single level of switching and a processor-memory relationship in much the way Fibre Channel settled. Buffering and contention become real issues when treating a network as a bus.
You can learn more for yourself from the Tech Field Day 27 videos and the other resources I’ve mentioned here.
This is what I’ve learned.
Thanks for reading Adventures with Andy! Subscribe for free to receive new posts and support my work.
Composability has been around in some form or fashion for a long time. Consider the Egenera BladeFrame, HP c-Class with Virtual Connect, and Cisco UCS. All had the ability to reallocate entire physical servers to different user personalities (MACs, WWPNs, etc.), creating a kind of physical cloud of compute. Need a high I/O, high memory configured physical server for a batch job from midnight to 3 am, no problem. Need to turn your daytime VDI servers into night time analytical servers, not problem.
CXL offers something similar, but more granular. Instead of swinging entire servers, you just swing the CXL attached GPUs from the VDI servers to the Deep Learning servers at 5pm, then back at 8am.
But what kind of server owner benefits the most from that? Cloud providers. They can use pricing to steer GPU accelerated VDI users to the peak VDI usage times, then offer reserved instances for Deep Learning during the overnight hours. And they are not constrained to the CPU and RAM configurations of the VDI or the DL servers.
I think the primary use case for CXL in traditional, enterprise, premises environments will be large memory configurations. These will use CXL to attach large amounts of memory, but will not really be composable in the sense of changing.
Having lived through the promise of InfiniBand, first with the Sun Microsystems proposed "Blue Moon" InfiniBand backplane blade servers (trying to do an Egenera or HP Virtual Connect style virtualized I/O), through the TopSpin (later Cisco) InfiniBand based "Multi-Fabric I/O" and "VFrame" software to manage it, I realize these esoteric fabrics rarely become mainstream. But they often find an almost embedded use case, such as the back-end memory replication fabric on EMC XtremIO arrays.