CXL: Composable Infrastructure’s missing link

The Piltdown Man remains were supposedly discovered in Sussex, England in the early 1900s. The discoverer claimed it was the remains of a half-million-year-old missing link between ape and human.

It took 40 years to be thoroughly debunked but it was, beyond any doubt, a hoax.

When I first heard about CXL (Compute Express Link) a couple years ago, I was skeptical.  I often have a rather cynical view of new technology, having seen a lot debut with no reasonable purpose. Does it add any usable customer value?  Or is it just some vendors’ efforts to foist high-margin hardware on budget-flush buyers? FCoE is a prime example of the latter.

I wasn’t sure high-end computing needed a new device interconnect.  It smelled like a hoax.

I attended Tech Field Day 27 as a delegate in Milpitas earlier this year.  The first day was chock full of CXL discussion, starting with a CXL roundtable discussion in the late morning. This was followed by a presentation from Memverge (with segments from Astera Labs and XConn—possibly one of the most poorly conceived names in the tech industry). We wrapped up with a CXL update from the former head of the CXL Consortium, Siamak Tavallaei.

This is not going to be a CXL tutorial.  The link to the consortium above is going to be far better at that.  A summary is that CXL is a nascent memory interconnect that attaches via PCIexpress (PCIe). It allows memory to be installed internally via a PCIe slot, or by connection outside the box through a CXL adapter.

The current version of CXL with released products is CXL 1.1.  This works with PCIe 5.0, available on some of the latest generation of Intel and AMD processors.  The CXL wikipedia entry is almost certainly kept up to date and provides compatibility info for the various versions.

The variants in the graphic show some of the ways CXL can be applied.

CXL version 2.0 is specified at this point, with hardware support underway from several companies.  A primary new feature is the ability to have a single switching layer between the processor and the memory.  Another main feature is the ability to divvy up a memory device to various different processors.

The nice thing about CXL-connected memory is that the CXL connection can just look like a laggy NUMA bridge to a processor. Instead of the memory being connected to another processor, the memory is on the potentially long and winding road of a PCIe connection. But at least it ends up looking like a memory access, not an off-host I/O request.

I’ve spent a long time in the computing high-tech industry.  Over that time, I’ve watched the continued modularization and commoditization of compute resources.  From stand-alone computers, the industry went to networks. Internal storage got broken out to external, and then shared external, storage.

A parallel advancement was going from single processor to multiprocessor systems.  This pair of co-incidental technology progressions crapped out the child, virtualization, which brought with it continuing evolution.

(image source: Greg Perry, Toronto Star)

Virtualization gave rise to virtual storage and virtual networking, meaning that in addition to slicing and dicing with hardware, you could now do (some of) the same with software. Virtualization also found a natural home in converged and hyper-converged systems, meaning you could divvy up resources and configure hardware in more variable ways. This was eventually called the Software-Defined Data Center (SDDC).

SDDC evolved (and devolved in many cases) into the idea of hardware composable infrastructure.  The thinking with composable infrastructure is that you could use hardware building blocks to build any system you want.  It could be a single-purpose system, but more likely it would be the underpinning of virtualized systems.

What I realized at the Tech Field Day discussions was that CXL supplied the next step in composable infrastructure.  Until this concept came up, the ratio and relationship between processor and memory were always fixed. There was no way to turn the knob to adjust that ratio.  CXL provides the ability to add or subtract physical memory for a processor or set of processors, much the way virtualization provisions between virtual memory and virtual processors.  It puts these two components into the mix of flexible resources. CXL is a missing link in the evolution of composable infrastructure. It allows memory and processors to be scaled independently.

So I might have been the Encino Man in seeing the value of CXL. CXL probably isn’t for every configuration, especially anything other than top, high-end compute systems. But it’s no hoax.  It ain’t the Piltdown Man.

I’ll end with the comment that CXL version 3.0, with multi-tier switching and peer-to-peer communications, has lots of challenges in front of it. Attempting to treat a network as a bus is fraught with difficulties.  In the end, I foresee CXL leveling out at a single level of switching and a processor-memory relationship in much the way Fibre Channel settled.  Buffering and contention become real issues when treating a network as a bus.

You can learn more for yourself from the Tech Field Day 27 videos and the other resources I’ve mentioned here.

This is what I’ve learned.