Beyond the Backplane: a Chip-to-Chip Design Solution

By Gordon Vinther, Ardent Concepts, Inc.

I am sitting in my office on a Saturday while hurricane Irene passes over New Hampshire. As the lights flicker and the world goes black momentarily, all of the fire doors around our command post in New England slam shut in unison as the magnetic doorstops lose power. In a moment, I hear the distant belt squeal of a generator coming online. Within seconds, lights are restored and my office beeps and burrs with office equipment coming back to life. As I walk down the hall of closed doors, resetting the heavy fire doors against electric magnets that draw power 24/7, bringing the office back to a recognizable state, I can hear the fans in our server room running in unison in a constant dueling match between heaters and exchangers. The noise is so loud that with everything else in the office shut down I consider my small connector business and the computing power it consumes, even on a Saturday. In a larger context our usage is very modest, even quaint when compared to supercomputers and large data centers that power the internet today. Then, from thoughts of my business to usage on a grander scale, I come to this question: How is power usage and efficiency in supercomputers and large server farms being addressed as the expectant speeds of the systems increase?

A Google search partially answered this question. Getting high-speed signals out of the backplane of large computing centers is one way - short haul, chip-to-chip interconnect. Super computers are made up of several processing and peripheral racks or blades all interconnected through the backplane of the computer. The standard backplane, along with all the traces, vias, and interconnects within it, gobbles up bandwidth, slows signal speeds, and requires extra power and signal conditioning to bring signals back up to the speeds of the drivers to maintain the speed of the system.

However, there are a few methods of achieving high speed, short haul, chip-to-chip interconnect outside the backplane of large computing systems, thus requiring less energy and increasing performance of the system. This has come to light when customers approached us to help them with high-speed interconnects solutions. These include, but are not limited to, optical and electrical transmission lines. Optical communication involves a transmitter and receiver on or very close to the chip, whereas electrical is simply running connections between the chips through high quality, low-loss coaxial cables and flex circuits (Figure 1). Each option has its advantages and disadvantages depending on the requirements of the system and the length of the interconnect.

It is widely believed that optical transmission will ultimately be the solution for short haul, chip-to-chip transmission. It is capable of lower loss at any distance due to its X-talk immunity, and it can also enable higher channel density than the electrical solution above certain lengths. However, one issue with optical today is that it is difficult to integrate transmitter and receiver manufacturing into the standard CMOS process and thus requires separate transmitters and receivers, ICs to and from the CPUs. These transmitters and receivers still require another short haul electrical interconnect to access them and also require extra power to run them.

Figure 1: Flex circuit chip to chip interconnection on a backplane.

In light of this, much research has been done on short haul electrical interconnect. Electrical is more efficient below certain distance (1m-3m) at which point signal loss becomes too high. Channel densities can be high, but are greatly a function of achievable cable densities; it doesn’t help to have the interconnect density higher than the cable density, resulting in a big bunch of cables running into a small connector. Today, communication rates of 16Gb/s have been achieved with this electrical short haul using less power than conventional backplane systems. A recent IEEE presentation by Bryan Casper, Energy Efficient Multi-Gb/s I/O:Circuit and System Design Techniques demonstrates how an electrical link comprising a 50ohm high quality twin-ax cable can bring transmission line bandwidth from 6Gb/s in a traditional backplane design to more than 50Gp/s at the same 20 pj/bit power requirement. This type of system requires high bandwidth interconnects and low loss interfaces between the 50ohm cables and their mating PCBs. To make this happen at the highest efficiency and speed, the chip manufacturers must provide topside access in the substrate or die design (Figure 2). As such, it seems reasonable that if optical transmitters and receivers - when placed close to the CPU - can improve the system efficiency over a backplane system, then the same electrical connectors should also improve the performance over a backplane system when placed in close proximity to the CPU but not on the actual CPU substrate.

Figure 2: System design for chip to chip interconnect solution.

Conventional FR4 backplanes systems stymie computer performance. One method of increasing the performance of supercomputers is to get high-speed transmission lines out of the backplane. Chip-to-chip communication via optical or electrical will increase speed and/or efficiency of the supercomputers that power the internet today. Until backplanes incorporate wave guides for optical transmission, and optical receivers and transmitters can be integrated into the CPU's functionality in a single die, electrical short haul electrical interconnect will be used with success in improving performance and power efficiency for short haul, chip-to-chip communication.