Nanostore paper continues: The proposed architecture consists of lots of cores, basically, lots of chips. The motivation behind having lots of chips is;

  1. Want to have high memory capacity
  2. Don’t want to use disk

So, one way to satisfy both of those requirements is to use nonvolatile memory in memory hierarchy. There are two ways to do it.

  1. Use one processor and connect several nonvolatile memories to it.
  2. If the first solution does not give enough capacity, replicate the same architecture.

However, CPU doesn’t like the speed of nonvolatile memory, and want the speed of DRAM. We need to find a way to compensate for the requirements of CPU which wants to have high bandwidth, high throughput and low latency. So, we want to make nonvolatile memory close to CPU.

If we use high speed processor with high latency memory, it won’t work well. So, we need to compensate that difference. Caching is one general way to solve it to some extent, but generally they are not enough to hide that latency. We need parallelism in the workload, so that, if one thread stalls due to memory latency, there should be another context than can continue its execution to compensate the memory latency. For datacenter workloads, instead of high speed processor, we need a processor which is capable of doing multiple tasks at the same time. There are two ways to do that;

  1. Hardware multi-threading: One pipeline with an efficient support for multiple contexts
    1. Motivation:
      1. Full use of hardware resources
      2. Need parallelism within code to hide the memory latency
      3. Need multiple independent context
      4. small number of thread, but parallelism inside the thread
    2. Disadvantages:
      1. Resources are shared among different contexts
      2. Need a structure to make resource sharing available, and efficient
      3. Make pipeline more complicated
  2. Multicore architecture: Several cores; so pipeline.
    1. Motivation:
      1. Each thread does not have high parallelism in their task.
      2. There are several available independent threads in the system
    2. Disadvantages:
      1. Even if you get high throughput, it won’t tolerate the memory latency.

Fundamental difference between hardware multi-threading and multicore architecture is that; we have a small number of threads but more parallelism inside the thread (instruction level parallelism) while having several independent threads running on multicore processor (thread level parallelism).

How to connect cores to memory?

  • Cache hierarchy
  • Also need to connect cache hierarchy and memory to multiple chips

If the workload is memory intensive, the amount that we can process is bounded to the amount that we can get from memory, memory bandwidth. So, we need to get high bandwidth from memory, which basically means more pins, banks, channels … etc.

If I have a single core and want to utilize memory, you need to generate memory requests fast enough to keep pipeline full. If we connect all cores to all available memories, it will require a good interconnect structure to utilize them efficiently. On the other hand, we can also distribute channels and memories among cores. However, it will require explicit communication between cores to access different memory spaces.

Ex: Intel “Single Chip Cloud Processor”

  • 48 cores in one chip, server on one chip
  • Each core has its own memory system
  • Interconnect between cores
  • Communication between cores is not through shared memory, but explicit message passing.

How many pins do we have to memory? ~1000 pins. Half of the pins used for power/ground and high-end packages have >2000 pins.

One pin can easily carry 5Gbps, especially if using differential signaling across two pins. Let’s assume one pin carry ~10Gbps. If we have 500 pins, it means roughly 5Tbps. This number is quite high and unrealistic, since not all the pins carry 10Gbps in a package. Not all pins are used for memory, some of them used for I/O and such. We can achieve 400Gbps, which is reasonable for an expensive package.

  • Xeon: 3 memory channel DDR3, 50Gbps
  • NVidia GPU: 6–8 memory channel, 50% faster channels, over 200Gbps

What limits the bandwidth? Generally, power limits the bandwidth, which is approximately “10 mW/Ggps” as current power usage. Let’s assume we have 5Tbps bandwidth, it will consume 50 watts just to move data around. So we need a low power signaling, not just large number of pins.

How to reduce power?

  1. Go slower
  2. Go shorter
    1. How to reduce distance?
      • Put them closer
      • Put them into the same package
      • Put them as stack to make them even more closer

“Stacked design”

  • High communication bandwidth
  • Reduce power consumption by going slower
  • Decrease the distance to travel
  • May increase the temperature; cannot have high power CPU, then cooling will be problem

We can have large silicon interposer with bunch of wires in it between CPU and pins, but need to traverse wires. It is a question of which one is better;

  1. Connect through pins, and have temperature problem
  2. Have interposer, and go through long trace

Back to paper discussion: So the idea is; lots of memory bandwidth to many cores with reasonable amount of power. They didn’t talk about heat problem at all. Here is the summary;

  • Lots of core for more parallelism
  • Tolerate latency
  • Stacked design for more bandwidth
  • Don\’t need expensive package
  • Reduce power due to slower communication
  • Lower cost, so can put more stuff into package

How did they evaluate their idea?

  • What application did they use?
    • Not SPEC because it is not parallel enough, not a data centric workload
    • They used:
      • Sort: bin sort , n chunks b bins  reduce
      • Ranking: recommender (ignored in class discussion)
      • Checksum: n files, compute checksum for each file
      • Video: n files to decode/encode
      • Search: (ignored in class discussion)
  • How to simulate 1B core?
    • Do for 1 core and multiply by # of cores
    • Unrealistic: shared stuff (which means contention), and load imbalance. So, the actual formulation becomes;
      • Overall execution time = Execution time of one core * # of cores + sharing effect + load imbalance
  • How did they make those configurations? There are several weird looking system configurations like 28-core 2-way, 22-core 2-way, 56-core 4-way …etc.
    • Have max resource for setup, and then underutilize the resource for evaluation.
  • EDP is used as evaluation metric.
  • Why EDP is good metric?
    • Going slower means energy efficiency. However, optimizing for energy is not good since you cannot wait infinite amount of time for computation to be completed.
    • EDP = Energy for computation * delay to complete
    • Energy = Power for computation * delay to complete
    • EDP = Power * (Delay)2
  • Is EDP only metric doing that?
  1. Power – performance: pα x Tb where α and b can be changed according to which metric is more important
  2. ED, ED2
  • We saw in DVFS paper that Power = αCV2f + Ps
    • ED
      • Not true actually, since V2 should be V∆V
      • f proportional to V (roughly)
      • Power ~ V3
      • Time ~ 1/V
      • ED = V3 * 1/V2 = ~V
      • So use DVFS for better EDP
  • ED2
    • ED2= P \* D \* D2 = PD<3 = V3 / V3 = ~Constant
  • There are two metric that are more meaningful than EDP;
    • Energy to fixed delay: with fixed time for computation, how much energy you will use to complete the task.
    • Delay to fixed energy: performance that you can get with your battery

EDP does not say that system A is better than system B for all time. It can only say EDP is better on system A than system B. Moreover, EDP does not assign values; it just compares which one is better without bothering “how much?”. So, pα x Tb is correct and useful, but ED calculation is more intuitive. ED & ED2 are nice but “A to fixed B” kind of metrics described above are better.

Back to paper again, the authors used EDP for their performance evaluation metric, it shows that their technique has better EDP, but does not say if they meet the performance requirement or how much better their technique is.