Parallelism

What is Parallelism?

  • What comprises parallelism?
* Multiple alu’s
* Multiple cores
* Pipelining
* Circuits and component concurrency
  • Parallelism is inherent
* 32 bit processing
* VHDL/Verilog are parallel
* Hardware is already parallelized, impossible to ignore
  • Mechanism to express parallelism in software
* Multiple threads
* Multiple instructions (SIMD)
* Instruction level parallelism (OOO)
* Pipeling
  • Concurrency in the OS does not imply parallelism.
* Can have concurrent processes in the OS running on a single processor.
* These multiple processes are active at the same time from the OS perspective, but are not executing at the same time from the HW perspective.
* From the logical perspective (programmer), there is little difference between the two.
* Many of the problems with concurrent systems are shared with parallel systems.

How does Parallelism help with the challenges?

  • Performance
* Higher throughput due to multiple compute resources
* Helps in sustaining high bandwidth, both on and off chip, through wide interconnect paths, pipelining, and efficient scheduling and communication resource sharing.
  • Power Efficiency
* Power = Static Power + Dynamic Power
* Parallelism does not help reduce the Static Power (leakage)
* Dynamic power = 0.5 x C x V2DD x f
* CMOS has a non-linear relation between power and the speed at which they operate, with power increasing super-linearly with speed.
* If we can slow down the clock, we can reduce voltage
* There is a win using many slower and lower power parallel units than a fast serial unit.
  • Designability
* Modular design make design easier.
  • Programmability
* Programming becomes difficult but not undoable
* Debugging becomes significantly harder
  • Reliability
* Reliability improves using multiple units
* However there is a trade off between reliability and efficiency

Locality

What is Locality?

  • Spatial locality — two memory locations that are close to one another in the memory namespace will likely be accessed within a certain period of time.
  • Temporal locality — a memory location is likely to be accessed multiple times within a certain period.
  • Physical locality — actual physical structures that are close to one another in space. For example, an ALU and a directly connected register file are local to one another, as are two operands in the same register file.
  • Producer-consumer locality — a limited form of temporal locality, where a location is written and then read soon (and usually not read again).

Locality in a CPU

  • There is inherent locality in various places in a CPU. You cannot build a machine without locality.
* Locality in registers
* Caches
* Data on the interconnect
* Bypass and forwarding networks
* In Reservation Stations

How does locality help with the challenges?

  • Performance
* Physical locality improves latency
* Temporal locality enables physical locality
* Improved memory access with spatial locality
* Short wires (higher bandwidth density) enables more communication
  • Power Efficiency
* Communication energy increases with distance
* Today interconnect power is 50–80% of the power
* There is significant power benefit from exploiting locality
  • Designability
* Makes design more difficult because we have to meet faster timing closure
* However, it encourages modular design and hence improves designability
  • Programmability
* Programming difficulty increases
* Parallelism and Locality pull the programmer in different directions and hence we have to balance parallelism and locality.
  • Reliability
* Reliability improves because with less communication there is less chance of breakdown

Hierarchy

What is hierarchy?

  • Basic engineering design principle that lets us tackle complex systems by the divide and conquer approach
  • We benefit because we amortize cost of control, our control state is significantly reduced
  • As an example we cannot control every transistor in the chip, but rather local control happens.
  • Thus we create hierarchies such as SIMD, cache lines, collections of threads etc.

How does hierarchy help with the challenges?

  • Performance
* Hierarchy enables asynchronous and parallel subsystems.
* Hierarchy improves locality.
  • Power Efficiency
* Efficiency improves as architectures become easier to manage
  • Designability
* Makes design easier, helps handle the complexity of multi-billion transistor chips.
  • Programmability
* Hierarchy encapsulates complex subsystems and can present a simple and effective interface that can be used by the programming system or programmer.
  • Reliability
* Encapsulate and protect subsystems — fault isolation and repair.