Defect Tolerance on the Teramac Custom Computer

Background

  • With newer technology, likelihood of having faults is going up
  • Design for manufacturing getting more important
  • Teramac was meant to be a design evaluation system, rather than a production system. It ran at 1MHz, compared to GPPs which ran at 100MHz - 200MHz
  • Lots of programmable logic
  • Philosophy : “Reliable system from unreliable parts”
  • This philosophy is valid even today

What is the problem the paper attempts to solve?

  • Tolerate hard faults
    • Hardware faults that cannot be fixed and will always happen (as opposed to soft errors which happen sporadically during runtime)
  • Reduce system manufacturing cost
    • Buy cheap parts that aren’t perfect that would not be used under normal circumstances
  • Targeting FPGAs in specific
  • Map any user design on faulty system
  • Hide the mapping process from the user. Also, mapping should work for any faulty system
  • Increase system lifetime
  • Fine-grained fault circumvention - Exactly identify fault location
  • Handle not just chip/logic faults but also system-level interconnect (MCM,PCB, system cables) faults
  • Wanted to build a large custom computer (not explicitly stated in paper) using FPGAs

Who are the Users?

  • High performance computing - Fairly specific - Mostly system/circuit simulation
  • Reconfigurable general purpose computing
  • Not targeting traditional “reliability” because if there are faults, system has to be restarted and reconfigured and cannot be resolved dynamically

What is unique about the presented solution?

  • Previous techniques used sparing
    • Put some spare units onto a chip and not use dead rows/columns.
    • Memories, cache use this a lot.
    • Easy mechanism
    • Basic idea : Direct replacement or binning
    • Perhaps unfair comparison with wasteful sparing techniques in paper. There are more efficient sparing techniques today.
  • In Teramac : Any two chips are different from each other and the software is aware of this

Comments:

  • Today’s equivalent would be for software to diagnose faulty components at boot and map them so that they will not be used
  • This technique is not good for fixed-function circuits
  • Also, this is not good for general FPGAs - takes too long to test, binning logistics are problematic except for custom computers
  • Having idle defective parts would not work too well today because they would consume power. This was not a concern in the 90s
  • This is a very forward looking paper

What does Teramac look like?

  • Followed Rent’s rule (IBM) - Power law relation between number of gates and number of I/O signals in a functional block. IO = sqrt(GATES)
  • Fat tree interconnect - Capacity of parent path = Sum of bandwidths of child paths
  • This is a bad design today, because designs are wire-limited (as opposed to transistor-limited in the 90s)
  • Other topologies can be used, but mapping would be significantly more complicated
  • They did not spare wires enough (can be inferred from results section). Defective wires made it hard to follow Rent’s rule.
  • On the software side, Teramac used several testing techniques
    • Redundant testing
    • Fine grained fault isolation
    • Pseudo-random number generator with groups of logic blocks forming a circuit. Verified checksums.
    • Verified both functional and timing correctness using counters on the chip

How did they evaluate the idea?

  • Built the system!
  • Did not show much data regarding designs that do/do not work well with this system
  • Did not evaluate performance with respect to fully functional machine
  • Did not evaluate power, but this was not a concern in the 90s
  • They changed FPGA design and did not use off-the-shelf FPGAs. But they did not evaluate the costs for these.
  • Lesson : Building a machine is not a substitute for a thorough evaluation and discussion of tradeoffs