The Problem
- A large system not performing as expected
- “System noise” reduces performance
- OS kernels, background processes, context switching
- Insufficient prior effort
Users
- ASCI Q users (almost of SAGE)
- Supercomputing
- One single big computation on highly parallel system
- Dedicated system
- Lots of control
- High performance (tuned) communication infrastructure with known expected behavior
- Cluster computing
- Building a supercomputer with commodity parts: Beowulf cluster
- Ethernet for connectivity
- Infiniband
Solution
- Based on application model
- Series of hypotheses/test
- Vary number of nodes
- Vary number of processors per node
- Examine SAGE cycle time
- Break into individual components
- Speed up allreduce execution time
- Identified external noise
- Periodic OS activity
- Things that happen frequently and are short are worse than things that happen infrequently and are long, for their application
Other Users
- Exact solutions won’t be the same, but other applications may react differently to solution
Broader Context in Class
- Simulators commonly abstract issues such as OS noise, TLB misses, etc. Can we actually trust the simulator results?
- E.G. FLASH vs. Simulator FLASH
