Background
The FLASH project at Stanford involved building an actual multiprocessor system to be able to experiment with Cache Coherence protocols. The system consisted of processor running a modified off the shelf OS and the “Magic Chips” (programmable components to deal with the coherence traffic in order to experiment with different protocols). A few projects of the same nature were running in academia at that time. The research goal was to try to change adapt the coherence protocol to the application. The conclusions they reached was that adapting the coherence protocol is useful but verifying coherence protocols is difficult.
What is the problem being solved?
- The main objective of the authors is to convince people that simulation and its associated results are worst that what people think it to be.
- Demonstrate that simulation parameters and small details can have significant impact on results.
- Explain that simulation inaccuracies are difficult to detect until you have the real system. You can verify the functional performance correctness but timing correctness is very difficult to model.
Examples of sources of simulation inaccuracies
- TLB: The lack of modeling of the TLB were significant in the simulation of the MIPS R10000. This was particularly true because the R10000 has only a 64 entry TLB which is small for their applications sizes. An actual system had a significant number of TLB misses and the associated penalties were significant. Many architects view the effects of TLB as small but the authors showed that such lack in detail had significant impact on the results.
- Page Coloring: Usually simulators ignore the effects of the the OS. They just model the timing e.g. an OS call such as malloc would be equivalent to say 500 processor cycles. Even when the simulator tries to model OS effects inaccuracies exist because the simulator and the OS could be using different schemes and algorithms. With malloc, when the OS allocates physical pages, it allocates using schemes to try to minimize conflict with other pages, but the simulator might simply implement a scheme to return a pointer to a large contiguous memory space.
- Modeling Memory Latencies / Latency Hiding: It is difficult to accurately (and simplistically) simulate the exact behavior of out of order processors with respect of memory latencies. For example, if we hit a load statement for simplicity we could just induce a wait till the memory operation is finished. However most processors today start the of execution of a later instruction (non-blocking loads) or context switch. Modeling such behavior is essential today but makes for more complicated simulators.
- Prefetch Timing / Cache Pollution Effects: Cache pollution describes situations where an executing computer program loads data into cache, thus causing other needed data to be evicted from the cache causing performance loss. Modeling of such effects is difficult and even when modeled there could be a disconnect as compared to the actual schemes used by the OS.
- Exceptions: Most simulations do not model exceptions but depending on the architecture and application the impact could be significant.
- Environment Variables: Even seemingly simple details such as OS environment variables can cause performance differences of ~10%.
Tradeoffs in Simulation / Important Considerations
- Building system is too time/money consuming. Simulation needs to be done. We cannot do away with errors but its important to minimize them. Details are important and diligence is required on the part of the experimenters.
- Memory system has a huge impact on performance. Not modeling the memory system or modeling a different memory system can completely invalidate results. Its important to model as close as possible by using techniques such as tracing or varying parameters to mimic the actual system.
- Having a hypothesis is very very important for any experiment to succeed. The experimenters should try to predict results by intuition and then be able to argue on the correctness of their simulation results.
- Bottom line is to be as careful as possible and take simulation results with a grain of salt because accurate modeling is not possible by leaving out details.
