Overview

The latest HMC-Sim development has focused on improving our integration with the Sandia SST simulation infrastructure.  This tutorial presents a known functional recipe for building, integrating and utilizing the SST simulation platform with the GC64 HMC-Sim.

First, a few assumptions.  This tutorial assumes that we’re utilizing specific versions of the SST and HMC-Sim packages.  Specifically, this recipe assumes the following:

• SST Core version 6.0.0+ (Top of Tree from Github is also known to work)
• SST Elements 6.1.0+ (Top of Tree from Github is also known to work)
• HMC-Sim 3.0 Release+

All combinations of the aforementioned packages are currently tested in a continuous integration suite in order to verify continuity.  For the purposes of this tutorial, we’ll focus on building and using the SST and HMC-Sim top of tree.

Building & Installing

This build guide is known to work on Ubuntu 14.04, Ubuntu 16.04 and OSX El Capitan.  Please keep in mind that you must first satisfy the SST build requirements as noted in the SST documentation.

First things, first, we need to pull the latest HMC-Sim 3.0 from the TactCompLabs Github repo.  You can do so as follows:

$> git clone https://github.com/tactcomplabs/gc64-hmcsim.git

This will pull all the necessary source to build HMC-Sim.  Now that we have the source, we need to select an appropriate build recipe that includes an installation location.  The default Makefile.inc sets the PREFIX installation path to /opt/hmcsim.  Change this to suit your needs.  You can also see some additional examples of Makefile templates in ~/gc64-hmcsim/configs/.

Now that you have your build environment ready, you need to build and install the source:

$> make

$> make install

This will install all the necessary HMC-Sim headers, libraries (*.so,*.a) and sample CMC libraries to PREFIX.  Now you need to ensure that the libhmcsim.so shared object can be found by SST by adding it to your LD_LIBRARY_PATH as follows:

$> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/hmcsim/install

The next step in this process is to pull the appropriate SST Core source code.  First, clone the latest stable version of SST Core as follows:

$> git clone https://github.com/sstsimulator/sst-core.git

Now we need to build and install it (remember to satisfy all the SST build requirements):

$> cd sst-core

$> ./autogen.sh

$> ./configure --prefix=/path/to/sst/install

$> make

$> make install

Now we can prepare our SST-Elements tree as follows (remember the installation location of HMC-Sim!).  Optional build flags are outlined below:

$> mkdir /path/to/sst/install/elements

$> git clone https://github.com/sstsimulator/sst-elements.git

$> cd sst-elements

$> ./autogen.sh

$> ./configure --prefix=/path/to/sst/install/elements --with-goblin-hmcsim=/path/to/hmcsim/install

$> make

$> make install

Now add the SST environment to your path and library path:

$> export PATH=$PATH:/path/to/sst/install/bin

$> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/sst/install/lib:/path/to/sst/install/elements/lib/sst-elements-library

Optional SST-Elements Build Flags

Additional build options may be specified for the SST-Elements package in order to enable certain functionality for the goblinHMCSim memory backend.  These special build options must be added a preprocessor definitions prior to executing the configure step for the sst-elements package.  Adding these options can be performed as follows (note that we specify both the CFLAGS and CXXFLAGS build flags):

$> export CFLAGS=-DMY_OPTION

$> export CXXFLAGS=-DMYOPTION

A summary of the available options is as follows:

Configuring HMC-Sim with SST

Now that we have our build environment functional, we need to configure a simulation with the HMC-Sim memory backend.  We’ll leave the majority of the configuration items up to the readers to investigate via the standard SST documentation.

The basic simulation using HMC-Sim requires that the user specify the goblinHMCSim memory backend.  This is done by first configuring a memory controller component.  An example of doing so is as follows:

comp_memory = sst.Component("memory", "memHierarchy.MemController")

comp_memory.addParams({
      "coherence_protocol" : "MSI",
      "debug" : "1",
      "backend.access_time" : "1000 ns",
      "clock" : "1GHz",
      "backend" : "memHierarchy.goblinHMCSim",
      "verbose" : "1",
      "backend.mem_size" : "512MiB",
})

Note the use of the goblinHMCSim memory backend.  This is the most basic version of the configuration that will utilize the gc64-hmcsim simulation backend for the results.  Any time you utilize this backend, you’ll find an additional text file called trace.out that contains all the HMC-Sim trace data.  NOTE: the trace file will be overwritten for each new simulation.  If you want to save, move it or change the name between simulation runs.

If you only select the basic goblinHMCSim backend, the glue logic will select configure HMC-Sim to utilize a single, 4-link 4GB HMC device.  You can modify these parameters by specifying additional configuration parameters.  For example, in the sample config below, we configure an 8-link 8GB device.  The device has 128 crossbar queue slots (backend.xbar_depth), 64 vault queue slots (backend.queue_depth) and a maximum HMC request size of 128 bytes (backend.max_req_size).

comp_memory = sst.Component("memory", "memHierarchy.MemController")
comp_memory.addParams({
      "coherence_protocol" : "MSI",
      "debug" : "1",
      "backend.access_time" : "1000 ns",
      "clock" : "1GHz",
      "backend" : "memHierarchy.goblinHMCSim",
      "backend.device_count" : "1",
      "backend.link_count" : "8",
      "backend.vault_count" : "32",
      "backend.queue_depth" : "64",
      "backend.bank_count" : "16",
      "backend.dram_count" : "20",
      "backend.capacity_per_device" : "8",
      "backend.xbar_depth" : "128",
      "backend.max_req_size" : "128",
      "verbose" : "1",
      "backend.mem_size" : "512MiB",
})

In addition to configuring the basic simulation environment, you can also enable additional tracing output by adding the following parameters.  Setting the trace value to “1” enables tracing for that type of value.  NOTE: the more tracing you enable, the larger the trace file.  Power tracing will enable both power measurement tracing and thermal tracing for each subcomponent.

"backend.trace-banks" : "1", "backend.trace-queue" : "1", "backend.trace-cmds" : "1", "backend.trace-latency" : "1", "backend.trace-stalls" : "1",
"backend.trace-power" : "1"

One of the unique features in HMC-Sim 3.0+ is the ability to record sub-device component power and thermal metrics.  For each sub-device component (links, vaults, queue slots, etc) we record power metrics in terms of milliwatts.  We also do a basic conversion to BTU’s.  The power and thermal metrics are recorded for each sub-component on each clock cycle.  The tracing reports the individual per-clock metrics as well as a summation across time.  The default configuration for the power metrics sets the per-clock power requirements for each sub-component to 0.1 milliwatts.  If you want to change these values, you may also do so in the configuration as follows.  The values accept single precision floating point values in milliwatts.  This requires you to also enable “backend.trace-power” : “1”.

"backend.link_phy_power" : "0.11",
"backend.link_local_route_power" : "0.12",
"backend.link_remote_route_power" : "0.13",
"backend.xbar_rqst_slot_power" : "0.14",
"backend.xbar_rsp_slot_power" : "0.15",
"backend.xbar_route_extern_power" : "0.16",
"backend.vault_rqst_slot_power" : "0.17",
"backend.vault_rsp_slot_power" : "0.18",
"backend.vault_ctrl_power" : "0.19",
"backend.row_access_power" : "0.20"

References

Here are a few good references to take note of:

• SST Quick Start Guide
• SST External Components Instructions
• SST Elements HMC-Sim Glue Logic

Here are a few sample tests from the HMC-Sim source tree to examine for questions on configuring the simulation environment:

• ~/gc64-hmcsim/test/sst/6.1.0/goblinHMC1.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblinHMC2.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblinHMC3.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream1.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream2.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream3.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream1-trace.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream2-trace.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream3-trace.py
• ~/gc64-hmcsim/test/sst/6.1.0/goblin_singlestream4-trace.py