Tool Learning 3 - SPEC CPU 2006 | Gary's Personal Page

Introduction

SPEC CPU 2006 (documentation here) is a frequently used benchmark in low-level security. The SPEC CPU Integer Benchmarks are listed below:

Number	Name	Language	Description
400	perlbench	C	PERL Programming Language
401	bzip2	C	Compression
403	gcc	C	C Compiler
429	mcf	C	Combinatorial Optimization
445	gobmk	C	Artificial Intelligence: go
456	hmmer	C	Search Gene Sequence
458	sjeng	C	Artificial Intelligence: chess
462	libquantum	C	Physics: Quantum Computing
464	h264ref	C	Video Compression
471	omnetpp	C++	Discrete Event Simulation
473	astar	C++	Path-finding Algorithms
483	xalancbmk	C++	XML Processing

This table lists the floating-point benchmarks:

Number	Name	Language	Description
410	bwaves	Fortran	Fluid Dynamics
416	gamess	Fortran	Quantum Chemistry
433	milc	C	Physics: Quantum Chromodynamics
434	zeusmp	Fortran	Physics / CFD
435	gromacs	C/Fortran	Biochemistry/Molecular Dynamics
436	cactusADM	C/Fortran	Physics / General Relativity
437	leslie3d	Fortran	Fluid Dynamics
444	namd	C++	Biology / Molecular Dynamics
447	dealII	C++	Finite Element Analysis
450	soplex	C++	Linear Programming, Optimization
453	povray	C++	Image Ray-tracing
454	calculix	C/Fortran	Structural Mechanics
459	GemsFDTD	Fortran	Computational Electromagnetics
465	tonto	Fortran	Quantum Chemistry
470	lbm	C	Fluid Dynamics
481	wrf	C/Fortran	Weather Prediction
482	sphinx3	C	Speech recognition

Here is the official documentation list:

Read Me First (Q&A about SPEC CPU 2006, including background, scope, metrics, and more)
Installation Tutorial (Installation guide for *NIX systems, including how to run a simple benchmark)
Get Familiar with Runspec (Common and less common runspec options)
Run and Reporting Rules (Read this before citing results)

Installation Instructions

My Ubuntu environment:

After downloading the file, run the following command to install it:

1
2
3

cd /path/to/spec
./install.sh
source shrc #or . ./shrc

After this, modify the configuration file in SPEC_CPU2006-103/config/. Make sure the compiler is linked through update-alternative --config. I configured it with gcc, but it can also be replaced with clang. Installing fortran is also necessary.

My gcc version:

My g++ version:

You can also try to optimize by changing -o2 to -o3 or another optimization level. Adding -std=c++98 to CXXOPTIMIZE can solve some compilation errors.

Remember to modify the configuration file in the 483 section; otherwise, a compilation error will appear when compiling 483:

Some people also face issues when compiling 400; it can be fixed like this:

The configuration-file documentation is here.

Hello, SPEC CPU

Example 1

The first step in using SPEC is to try a simple example. Here is example 1:

1	runspec -config=XX.cfg -size=ref -tune=base -noreportable -o text,screen,pdf -I -iterations=3 all &>log

runspec is a crucial command that every user needs to understand. Here, -o stands for output_format.

Configuration file: xx.cfg. Remember to replace it with your own configuration file.
Test scale: ref. It can be set to test, ref, or train. test has the shortest running time and can be used to check correctness.
Test iterations: -iterations=3. This can be set to 1 if the compiler is stable.
Test range: all. This can be replaced by a benchmark number, such as 400 or 401.

There are also other important parameters and tools, such as icc, -pg, and gprof.

Example 2

Here is example 2:

runspec --iterations 1 --size ref \
        --action onlyrun \
        --config Example-linux-ia64-gcc.cfg \
        --noreportable mcf

Regarding --action, many flags can be used in runspec. --noreportable means it will run the number of times set by --iterations. With --reportable, add strict_rundir_verify if you want to validate the content during running and compilation.

Example 3

Here is example 3:

1	runspec -c test.cfg -a run --rebuild -i test -n 1 --noreportable 433

-c stands for --config or --conf
-a stands for --action. Common values include run and build.
-D stands for --rebuild, which deletes the previously built files and rebuilds the benchmark. If you have modified xx.cfg, add this option. If the benchmark has already built successfully and you only need to run the test, you can omit it. Note: if you modified xx.cfg but do not want to build again, use --nobuild.
-i stands for --size, which includes test, train, and ref.
output_format stands for the output format, which can be txt, html, cfg, pdf, or csv.

Example 4

1 2	runspec -c test.cfg -i ref -n 1 all -I runspec -c test.cfg -i ref -r 8 -n 3 all -I

--rate(r) stands for the number of cores.
-I stands for --ignore_errors

Example 5

1 2	runspec --config=my.config -T base --action=build int runspec -config=XX.cfg -size=ref -r 4 -tune=base -noreportable -o text,screen,pdf -I -iterations=3 all

-T stands for --tune
noreportable and reportable control whether to detect if the binary file has been modified.

runspec -c test.cfg -i test -I all # Validate the environment quickly
runspec -c test.cfg -i ref -n 3 -I all # Test the single core CPU
runspec -c test.cfg –r ref –n 3 fp  # -r stand for -rate, only test fp
runspec -c test.cfg –r ref –n 3 int # only test int
runspec -c test.cfg -i ref 473.astar # only execute the single test (473)
runspec -c test.cfg -i ref --rate 4 int

Other commands (for more information about action, refer to 3.3 Actions):

--action build: Create the test project.
--action onlyrun: Run the project without validating the result.
--action report: Run the standard test.
--action clean: Clean the already built test project.

Test the Number of Cores in a CPU

Test the number of logical CPU cores:

1	cat /proc/cpuinfo \|grep "processor"\|sort -u\|wc -l

Test the number of physical CPUs:

1	grep "physical id" /proc/cpuinfo\|sort -u\|wc -l

Or:

1	grep "physical id" /proc/cpuinfo\|sort -u

Number of physical cores per physical CPU:

1	grep "cpu cores" /proc/cpuinfo\|uniq

Number of logical cores per physical CPU:

1	grep "siblings" /proc/cpuinfo\|uniq

Note: If the number of logical cores is larger than the number of physical cores, the CPU has Hyper-Threading enabled.

Some Questions You May Ask

Q: What's the difference between --nobuild, run, and onlyrun?
- A: --nobuild is an option of runspec. run and onlyrun are not options of runspec. For a quick reference of runspec, refer to here.
Q: How do I run an instrumented program?
- A: Replace the MD5 value in xx.cfg with the new one. You can also add --nobuild to prevent SPEC from building the program again.
Q: What if I did not assign rate and copies?
- A: SPEC CPU 2006 will test in speed mode, which only tests single-core performance.
Q: What is the difference between a "rate" and a "speed" metric?
- A: Please refer to here.

File Structure

The root directory has the following file structure:

. |-- Docs |-- Docs.txt |-- LICENSE |-- LICENSE.txt |-- MANIFEST |-- README |-- README.txt |-- Revisions |-- SUMS.tools |-- benchspec |-- bin |-- config |-- cshrc |-- install.bat |-- install.sh |-- redistributable_sources |-- result |-- shrc |-- shrc.bat |-- tmp |-- tools |-- uninstall.sh `-- version.txt

In SPEC_CPU2006-103/benchspec/CPU2006:

. |-- Docs |-- Spec |-- data |-- exe |-- run |-- src `-- version.txt

src contains the source code, run contains run directories, exe contains executable files, and data contains datasets.

Cross Compilation

To run SPEC CPU 2006 in a simulation environment such as GEM5, or on an embedded board such as a Raspberry Pi, we need cross compilation.

Open arm.cfg and change these values to your cross compiler:

CC = your-cross-compiler-gcc

CXX = your-cross-compiler-g++

FC = your-cross-compiler-gfortran

Then save and quit, and try to compile SPEC CPU. The command below only builds the integer benchmarks without running them:

1	runspec --action=build --config=arm.cfg --tune=base int

Then move the binary executable, which is in the source folder, for example benchspec/CPU2006/400.perlbench/build/, to GEM5 or another platform.