Tool Learning 3 - SPEC CPU 2006 | Gary's Personal Page

Introduction

SPEC CPU 2006 (document is here) is a frequently used benchmark in low level security. Here is the benchmark list of SPEC CPU Integer Benchmarks:

Number	Name	Language	Description
400	perlbench	C	PERL Programming Language
401	bzip2	C	Compression
403	gcc	C	C Compiler
429	mcf	C	Combinatorial Optimization
445	gobmk	C	Artificial Intelligence: go
456	hmmer	C	Search Gene Sequence
458	sjeng	C	Artificial Intelligence: chess
462	libquantum	C	Physics: Quantum Computing
464	h264ref	C	Video Compression
471	omnetpp	C++	Discrete Event Simulation
473	astar	C++	Path-finding Algorithms
483	xalancbmk	C++	XML Processing

This table is Floating Point Benchmarks:

Number	Name	Language	Description
410	bwaves	Fortran	Fluid Dynamics
416	gamess	Fortran	Quantum Chemistry
433	milc	C	Physics: Quantum Chromodynamics
434	zeusmp	Fortran	Physics / CFD
435	gromacs	C/Fortran	Biochemistry/Molecular Dynamics
436	cactusADM	C/Fortran	Physics / General Relativity
437	leslie3d	Fortran	Fluid Dynamics
444	namd	C++	Biology / Molecular Dynamics
447	dealII	C++	Finite Element Analysis
450	soplex	C++	Linear Programming, Optimization
453	povray	C++	Image Ray-tracing
454	calculix	C/Fortran	Structural Mechanics
459	GemsFDTD	Fortran	Computational Electromagnetics
465	tonto	Fortran	Quantum Chemistry
470	lbm	C	Fluid Dynamics
481	wrf	C/Fortran	Weather Prediction
482	sphinx3	C	Speech recognition

Here is the official instruction list

Read Me First (This is full of Q&A in spec cpu 2006. including background, scope, metrics and something else)
Installation Tutorial (This is a installation guide for *NIX system only, including how to simply run a benchmark)
Get Familiar with Runspec (Introduce the commonly and less commonly used options in runspec)
Run and Reporting Rules (Run this before cite this)

Install Instruction

My ubuntu environment:

After downloading the file, input the following command to install.

1
2
3

cd /path/to/spec
./install.sh
source shrc #or . ./shrc

After this, modify the configuration file in SPEC_CPU2006-103/config/. Ensure link the program to update-alternative --config already. Here I configured with gcc, it also can be replaced by clang. The installation of fortran is also necessary.

My gcc version:

My g++ version:

Here's also can try to optimize by change the -o2 to -o3 or something else. Adding -std=c++98 to CXXOPTIMIZE can solve some compilation errors.

Remember to modify the configuration file on 483 section, otherwise it would pop up compilation error when compiling 483:

Some people also faced issues when they compile 400, it can be fixed by this:

The configuration file doc is here.

Hello, SPEC CPU

Example 1

The first phrase to use spec is to try it with simple example. Here's the example 1:

1	runspec -config=XX.cfg -size=ref -tune=base -noreportable -o text,screen,pdf -I -iterations=3 all &>log

runspec is a very crucial command that every user need to get familiar with (here -o stand for output_format).

Configuration File: xx.cfg (remember to replace it with your own configuration file.)
Test Scale: ref (it can be set with test, ref and train, test has the least running time, which could be leveraged to test correctness)
Test Iteration: -iterations=3 (It could be 1 if the compiler is stable)
Test Range: all (This can be replaced by the number of benchmark, like 400, 401 or something else)

There also exists some other important parameter, such as icc, -pg, gprof.

Example 2

Here's example 2:

runspec --iterations 1 --size ref \
        --action onlyrun \
        --config Example-linux-ia64-gcc.cfg \
        --noreportable mcf

To talk about the --action, many flag can be learned to use in runspec. --noreportable means it would run the times we set on --interation. For --reportable, if user want to validate the content during running and compilation, user should add strict_rundir_verify.

Example 3

Here's example 3:

1	runspec -c test.cfg -a run --rebuild -i test -n 1 --noreportable 433

-c stand for --config or --conf
-a stand for --action (there are some common parameter like run and build)
-D stand for --rebuild, which means deleted the file (which has been built) and rebuild the content. If you have modified the xx.cfg, you should add this. If you have successfully built and only need to run the test, you may not add this. (Attention: if you have modified the xx.cfg but you don't want to build again, you can change to --nobuild)
-i stand for --size, which content test, train and ref
output_format stand for the output format, which contain txt, html, cfg, pdf, csv.

Example 4

1 2	runspec -c test.cfg -i ref -n 1 all -I runspec -c test.cfg -i ref -r 8 -n 3 all -I

--rate(r) stand for the number of core
-I stand for --ignore_errors

Example 5

1 2	runspec --config=my.config -T base --action=build int runspec -config=XX.cfg -size=ref -r 4 -tune=base -noreportable -o text,screen,pdf -I -iterations=3 all

-T stand for --tune
noreportable && reportable means detect or not detect the binary file has been modified or not.

runspec -c test.cfg -i test -I all # Validate the environment quickly
runspec -c test.cfg -i ref -n 3 -I all # Test the single core CPU
runspec -c test.cfg –r ref –n 3 fp  # -r stand for -rate, only test fp
runspec -c test.cfg –r ref –n 3 int # only test int
runspec -c test.cfg -i ref 473.astar # only execute the single test (473)
runspec -c test.cfg -i ref --rate 4 int

Other command (for more information of action, we can refer to 3.3 Actions):

--action build: Create test project.
--action onlyrun: Only run the project and not validate the result.
--action report: Standard test.
--action clean: Clean the already built test project.

Test the Number of Core in CPU

Test the logical CPU core:

1	cat /proc/cpuinfo \|grep "processor"\|sort -u\|wc -l

Test the number of physical CPU:

1	grep "physical id" /proc/cpuinfo\|sort -u\|wc -l

Or:

1	grep "physical id" /proc/cpuinfo\|sort -u

Real core number in every physical cpu:

1	grep "cpu cores" /proc/cpuinfo\|uniq

Logic core number in every physical cpu:

1	grep "siblings" /proc/cpuinfo\|uniq

Note: If the number of logic core larger than the real core number, this means CPU has open Hyper-Threading.

Some Question You may Ask

Q: What's the different between --nobuild and run and onlyrun ?
- A: --nobuild is a option of runspec. run and onlyrun they are not the option of runspec. For quick reference of runspec, we can refer to here.
Q: How to run the program which has instrumentation?
- A: Modify the MD5 value in xx.cfg with the new MD5 value. In addition, also can add --nobuild to prevent spec from building again the program.
Q: What if I did not assign rate and copies ?
- A: The SPEC CPU 2006 will test in speed, which will only test the single core performance.
Q: What is the different between a "rate" and a "speed" metric?
- A: Please refer to here.

File Structure

The file structure of root is like this.

. |-- Docs |-- Docs.txt |-- LICENSE |-- LICENSE.txt |-- MANIFEST |-- README |-- README.txt |-- Revisions |-- SUMS.tools |-- benchspec |-- bin |-- config |-- cshrc |-- install.bat |-- install.sh |-- redistributable_sources |-- result |-- shrc |-- shrc.bat |-- tmp |-- tools |-- uninstall.sh `-- version.txt

In SPEC_CPU2006-103/benchspec/CPU2006:

. |-- Docs |-- Spec |-- data |-- exe |-- run |-- src `-- version.txt

src is the folder of source code. run is the running folder, exe is the executable file folder. data is the folder of dataset.

## Cross Compilation

To run SPEC CPU 2006 on simulation environment (GEM5, for instance) or the other embedded board (such as raspberry pi), we need cross compilation.

Open arm.cfg, change them to your cross compiler:

CC = your-cross-compiler-gcc

CXX = your-cross-compiler-g++

FC = your-cross-compiler-gfortran

Then save and quit, trying to compile spec cpu (only build the int benchmark, and not run):

1	runspec --action=build --config=arm.cfg --tune=base int

Then move the binary executable file which at the source folder (for example, benchspec/CPU2006/400.perlbench/build/) to GEM5 or other platform.