SPEC benchmark 测试程序使用教程

• 例子
  output_format = asc,ps
  tune = base
  reportable = 1
  runlist = fp
  
• 设置只执行特定的benchmark
  runlist = sjeng
  
• 设置输入源
  size = test/ref
  
• 设置迭代执行次数
  iterations = 1
  
• 设置输出格式
  output_format = text
  
• 设置tune级别
  tune = base
  
• 设置将编译信息打印到屏幕
  teeout = 1
  

benchmark[,…]=tuning[,…]=extension[,…]=machine[,…]:

benchmark: 要编译的测试文件
tuning类型
extension：标志某条规则。runspec命令行可以指定extension,执行特定规则

例子，下面三句等价
465.tonto=base=default=default:
465.tonto=base=default:
465.tonto=base:

该部分是spec运行过程中自动生成的，不用管

1. 下载使用spec默认工具获取系统信息脚本
   $ . ./shrc
   $ mkdir update
   $ cd update
   $ curl -s -O http://www.spec.org/cpu2006/Docs/sysinfo
   $ chmod +x sysinfo
   $ ./sysinfo -p | grep Rev | tr $ " "
   
2. 在配置文件中加上该脚本
   sysinfo_program = specperl $[top]/Docs/sysinfo
   

使用下面的变量
V. Config file options for readers

$[variable]

$variable, ${variable}

include: SUT.inc

#开头

$ cd $SPEC/config
$ cp Example-macosx-gcc421.cfg Khushboo-macosx.cfg
$ runspec –config=Khushboo-macosx.cfg –action=build –tune=base bzip2

成功标志:
Build successes: 401.bzip2(base) <<– what we want to see

The base metrics (e.g. SPECint_base2006) are required for all reported results and have stricter guidelines for compilation. For example, the same flags must be used in the same order for all benchmarks of a given language. This is the point closer to those who might prefer a relatively simple build process.

The peak metrics (e.g. SPECint2006) are optional and have less strict requirements. For example, different compiler options may be used on each benchmark, and feedback-directed optimization is allowed. This point is closer to those who may be willing to invest more time and effort in development of build procedures.

runspec (CPU2006)

$ runspec –config=Khushboo-macosx.cfg –size=test –noreportable –tune=base –iterations=1 bzip2

$SPEC/result

1. 最小测试集
   –size=test
   test - data for a simple test that an executable is functional
   
2. 真实测试集
   –size=ref
   ref - the real data set, required for all result reporting
   
3. all - data used by all runs (if needed by the benchmark)
   
4. 训练数据集
   train - data for feedback-directed optimization
   

1. 仅仅运行某一个测试，而不是所有测试
   –noreportable
   
2. 运行所有测试
   runspec –tune=base –config=Khushboo-macosx.cfg int
   

–iterations=1

–nobuild
bash-2.05$ runspec –config newint.cfg –nobuild –reportable int

peak和base的优先级一样，都会执行
$ cat tmp.cfg
runlist = sjeng
size = test
iterations = 1
tune = base,peak
output_format = text
teeout = 1

default=default=default=default:
CC = /opt/SUNWspro/bin/cc -w

default=base=default=default: #执行
CC = /update1/bin/cc -w

default=peak=default=default: #执行
CC = /update2/bin/cc -w

$ runspec –config=tmp | grep sjeng.c
/update1/bin/cc -w -c -o sjeng.o -DSPEC_CPU -DNDEBUG sjeng.c
/update2/bin/cc -w -c -o sjeng.o -DSPEC_CPU -DNDEBUG sjeng.c

$ cat tmp.cfg
runlist = sjeng
size = test
iterations = 1
tune = base
output_format = text
teeout = 1

default=default=default:
LIBS = -lslowmalloc

default=default=myke:
LIBS = -lbsdmalloc

default=default=yusuf:
LIBS = -lthread -lmtmalloc
$
$ runspec –config=tmp –extension=myke | grep sjeng.o
cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG sjeng.c
cc attacks.o book.o crazy.o draw.o ecache.o epd.o eval.o leval.o moves.o
neval.o partner.o proof.o rcfile.o search.o see.o seval.o sjeng.o ttable.o
utils.o -lbsdmalloc -o sjeng
$
$ runspec –config=tmp –extension=yusuf | grep sjeng.o
cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG sjeng.c
cc attacks.o book.o crazy.o draw.o ecache.o epd.o eval.o leval.o moves.o
neval.o partner.o proof.o rcfile.o search.o see.o seval.o sjeng.o ttable.o
utils.o -lthread -lmtmalloc -o sjeng
$
$ cd $SPEC/benchspec/CPU2006/458.sjeng/exe
$ ls -lt | head -3
total 5888
-rwxrwxr-x 1 alan staff 244688 May 11 16:32 sjeng_base.yusuf
-rwxrwxr-x 1 alan staff 244628 May 11 16:31 sjeng_base.myke
$

benchmark>suite>tuning>extension

$ cat tmp.cfg
runlist = sjeng
size = test
iterations = 1
tune = base
output_format = text
teeout = 1

default=default=default=default:
OPTIMIZE = -xO2 -w
CC = /opt/SUNWspro/bin/cc
LIBS = -lbsdmalloc

458.sjeng=default=default=default:
OPTIMIZE = -xO4 -w

default=peak=default=default:
CC = /update1/bin/cc

default=default=mt=default:
LIBS = -lthread -lmtmalloc

$ runspec –config=tmp –tune=peak –ext=mt | grep sjeng.o
/update1/bin/cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG -xO4 -w sjeng.c
/update1/bin/cc -xO4 -w attacks.o book.o crazy.o draw.o ecache.o epd.o
eval.o leval.o moves.o neval.o partner.o proof.o rcfile.o search.o see.o
seval.o sjeng.o ttable.o utils.o -lthread -lmtmalloc -o sjeng
$

Notice above that all three sections applied: the section specifier for 458.sjeng, the specifier for peak tuning, and the specifier for extension mt.