Archive for May 18th, 2009

Performance Testing

May 18, 2009

For no particularly good reason, I decided to clean up the script I used recently to get my performance testing data, and thought I’d share. The script is a bash script, to be run on linux machines. The way I’ve been setting things up (inspired by what Eric said he was doing), I make a directory (folder) for each problem number, and in that directory I have a bunch of .py files, one for each different version of the program. I call my scripts, e.g., prob1-v1.py, prob1-v2.py, and so on. Each of the scripts (that I want to test anyway) is set up to take command line arguments (see my other post for how to do that). The script below assumes that the argument that is changing comes first, and any other arguments are fixed, and come after that. Also, it only works for pure python scripts currently, no sage.

The output of my script is (when things work) a URL that uses the Google Charts API to draw a graph. Visiting that output URL using your favorite (open source) browser should turn up a sweet graph (hopefully). To include the graph in a post here, click the button next to ‘Upload/Insert’ above the editor box (when you mouseover the button, it says ‘Add an Image’), then click ‘From URL’ in the top of the “popup” and go from there.

There’s as much flexibility built in to this script as I could handle this evening. You’ll likely have to tinker with some of the lines to get them working with whatever scheme you’ve got going at home. You’ll also have to tinker with them for different problems, because (at least) the command line arguments will change. Sections you are most likely to have to change have been noted. As things are currently shown below, I am comparing scripts prob1-v3.py and prob1-v6.py (designed to solve problem 1) with inputs ranging from 1000 to 10000 in multiples of 1000. Both of these scripts also take the list of numbers you want to sum multiples of, in the base case of the problem this is “3 5”, which is used below (variable E).

#/bin/bash

###################################
### initial setup block. mess with these values
###################################

# assume scripts have names "probP-vN.py". e.g., "prob1-v3.py"
P=1 # the problem number
V="3 6" # the versions to compare

T=50 # number of times to run each program with each input for averaging

# I should be the string of values to use as inputs
# this will be the changing argument that performance is tested against,
# which scripts should be expecting as the first argument
I="$(seq 1000 1000 10000)"

# the extra arguments that should get passed each time, not varying
E="3 5"

##################################
### url output block. may want to mess with it
##################################
# documentation available at:
# http://code.google.com/apis/chart/
##################################
# set things up here, or, after completion,
# mess with just these parts of the output
##################################
echo -n "http://chart.apis.google.com/chart?"
echo -n "cht=lxy&" # type
echo -n "chs=300x200&" # size
echo -n "chxt=x,y&" # place tickmarks on x and y axis
echo -n "chxr=0,1,10,1|1,0,5,1&" # the tickmark numbers on the axes
echo -n "chco=FF0000,0000FF&" # color for each graph
echo -n "chdl=Sets|Sums&" # legend
echo -n "chds=1000,10000,0,5,1000,10000,0,5&" # pairs of min/max for each data set
echo -n "chd=t:" # data, to be output below

##################################
### generating data block. probably mostly safe to not mess with
##################################

# generate the string of x values, from I (just put , between values)
X="$(for J in ${I}; do echo -n "${J},"; done; echo -n "|")"

( # wrap up all the plot data output, for some post-processing

# start testing
for W in ${V}; do

    # print the string of "x" values for the Google Chart data chd=t:
    echo -n ${X}

    # loop through each input to be tested
    for J in ${I}; do
	C="scale=10;(" # string we'll pass to bc to calculate average

	# run the program ${T} times, getting runtime each time
	for A in $(seq 1 ${T}); do
	    O="$(python prob${P}-v${W}.py ${J} ${E})" # program's output

	    # pull out just the runtime of the output
	    R="$(echo ${O} | awk '{print $3}')"
	    # bc doesn't like scientific notation
	    R="$(echo ${R} | sed -s 's/e/*10^/')"
	    # scale by 1000 for no particularly good reason
	    # except if runtimes are along the lines of milliseconds...
	    R="$(echo "scale=10;1000*${R}" | bc)"

	    # concatenate R to C, and a plus sign
	    C="${C}${R}+"
	done

	# cap off C, compute the average, truncate to 4 characters
	echo -n "$(echo "${C}0)/${T}" | bc | head -c 4),"
    done

    echo -n "|"
done

# all done, pull out extraneous symbols
) | sed -s 's/,|/|/g' | sed -s 's/|$//'

echo

Anyway, if you feel like using it, go right ahead. If not, no worries. It remains to be seen how much I’ll use it myself. Please share success/failure stories. If you’d like to rearrange the script to accomodate your home setup, I’m happy to try to help. I’m sure there’s all sorts of room for improvement, but I hope it’s good enough for now.

Now, what was I supposed to be doing today?

Update: Just after posting, I realized that I failed to mention that this script makes an assumption about the output of the python scripts it is running. It expects the output to look like “1234 in 0.00456587 sec”, where 1234 is the answer calculated, and 0.00456587 is the time as calculated in, e.g., the scripts of my original comparison of solutions for problem 1.