[Beowulf] Re: Why Do Clusters Suck?

[Joe Landman]

Robert G. Brown wrote:
> On Tue, 22 Mar 2005, Andrew D. Fant wrote:

[...]

> Hmmm.  OK, how's this.  Just supposing that I finish building xmlbenchd
> before an infinite amount of time elapses (I've once again gotten mired
> in teaching and haven't had time to work on it for a week+).  Suppose
> xmlbenchd can run any given program inside a fairly standard timing
> wrapper (probably a perl script for maximum portability and ease of
> use).  Suppose that the perl script, which will certainly contain the
> command line for the application which therefore will (for fixed random
> number seeds where appropriate) produce some sort of fixed output.

Hmmm.  Have you seen the input and output to BBS (in retrospect, a 
poorly named tool)?

In BBS (bioinformatics benchmark system), you create input experiments 
using simple XML documents that describe the binary, the arguments, 
where STD* should go or come from.  You get output in XML format for the 
experiment in the form of timing.  It would not be too difficult a 
stretch to include regex pattern matchers on the output, or similar, to 
specifically test for "goodness".

That is, suppose I want to run a nice simple GAMESS run:

<?xml version="1.0" ?>
<package name='gamess'>
  <description>
   This package implements a GAMESS benchmark we have found useful for
   testing compiler and machine options and configurations.
  </description>
  <author name='Joe Landman' email='landman at scalableinformatics.com'
   web='http://www.scalableinformatics.com' />
<license>GPL 2.0</license>
  <home 
url="http://downloads.scalableinformatics.com/downloads/bbs/baseline/" />
  <global>
   <!-- package defaults, override if needed in each experiment -->
   <stdout     path="STDOUT.%s.%s"    parameters="$instance,$thread" 
eval="true" />
   <stderr     path="STDERR.%s.%s"    parameters="$instance,$thread" 
eval="true" />
   <stdin      path=""    eval="false" 
/>
   <exec       parameters= "$options"
	      shell= '/bin/bash'
	      environment = ''
   />
  </global>
  <!-- experiment(s), inherit the global variables,
       and can make local modifications as needed
    -->
  <experiment 	name 		= 'test1-1CPU'		order = '1'		 >
   <program	binary 		= '/big/chem/build/gamess/rungms '  />
   <options>
    <option value = ' test1 02 1 '     />
   </options>	
   <run 		instances	= "1"		/>
  </experiment>
  <experiment 	name 		= 'test1-2CPU'		order = '2'		 >
   <program	binary 		= '/big/chem/build/gamess/rungms '  />
   <options>
    <option value = ' test1 02 2 '     />
   </options>	
   <run 		instances	= "1"		/>
  </experiment>
</package>

This will run 2 GAMESS timing experiments (benchmarks), with names of 
test1-1CPU, and test1-2CPU, hand the appropriate arguments to the 
program and run each test in order.  We can launch multiple instances, 
and simultaneous invocations of multiple instances with different 
experiments.

The output from this looks like this (after filtering through simplexml, 
included in the BBS package):

<?xml version="1.0" ?>
  <package name="gamess">
   <experiment name="test1-1CPU">
    <thread number="1" runtime="1.158708" />
   </experiment>
   <experiment name="test1-2CPU">
    <thread number="1" runtime="1.139958" />
   </experiment>
  </package>

We could add in some tags like

   <check>
    <compare output=stdout known_good=good_file method=diff ... />
    <compare output=stdout method=regex regex="..."  ... />
    ...
   </check>

or similar.

FWIW:  BBS is at http://www.scalableinformatics.com/BBS , is GPL, and is 
in active use by a number of groups/companies for testing purposes.

> 
> Then it would be trivial to add a segment to at LEAST diff the output
> with the expected output, and not a horrible amount of work to actually
> compute a chisq difference between the two.  I can easily introduce xml
> tags for returning a validation score on the actual result (or even a
> set of such scores) because extensibility IS useful during the time a
> new thing is being invented (sorry, Don:-) if not beyond.  This would
> permit the best of both worlds:

Could leverage what exists rather than re-inventing this particular 
wheel.  Let me know if there are particular things you would like to see 
in the output comparison.  Could do a chi-square, but this makes more 
sense for numerical bits than non-numerical bits (BBS doesn't care, and 
may the solution to this are analysis plug-ins that implement the 
appropriate tests).

> 
>   a) I expect to assemble a set of macro-level applications to function
> somewhat like the spec suite does today but without the "corporation"
> baggage, for distribution WITH the package.  At that point I will
> actually solicit this list for good candidates for primary inclusion.
> This set can actually be quite large, permitting users to preselect at
> configuration time the ones to run for their particular site.  For the
> ones that are selected, I will go ahead and do the validation test for
> when I wrap them up in the timing script.

Heh...

	bbsrun --experiment "test1-1CPU" --debug < gamess.xml

Include the XML with your package, and bbs can (largely) do the rest.

> 
>   b) Users who want to wrap their OWN application set up for automated
> benchmarking inside the provided template script will then be able to
> follow fairly simple instructions and (presuming that they know enough
> perl to be able to parse their application's output file(s)) validate as
> well as time to their heart's content.
> 
> This may not be sufficient for all users -- I'm probably not going to
> write a core loop that would permit a sweep of an input parameter in the
> command line, for example, and to test e.g. special function calls in
> the GSL that change algorithms at certain breakpoints that kind of thing
> is really necessary.  However, folks with more advanced needs will
> presumably be more advanced programmers and the perl to add such a sweep
> and generate a more complex validation isn't terrribly challenging.

:)

> 
> Would that do?

As most of this exists in BBS now, and it is in active use, I would say 
yes. :)

> 
>     rgb
> 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615