[Beowulf] gpu benchmark

[Michael Di Domenico]

I'm looking for a benchmark that can keep a gpu busy (ideally both
compute and memory) for 2 or 3 hours.  but here's the kick, at the end
of the benchmark it needs to check it's answers

i'm trying to hunt down some potentially bad hardware.  linpack works
great for the bulk of it, but the nodes i have, have more gpu power
then ram available, so a linpack run using the full ram of a single
box doesn't run long enough.  and successive runs one after another
doesn't seem to trigger it

i can trigger the error using large mpi jobs across many nodes, but
that doesn't let me isolate which gpu was at fault.  and since these
are GTX level cards, no ecc and no error diagnostics on the console...

i'm going through my bag of tricks, but haven't come up with anything just yet.