[Beowulf] How to find a swapped out, runnable process?

[Matt Phillips]

I am running RH9 (2.4.20-9SGI_XFS_1.2.0smp) on a 16-node cluster. I noticed 
the load on the I/O node to be consistently high after one of the clients 
crashed during rsync. I did vmstat and found that 1 or more process are 
always in the runnable but swapped out queue.. Here's a sample output of 
vmstat

procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 1 30120 9508 176 1609060 0 0 1 1 0 2 0 0 0
0 0 1 30120 9508 176 1609060 0 0 0 0 118 237 0 3 97
0 0 2 30120 9508 176 1609060 0 0 0 1064 252 670 0 1 99
0 0 1 30120 9508 176 1609060 0 0 0 36 156 330 0 0 100
0 0 1 30120 9508 176 1609060 0 0 0 268 157 305 0 0 100
0 0 1 30120 9508 176 1609060 0 0 0 8 129 245 0 0 100

As you can see, there is always one process in procs/w queue.. How do I find 
which process is this? I tried various combos of ps (like looking at wchan, 
stat outputs etc, variations of top).. but ps/top only show 1-2 process in 
the runnable queue and doesnt indicated if they are swapped. Maybe I am 
reading the man pages incorrectly.

Anyone has ideas how I can catch this errant process?

TIA,
Matt

_________________________________________________________________
On the road to retirement? Check out MSN Life Events for advice on how to 
get there! http://lifeevents.msn.com/category.aspx?cid=Retirement