Newbie question

[Rayson Ho]

--- Mike <mladwig at comcast.net> wrote:
> An optimization would be to prelocate the constants to the node(s)
> that will be using them, keep them in memory or on local disk, and 
> not need to transfer them while a work unit is ongoing.   When I've 
> looked at some of the batch processing tools (e.g. SGE - thanks for 
> the suggestion Rayson), I don't see a  way to support this kind of 
> resource "affinity".  Did I miss something?

You can define what resource (which cache files) you have on each node.
You then submit your jobs with the "-l" option, which tells SGE what
type of resources the jobs need.

Another way (a dynamic way) to use the above optimization is:

1) setup a cache directory, and set up a load sensor to report what's
available in the cache directory on each node.

2) You submit the jobs with "soft" requesting the resource (i.e. if all
nodes are in use, SGE still schedules your jobs to the nodes without
the cache files even if the resources are not available, something like
best-effort).

3) In your jobscript, you check if the files are in the cache
directory, if not, transfer from the file server.

4) If the cache directory is full, you delete the oldest cached files.
(the load sensor will take care of updating the resource availability
list)

I am not an SGE expert, you can get more details from the SGE mailing
list...

Rayson



__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com