From ispmarin at gmail.com Mon Oct 1 05:34:46 2007 From: ispmarin at gmail.com (Ivan Paganini) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <200709301202.01236.csamuel@vpac.org> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> Message-ID: <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> Hello Chris, everybody: I am not using jumbo frames, and I'm now considering this option, but first I wanted to know for sure that there is no other problem before, just to control the number of variables at hand. But thanks for your help. I did a strace on the hanged process, and the output is this: ______________________________________________ mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x401 76000 read(4, "#\n# hosts This file desc"..., 4096) = 4096 read(4, "yriBlade077\n192.168.30.178 myri"..., 4096) = 4096 read(4, " blade067 blade067.lcca.usp.br\n1"..., 4096) = 2055 read(4, "", 4096) = 0 close(4) = 0 munmap(0x40176000, 4096) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil d_tidptr=0x40046f68) = 25994 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil d_tidptr=0x40046f68) = 25995 brk(0x102ab000) = 0x102ab000 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil d_tidptr=0x40046f68) = 25996 waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) --- SIGWINCH (Window changed) @ 0 (0) --- waitpid(-1, ______________________________________________ and just that. I'm now trying to make a better undestanding that what is happening. Thank you. Ivan 2007/9/29, Chris Samuel : > On Sat, 29 Sep 2007, Ivan Paganini wrote: > > > I sniffed the network in the store nodes interface, and i got lots > > of TCP lost fragment, previos lost fragments, ack lost fragments > > and TCP window size full. > > Some suggestions would be to check that all network interfaces are > negotiating gigabit back to the switch, and that if you are using > jumbo frames then all interfaces are indeed using jumbo frames. > > A useful check to verify 2 way jumbo frames connectivity is by using > the ping command, doing: > > ping -c 1 -M do -s 8900 $hostname > > should tell you whether or not it is working. > > Best of luck! > Chris > -- > Christopher Samuel - (03) 9925 4751 - Systems Manager > The Victorian Partnership for Advanced Computing > P.O. Box 201, Carlton South, VIC 3053, Australia > VPAC is a not-for-profit Registered Research Agency > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ----------------------------------------------------------- Ivan S. P. Marin ---------------------------------------------------------- From ispmarin at gmail.com Mon Oct 1 06:05:43 2007 From: ispmarin at gmail.com (Ivan Paganini) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> Message-ID: <751c63ee0710010605g2a26bce8udbd955741b42650e@mail.gmail.com> Just a update: trying several times, the strace stops in different points, the speficied in the other email and here: _______________________________________________ munmap(0x40176000, 4096) = 0 time([1191243868]) = 1191243868 open("/etc/hosts", O_RDONLY) = 4 fcntl64(4, F_GETFD) = 0 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 fstat64(4, {st_mode=S_IFREG|0644, st_size=10247, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40176000 read(4, "#\n# hosts This file desc"..., 4096) = 4096 read(4, "yriBlade077\n192.168.30.178 myri"..., 4096) = 4096 read(4, " blade067 blade067.lcca.usp.br\n1"..., 4096) = 2055 read(4, "", 4096) = 0 close(4) = 0 munmap(0x40176000, 4096) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x40046f68) = 31382 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x40046f68) = 31383 brk(0x102ab000) = 0x102ab000 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x40046f68) = 31384 waitpid(-1, _______________________________________________ Thanks. 2007/10/1, Ivan Paganini : > Hello Chris, everybody: > > I am not using jumbo frames, and I'm now considering this option, but > first I wanted to know for sure that there is no other problem before, > just to control the number of variables at hand. But thanks for your > help. > > I did a strace on the hanged process, and the output is this: > ______________________________________________ > > mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x401 > 76000 > read(4, "#\n# hosts This file desc"..., 4096) = 4096 > read(4, "yriBlade077\n192.168.30.178 myri"..., 4096) = 4096 > read(4, " blade067 blade067.lcca.usp.br\n1"..., 4096) = 2055 > read(4, "", 4096) = 0 > close(4) = 0 > munmap(0x40176000, 4096) = 0 > clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil > d_tidptr=0x40046f68) = 25994 > clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil > d_tidptr=0x40046f68) = 25995 > brk(0x102ab000) = 0x102ab000 > clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, chil > d_tidptr=0x40046f68) = 25996 > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, 0xffffdbc8, 0) = ? ERESTARTSYS (To be restarted) > --- SIGWINCH (Window changed) @ 0 (0) --- > waitpid(-1, > > ______________________________________________ > and just that. I'm now trying to make a better undestanding that what > is happening. > > Thank you. > > Ivan > > > 2007/9/29, Chris Samuel : > > On Sat, 29 Sep 2007, Ivan Paganini wrote: > > > > > I sniffed the network in the store nodes interface, and i got lots > > > of TCP lost fragment, previos lost fragments, ack lost fragments > > > and TCP window size full. > > > > Some suggestions would be to check that all network interfaces are > > negotiating gigabit back to the switch, and that if you are using > > jumbo frames then all interfaces are indeed using jumbo frames. > > > > A useful check to verify 2 way jumbo frames connectivity is by using > > the ping command, doing: > > > > ping -c 1 -M do -s 8900 $hostname > > > > should tell you whether or not it is working. > > > > Best of luck! > > Chris > > -- > > Christopher Samuel - (03) 9925 4751 - Systems Manager > > The Victorian Partnership for Advanced Computing > > P.O. Box 201, Carlton South, VIC 3053, Australia > > VPAC is a not-for-profit Registered Research Agency > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > > -- > ----------------------------------------------------------- > Ivan S. P. Marin > ---------------------------------------------------------- > -- ----------------------------------------------------------- Ivan S. P. Marin ---------------------------------------------------------- From patrick at myri.com Mon Oct 1 06:35:38 2007 From: patrick at myri.com (Patrick Geoffray) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> Message-ID: <4700F7AA.9020407@myri.com> Hi Ivan, Ivan Paganini wrote: > The myrinet connection was working right, but sometimes a user program > just got stuck - one of the processes was sleeping, and all others > were running. Then, the program hangs. Investigating this further, Unless you are using bocking receives ("--mx-recv blocking" or "--mx-recv hybrid"), the default mode is polling. So, a process will only sleep if it is still in the spawning phase (in MPI_Init) or if it's blocking on something outside MPI (like disk IO). > overheat. mpirun.ch_mx -v shows that all the processes are issued ok > to the nodes, but somehow one (or more) process go to sleep or never > starts, and all the other processes just hangs. The mx diagnose tools All processes wait on everybody at spawn time, so if one process never starts, the rest of the MPI world will wait for it, possibly forever. The root problem is the process not starting. The spawning phase in MPICH-MX uses socket and ssh (or rsh). Usually, ssh uses native Ethernet, but it could also use IPoM (Ethernet over Myrinet). Which case is it for you ? Patrick From patrick at myri.com Mon Oct 1 06:38:34 2007 From: patrick at myri.com (Patrick Geoffray) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> Message-ID: <4700F85A.2030300@myri.com> Mark Hahn wrote: > here's an idea: configure ip-over-myrinet, and use it exclusively > to start the jobs. if that works, then you know for sure that the > problem is solely on the eth side (switch, perhaps, or maybe a nic > that's jabbering or otherwise misbehaving?) Ivan may have to stage the binary on local disk prior to spawning, to not rely on GPFS over Ethernet to serve it. Or even run GFPS over IPoM too. Patrick -- Patrick Geoffray Myricom, Inc. http://www.myri.com From patrick at myri.com Mon Oct 1 06:42:42 2007 From: patrick at myri.com (Patrick Geoffray) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> Message-ID: <4700F952.10307@myri.com> Ivan, Ivan Paganini wrote: > I did a strace on the hanged process, and the output is this: "strace -f" to trace the children as well. Could you send the output of mpirun.ch_mx -v also, to see if the process starts and send some info to the mpirun perl script and hangs later or never really starts. What is your Myricom Tech Support ticket number ? Patrick -- Patrick Geoffray Myricom, Inc. http://www.myri.com From hahn at mcmaster.ca Mon Oct 1 06:44:29 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <751c63ee0710010605g2a26bce8udbd955741b42650e@mail.gmail.com> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> <751c63ee0710010605g2a26bce8udbd955741b42650e@mail.gmail.com> Message-ID: > clone(child_stack=0, > flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, > child_tidptr=0x40046f68) = 31384 > waitpid(-1, this looks like a fork/exec that's failing. as you might expect if, for instance, your shared FS doesn't supply a binary successfully. note also that ltrace -S often provides somewhat more intelligible diags for this kind of thing (since it might show what's actually being exec'ed.) From ispmarin at gmail.com Mon Oct 1 07:06:45 2007 From: ispmarin at gmail.com (Ivan Paganini) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> <751c63ee0710010605g2a26bce8udbd955741b42650e@mail.gmail.com> Message-ID: <751c63ee0710010706x3444feeem77e560c82df05d07@mail.gmail.com> Hello Mark, Patrick, >>The spawning phase in MPICH-MX uses socket and ssh (or rsh). Usually, >>ssh uses native Ethernet, but it could also use IPoM (Ethernet over >>Myrinet). Which case is it for you ? As I said before, I'm also experiencing some ether problems (in the service network) like TCP window full, lost segments, ack lost segments, and trying to rule this out too. I'm using the IPoM, as the manual says, because I configured each node with ifconfig myri0 192.168.30. and associated this number on the /etc/hosts with a hostname, like myriBlade. I am also using ssh and polling method. the mpirun.ch_mx -v with a hanged process is below: ___________________________________________ ivan@mamute:~/lib/mpich-mx-1.2.7-5-xl/examples> mpirun.ch_mx -v --mx-label --mx-kill 30 -machinefile list -np 3 ./cpi Program binary is: /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel Program binary is: /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi Machines file is /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/list Processes will be killed 30 after first exits. mx receive mode used: polling. 3 processes will be spawned: Process 0 (/mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi ) on mamute Process 1 (/mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi ) on mamute Process 2 (/mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi ) on myriBlade109 Open a socket on mamute... Got a first socket opened on port 55353. ssh mamute "cd /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples && exec env MXMPI_MAGIC=3366365 MXMPI_MASTER=mamute MXMPI_PORT=55353 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=0 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.15.1 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi " ssh mamute -n "cd /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples && exec env MXMPI_MAGIC=3366365 MXMPI_MASTER=mamute MXMPI_PORT=55353 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=1 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.15.1 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi " ssh myriBlade109 -n "cd /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples && exec env MXMPI_MAGIC=3366365 MXMPI_MASTER=mamute MXMPI_PORT=55353 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=2 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.30.209 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/examples/./cpi " All processes have been spawned MPI Id 0 is using mx port 0, board 0 (MAC 0060dd47afe7). MPI Id 2 is using mx port 0, board 0 (MAC 0060dd478aff). MPI Id 1 is using mx port 1, board 0 (MAC 0060dd47afe7). Received data from all 3 MPI processes. Sending mapping to MPI Id 0. Sending mapping to MPI Id 1. Sending mapping to MPI Id 2. Data sent to all processes. ___________________________________________ and hanged. The list file includes mamute:2 myriBlade109:4 myriBlade108:4 where mamute is my headnode, so I can do all the traces. >>Ivan may have to stage the binary on local disk prior to spawning, to >>not rely on GPFS over Ethernet to serve it. Or even run GFPS over IPoM too. GPFS over myri now is not an option. I compiled the executable staticaly and tested it. Same problem. Now I staged the binary in the scrath partition in each node, and the process hanged the same way: __________________________________________ ivan@mamute:/home/ivan> mpirun.ch_mx -v --mx-label --mx-kill 30 -machinefile list -np 3 ./cpi Program binary is: /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel Program binary is: /home/ivan/./cpi Machines file is /home/ivan/list Processes will be killed 30 after first exits. mx receive mode used: polling. 3 processes will be spawned: Process 0 (/home/ivan/./cpi ) on mamute Process 1 (/home/ivan/./cpi ) on mamute Process 2 (/home/ivan/./cpi ) on myriBlade109 Open a socket on mamute... Got a first socket opened on port 55684. ssh mamute "cd /home/ivan && exec env MXMPI_MAGIC=1802255 MXMPI_MASTER=mamute MXMPI_PORT=55684 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=0 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.15.1 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /home/ivan/./cpi " ssh mamute -n "cd /home/ivan && exec env MXMPI_MAGIC=1802255 MXMPI_MASTER=mamute MXMPI_PORT=55684 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=1 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.15.1 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /home/ivan/./cpi " ssh myriBlade109 -n "cd /home/ivan && exec env MXMPI_MAGIC=1802255 MXMPI_MASTER=mamute MXMPI_PORT=55684 MX_DISABLE_SHMEM=0 MXMPI_VERBOSE=1 MXMPI_SIGCATCH=1 LD_LIBRARY_PATH=/usr/lib:/usr/lib64 MXMPI_ID=2 MXMPI_NP=3 MXMPI_BOARD=-1 MXMPI_SLAVE=192.168.30.209 /mamuteData/ivan/lib/mpich-mx-1.2.7-5-xl/bin/mpimxlabel /home/ivan/./cpi " All processes have been spawned MPI Id 1 is using mx port 0, board 0 (MAC 0060dd47afe7). MPI Id 2 is using mx port 0, board 0 (MAC 0060dd478aff). MPI Id 0 is using mx port 1, board 0 (MAC 0060dd47afe7). Received data from all 3 MPI processes. Sending mapping to MPI Id 0. Sending mapping to MPI Id 1. Sending mapping to MPI Id 2. Data sent to all processes. __________________________________________ I notice, thought, that the spawing is _much_ faster than firing the process from the GPFS partition. This is the output of strace -f (lots of things here!): ________________________________________ [pid 7498] ioctl(4, TCGETS or TCGETS, 0xffffda30) = -1 EINVAL (Invalid argument) [pid 7498] _llseek(4, 0, 0xffffda98, SEEK_CUR) = -1 ESPIPE (Illegal seek) [pid 7498] fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 [pid 7498] setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 [pid 7498] connect(4, {sa_family=AF_INET, sin_port=htons(55787), sin_addr=inet_addr("192.168.15.1")}, 16) = 0 [pid 7498] write(1, "Sending mapping to MPI Id 1.\n", 29Sending mapping to MPI Id 1. ) = 29 [pid 7498] send(4, "[[[<0:96:3712462823:0><1:96:3712"..., 72, 0) = 72 [pid 7498] close(4) = 0 [pid 7498] time([1191247146]) = 1191247146 [pid 7498] open("/etc/hosts", O_RDONLY) = 4 [pid 7498] fcntl64(4, F_GETFD) = 0 [pid 7498] fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 [pid 7498] fstat64(4, {st_mode=S_IFREG|0644, st_size=10247, ...}) = 0 [pid 7498] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000 [pid 7498] read(4, "#\n# hosts This file desc"..., 4096) = 4096 [pid 7498] read(4, "yriBlade077\n192.168.30.178 myri"..., 4096) = 4096 [pid 7498] read(4, " blade067 blade067.lcca.usp.br\n1"..., 4096) = 2055 [pid 7498] read(4, "", 4096) = 0 [pid 7498] close(4) = 0 [pid 7498] munmap(0x40018000, 4096) = 0 [pid 7498] open("/etc/protocols", O_RDONLY) = 4 [pid 7498] fcntl64(4, F_GETFD) = 0 [pid 7498] fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 [pid 7498] fstat64(4, {st_mode=S_IFREG|0644, st_size=6561, ...}) = 0 [pid 7498] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000 [pid 7498] read(4, "#\n# protocols\tThis file describe"..., 4096) = 4096 [pid 7498] close(4) = 0 [pid 7498] munmap(0x40018000, 4096) = 0 [pid 7498] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4 [pid 7498] ioctl(4, TCGETS or TCGETS, 0xffffda30) = -1 EINVAL (Invalid argument) [pid 7498] _llseek(4, 0, 0xffffda98, SEEK_CUR) = -1 ESPIPE (Illegal seek) [pid 7498] ioctl(4, TCGETS or TCGETS, 0xffffda30) = -1 EINVAL (Invalid argument) [pid 7498] _llseek(4, 0, 0xffffda98, SEEK_CUR) = -1 ESPIPE (Illegal seek) [pid 7498] fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 [pid 7498] setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 [pid 7498] connect(4, {sa_family=AF_INET, sin_port=htons(45412), sin_addr=inet_addr("192.168.30.209")}, 16) = 0 [pid 7498] write(1, "Sending mapping to MPI Id 2.\n", 29Sending mapping to MPI Id 2. ) = 29 [pid 7498] send(4, "[[[<0:96:3712462823:0><1:96:3712"..., 69, 0) = 69 [pid 7498] close(4) = 0 [pid 7498] alarm(0) = 0 [pid 7498] write(1, "Data sent to all processes.\n", 28Data sent to all processes. ) = 28 [pid 7498] accept(3, [pid 7499] <... select resumed> ) = 1 (in [3]) [pid 7499] read(3, "\302\317\32\275\357jD\230\222=\270N\341F\237\326@]\4\4"..., 8192) = 80 [pid 7499] select(7, [3 4], [6], NULL, NULL) = 1 (out [6]) [pid 7499] write(6, "0: Process 0 on mamute.lcca.usp."..., 350: Process 0 on mamute.lcca.usp.br ) = 35 [pid 7499] select(7, [3 4], [], NULL, NULL ________________________________________ and hangs. This was with the binary _out_ of GPFS and statically compiled. My ticket number is 53912, and Ruth and Scott are helping me. Mark, ltrace does not accepts the mpirun.ch_mx as a valid elf binary... it was compiled using the xlc compiler. Strange, because it works with other system binaries (like ls...) Thank you very much!! Ivan 2007/10/1, Mark Hahn : > > clone(child_stack=0, > > flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, > > child_tidptr=0x40046f68) = 31384 > > waitpid(-1, > > this looks like a fork/exec that's failing. as you might expect > if, for instance, your shared FS doesn't supply a binary successfully. > note also that ltrace -S often provides somewhat more intelligible > diags for this kind of thing (since it might show what's actually > being exec'ed.) > -- ----------------------------------------------------------- Ivan S. P. Marin ---------------------------------------------------------- From scheinin at crs4.it Mon Oct 1 10:11:32 2007 From: scheinin at crs4.it (Alan Louis Scheinine) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Problems with a JS21 - Ah, the networking... In-Reply-To: <751c63ee0710010706x3444feeem77e560c82df05d07@mail.gmail.com> References: <751c63ee0709281343r535364d2rdfa8db9aed8426c@mail.gmail.com> <200709301202.01236.csamuel@vpac.org> <751c63ee0710010534u1c611f03x8d31bb1fa58165b9@mail.gmail.com> <751c63ee0710010605g2a26bce8udbd955741b42650e@mail.gmail.com> <751c63ee0710010706x3444feeem77e560c82df05d07@mail.gmail.com> Message-ID: <47012A44.6090209@crs4.it> I found that the blocking send of MPI blocks for the version of MPICH compiled for Myrinet (at least w.r.t. the old GM) but does not block for the MPICH from Argonne compile for GCC and PGI. Or was it the other way around, I don't recall. Assuming it was the first case, it might be relavent. Someone wrote a program that had deadlock, a simple send() ; recv(); in the same order for two processes will cause deadlock. But the program ran always until he changed the connection and hence the underlying drivers. From X.Jin at Bradford.ac.uk Mon Oct 1 08:22:36 2007 From: X.Jin at Bradford.ac.uk (X.Jin) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] CFP -- PMEO-UCNS'08 to be held with IPDPS'08 Message-ID: <200710011522.l91FMabL016908@radon.cen.brad.ac.uk> [Please accept our apologies if you receive multiple copies of this emails] ************************************************************** (Selected high-quality papers from the PMEO-UCNS'08 workshop will appear in a Special Issue of the journal - International Journal of Parallel, Emergent, and Distributed Systems (IJPEDS), Taylor & Francis - http://www.tandf.co.uk/journals/titles/17445760.asp) ************************************************************** CALL FOR PAPERS 7th International Workshop on Performance Modeling, Evaluation, and Optimization of Ubiquitous Computing and Networked Systems (PMEO-UCNSĄŻ08) To be held in conjunction with IPDPS'08 (Supported by IEEE Computer Society in cooperation with ACM SIGARCH), 14 -18 April, 2008, Miami, Florida, USA http://www.inf.brad.ac.uk/~gmin/PMEO-08.html http://www.ipdps.org/ipdps2008/2008_workshops.html SCOPE: The performance modeling, evaluation, and optimization of ubiquitous computing and networked systems have been an important research topic over the past years and poses challenging problems that require new tools and methods to keep up with the rapid evolution and increasing complexity of such systems. This workshop will bring together scientists, engineers, practitioners, and computer users to share and exchange their experiences, discuss challenges, and report state-of-the-art and in-progress research on all aspects of performance modeling, evaluation, and optimization of ubiquitous computing and networked systems. The topics of interest include, but are not limited to: -- Predictive performance models of ubiquitous computing systems -- Predictive performance models of wired and wireless networks -- Performance measurement and monitoring tools -- Tracing and trace analysis -- Simulation -- Analytical modeling -- Software tools for system performance and evaluation -- Automatic performance analysis -- Performance comparison -- Performance of memory and I/O interconnect -- Performance of communication networks -- Performance of mobile distributed systems -- Performance analysis and evaluation of ubiquitous computing and networked applications -- Improvement in system performance through optimization and tuning -- Case studies showing the role of evaluation in the design of systems WORKSHOP CO-CHAIRS: Geyong Min Department of Computing University of Bradford Bradford, BD7 1DP, U.K. E-mail: g.min@brad.ac.uk Mohamed Ould-Khaoua Department of Computing Science University of Glasgow Glasgow, G12 8RZ, U.K. E-mail: mohamed@dcs.gla.ac.uk PROGRAM CHAIR: Ahmed Y. Al-Dubai School of Computing Napier University Edinburgh, EH10 5DT, U.K. E-mail: A.Al-Dubai@napier.ac.uk PUBLICITY CO-CHAIRS: Xiaolong Jin Department of Computing University of Bradford Bradford, BD7 1DP, U.K. E-mail: x.jin@brad.ac.uk Mirela Sechi Moretti Annoni Notare Barddal University Florianopolis, SC Brazil Email: mirela@barddal.br PROGRAM COMMITTEE: K. Al-Begain, Univ. of Glamorgan (UK) I. Romdhani, Napier University (UK) H. R. Arabnia, Univ. of Georgia (USA) D. K. Arvind, Edinburgh University (UK) I. Awan, Univ. of Bradford (UK) A. Boukerche, Univ. of North Texas (USA) J. Bradley, Imperial College London (UK) P. Cockshott, Univ. of Glasgow (UK) M. Colajanni, Univ. of Modena (Italy) K. Day, Sultan Qaboos Univ. (Oman) K. Djemame, Univ. of Leeds (UK) T. El-Ghazawi, George Washington University (USA) R. Fatoohi, San Jose State University (USA) M. Gueroui, University of Cergy-Pontoise (France) S. Helal, University of Florida (USA) H. Hassanein, Queen's University (Canada) S. Jarvis, Univ. of Warwick (UK) H. Karatza, Univ. of Thessaloniki (Greece) A. Katangur, Texas A&M Univ. (USA) A. Khonsari, IPM (Iran) W. Knottenbelt, Imperial College London (UK) K. Li, State Univ. of New York at New Paltz (USA) H. Liu, Huazhong Univ. of Science and Technology (China) S. Loucif, Moncton University, (Canada) L.M. Mackenzie, Univ. of Glasgow (UK) Y. Pan, Georgia State Univ. (USA) D. K. Pradhan, Univ. of Bristol (UK) H. Sarbazi-Azad, Sharif Univ. & IPM (Iran) A. Shahrabi, Glasgow Caledonian Univ. (UK) E. Song, Huazhong Univ. of Science and Technology (China) N. Thomas, Univ. of Newcastle (UK) A. Touzene, Sultan Qaboos Univ. (Oman) M. Woodward, Univ. of Bradford (UK) J. Wu, Florida Atlantic Univ. (USA) L. Xiao, Michigan State Univ. (USA) T. Xie, San Diego State University (USA) L. T. Yang, St Francis Xavier Univ. (Canada) W. Vanderbauwhede, University of Glasgow (UK) W. Buchanan, Napier University (UK) A. Zomaya, Univ. of Sydney (Australia) PAPER SUBMISSION: Authors are invited to submit manuscripts reporting original unpublished research and recent developments in the topics related to the workshop. The length of the papers should not exceed 8 pages (IEEE Computer Society Proceedings Manuscripts style: two columns, single-spaced), including figures and references, using 10 fonts, and number each page. Papers should be submitted electronically in PDF format (or postscript) by sending it as an e-mail attachment to A.Al-Dubai@napier.ac.uk. All papers will be peer reviewed and the comments will be provided to the authors. The accepted papers will be published together with those of other IPDPS'08 workshops by the IEEE Computer Society Press. IMPORTANT DATES: Submission Deadline: October 21, 2007 Author Notification: December 11, 2007 Final Manuscript Due: January 28, 2008 ************************************************************** From hcxckwk at hkucc.hku.hk Tue Oct 2 04:10:16 2007 From: hcxckwk at hkucc.hku.hk (Kwan Wing Keung) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs Message-ID: This is perhap a naive question. 10 years before we started using the SP2, but we later changed to Intel based linux beowulf in 2001. In our University there are quite a no. of mpi-based parallel programs running in a 178 node dual-Xeon PC cluster that was installed 4 years ago. We are now planning to upgrade our cluster in the coming year. Very likely blade servers with multi-core CPUs will be used. To port these mpi-based parallel programs to a multi-core CPU environment, someone suggested that OpenMP should be used, such that the programs can be converted to a multi-thread version. Nevertheless it may take time, and the users may be reluctant to do so. Also for some of the installed programs, we don't have the source code. Another user suggested that we may change slightly on the .machinefile before executing the "mpirun" command. Suppose we are going to run a 8 mpi-task program on a quad-core cluster, then only 2 CPUs should be selected, with the ".machinefile" looks like "cpu0 cpu1 cpu0 cpu1 cpu0 cpu1 cpu0 cpu1" created, i.e. 4 mpi-tasks will be spooled to CPU0 and 4 mpi-tasks will be spooled to CPU1. But the REAL question will be: Will EACH mpi-task be executed on ONE single core? If not, then could there be any Linux utility program to help? I asked this question to one of the potential vendor, and the sales suddenly suggested "Well, you can buy VMWARE to create virtual CPUs to do so." Do you think it is logical? Thanks in advance. W.K. Kwan Computer Centre University of Hongkong From Li at mx2.buaa.edu.cn Tue Oct 2 07:40:57 2007 From: Li at mx2.buaa.edu.cn (Li@mx2.buaa.edu.cn) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs Message-ID: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> Hello, ---------------------------------------------------- > >This is perhap a naive question. > >10 years before we started using the SP2, but we later changed to Intel >based linux beowulf in 2001. In our University there are quite a no. of >mpi-based parallel programs running in a 178 node dual-Xeon PC cluster >that was installed 4 years ago. > >We are now planning to upgrade our cluster in the coming year. Very >likely blade servers with multi-core CPUs will be used. To port these >mpi-based parallel programs to a multi-core CPU environment, someone >suggested that OpenMP should be used, such that the programs can be >converted to a multi-thread version. Nevertheless it may take time, and >the users may be reluctant to do so. Also for some of the installed >programs, we don't have the source code. > >Another user suggested that we may change slightly on the .machinefile >before executing the "mpirun" command. > >Suppose we are going to run a 8 mpi-task program on a quad-core cluster, >then only 2 CPUs should be selected, with the ".machinefile" looks like >"cpu0 cpu1 cpu0 cpu1 cpu0 cpu1 cpu0 cpu1" created, i.e. 4 mpi-tasks will >be spooled to CPU0 and 4 mpi-tasks will be spooled to CPU1. But the REAL >question will be: > Will EACH mpi-task be executed on ONE single core? > If not, then could there be any Linux utility program to help? > Generally, each mpi-task should be executed on a single core, and if not, you can run 4 mpid on a single node. >I asked this question to one of the potential vendor, and the sales >suddenly suggested "Well, you can buy VMWARE to create virtual CPUs to do >so." Do you think it is logical? > It was just a kidding. I suggest to use openmp on a single node and mpi among nodes for performance issues. And you may need to find a good mpi for your purpose. Regards, Li, Bo From gerry.creager at tamu.edu Tue Oct 2 07:50:18 2007 From: gerry.creager at tamu.edu (Gerry Creager) Date: Sat Jul 19 01:06:25 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> References: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> Message-ID: <47025AAA.9020108@tamu.edu> Just for the record, I hate HTML-encoded e-mails. Li@mx2.buaa.edu.cn wrote: > Hello, > ---------------------------------------------------- >> This is perhap a naive question. >> >> 10 years before we started using the SP2, but we later changed to Intel >> based linux beowulf in 2001. In our University there are quite a no. of >> mpi-based parallel programs running in a 178 node dual-Xeon PC cluster >> that was installed 4 years ago. >> >> We are now planning to upgrade our cluster in the coming year. Very >> likely blade servers with multi-core CPUs will be used. To port these >> mpi-based parallel programs to a multi-core CPU environment, someone >> suggested that OpenMP should be used, such that the programs can be >> converted to a multi-thread version. Nevertheless it may take time, and >> the users may be reluctant to do so. Also for some of the installed >> programs, we don't have the source code. >> >> Another user suggested that we may change slightly on the .machinefile >> before executing the "mpirun" command. >> >> Suppose we are going to run a 8 mpi-task program on a quad-core cluster, >> then only 2 CPUs should be selected, with the ".machinefile" looks like >> "cpu0 cpu1 cpu0 cpu1 cpu0 cpu1 cpu0 cpu1" created, i.e. 4 mpi-tasks will >> be spooled to CPU0 and 4 mpi-tasks will be spooled to CPU1. But the REAL >> question will be: >> Will EACH mpi-task be executed on ONE single core? >> If not, then could there be any Linux utility program to help? >> > Generally, each mpi-task should be executed on a single core, and if not, you can run 4 mpid on a single node. >> I asked this question to one of the potential vendor, and the sales >> suddenly suggested "Well, you can buy VMWARE to create virtual CPUs to do >> so." Do you think it is logical? Selection of OpenMP vs MPI, or the combination of the two, depends considerably on how your code funcitons. We have been working with dual-core, dual-processor (4-cores/node) machines for a couple of years, running exclusively MPI codes, and seen very good performance. Similarly, (yeah, guys, I know it's not a Beowulf but...) I run weather forecast (WRF) codes on an IBM Cluster 1600 ) (p575) system using 8 of 16 coreres and 32 nodes (don't ask why: silly sysadmin tricks). The WRF codes will run as SMP, DM or a combination of both but, for the domains I tend to forecast over, are more efficient using just MPI. We see no problems with multicore machines running MPI. It would be worth encouraging your users to evaluate how their codes would run if enabled for a combination of OpenMP and MPI but not mandatory. gerry -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From ntmoore at gmail.com Tue Oct 2 07:20:58 2007 From: ntmoore at gmail.com (Nathan Moore) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: References: Message-ID: <6009416b0710020720p2c8b94a3ifcce5e9640e43938@mail.gmail.com> My understanding is that on a multi-core machine, mpi communication routines (MPI_SEND, etc) are implemented as memory copy instructions. Accordingly, message passing within a multi-core node should be very fast compared to your present cluster. That said, It seems like all the performance benchmarks suggest that dual-core chips have the performances of 1.5-1.7 single core chips, so for the same number of nodes (defined as a CPU core) you wouldn't see the same output. All of this course depends on the structure of the code, memory usage, etc - these are just scaling estimates on my part. regards, Nathan On 10/2/07, Kwan Wing Keung wrote: > > > This is perhap a naive question. > > 10 years before we started using the SP2, but we later changed to Intel > based linux beowulf in 2001. In our University there are quite a no. of > mpi-based parallel programs running in a 178 node dual-Xeon PC cluster > that was installed 4 years ago. > > We are now planning to upgrade our cluster in the coming year. Very > likely blade servers with multi-core CPUs will be used. To port these > mpi-based parallel programs to a multi-core CPU environment, someone > suggested that OpenMP should be used, such that the programs can be > converted to a multi-thread version. Nevertheless it may take time, and > the users may be reluctant to do so. Also for some of the installed > programs, we don't have the source code. > > Another user suggested that we may change slightly on the .machinefile > before executing the "mpirun" command. > > Suppose we are going to run a 8 mpi-task program on a quad-core cluster, > then only 2 CPUs should be selected, with the ".machinefile" looks like > "cpu0 cpu1 cpu0 cpu1 cpu0 cpu1 cpu0 cpu1" created, i.e. 4 mpi-tasks will > be spooled to CPU0 and 4 mpi-tasks will be spooled to CPU1. But the REAL > question will be: > Will EACH mpi-task be executed on ONE single core? > If not, then could there be any Linux utility program to help? > > I asked this question to one of the potential vendor, and the sales > suddenly suggested "Well, you can buy VMWARE to create virtual CPUs to do > so." Do you think it is logical? > > Thanks in advance. > > W.K. Kwan > Computer Centre > University of Hongkong > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- - - - - - - - - - - - - - - - - - - - - - Nathan Moore Assistant Professor, Physics Winona State University AIM: nmoorewsu - - - - - - - - - - - - - - - - - - - - - -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071002/6d71e2a9/attachment.html From gdjacobs at gmail.com Tue Oct 2 08:24:43 2007 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> References: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> Message-ID: <470262BB.7070500@gmail.com> Li@mx2.buaa.edu.cn wrote: Using OpenMP per node is going to be much less portable and much more complex to implement. It's better to let your MPI library handle things the smart way. -- Geoffrey D. Jacobs To have no errors would be life without meaning No struggle, no joy From tom.elken at qlogic.com Tue Oct 2 10:07:27 2007 From: tom.elken at qlogic.com (Tom Elken) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <470262BB.7070500@gmail.com> References: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> <470262BB.7070500@gmail.com> Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A01594BF1@AVEXCH1.qlogic.org> > -----Original Message----- > From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Geoff Jacobs > > Using OpenMP per node is going to be much less portable and > much more complex to implement. It's better to let your MPI > library handle things the smart way. At QLogic we're quite familiar with benchmarking with good OpenMP compilers (esp. the PathScale compiler, which used to be part of QLogic) and with various modern MPIs (InfiniPath MPI, MVAPICH, Open MPI, HP-MPI, Scali MPI), and I totally agree with the previous two answers. MPI developers pay a large and increasing amount of performance-tuning attention to the larger core counts, NUMA and SMP issues on cluster nodes. Even if you underwent the large effort to re-code an application for which you had the source to an OpenMP / MPI hybrid approach, it is as likely that you would reduce your application's performance as to improve it. That said, there is continuing interest in the Hybrid approach, and large SMP nodes will keep that interest going. A Google on "Hybrid OpenMP MPI" will provide you with some presentations and papers on recent work in this area. You will find that wins with the Hybrid approach are possible, but it is a lot of work, and by no means required, as you move towards clusters with multi-core CPUs. -Tom > > -- > Geoffrey D. Jacobs > > To have no errors > would be life without meaning > No struggle, no joy > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org To change your > subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From larry.stewart at sicortex.com Tue Oct 2 11:52:21 2007 From: larry.stewart at sicortex.com (Larry Stewart) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> References: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> Message-ID: <47029365.8090606@sicortex.com> The question of OpenMP vs MPI has been around for a long time, for example: http://www.beowulf.org/archive/2001-March/002718.html My general impression is that it is a waste of time to convert from pure MPI to a hybrid approach. For example: www.sc2000.org/techpapr/papers/pap.pap214.pdf On the other hand, here's a fellow who got a 4X speedup by going to hybrid: www.nersc.gov/nusers/services/training/classes/NUG/Jun04/NUG2004_yhe_hybrid.ppt My own view is that with a modern cluster with fast processors and with inter-node communications not that much slower than a cache miss to main memory, the unified MPI model makes more sense, but there are many many papers arguing about this topic. -L -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071002/567c759a/attachment.html From lindahl at pbm.com Tue Oct 2 14:16:00 2007 From: lindahl at pbm.com (Greg Lindahl) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <47029365.8090606@sicortex.com> References: <20071002144057.D417A37F7E@mx2.buaa.edu.cn> <47029365.8090606@sicortex.com> Message-ID: <20071002211559.GA1115@bx9.net> On Tue, Oct 02, 2007 at 02:52:21PM -0400, Larry Stewart wrote: > On the other hand, here's a fellow who got a 4X speedup by going to hybrid: > > www.nersc.gov/nusers/services/training/classes/NUG/Jun04/NUG2004_yhe_hybrid.ppt It's always hard to evaluate how real such claims are. But I will note that this guy is on an IBM SP, which for a while (dunno if it's fixed now) had some really horribly slow MPI library code for in-box communications. Code so bad that it made OpenMP and hybrid programming look good. He also claims MM5 gets a win with hybrid programming, but the numbers I've seen say that pure MPI is best on that code. -- greg From gdjacobs at gmail.com Wed Oct 3 07:31:38 2007 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <6009416b0710020720p2c8b94a3ifcce5e9640e43938@mail.gmail.com> References: <6009416b0710020720p2c8b94a3ifcce5e9640e43938@mail.gmail.com> Message-ID: <4703A7CA.6010604@gmail.com> Nathan Moore wrote: > My understanding is that on a multi-core machine, mpi communication > routines (MPI_SEND, etc) are implemented as memory copy instructions. > Accordingly, message passing within a multi-core node should be very > fast compared to your present cluster. > > That said, It seems like all the performance benchmarks suggest that > dual-core chips have the performances of 1.5-1.7 single core chips, so > for the same number of nodes (defined as a CPU core) you wouldn't see > the same output. > > All of this course depends on the structure of the code, memory usage, > etc - these are just scaling estimates on my part. > > regards, > > Nathan Aren't there two ways it's done? IARC, MPICH2 has (1) standard ch3 TCP/IP for all coms, but local coms bounced off a loopback interface (2) Nemesis TCP/IP for inter node coms, but intra node coms use shared memory. And dual core chip performance is, again, totally dependent on the code in question. If the code is highly pipelined and can run in cache, an embarrassingly parallel algorithm will scale linearly with cores. Code limited by memory bandwidth, I/O, or (in a cluster) network performance will not scale as well. -- Geoffrey D. Jacobs To have no errors would be life without meaning No struggle, no joy From herbert.fruchtl at st-andrews.ac.uk Wed Oct 3 01:28:05 2007 From: herbert.fruchtl at st-andrews.ac.uk (Herbert Fruchtl) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Re: Naive question: mpi-parallel program in multicore CPUs In-Reply-To: <200710021900.l92J08Up016243@bluewest.scyld.com> References: <200710021900.l92J08Up016243@bluewest.scyld.com> Message-ID: <47035295.5060501@st-andrews.ac.uk> What is really difficult with MPI is data distribution. A lot of applications are parallelised using replicated data. That's fine if you are CPU bound, but it you are limited by the amount of memory per processor, a shared memory approach (which is the default with OpenMP) is the easiest way of using all the memory. MPI may also add a lot of overhead if you parallelise inner loops, which is easy and cheap with OpenMP. OTOH, coarse-grain parallelism with OpenMP is difficult; MPI is usually more suitable here. It depends on your application, and you may find candidates for both approaches in the same application. Herbert beowulf-request@beowulf.org wrote: > Send Beowulf mailing list submissions to > beowulf@beowulf.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.beowulf.org/mailman/listinfo/beowulf > or, via email, send a message with subject or body 'help' to > beowulf-request@beowulf.org > > You can reach the person managing the list at > beowulf-owner@beowulf.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Beowulf digest..." > > > Today's Topics: > > 1. Re: Naive question: mpi-parallel program in multicore CPUs > (Larry Stewart) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 02 Oct 2007 14:52:21 -0400 > From: Larry Stewart > Subject: Re: [Beowulf] Naive question: mpi-parallel program in > multicore CPUs > To: Li@mx2.buaa.edu.cn, Bo > Cc: beowulf@beowulf.org, Kwan Wing Keung > Message-ID: <47029365.8090606@sicortex.com> > Content-Type: text/plain; charset="gb2312" > > The question of OpenMP vs MPI has been around for a long time, > for example: > > http://www.beowulf.org/archive/2001-March/002718.html > > My general impression is that it is a waste of time to convert from pure > MPI to > a hybrid approach. For example: > > www.sc2000.org/techpapr/papers/pap.pap214.pdf > > On the other hand, here's a fellow who got a 4X speedup by going to hybrid: > > www.nersc.gov/nusers/services/training/classes/NUG/Jun04/NUG2004_yhe_hybrid.ppt > > My own view is that with a modern cluster with fast processors and with > inter-node communications not > that much slower than a cache miss to main memory, the unified MPI model > makes more sense, but > there are many many papers arguing about this topic. > > -L > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://www.scyld.com/pipermail/beowulf/attachments/20071002/567c759a/attachment-0001.html > > ------------------------------ > > _______________________________________________ > Beowulf mailing list > Beowulf@beowulf.org > http://www.beowulf.org/mailman/listinfo/beowulf > > > End of Beowulf Digest, Vol 44, Issue 4 > ************************************** -- Herbert Fruchtl EaStCHEM Fellow School of Chemistry University of St Andrews From anandvaidya.ml at gmail.com Wed Oct 3 04:29:23 2007 From: anandvaidya.ml at gmail.com (Anand Vaidya) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux Message-ID: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Dear Beowulfers, We ran into a problem with large disks which I suspect is fairly common, however the usual solutions are not working. IBM, RedHat have not been able to provide any useful answers so I am turning to this list for help. (Emulex is still helping, but I am not sure how far they can go without access to the hardware) Details: * Linux Cluster for Weather modelling * IBM Bladecenter blades and an IBM x3655 Opteron head node FC attached to a Hitachi Tagmastore SAN storage, Emulex LightPulse FC HBA, PCI-Express, Dual port * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided Emulex driver (lpfc) and lpfcdfc also installed * GPT partition created with parted There is one 2TB LUN, works fine. There is a 3TB LUN on the Hitachi SAN which is reported as "only" 2199GB ( 2.1TB) , We noticed that, when the emulex driver loads, the following error message is reported: Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 Copyright(c) 2003-2007 Emulex. All rights reserved. ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, low) -> IRQ 185 PCI: Setting latency timer of device 0000:2d:00.0 to 64 lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: x2 x4 x1000 lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: x2 x4 x1000 lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received Data: x3 x1 x10 x0 scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x on PCI bus 2d device 00 irq 185 port 0 Vendor: HITACHI Model: OPEN-V*3 Rev: 5007 Type: Direct-Access ANSI SCSI revision: 03 sdb : very big device. try to use READ CAPACITY(16). sdb : READ CAPACITY(16) failed. sdb : status=1, message=00, host=0, driver=08 sdb : use 0xffffffff as device size SCSI device sdb: 4294967296 512-byte hdwr sectors (2199023 MB) SCSI device sdb: drive cache: write back sdb : very big device. try to use READ CAPACITY(16). sdb : READ CAPACITY(16) failed. sdb : status=1, message=00, host=0, driver=08 sdb : use 0xffffffff as device size SCSI device sdb: 4294967296 512-byte hdwr sectors (2199023 MB) SCSI device sdb: drive cache: write back The problem is with the READ CAPACITY(16) failed, but we are unable to find the source of this error. We conducted several experiments without success: - Tried compiling the latest driver from Emulex (8.0.16.32) - same error - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19) , and CentOS 4.4 - same error - Tried to boot Belenix (Solaris 32 bit live), failed to boot completely (may be unrelated issue) We have a temporary workaround in place: We created 3x1TB disks and used LVM to create a striped 3TB volume with ext3 FS. This works fine. RedHat claims ext3 and RHEL4 supports disks upto 8TB and 16TB respectively (since RHEL4u2) I would like to know if anyone on the list has any pointers that can help us solve the issue. Regards Anand Vaidya -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071003/118f7440/attachment.html From atchley at myri.com Wed Oct 3 08:14:05 2007 From: atchley at myri.com (Scott Atchley) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Message-ID: Is someone using a signed int to represent the 1 KB blocks? 2 * 1024 * 1024 * 1024 * 1024 = 2199023255552 Scott On Oct 3, 2007, at 7:29 AM, Anand Vaidya wrote: > Dear Beowulfers, > > We ran into a problem with large disks which I suspect is fairly > common, however the usual solutions are not working. IBM, RedHat > have not been able to provide any useful answers so I am turning to > this list for help. (Emulex is still helping, but I am not sure how > far they can go without access to the hardware) > > Details: > > * Linux Cluster for Weather modelling > > * IBM Bladecenter blades and an IBM x3655 Opteron head node FC > attached to a Hitachi Tagmastore SAN storage, Emulex LightPulse FC > HBA, PCI-Express, Dual port > > * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided > Emulex driver (lpfc) and lpfcdfc also installed > > * GPT partition created with parted > > There is one 2TB LUN, works fine. > > There is a 3TB LUN on the Hitachi SAN which is reported as "only" > 2199GB ( 2.1TB) , > > We noticed that, when the emulex driver loads, the following error > message is reported: > > Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 > Copyright(c) 2003-2007 Emulex. All rights reserved. > ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, > low) -> IRQ 185 > PCI: Setting latency timer of device 0000:2d:00.0 to 64 > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received > Data: x2 x4 x1000 > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received > Data: x2 x4 x1000 > lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received > Data: x3 x1 x10 x0 > scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x > on PCI bus 2d device 00 irq 185 port 0 > Vendor: HITACHI Model: OPEN-V*3 Rev: 5007 > Type: Direct-Access ANSI SCSI > revision: 03 > sdb : very big device. try to use READ CAPACITY(16). > sdb : READ CAPACITY(16) failed. > sdb : status=1, message=00, host=0, driver=08 > sdb : use 0xffffffff as device size > SCSI device sdb: 4294967296 512-byte hdwr sectors > (2199023 MB) > SCSI device sdb: drive cache: write back > sdb : very big device. try to use READ CAPACITY(16). > sdb : READ CAPACITY(16) failed. > sdb : status=1, message=00, host=0, driver=08 > sdb : use 0xffffffff as device size > SCSI device sdb: 4294967296 512-byte hdwr sectors > (2199023 MB) > SCSI device sdb: drive cache: write back > > The problem is with the READ CAPACITY(16) failed, but we are unable > to find the source of this error. > > We conducted several experiments without success: > > - Tried compiling the latest driver from Emulex (8.0.16.32) - same > error > - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19 ) , and CentOS > 4.4 - same error > - Tried to boot Belenix (Solaris 32 bit live), failed to boot > completely (may be unrelated issue) > > We have a temporary workaround in place: We created 3x1TB disks and > used LVM to create a striped 3TB volume with ext3 FS. This works > fine. > > RedHat claims ext3 and RHEL4 supports disks upto 8TB and 16TB > respectively (since RHEL4u2) > > I would like to know if anyone on the list has any pointers that > can help us solve the issue. > > Regards > Anand Vaidya > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Wed Oct 3 08:49:42 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Message-ID: > Is someone using a signed int to represent the 1 KB blocks? > 2 * 1024 * 1024 * 1024 * 1024 = 2199023255552 yes - you can see from the message that the kernel is trying to use a 16-byte read-capacity command, but it's failing. these days, scsi and especially fc and super-especially some obscure driver like this are less heavily scrutinized, so this is not hugely surprising. I don't know whether the error status indicates a flaw in the driver or perhaps in the target (which might not support large luns). >> We ran into a problem with large disks which I suspect is fairly common, in my world, large == sata raid, and scsi/fc is a vanishing breed, which also coincides with the stunted size of scsi/fc disks. that said, the only scsi my organization has (embedded in HP SFS clusters) is (annoyingly) chopped up into piddly little 2TB luns. >> however the usual solutions are not working. IBM, RedHat have not been >> able to provide any useful answers so I am turning to this list for help. >> (Emulex is still helping, but I am not sure how far they can go without >> access to the hardware) I think you should ask them whether the driver supports the 16-byte read-capacity. and you should ask the target provider (hitachi) whether they support >2TB luns and whether they implement 16-byte commands. you paid through the nose for FC hardware; you should expect a high level of service. >> >> Details: >> >> * Linux Cluster for Weather modelling >> >> * IBM Bladecenter blades and an IBM x3655 Opteron head node FC attached to >> a Hitachi Tagmastore SAN storage, Emulex LightPulse FC HBA, PCI-Express, >> Dual port >> >> * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided Emulex driver >> (lpfc) and lpfcdfc also installed >> >> * GPT partition created with parted >> >> There is one 2TB LUN, works fine. >> >> There is a 3TB LUN on the Hitachi SAN which is reported as "only" 2199GB ( >> 2.1TB) , >> >> We noticed that, when the emulex driver loads, the following error message >> is reported: >> >> Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 >> Copyright(c) 2003-2007 Emulex. All rights reserved. >> ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, low) -> >> IRQ 185 >> PCI: Setting latency timer of device 0000:2d:00.0 to 64 >> lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: x2 >> x4 x1000 >> lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: x2 >> x4 x1000 >> lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received Data: x3 x1 >> x10 x0 >> scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x on PCI >> bus 2d device 00 irq 185 port 0 >> Vendor: HITACHI Model: OPEN-V*3 Rev: 5007 >> Type: Direct-Access ANSI SCSI revision: >> 03 >> sdb : very big device. try to use READ CAPACITY(16). >> sdb : READ CAPACITY(16) failed. >> sdb : status=1, message=00, host=0, driver=08 >> sdb : use 0xffffffff as device size >> SCSI device sdb: 4294967296 512-byte hdwr sectors (2199023 MB) >> SCSI device sdb: drive cache: write back >> sdb : very big device. try to use READ CAPACITY(16). >> sdb : READ CAPACITY(16) failed. >> sdb : status=1, message=00, host=0, driver=08 >> sdb : use 0xffffffff as device size >> SCSI device sdb: 4294967296 512-byte hdwr sectors (2199023 MB) >> SCSI device sdb: drive cache: write back >> >> The problem is with the READ CAPACITY(16) failed, but we are unable to find >> the source of this error. >> >> We conducted several experiments without success: >> >> - Tried compiling the latest driver from Emulex (8.0.16.32) - same error >> - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19 ) , and CentOS 4.4 - >> same error >> - Tried to boot Belenix (Solaris 32 bit live), failed to boot completely >> (may be unrelated issue) >> >> We have a temporary workaround in place: We created 3x1TB disks and used >> LVM to create a striped 3TB volume with ext3 FS. This works fine. >> >> RedHat claims ext3 and RHEL4 supports disks upto 8TB and 16TB respectively >> (since RHEL4u2) >> >> I would like to know if anyone on the list has any pointers that can help >> us solve the issue. >> >> Regards >> Anand Vaidya >> >> >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- operator may differ from spokesperson. hahn@mcmaster.ca From gmpc at sanger.ac.uk Wed Oct 3 09:03:29 2007 From: gmpc at sanger.ac.uk (Guy Coates) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Message-ID: <4703BD51.9020001@sanger.ac.uk> Luns over 2TB are a bad idea. There are just too many reasons why they might not work, and trying to track down the right one is a pain. Your workaround to use the LVM to stripe 3x1TB luns together is the way to go. (You really want to use LVM anyhow, as trying to do SAN storage without it is a pain.) You've covered the usual Gotchas with 2TB luns (dos partition tables and 32 bit kernels), so I'd suspect your storage array. Check with Hitachi that the array supports creating luns over 2TB. Many don't, and for maximum confusion, some allow you to create >2TB luns, even though you can't subsequently use them. Cheers, Guy -- Dr. Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From jlforrest at berkeley.edu Wed Oct 3 09:13:44 2007 From: jlforrest at berkeley.edu (Jon Forrest) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <4703BD51.9020001@sanger.ac.uk> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> <4703BD51.9020001@sanger.ac.uk> Message-ID: <4703BFB8.1020606@berkeley.edu> Guy Coates wrote: > Luns over 2TB are a bad idea. There are just too many reasons why they might not > work, and trying to track down the right one is a pain. Other than the usual putting all your eggs into one basket problem, why should a lun over 2TB containing an ext3 file system, be any more risky than those under 2TB, assuming all the partition table issues are resolved? Are there performance, reliability, compatibility, or other problems? (I'm not saying you're wrong but I'm always suspicious of claims like these without specific examples of why something is a bad idea.) Cordially, -- Jon Forrest Unix Computing Support College of Chemistry 173 Tan Hall University of California Berkeley Berkeley, CA 94720-1460 510-643-1032 jlforrest@berkeley.edu From gmpc at sanger.ac.uk Wed Oct 3 09:27:35 2007 From: gmpc at sanger.ac.uk (Guy Coates) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <4703BFB8.1020606@berkeley.edu> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> <4703BD51.9020001@sanger.ac.uk> <4703BFB8.1020606@berkeley.edu> Message-ID: <4703C2F7.2010100@sanger.ac.uk> Jon Forrest wrote: > Guy Coates wrote: >> Luns over 2TB are a bad idea. There are just too many reasons why they >> might not >> work, and trying to track down the right one is a pain. > > Other than the usual putting all your eggs into one basket problem, > why should a lun over 2TB containing an ext3 file system, be any > more risky than those under 2TB, assuming all the partition table > issues are resolved? Are there performance, reliability, > compatibility, or other problems? You are right; luns over 2TB are not any more risky than luns under 2TB. If they work "out of the box" on your particular setup, great. However, if they don't work then you are into a world of pain working out whether it is your storage controller, SAN fabric, HBA, multipath implementation, kernel, partition table or filesystem which is broken. It gets especially bad in SAN environments where you have multiple kernel/storage combinations. Keeping luns < 2TB means that there is a reasonable expectation that things will keep on working when you change the storage presentation. Cheers, Guy -- Dr. Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From landman at scalableinformatics.com Wed Oct 3 09:46:43 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Message-ID: <4703C773.2040102@scalableinformatics.com> Hi Anand: Anand Vaidya wrote: > Dear Beowulfers, > > We ran into a problem with large disks which I suspect is fairly common, > however the usual solutions are not working. IBM, RedHat have not been > able to provide any useful answers so I am turning to this list for > help. (Emulex is still helping, but I am not sure how far they can go > without access to the hardware) > > Details: > > * Linux Cluster for Weather modelling > > * IBM Bladecenter blades and an IBM x3655 Opteron head node FC attached > to a Hitachi Tagmastore SAN storage, Emulex LightPulse FC HBA, > PCI-Express, Dual port > > * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided Emulex > driver (lpfc) and lpfcdfc also installed There is a problem in some parted versions (prior to 1.8.x) where they munge the partition table on gpt/large disks. Apart from suggesting a more modern kernel (2.6.22.6 or so), there may be other things you can do. > > * GPT partition created with parted I presume this is 1.6.19 parted? > > There is one 2TB LUN, works fine. > > There is a 3TB LUN on the Hitachi SAN which is reported as "only" 2199GB > ( 2.1TB) , Yup. Sounds like either the parted problem, or a driver issue. What does parted /dev/3TBlun/ print report where /dev/3TBlun is the device containing the 3TB lun? We had seen this behavior with some 1.6.9 and 1.7.x versions of parted. The only way to fix it was to rebuild parted with 1.8.x > > We noticed that, when the emulex driver loads, the following error > message is reported: > > Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 > > Copyright(c) 2003-2007 Emulex. All rights reserved. > ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, low) > -> IRQ 185 > PCI: Setting latency timer of device 0000:2d:00.0 to 64 > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: > x2 x4 x1000 > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: > x2 x4 x1000 > lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received Data: x3 > x1 x10 x0 > scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x on > PCI bus 2d device 00 irq 185 port 0 > Vendor: HITACHI Model: OPEN-V*3 Rev: 5007 > Type: Direct-Access ANSI SCSI > revision: 03 > sdb : very big device. try to use READ CAPACITY(16). This is what our JackRabbit reports ... > sdb : READ CAPACITY(16) failed. This is not what our JackRabbit reports. [...] > The problem is with the READ CAPACITY(16) failed, but we are unable to > find the source of this error. > > We conducted several experiments without success: > > - Tried compiling the latest driver from Emulex (8.0.16.32 > ) - same error > - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19 ) , and CentOS 4.4 > - same error Sounds a great deal like parted. > - Tried to boot Belenix (Solaris 32 bit live), failed to boot completely > (may be unrelated issue) > > We have a temporary workaround in place: We created 3x1TB disks and used > LVM to create a striped 3TB volume with ext3 FS. This works fine. > > RedHat claims ext3 and RHEL4 supports disks upto 8TB and 16TB > respectively (since RHEL4u2) ... yeah. > > I would like to know if anyone on the list has any pointers that can > help us solve the issue. Please run the parted command as indicated. Lets see what the partition table thinks it is. Do you have data on that partition? Can you remake the label on that device with a new version of parted? > > Regards > Anand Vaidya > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From nixon at nsc.liu.se Wed Oct 3 14:13:48 2007 From: nixon at nsc.liu.se (Leif Nixon) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: (Mark Hahn's message of "Wed, 3 Oct 2007 11:49:42 -0400 (EDT)") References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> Message-ID: Mark Hahn writes: > I think you should ask them whether the driver supports the 16-byte > read-capacity. and you should ask the target provider (hitachi) > whether > they support >2TB luns and whether they implement 16-byte commands. As far as I know, neither the Hitachi AMS series nor the Hitachi NSC series support LUNs > 2TB. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From xingqiuyuan at gmail.com Wed Oct 3 09:00:12 2007 From: xingqiuyuan at gmail.com (xingqiu yuan) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price Message-ID: Dear All We are going to buy a small size beowulf cluster with 32 multicore processors (INTEL or AMD multicore processor). We don't care the network, but we need a big storage disks. Anybody can give us some suggestions about the hardware and the price? best regards Xingqiu Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071003/27c000d0/attachment.html From merc4krugger at gmail.com Wed Oct 3 09:59:18 2007 From: merc4krugger at gmail.com (Krugger) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <4703C2F7.2010100@sanger.ac.uk> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> <4703BD51.9020001@sanger.ac.uk> <4703BFB8.1020606@berkeley.edu> <4703C2F7.2010100@sanger.ac.uk> Message-ID: Hi, It usually takes some more time to get a stable storage with over 2 TB partitions. In my personal experience we had to update the SAN firmware, after that we figured out that the SCSI board that connected to the SAN hardware also was bad and had to have it replaced. Also remember to enable large block device support. But after this the storage has been stable with over 2TB partitions. Krugger > You are right; luns over 2TB are not any more risky than luns under 2TB. If they > work "out of the box" on your particular setup, great. > > However, if they don't work then you are into a world of pain working out > whether it is your storage controller, SAN fabric, HBA, multipath > implementation, kernel, partition table or filesystem which is broken. > > It gets especially bad in SAN environments where you have multiple > kernel/storage combinations. Keeping luns < 2TB means that there is a reasonable > expectation that things will keep on working when you change the storage > presentation. > > Cheers, > > Guy From zahir.tari at rmit.edu.au Wed Oct 3 19:18:44 2007 From: zahir.tari at rmit.edu.au (Zahir Tari) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] OTM 07 Call for Participation Message-ID: <8EE2E0AC-69D5-4F27-9E9F-CE1E8F62CE47@rmit.edu.au> Dear all, This is to let you know that the selection process for the OTM 2007 Federated event just finished. This event will be held in Algarve (Portugal) during the period of 25-30 November 2007, and it involves the following major conferences: - International Symposium on Distributed Objects and Applications (DOA'07) - International Conference on Cooperative Information Systems (CoopIS'07) - International Symposium on Grid computing, high-performAnce and Distributed Applications (GADA'07) - International Conference on Ontologies, Databases and Applications of Semantics (ODBASE'07) - International Symposium on Information Security (IS'07) We are proud to announce you the four exceptional keynote speakers for the OTM 2007 event: - Donald Ferguson (Microsoft) on "The Internet Service Bus" - Mark Little (Red Hat) on "Transaction Processing in a Service Oriented Architecture" - York Sure (SAP) on "Towards the next Generation Value Networks" - Dennis Gannon (Indiana University) on "A Service Architecture for eScience Grid Gateways" - Whitfield Diffie (Sun Microsystems) on "Cryptography: Past, Present and Future" Details about the list of accepted papers can be found at http://www.cs.rmit.edu.au/fedconf/index.html?page=accepted In additional to the conferences, OTM 2007 is also running exceptional workshops covering various topics (e.g. agents in web services, privacy, context-aware mobile services...). Hope to see you at OTM 2007 in Portugal! Please send us an e-mail to fedconf@cs.rmit.edu.au for any information. Regards, Zahir [Tari] & Robert [Meersman] -- OTM 2007 General Co-Chairs -- http://www.cs.rmit.edu.au/fedconf/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071004/36756fe0/attachment.html From hahn at mcmaster.ca Thu Oct 4 07:11:27 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: Message-ID: > We are going to buy a small size beowulf cluster with 32 multicore > processors (INTEL or AMD multicore processor). does "multi" mean 2 or 4 to you? also, are you talking dual-socket or single (or quad)? > We don't care the network, but we need a big storage disks. that's even more poorly defined. how big is big, and does it need to be raid, and if so what level of redundancy and hotpluggability? by big do you mean capacity, or do you also mean sustained bandwidth (presumably you don't mean to maximize random seeks, or do you?) what space footprint is acceptable? are you sure you can't tolerate shared storage of any sort, even if over a high-bandwidth media? > Anybody can give us some suggestions about > the hardware and the price? no. your query could mean "1-socket 1U server with 2x 750G local" (probably around $1500), or it could mean "4-socket quadcore with 128G ram and 12 15K rpm SAS disks" (more like $50k). if you're really taking the extreme of high disk-to-cpu ratio, HP has a product which puts 14xSATA disks in 2U with a single socket for about $11k (US list). sun's thumper is 48x750 in 4U, I think; I don't know what kind of cpu it has locally. From anandvaidya.ml at gmail.com Wed Oct 3 21:36:56 2007 From: anandvaidya.ml at gmail.com (Anand Vaidya) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] Problem with Single RAID disk larger than 2TB and Linux In-Reply-To: <4703C773.2040102@scalableinformatics.com> References: <6f99ad660710030429h2dd12e4eicb2f999d2bb0531c@mail.gmail.com> <4703C773.2040102@scalableinformatics.com> Message-ID: <200710041236.56399.anandvaidya.ml@gmail.com> My apologies for top posting. Thanks to all the respondents. I have collated your comments and questions into one response. Hope it is more useful... [Joe Landman]: recommended using a newer version of parted [My Reply]: We are seeing the error right when the driver loads, even before Linux is fully booted up. So I guess parted versions etc have no role to play. [Leif Nixon]: As far as I know, neither the Hitachi AMS series nor the Hitachi NSC series support LUNs > 2TB. [My Reply]: The Hitachi folks here claims any LUN size (< physical capacity) is configurable, 3TB definitely should be fine on the Hitachi storage side, but cannot comment on OS / HBA-drivers etc. I did some search and found a document online that stated 2TB is max LUN size for some Enterprise storage , but I am still trying to ascertain the fact for Hitachi. However, we have seen the HDS Storage config GUI where a 3TB LUN was created and zoned correctly. [Guy Coates]: Recommended staying with 2TB LUNs. [My Reply]: While I agree that staying under 2TB and using LVM to stripe will avoid a lot of issues, this sizing is needed by the end customer, .... but I thought with x86_64 linux is 64-bit clean everywhere! (SCSI LBA atleast?) Moreover, I can earn karma points if I can find a bug or two in linux or linux driver :-) and contribute to the improvement of GNU/Linux. So while there is a temporary solution, I intend to pursue the matter to completion. [Mark Hahn]: Suggested checking whether driver supports 16-byte READ CAPACITY. [My Reply]: I did check with Emulex. They say, "lpfc" driver does use 16-byte read cap. They think the Linux SCSI layer is the underlying cause. I am still working with their support staff on gathering detailed info via "System Grab Diag Tool" They say "The error would occur immediately after the lpfc driver loads because the target is then available (the Fibre Channel controller of the HDS storage) and the SCSI mid-layer has queried the LUNs, that then generates the error. The lpfc driver does not query the LUNs and so would never request a capacity nor any other inforamation. The SCSI mid-layer and higher levels would perform this function." One thing I am very happy about is the level of support offered by Emulex, even though I have not purchased the product from them directly and cannot even furnish an Emulex partno. for the HBA. I wish other companies were as good at support. Regards Anand On Thursday 04 October 2007 00:46:43 Joe Landman wrote: > Hi Anand: > > Anand Vaidya wrote: > > Dear Beowulfers, > > > > We ran into a problem with large disks which I suspect is fairly common, > > however the usual solutions are not working. IBM, RedHat have not been > > able to provide any useful answers so I am turning to this list for > > help. (Emulex is still helping, but I am not sure how far they can go > > without access to the hardware) > > > > Details: > > > > * Linux Cluster for Weather modelling > > > > * IBM Bladecenter blades and an IBM x3655 Opteron head node FC attached > > to a Hitachi Tagmastore SAN storage, Emulex LightPulse FC HBA, > > PCI-Express, Dual port > > > > * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided Emulex > > driver (lpfc) and lpfcdfc also installed > > There is a problem in some parted versions (prior to 1.8.x) where they > munge the partition table on gpt/large disks. > > Apart from suggesting a more modern kernel (2.6.22.6 or so), there may > be other things you can do. > > > * GPT partition created with parted > > I presume this is 1.6.19 parted? > > > There is one 2TB LUN, works fine. > > > > There is a 3TB LUN on the Hitachi SAN which is reported as "only" 2199GB > > ( 2.1TB) , > > Yup. Sounds like either the parted problem, or a driver issue. > > What does > > parted /dev/3TBlun/ print > > report where /dev/3TBlun is the device containing the 3TB lun? > > We had seen this behavior with some 1.6.9 and 1.7.x versions of parted. > The only way to fix it was to rebuild parted with 1.8.x > > > We noticed that, when the emulex driver loads, the following error > > message is reported: > > > > Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 > > > > Copyright(c) 2003-2007 Emulex. All rights reserved. > > ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, low) > > -> IRQ 185 > > PCI: Setting latency timer of device 0000:2d:00.0 to 64 > > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: > > x2 x4 x1000 > > lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: > > x2 x4 x1000 > > lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received Data: x3 > > x1 x10 x0 > > scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x on > > PCI bus 2d device 00 irq 185 port 0 > > Vendor: HITACHI Model: OPEN-V*3 Rev: 5007 > > Type: Direct-Access ANSI SCSI > > revision: 03 > > sdb : very big device. try to use READ CAPACITY(16). > > This is what our JackRabbit reports ... > > > sdb : READ CAPACITY(16) failed. > > This is not what our JackRabbit reports. > > [...] > > > The problem is with the READ CAPACITY(16) failed, but we are unable to > > find the source of this error. > > > > We conducted several experiments without success: > > > > - Tried compiling the latest driver from Emulex (8.0.16.32 > > ) - same error > > - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19 ) , and CentOS 4.4 > > - same error > > Sounds a great deal like parted. > > > - Tried to boot Belenix (Solaris 32 bit live), failed to boot completely > > (may be unrelated issue) > > > > We have a temporary workaround in place: We created 3x1TB disks and used > > LVM to create a striped 3TB volume with ext3 FS. This works fine. > > > > RedHat claims ext3 and RHEL4 supports disks upto 8TB and 16TB > > respectively (since RHEL4u2) > > ... yeah. > > > I would like to know if anyone on the list has any pointers that can > > help us solve the issue. > > Please run the parted command as indicated. Lets see what the partition > table thinks it is. > > Do you have data on that partition? Can you remake the label on that > device with a new version of parted? > > > Regards > > Anand Vaidya > > > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf From rickyfingers2000 at gmail.com Thu Oct 4 07:14:04 2007 From: rickyfingers2000 at gmail.com (John Hancock) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] no shared state, shared state with explicit locking, shared state without explicit locks Message-ID: <222fc29f0710040714h66579620r26ac383379d3f01d@mail.gmail.com> Just ran across this piece of advice on slashdot: "I'm not familiar with all of those libraries mentioned in the story, but I'll bet that those three classifications (no shared state, shared state with explicit locking, shared state without explicit locks) probably cover the models used by most if not all of them. If you understand the trade-offs in those, you can produce a sensible design, and then the toolkit or framework you use to code it up is mostly just an implementation detail." -Anonymous Brave Guy (457657) A quick google and a search of the mail list archives did not yield anything that looked like a good explanation of what the differences are between shared state, shared state with explicit locking, and a shared state without explicit locks libraries. Is Anonymous Brave Guy using unusual terminology for concepts that usually go by other names? If anyone on the list would care to share his/her explanation/opinion about what Anonymous Brave Guy wrote, I'd be most honored. -j From hahn at mcmaster.ca Thu Oct 4 13:02:39 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] no shared state, shared state with explicit locking, shared state without explicit locks In-Reply-To: <222fc29f0710040714h66579620r26ac383379d3f01d@mail.gmail.com> References: <222fc29f0710040714h66579620r26ac383379d3f01d@mail.gmail.com> Message-ID: this question is referring to a thread here: http://developers.slashdot.org/article.pl?sid=07/10/03/0021253&from=rss > but I'll bet that those three classifications (no shared state, shared > state with explicit locking, shared state without explicit locks) I think there's one big distinction: shared or not shared. I figured the author was talking about functional/dataflow languages which don't have shared state (including MPI, in a sense) versus languages which assume shared memory such as OpenMP. within the shared-memory approach, explicit locking is pretty clear. I presume "without explicit locks" just means crude systems like Java's monitored data types. IMO, the penitude of parallel packages mainly shows that we don't have good answers yet... From landman at scalableinformatics.com Thu Oct 4 15:56:44 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: Message-ID: <47056FAC.5090405@scalableinformatics.com> Mark Hahn wrote: [...] > if you're really taking the extreme of high disk-to-cpu ratio, > HP has a product which puts 14xSATA disks in 2U with a single socket > for about $11k (US list). sun's thumper is 48x750 in 4U, I think; I don't > know what kind of cpu it has locally. ... FWIW our JackRabbit unit supports 48 x 1TB (1000GB) drives in 5U ... A terabyte here, a terabyte there, and soon you are talking about real storage ... :) Sun uses Opteron 280's, which are dual core socket 940 processors. They are limited to 4 cores and 16 GB ram. Again, FWIW, our JackRabbit can hit 16 cores and 64 GB ram. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From hunting at ix.netcom.com Thu Oct 4 18:16:10 2007 From: hunting at ix.netcom.com (Michael Huntingdon) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <47056FAC.5090405@scalableinformatics.com> References: <47056FAC.5090405@scalableinformatics.com> Message-ID: <200710050116.l951GLU9023688@bluewest.scyld.com> Actually Mark..I just did an HP DL320 (the system I believe you were talking about) with 9TB (12x750 SATA) for a client for under $8K. This include $400 or so for three years of on site services, before requesting any additional (special) discount. So your $11K estimate is more than a bit high. ~m At 03:56 PM 10/4/2007, Joe Landman wrote: >Mark Hahn wrote: > >[...] > > > if you're really taking the extreme of high disk-to-cpu ratio, > > HP has a product which puts 14xSATA disks in 2U with a single socket > > for about $11k (US list). sun's thumper is 48x750 in 4U, I think; I don't > > know what kind of cpu it has locally. > >... FWIW our JackRabbit unit supports 48 x 1TB (1000GB) drives in 5U ... > >A terabyte here, a terabyte there, and soon you are talking about real >storage ... :) > >Sun uses Opteron 280's, which are dual core socket 940 processors. They >are limited to 4 cores and 16 GB ram. Again, FWIW, our JackRabbit can >hit 16 cores and 64 GB ram. > >-- >Joseph Landman, Ph.D >Founder and CEO >Scalable Informatics LLC, >email: landman@scalableinformatics.com >web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com >phone: +1 734 786 8423 >fax : +1 866 888 3112 >cell : +1 734 612 4615 >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf ************************************************************************************************ Systems Performance Consultants Michael Huntingdon Higher Education Technology Office (408) 294-6811 1981 Randolph Dr. Cell (707) 478-0226 San Jose, CA 95128 fax (601) 510-3808 Web: <http://www.spcnet.com> hunting@ix.netcom.com ************************************************************************************************ From hahn at mcmaster.ca Thu Oct 4 20:39:35 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: > Actually Mark..I just did an HP DL320 (the system I believe you were talking > about) with 9TB (12x750 SATA) for a client for under $8K. This include $400 > or so for three years of on site services, before requesting any additional > (special) discount. So your $11K estimate is more than a bit high. well, it was 14 disks (it's a model that mounts two more disks in the back), and the price came straight off HP's website... (I may have included an IB card, too.) if anyone from HP is listening, a model with two quad-core-capable sockets would be a good move. think of it as the perfect map-reduce brick ;) regards, mark hahn. From lindahl at pbm.com Thu Oct 4 21:42:57 2007 From: lindahl at pbm.com (Greg Lindahl) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: <20071005044257.GA22517@bx9.net> On Thu, Oct 04, 2007 at 11:39:35PM -0400, Mark Hahn wrote: > if anyone from HP is listening, a model with two quad-core-capable > sockets would be a good move. think of it as the perfect map-reduce brick > ;) Quad cores use the same socket as duals. It's just a modest bios difference. I suspect map-reduce bricks are headed towards single-socket quad-cores, but what do I know? ;-) -- greg From nixon at nsc.liu.se Thu Oct 4 23:51:07 2007 From: nixon at nsc.liu.se (Leif Nixon) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: (Mark Hahn's message of "Thu, 4 Oct 2007 23:39:35 -0400 (EDT)") References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: Mark Hahn writes: >> Actually Mark..I just did an HP DL320 (the system I believe you were >> talking about) with 9TB (12x750 SATA) for a client for under $8K. >> This include $400 or so for three years of on site services, before >> requesting any additional (special) discount. So your $11K estimate >> is more than a bit high. > > well, it was 14 disks (it's a model that mounts two more disks in the back), > and the price came straight off HP's website... (I may have included an IB > card, too.) > > if anyone from HP is listening, a model with two quad-core-capable > sockets would be a good move. think of it as the perfect map-reduce > brick ;) I'd prefer they put their effort into supplying a raid controller that can actually detect corrupt raid sets. (cf. the recent thread "Big storage") -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From andrew at moonet.co.uk Fri Oct 5 01:30:40 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: I just did a quote for 10TB storage. 3Ware Escalade 9650SE + battery backup (16 port) 12 x Hitachi 1TB Enterprise SATAII in raid 6 2 x Intel Xeon Woodcrest 5160 - 3GHz 8 x 2048MB DDRII-667 All for ?6,886.00 Which I realise would buy a small house in the USA at the moment :) Ta Andy On 05/10/2007, Leif Nixon wrote: > Mark Hahn writes: > > >> Actually Mark..I just did an HP DL320 (the system I believe you were > >> talking about) with 9TB (12x750 SATA) for a client for under $8K. > >> This include $400 or so for three years of on site services, before > >> requesting any additional (special) discount. So your $11K estimate > >> is more than a bit high. > > > > well, it was 14 disks (it's a model that mounts two more disks in the back), > > and the price came straight off HP's website... (I may have included an IB > > card, too.) > > > > if anyone from HP is listening, a model with two quad-core-capable > > sockets would be a good move. think of it as the perfect map-reduce > > brick ;) > > I'd prefer they put their effort into supplying a raid controller that > can actually detect corrupt raid sets. (cf. the recent thread "Big storage") > > -- > Leif Nixon - Systems expert > ------------------------------------------------------------ > National Supercomputer Centre - Linkoping University > ------------------------------------------------------------ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From andrew at moonet.co.uk Fri Oct 5 01:31:05 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: Oh and this includes 3 years onsite On 05/10/2007, andrew holway wrote: > I just did a quote for 10TB storage. > > 3Ware Escalade 9650SE + battery backup (16 port) > 12 x Hitachi 1TB Enterprise SATAII in raid 6 > 2 x Intel Xeon Woodcrest 5160 - 3GHz > 8 x 2048MB DDRII-667 > > All for ?6,886.00 > > Which I realise would buy a small house in the USA at the moment :) > > Ta > > Andy > > > > On 05/10/2007, Leif Nixon wrote: > > Mark Hahn writes: > > > > >> Actually Mark..I just did an HP DL320 (the system I believe you were > > >> talking about) with 9TB (12x750 SATA) for a client for under $8K. > > >> This include $400 or so for three years of on site services, before > > >> requesting any additional (special) discount. So your $11K estimate > > >> is more than a bit high. > > > > > > well, it was 14 disks (it's a model that mounts two more disks in the back), > > > and the price came straight off HP's website... (I may have included an IB > > > card, too.) > > > > > > if anyone from HP is listening, a model with two quad-core-capable > > > sockets would be a good move. think of it as the perfect map-reduce > > > brick ;) > > > > I'd prefer they put their effort into supplying a raid controller that > > can actually detect corrupt raid sets. (cf. the recent thread "Big storage") > > > > -- > > Leif Nixon - Systems expert > > ------------------------------------------------------------ > > National Supercomputer Centre - Linkoping University > > ------------------------------------------------------------ > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > From hahn at mcmaster.ca Fri Oct 5 07:52:50 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: > I'd prefer they put their effort into supplying a raid controller that > can actually detect corrupt raid sets. (cf. the recent thread "Big storage") not me. raid is too important to be trusted to hardware - have you tried MD's check/scrub features? though afaik it doesn't have a way to switch on verification during normal reads (or, for that matter, verify-after-write.) in any case, I suspect that the trend is away from block-level raid... regards, mark hahn. From geoff at galitz.org Fri Oct 5 08:07:49 2007 From: geoff at galitz.org (Geoff Galitz) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: Message-ID: <200710051508.l95F7xvL030704@b.mail.sonic.net> Why do you automatically distrust hardware raid? -geoff -----Original Message----- not me. raid is too important to be trusted to hardware - have you tried MD's check/scrub features? though afaik it doesn't have a way to switch on verification during normal reads (or, for that matter, verify-after-write.) in any case, I suspect that the trend is away from block-level raid... From hahn at mcmaster.ca Fri Oct 5 09:08:25 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <200710051508.l95F7xvL030704@b.mail.sonic.net> References: <200710051508.l95F7xvL030704@b.mail.sonic.net> Message-ID: > Why do you automatically distrust hardware raid? it's that I trust software raid more. upgradability, ability to audit source code, portability to other machines, etc. there are certainly still some cases where the host cpu is scarce enough to want to avoid the processing overhead. and in principle, I believe HW raid _could_ be faster if the PCI* bus (or memory) is the bottleneck. let me put it this way: do you know what goes on in your HW raid's firmware? personally, I like sausage, but often avoid eating it because I hate to notice the bits of unidentified clearly-not-meat foo in it. imagine trying to verify that a HW raid controller actually does perform online parity verification. it would involve moving disks out of the array to be carefully corrupted on another machine before being returned to test. with SW raid, I can audit the code, even add counters and diagnostics, stop the raid, corrupt a parity block, and start it again to verify it all works... mostly though, it seems clear that a lot of the hardening aspects of ZFS are increasingly important. it's also a "doh!" kind of realization that raid should be done on a per-file basis. (consider the benefits of writing as raid1, which later transparently transforms to raid5 or some form of FEC. also consider that it's rare for user-level codes to choose the right blocksize and alignment to permit full-stripe raid5 writes, but easy to arrange if you're doign it in the filesystem.) regards, mark hahn. > -----Original Message----- > > > not me. raid is too important to be trusted to hardware - have you > tried MD's check/scrub features? though afaik it doesn't have a way > to switch on verification during normal reads (or, for that matter, > verify-after-write.) > > in any case, I suspect that the trend is away from block-level raid... From buccaneer at rocketmail.com Fri Oct 5 09:12:40 2007 From: buccaneer at rocketmail.com (Buccaneer for Hire.) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <47056FAC.5090405@scalableinformatics.com> Message-ID: <391984.76010.qm@web30612.mail.mud.yahoo.com> Please see below for my comments --- Joe Landman wrote: > Mark Hahn wrote: > > [...] > A terabyte here, a terabyte there, and soon you are > talking about real > storage ... :) > > Sun uses Opteron 280's, which are dual core socket > 940 processors. They > are limited to 4 cores and 16 GB ram. Again, FWIW, > our JackRabbit can > hit 16 cores and 64 GB ram. We tested a 24TB x4500 using the same methodology we use to test all of our storage and we were pleasantly surprised. ____________________________________________________________________________________ Moody friends. Drama queens. Your life? Nope! - their life, your story. Play Sims Stories at Yahoo! Games. http://sims.yahoo.com/ From xingqiuyuan at gmail.com Thu Oct 4 16:48:53 2007 From: xingqiuyuan at gmail.com (xingqiu yuan) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <47056FAC.5090405@scalableinformatics.com> References: <47056FAC.5090405@scalableinformatics.com> Message-ID: In deed, we understand that our question is too general, and it is really very hard to what the price is. But we really don't have any good idea about which type of cluster structure is suitable for us. We work on computational plasma physics, basically doing some magnetohydrodynamic (MHD) and particle simulations for fusion and space plasmas. Most of our codes were written in C++ and Fortran using MPI, so in this point of view, our codes can be run on any type of supercomputers, but the code use a lot of memory and output huge data. We want to buy a small cluster to fill in the gap between our Lab. and supercomputer center. On 10/5/07, Joe Landman wrote: > > Mark Hahn wrote: > > [...] > > > if you're really taking the extreme of high disk-to-cpu ratio, > > HP has a product which puts 14xSATA disks in 2U with a single socket > > for about $11k (US list). sun's thumper is 48x750 in 4U, I think; I > don't > > know what kind of cpu it has locally. > > ... FWIW our JackRabbit unit supports 48 x 1TB (1000GB) drives in 5U ... > > A terabyte here, a terabyte there, and soon you are talking about real > storage ... :) > > Sun uses Opteron 280's, which are dual core socket 940 processors. They > are limited to 4 cores and 16 GB ram. Again, FWIW, our JackRabbit can > hit 16 cores and 64 GB ram. > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071005/a1ab13fc/attachment.html From jdvw at tticluster.com Fri Oct 5 04:12:07 2007 From: jdvw at tticluster.com (John Van Workum) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: References: <47056FAC.5090405@scalableinformatics.com> <200710050116.l951GBpq011916@zafron7.UTS.McMaster.CA> Message-ID: We are pricing out a modest storage system too. We are looking at using Areca (http://www.areca.com.tw/products/pcie341.htm). Their new Intel 800MHz IOP341 I/O processor based RAID cards with 2GB cache look impressive. Anyone have experience with these? Seems 3Ware are more popular on this list. Regards, John On 10/5/07, andrew holway wrote: > > I just did a quote for 10TB storage. > > 3Ware Escalade 9650SE + battery backup (16 port) > 12 x Hitachi 1TB Enterprise SATAII in raid 6 > 2 x Intel Xeon Woodcrest 5160 - 3GHz > 8 x 2048MB DDRII-667 > > All for ?6,886.00 > > Which I realise would buy a small house in the USA at the moment :) > > Ta > > Andy > > > > On 05/10/2007, Leif Nixon wrote: > > Mark Hahn writes: > > > > >> Actually Mark..I just did an HP DL320 (the system I believe you were > > >> talking about) with 9TB (12x750 SATA) for a client for under $8K. > > >> This include $400 or so for three years of on site services, before > > >> requesting any additional (special) discount. So your $11K estimate > > >> is more than a bit high. > > > > > > well, it was 14 disks (it's a model that mounts two more disks in the > back), > > > and the price came straight off HP's website... (I may have included > an IB > > > card, too.) > > > > > > if anyone from HP is listening, a model with two quad-core-capable > > > sockets would be a good move. think of it as the perfect map-reduce > > > brick ;) > > > > I'd prefer they put their effort into supplying a raid controller that > > can actually detect corrupt raid sets. (cf. the recent thread "Big > storage") > > > > -- > > Leif Nixon - Systems expert > > ------------------------------------------------------------ > > National Supercomputer Centre - Linkoping University > > ----------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20071005/6f79082c/attachment.html From becker at scyld.com Fri Oct 5 11:42:19 2007 From: becker at scyld.com (Donald Becker) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement and sponsorship opportunity Message-ID: Ninth Annual Beowulf Bash We are putting together our annual Beowulf Bash. It will take place at its traditional place and time: during the week of the IEEE Supercomputing Conference. This year SC07 is in Reno NV during the week of Nov 10-15 We are trying to keep a casual event with broad sponsorship. The attraction is the other attendees, rather than high-end food or entertainment. Read that as "beer, snacks, and people you always wanted to talk to." We are tenatively planning the party for Tuesday or Wednesday evening 6-8pm in the downtown Reno area. Probably Third Street Blues (formerly The Blue Note) or a similar low-key bar. (BTW, if any reader knows of other vendor events that we should avoid, please let me know.) If your company (or even you as an individual) would like to help sponsor the event, please contact me, becker@beowulf.org Sponsors - get their name up in lights (Errrmm, or maybe just vinyl signs... you bring a sign, there will be some ambient light.) - are part of the brief greeting in the middle of the party. - have the opportunity for technical, hands-on demos at the bash - have their logos on the beowulf.org site through the end of SC07, and on the 2007 yearbook page after the event. -- Donald Becker becker@scyld.com Penguin Computing / Scyld Software www.penguincomputing.com www.scyld.com Annapolis MD and San Francisco CA From becker at scyld.com Fri Oct 5 12:26:22 2007 From: becker at scyld.com (Donald Becker) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] BWBUG / Beowulf Users Group meeting Oct 9 2007 at Georgetown Message-ID: Baltimore Washington Beowulf User Group Meeting Date: 9 Oct 2007 at 2:30 pm - 5:00pm. Location: Georgetown University at Whitehaven Street 3300 Whitehaven Street, Washington DC 20007 Speaker: Donald Becker, CTO of Scyld Software and Penguin Computing Host: Michael Fitzmaurice Here is the announcement from Mike: We are pleased to announce that Donald Becker the founder of the Beowulf Project at NASA, the founder of Scyld the world leader in Cluster Management Software and the CTO of Penguin Computing will be our special guest speaker. October 9th, 2007 at Georgetown University from 2:30 to 5:00 PM. Please join us for a very informative talk on High Performance Computing. Meeting Location: 3300 Whitehaven Street, Washington DC 20007. (This is NOT at the main Georgetown U campus. This is an off campus building one block from Wisconsin Avenue). If you have suggestions regarding speakers for the next meeting or meetings in the future please send them to beowulfUG@aol.com -- Donald Becker becker@scyld.com Penguin Computing / Scyld Software www.penguincomputing.com www.scyld.com Annapolis MD and San Francisco CA From nixon at nsc.liu.se Fri Oct 5 13:17:59 2007 From: nixon at nsc.liu.se (Leif Nixon) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 32 nodes cluster price In-Reply-To: <200710051508.l95F7xvL030704@b.mail.sonic.net> (Geoff Galitz's message of "Fri, 5 Oct 2007 17:07:49 +0200") References: <200710051508.l95F7xvL030704@b.mail.sonic.net> Message-ID: "Geoff Galitz" writes: > Why do you automatically distrust hardware raid? To some extent I share Mark's sentiment. I certainly trust the Linux kernel more than the firmware in a cheap raid controller. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From diep at xs4all.nl Sat Oct 6 15:24:09 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement References: Message-ID: <001001c80867$a1f48730$0900a8c0@objection> Any chance of us in Europe seeing something of this online in a live webcam broadcast? Thanks, Vincent ----- Original Message ----- From: "Donald Becker" To: "Beowulf Mailing List" ; "Beowulf Announcement mailing list" Sent: Friday, October 05, 2007 8:42 PM Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement and sponsorshipopportunity > > > > Ninth Annual Beowulf Bash > > We are putting together our annual Beowulf Bash. It will take place at > its traditional place and time: during the week of the IEEE Supercomputing > Conference. This year SC07 is in Reno NV during the week of Nov 10-15 > > We are trying to keep a casual event with broad sponsorship. > The attraction is the other attendees, rather than high-end food or > entertainment. Read that as "beer, snacks, and people you always wanted > to talk to." > > We are tenatively planning the party for Tuesday or Wednesday evening > 6-8pm in the downtown Reno area. Probably Third Street Blues (formerly > The Blue Note) or a similar low-key bar. (BTW, if any reader knows of > other vendor events that we should avoid, please let me know.) > > If your company (or even you as an individual) would like to help > sponsor the event, please contact me, becker@beowulf.org > > Sponsors > - get their name up in lights (Errrmm, or maybe just vinyl signs... > you bring a sign, there will be some ambient light.) > - are part of the brief greeting in the middle of the party. > - have the opportunity for technical, hands-on demos at the bash > - have their logos on the beowulf.org site through the end of SC07, > and on the 2007 yearbook page after the event. > > > -- > Donald Becker becker@scyld.com > Penguin Computing / Scyld Software > www.penguincomputing.com www.scyld.com > Annapolis MD and San Francisco CA > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From john.leidel at gmail.com Sat Oct 6 15:55:08 2007 From: john.leidel at gmail.com (John Leidel) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement In-Reply-To: <001001c80867$a1f48730$0900a8c0@objection> References: <001001c80867$a1f48730$0900a8c0@objection> Message-ID: <1191711308.697.75.camel@e521.site> Don't know if they have an internet connection at the bar... doesn't look like many wifi hotspots within a reasonable distance : http://www.wi-fihotspotlist.com/search.php?street=125+W+3rd +St&city=Reno&state=NV&zip=89501&country=US&network=&proximity=1&submit.x=100&submit.y=12&submit=Show+HotSpots On Sun, 2007-10-07 at 00:24 +0200, Vincent Diepeveen wrote: > Any chance of us in Europe seeing something of this online in a live webcam > broadcast? > > Thanks, > Vincent > ----- Original Message ----- > From: "Donald Becker" > To: "Beowulf Mailing List" ; "Beowulf Announcement > mailing list" > Sent: Friday, October 05, 2007 8:42 PM > Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement and > sponsorshipopportunity > > > > > > > > > > Ninth Annual Beowulf Bash > > > > We are putting together our annual Beowulf Bash. It will take place at > > its traditional place and time: during the week of the IEEE Supercomputing > > Conference. This year SC07 is in Reno NV during the week of Nov 10-15 > > > > We are trying to keep a casual event with broad sponsorship. > > The attraction is the other attendees, rather than high-end food or > > entertainment. Read that as "beer, snacks, and people you always wanted > > to talk to." > > > > We are tenatively planning the party for Tuesday or Wednesday evening > > 6-8pm in the downtown Reno area. Probably Third Street Blues (formerly > > The Blue Note) or a similar low-key bar. (BTW, if any reader knows of > > other vendor events that we should avoid, please let me know.) > > > > If your company (or even you as an individual) would like to help > > sponsor the event, please contact me, becker@beowulf.org > > > > Sponsors > > - get their name up in lights (Errrmm, or maybe just vinyl signs... > > you bring a sign, there will be some ambient light.) > > - are part of the brief greeting in the middle of the party. > > - have the opportunity for technical, hands-on demos at the bash > > - have their logos on the beowulf.org site through the end of SC07, > > and on the 2007 yearbook page after the event. > > > > > > -- > > Donald Becker becker@scyld.com > > Penguin Computing / Scyld Software > > www.penguincomputing.com www.scyld.com > > Annapolis MD and San Francisco CA > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Sat Oct 6 17:20:10 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement In-Reply-To: <1191711308.697.75.camel@e521.site> References: <001001c80867$a1f48730$0900a8c0@objection> <1191711308.697.75.camel@e521.site> Message-ID: <6.2.3.4.2.20071006170958.03118da0@mail.jpl.nasa.gov> At 03:55 PM 10/6/2007, John Leidel wrote: >Don't know if they have an internet connection at the bar... doesn't >look like many wifi hotspots within a reasonable distance : > >http://www.wi-fihotspotlist.com/search.php?street=125+W+3rd >+St&city=Reno&state=NV&zip=89501&country=US&network=&proximity=1&submit.x=100&submit.y=12&submit=Show+HotSpots Not to mention that most restaurants and bars, or businesses for that matter, do not allow any sort of photography or video from their premises. The reasons are varied, and not always legitimate, but, other than some sort of obvious "taking a picture of my friends at the table" will typically result in the manager asking you to stop (or worse). see, e.g., http://lessig.org/blog/2003/05/dear_starbucks_say_it_aint_tru.html >On Sun, 2007-10-07 at 00:24 +0200, Vincent Diepeveen wrote: > > Any chance of us in Europe seeing something of this online in a > live webcam > > broadcast? > > > > Thanks, > > Vincent > > ----- Original Message ----- > > From: "Donald Becker" > > To: "Beowulf Mailing List" ; "Beowulf Announcement > > mailing list" > > Sent: Friday, October 05, 2007 8:42 PM > > Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement and > > sponsorshipopportunity > > From gerry.creager at tamu.edu Sat Oct 6 20:07:06 2007 From: gerry.creager at tamu.edu (Gerry Creager) Date: Sat Jul 19 01:06:26 2008 Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement In-Reply-To: <1191711308.697.75.camel@e521.site> References: <001001c80867$a1f48730$0900a8c0@objection> <1191711308.697.75.camel@e521.site> Message-ID: <47084D5A.1030702@tamu.edu> I can bring my Sprint 3G card... John Leidel wrote: > Don't know if they have an internet connection at the bar... doesn't > look like many wifi hotspots within a reasonable distance : > > http://www.wi-fihotspotlist.com/search.php?street=125+W+3rd > +St&city=Reno&state=NV&zip=89501&country=US&network=&proximity=1&submit.x=100&submit.y=12&submit=Show+HotSpots > > On Sun, 2007-10-07 at 00:24 +0200, Vincent Diepeveen wrote: >> Any chance of us in Europe seeing something of this online in a live webcam >> broadcast? >> >> Thanks, >> Vincent >> ----- Original Message ----- >> From: "Donald Becker" >> To: "Beowulf Mailing List" ; "Beowulf Announcement >> mailing list" >> Sent: Friday, October 05, 2007 8:42 PM >> Subject: [Beowulf] 9th Annual Beowulf Bash: Announcement and >> sponsorshipopportunity >> >> >>> >>> >>> Ninth Annual Beowulf Bash >>> >>> We are putting together our annual Beowulf Bash. It will take place at >>> its traditional place and time: during the week of the IEEE Supercomputing >>> Conference. This year SC07 is in Reno NV during the week of Nov 10-15 >>> >>> We are trying to keep a casual event with broad sponsorship. >>> The attraction is the other attendees, rather than high-end food or >>> entertainment. Read that as "beer, snacks, and people you always wanted >>> to talk to." >>> >>> We are tenatively planning the party for Tuesday or Wednesday evening >>> 6-8pm in the downtown Reno area. Probably Third Street Blues (formerly >>> The Blue Note) or a similar low-key bar. (BTW, if any reader knows of >>> other vendor events that we should avoid, please let me know.) >>> >>> If your company (or even you as an individual) would like to help >>> sponsor the event, please contact me, becker@beowulf.org >>> >>> Sponsors >>> - get their name up in lights (Errrmm, or maybe just vinyl signs... >>> you bring a sign, there will be some ambient light.) >>> - are part of the brief greeting in the middle of the party. >>> - have the opportunity for technical, hands-on demos at the bash >>> - have their logos on the beowulf.org site through the end of SC07, >>> and on the 2007 yearbook page after the event. >>> >>> >>> -- >>> Donald Becker becker@scyld.com >>> Penguin Computing / Scyld Software >>> www.penguincomputing.com www.scyld.com >>> Annapolis MD and San Francisco CA >>> >>> >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit htt