[Beowulf] Problem with Single RAID disk larger than 2TB and Linux

[Joe Landman]

Hi Anand:

Anand Vaidya wrote:
> Dear Beowulfers,
> 
> We ran into a problem with large disks which I suspect is fairly common, 
> however the usual solutions are not working.  IBM, RedHat have not been 
> able to provide any useful answers so I am turning to this list for 
> help. (Emulex is still helping, but I am not sure how far they can go 
> without access to the hardware)
> 
> Details:
> 
> * Linux Cluster for Weather modelling
> 
> *  IBM Bladecenter blades and an IBM x3655 Opteron head node FC attached 
> to a Hitachi Tagmastore SAN storage, Emulex LightPulse FC HBA, 
> PCI-Express, Dual port
> 
> * RHEL 4update5, x86_64 kernel 2.6.9-55 SMP and RHEL provided Emulex 
> driver (lpfc) and lpfcdfc also installed

There is a problem in some parted versions (prior to 1.8.x) where they 
munge the partition table on gpt/large disks.

Apart from suggesting a more modern kernel (2.6.22.6 or so), there may 
be other things you can do.

> 
> * GPT partition created with parted

I presume this is 1.6.19 parted?

> 
> There is one 2TB LUN, works fine.
> 
> There is a 3TB LUN on the Hitachi SAN which is reported as "only" 2199GB 
> ( 2.1TB) ,

Yup.  Sounds like either the parted problem, or a driver issue.

What does

	parted /dev/3TBlun/ print

report where /dev/3TBlun  is the device containing the 3TB lun?

We had seen this behavior with some 1.6.9 and 1.7.x versions of parted. 
  The only way to fix it was to rebuild parted with 1.8.x

> 
> We noticed that, when the emulex driver loads, the following error 
> message is reported:
> 
>             Emulex LightPulse Fibre Channel SCSI driver 8.0.16.32 
> <http://8.0.16.32>
>             Copyright(c) 2003-2007 Emulex.  All rights reserved.
>             ACPI: PCI Interrupt 0000:2d:00.0[A] -> GSI 18 (level, low) 
> -> IRQ 185
>             PCI: Setting latency timer of device 0000:2d:00.0 to 64
>             lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: 
> x2 x4 x1000
>             lpfc 0000:2d:00.0: 0:1305 Link Down Event x2 received Data: 
> x2 x4 x1000
>             lpfc 0000:2d:00.0: 0:1303 Link Up Event x3 received Data: x3 
> x1 x10 x0
>             scsi5 : IBM 42C2071 4Gb 2-Port PCIe FC HBA for System x on 
> PCI bus 2d device 00 irq 185 port 0
>             Vendor: HITACHI   Model: OPEN-V*3          Rev: 5007
>             Type:   Direct-Access                      ANSI SCSI 
> revision: 03
>             sdb : very big device. try to use READ CAPACITY(16).

This is what our JackRabbit reports ...

>             sdb : READ CAPACITY(16) failed.

This is not what our JackRabbit reports.

[...]

> The problem is with the READ CAPACITY(16) failed, but we are unable to 
> find the source of this error.
> 
> We conducted several experiments without success:
> 
> - Tried compiling the latest driver from Emulex (8.0.16.32 
> <http://8.0.16.32>) - same error
> - Tried Knoppix (2.6.19) and Gentoo LiveCD (2.6.19 ) , and CentOS 4.4   
> - same error

Sounds a great deal like parted.

> - Tried to boot Belenix (Solaris 32 bit live), failed to boot completely 
> (may be unrelated issue)
> 
> We have a temporary workaround in place: We created 3x1TB disks and used 
> LVM to create a striped 3TB  volume with ext3 FS. This works fine.
> 
> RedHat claims ext3 and RHEL4  supports disks upto 8TB and 16TB 
> respectively (since RHEL4u2)

... yeah.

> 
> I would like to know if anyone on the list has any pointers that can 
> help us solve the issue.

Please run the parted command as indicated.  Lets see what the partition 
table thinks it is.

Do you have data on that partition?  Can you remake the label on that 
device with a new version of parted?

> 
> Regards
> Anand Vaidya
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


-- 

Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615