[Beowulf] hpl size problems

[Robert G. Brown]

Mark Hahn writes:

> On Sat, 1 Oct 2005, Robert [UTF-8] G. Brown wrote:
> ...
>> Seriously, how hard is it to move a code TOWARDS e.g. posix compliance?
> 
> I think that unfortunately several separate issues are being conflated:

Possibly, but I view them all as part of a large conflation, if you
like.  One of the things that makes porting software in general
difficult is a lack of posix (or other) compliance in the source
libraries, a lack of uniformity in how include files are managed.  Code
really should not need to be #ifdef'd to within an inch of its life in
order to achieve a portable build.  This is the way it was back in
/usr/local days, which is why I refer to it now.  It is an issue I
>>still<< face with legacy sources even now.  I use jove, for example,
because I'm too lazy or stubborn to learn a new editor.  I therefore
have to periodically expend at least enough effort to get it to build
clean under the latex linux revision and libraries.  The real killers
aren't the simple subroutines and C code, it is the system calls, the
deep stuff.  You have to hunt down the subroutine calls one at a time,
try to figure out what they were supposed to be doing, replace them with
a POSIX compliant thing that does the same thing, debug the result, test
the result, and finally smooth out all the ruffles caused by incorrect
or changed variable typing.

So the issue of making e.g. jove into a src rpm that will "just build"
on any rpm-based system without re-hacking it goes beyond "just" the
packaging, although that is an important part.  It also requires that
one face the issue of posix-compliant code and libraries (and compliance
with all sorts of OTHER related standards in both library, include and
application code alike) so that code doesn't have to be heavily
instrumented in order to build and run.  autoconf etc try to fix this
and can sometimes succeed, but all too often they fail.

This is precisely the stuff that keeps those ancient sources from being
ported to modern times much more so than "mere" packaging.  One can,
after all, package precompiled binaries and support files if one can't
do any better than that.  I just think that it ends up in practical
terms being very difficult to separate out the layers upon which long
term success depends.  But I do agree that it extends the discussion
beyond the PITA of packaging certain applications so they can easily be
distributed.

It's interesting that you cited ATLAS as an example, BTW, because it is
a very good one in both directions.  What exactly is something like
ATLAS?  It would require a lot more metadata than I think is currently
available in order to make it packageable and distributeable in a
plug-in fashion so that once it were installed all applications that do
linear algebra would "just use it" where appropriate.  Some of the
missing metadata involves the identification of the precise architecture
it was built for; more involves finding the right package to match the
architecture.  A third defines precisely what it can replace, and how.
I'm not certain about the fourth -- how to make "anything" that might
use it divert calls so that they do, at least not without making it
comply with a general ABI.

This is where I think things NEED to head.  RPM was and is in some ways
a lovely thing.  However, I personally think it's way short on the
metadata front, and woefully difficult to extend.  XML offers some hope
of extendibility without requiring that the final specification be
written a priori.  Perhaps the best approach will end up being XML plus
a style sheet (or more appropriately, data dictionary) used by the
package developer to generate the requisite metadata layer in a
reasonably portable and extendable way.

   rgb

> - package format.  as far as I can see, there is no real disagreement
> about what metadata needs to be attached to a package.  this means that 
> differences are just petty turf issues, and that it's a SMOP to write 
> a generic installer.  more importantly, to the app-vendor, any format will do.
> 
> - dependency-registry.  this is a bit sticky, since it requires a universal
> language/catalog of _capabilities_, that is a set of canonical names for 
> some standard of interface and behavior.  this is easy when there's a single 
> implementation (eg kde), but tougher for something like MTAs, where any
> number of implementations will do.
> 
> - interface/behavior spec.  typically an ABI, but will also include
> command-level things ("newaliases" works for both sendmail and postfix).
> 
> I think that efforts like LSB have not taken this approach mainly because
> it seems too intensive.  basically, every ABI function would have a 
> <capability, version, provider> entry, but traditionally it's whole 
> packages, not individual functions that are versioned.  (consider installing
> atlas, which does not provide full lapack).
> 
> obviously, there is also resistance to this because it inherently unifies
> interfaces, and makes it harder for vendors to distinguish themselves.
> (or, for the pessimists out there, to lock in apps/users to a particular 
> distro.)  I don't really see any reason in principle why Linux vendors 
> couldn't agree on this sort of standardization.  LSB might even be able 
> to do it, though I don't really see the point of mandating a particular 
> package format.  (the capability names, behavior, interface and versioning
> have to be agreed on, but as long as there is some ABI for querying the 
> per-machine capability registry, any packaging format could be used.)
> 
> imagine if there was a standard way to enumerate whether a machine has an 
> implementation of <daxpy,2.3.0,x86-64>, and to not only get the answer,
> but the name of the library to link with.  (libtool is related to this,
> but I think it's mainly a workaround for the lack of dependency-registry,
> and is probably crippled by lack of canonical capabilities (standard).)
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20051001/c6771060/attachment.sig>