[Beowulf] Linux quality vs. defects

[Lux, Jim (337C)]

I'm involved in some research with defect rates (on software that goes in to spacecraft), and more interestingly the rate of change of defect density.

I would guess that defect densities driven by a lot of factors besides the skill of the practitioners, the language chosen, or the schedule pressure.

We see jumps in defect density just before major reviews and deliveries, but that is probably an artifact of "getting the tests run", because the typical statistics are "when was the defect discovered" not "when was the defect created".  Those times are also typically when big integration efforts take place, which is historically a time when you find out that the requirements and interface specifications do not fully specify the behavior of the boxes being integrated.  Here, the "defect" is really not properly assigned to a particular block of code: both sides met their requirements and are defect free from that standpoint.  It's more a failing at a higher level in the process.  Is it fair to tag one block or the other with the defect (increasing the defect density of those blocks)?

We have several folks here at JPL who have studied things like "defects discovered over time" with an objective to predict when you've reached the asymptotic defect density (I think Brooks's book first mentioned this phenomenon, in connection with OS/360) , so you can stop the polishing of the cannonball that will never be round.  They have found that for a particular team in a particular project environment, over time you can develop calibration factors, but that you can't transfer those calibrations to other teams or projects.  The same sort of phenomenon occurs with predictions of effort or schedule requirement: any one team produces consistent estimates for their own work on their product (with some calibration factor between the estimate and actual) but the multiplier is different for every instance.

Another can of worms is the whole problem of the denominator in the defect rate calculation: what's a "line of code".  That in itself makes it difficult to compare languages.  (FORTRAN might be easiest. Pull out the comment cards, count the rest in your trusty card counting machine, and you're done.. Heck, you could do it in a few minutes with minimum programming of the plug board)

We have a source code counting tool called SLIC which attempts to count logical lines of code, and for which they've made a good effort to get numbers that are consistent across some languages.  But still, how do you deal with the "lines of code" that are things like boilerplate wrappers?  Do you count lines in .h files?  They're non executable, but I've found that significant numbers of defects arise from errors in those files.

I'm not sure that defect rate is highly correlated with experience.  It's probably more correlated with "experience with this specific code". Junior coders don't produce significantly more defects than senior coders, but they tend to be easier to find and more spectacular in their effects.  Senior coder defects, I would guess, are more expensive to handle: they tend to be more subtle, edge case kinds of things.  The senior coder tends to use "more" of a language.

The lowest "serious" defect rates probably come from people who are experienced in one language, and coding in another.  We've all seen folks who write FORTRAN style in C, for instance (that would be me).  So the errors tend to be pretty obvious ones resulting from a fundamental difference in how the two languages work, and the coder tends to not use much of the new language, and where they do, they tend to have the reference manual open at their side.



Jim Lux

From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Peter St. John
Sent: Monday, May 13, 2013 9:30 AM
To: Max R. Dechantsreiter
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] Linux quality vs. defects

re "...written in Linux, PHP, and Apache..."'; PHP is a language, but Apache is a web server (with a development environment) and Linux is an Operating System (which can be considered as a development environment). So this seems like comparing apples and oranges. If I write PHP via Apache running on Linux, as in LAMP, and I have a defect, does it count as PHP, or Apache, or Linux? or all three, and they are doing correlation analysis?

It's not easy to compare even languages by defect rate (although it's useful). C++ is easy to mess up, but also it's ubiquitous, many people who would rather write in something else are stuck with it (ditto Java). Lisp can be pretty confusing and you can have highway-blindness from the parentheses, but I bet it has a low defect rate, if only because the only people who write in it are professionals who choose it as the vernacular for their application domain (AI).

My bet is that Fortran has the lowest defect rate, because no-where on the planet is any inexperienced kid being rushed to meet a deadline using fortran.

Peter
On Mon, May 13, 2013 at 11:53 AM, Max R. Dechantsreiter <max at performancejones.com<mailto:max at performancejones.com>> wrote:
Linux Leads in Open Source Quality, but Risky Defects Lurk
Government Computer News (05/11/13) Paul McCloskey

Linux topped open source software in quality in a study of the defects that occur in the software development process.  For more than seven years, Coverity Scan Service analyzed 850 million lines of code from more than 300 open source projects, including those written in Linux, PHP, and Apache.  Using a measure of defects per 1,000 lines of code, the study found that Linux consistently recorded defect densities of less than 1.0, with versions scanned between 2011 and 2012 having defect rates below 0.7.  The study also found that high-risk defects were prevalent in the software development process, with 36 percent of defects classified as a "threat to overall software quality and security if undetected."  The most common high-risk defects included memory corruption, illegal memory access, and resource leaks, which the study's report says are "all difficult to detect without automated code analysis."  The study also found that the average quality of open source software was virtually equal to that of proprietary software.
http://gcn.com/blogs/pulse/2013/05/linux-leads-in-open-source-quality-but-risky-defects-lurk.aspx
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20130513/836d137e/attachment.html>