<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’m involved in some research with defect rates (on software that goes in to spacecraft), and more interestingly the rate of change of defect density.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I would guess that defect densities driven by a lot of factors besides the skill of the practitioners, the language chosen, or the schedule pressure.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We see jumps in defect density just before major reviews and deliveries, but that is probably an artifact of “getting the tests run”, because the typical statistics
are “when was the defect discovered” not “when was the defect created”. Those times are also typically when big integration efforts take place, which is historically a time when you find out that the requirements and interface specifications do not fully
specify the behavior of the boxes being integrated. Here, the “defect” is really not properly assigned to a particular block of code: both sides met their requirements and are defect free from that standpoint. It’s more a failing at a higher level in the
process. Is it fair to tag one block or the other with the defect (increasing the defect density of those blocks)?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We have several folks here at JPL who have studied things like “defects discovered over time” with an objective to predict when you’ve reached the asymptotic
defect density (I think Brooks’s book first mentioned this phenomenon, in connection with OS/360) , so you can stop the polishing of the cannonball that will never be round. They have found that for a particular team in a particular project environment, over
time you can develop calibration factors, but that you can’t transfer those calibrations to other teams or projects. The same sort of phenomenon occurs with predictions of effort or schedule requirement: any one team produces consistent estimates for their
own work on their product (with some calibration factor between the estimate and actual) but the multiplier is different for every instance.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Another can of worms is the whole problem of the denominator in the defect rate calculation: what’s a “line of code”. That in itself makes it difficult to
compare languages. (FORTRAN might be easiest. Pull out the comment cards, count the rest in your trusty card counting machine, and you’re done.. Heck, you could do it in a few minutes with minimum programming of the plug board)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We have a source code counting tool called SLIC which attempts to count logical lines of code, and for which they’ve made a good effort to get numbers that
are consistent across some languages. But still, how do you deal with the “lines of code” that are things like boilerplate wrappers? Do you count lines in .h files? They’re non executable, but I’ve found that significant numbers of defects arise from errors
in those files.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’m not sure that defect rate is highly correlated with experience. It’s probably more correlated with “experience with this specific code”. Junior coders
don’t produce significantly more defects than senior coders, but they tend to be easier to find and more spectacular in their effects. Senior coder defects, I would guess, are more expensive to handle: they tend to be more subtle, edge case kinds of things.
The senior coder tends to use “more” of a language.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The lowest “serious” defect rates probably come from people who are experienced in one language, and coding in another. We’ve all seen folks who write FORTRAN
style in C, for instance (that would be me). So the errors tend to be pretty obvious ones resulting from a fundamental difference in how the two languages work, and the coder tends to not use much of the new language, and where they do, they tend to have
the reference manual open at their side.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Jim Lux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org]
<b>On Behalf Of </b>Peter St. John<br>
<b>Sent:</b> Monday, May 13, 2013 9:30 AM<br>
<b>To:</b> Max R. Dechantsreiter<br>
<b>Cc:</b> beowulf@beowulf.org<br>
<b>Subject:</b> Re: [Beowulf] Linux quality vs. defects<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">re "...<span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#222222;background:white">written in Linux, PHP, and Apache..."'; PHP is a language, but Apache is a web server (with a development environment) and Linux is an
Operating System (which can be considered as a development environment). So this seems like comparing apples and oranges. If I write PHP via Apache running on Linux, as in LAMP, and I have a defect, does it count as PHP, or Apache, or Linux? or all three,
and they are doing correlation analysis?</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";color:#222222">It's not easy to compare even languages by defect rate (although it's useful). C++ is easy to mess up, but also it's ubiquitous, many people who would rather write in something
else are stuck with it (ditto Java). Lisp can be pretty confusing and you can have highway-blindness from the parentheses, but I bet it has a low defect rate, if only because the only people who write in it are professionals who choose it as the vernacular
for their application domain (AI).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";color:#222222">My bet is that Fortran has the lowest defect rate, because no-where on the planet is any inexperienced kid being rushed to meet a deadline using fortran.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Arial","sans-serif";color:#222222">Peter</span><o:p></o:p></p>
<div>
<p class="MsoNormal">On Mon, May 13, 2013 at 11:53 AM, Max R. Dechantsreiter <<a href="mailto:max@performancejones.com" target="_blank">max@performancejones.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Linux Leads in Open Source Quality, but Risky Defects Lurk<br>
Government Computer News (05/11/13) Paul McCloskey<br>
<br>
Linux topped open source software in quality in a study of the defects that occur in the software development process. For more than seven years, Coverity Scan Service analyzed 850 million lines of code from more than 300 open source projects, including those
written in Linux, PHP, and Apache. Using a measure of defects per 1,000 lines of code, the study found that Linux consistently recorded defect densities of less than 1.0, with versions scanned between 2011 and 2012 having defect rates below 0.7. The study
also found that high-risk defects were prevalent in the software development process, with 36 percent of defects classified as a "threat to overall software quality and security if undetected." The most common high-risk defects included memory corruption,
illegal memory access, and resource leaks, which the study's report says are "all difficult to detect without automated code analysis." The study also found that the average quality of open source software was virtually equal to that of proprietary software.<br>
<a href="http://gcn.com/blogs/pulse/2013/05/linux-leads-in-open-source-quality-but-risky-defects-lurk.aspx" target="_blank">http://gcn.com/blogs/pulse/2013/05/linux-leads-in-open-source-quality-but-risky-defects-lurk.aspx</a><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">
http://www.beowulf.org/mailman/listinfo/beowulf</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>