Normal distribution in action: defect distribution modeling and prediction

Preface: to be on the same page, it's recommended to review the following WP articles:
http://en.wikipedia.org/wiki/Normal_distribution

http://en.wikipedia.org/wiki/Theory_of_errors

http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics

Though, the notion normal (Gaussian) distribution appears potentially helpful for s/w project planning and analysis, but it's strongly recommended to play with this tool carefully and what more important - with statistically sufficient and significant data set. Yes there is a constraint - the project should be big enough. However, does it really important to forecast new defects and persistent ones on a project having 5000 SLOC? Don't think so.

Obviously this theory may help to:
- estimate volume of non-covered/discovered defects. By this - rebuild test plan to achieve proper test coverage.
- predict volume and distribution of newly revealed product defects. By this - come up and/or numerically adjust with project sign off date. Somehow metrics could shift project release or may play start point of more resources negotiation.
- model efficient test automation (functional and unit) coverage. By this - achieve high ROI on automation.

All you need is to:
- select a sampling variable. In our case likely it is number of valid reported defects, e.g. weekly/daily. It depends periods you are going to operate on.
- calculate mean over this variable
- calculate variance of this sampling variable
- Then build and graph Probability density function (PDF)


This graph as well as mean and variance calculations are easy task using Excel formulas and graphs. The resulting graph should be build together with original sample in time series (histogram). The view of the curve itself shows "normality» of this sample. Overlapping of two graphs shows divergence of normality with real state. But don't hurry to make judgments on this curve it's just tip for you to feel confidence of project control. E.g. the curve with open tail (end in time series) may signal that testing should be prolonged as there are undiscovered defects as expected statistically.

The next advanced application is prediction. To make it you need to build either ideal normal distribution or use existing one. Then the restore function will give you remaining sample sub-set (show future). So that, you may say how many defects will be found for example week by week. Or what functionality needs to be tested with more effort

Finally to calculate precision over your calculations you have to come up with confidence intervals of your observations.

Myths and silly Viewpoints about Test Automation

1. Test automation is just spending money without return
2. Automated testing is always profitable. Test automation fits to any kind of project
3. Unit testing is better and finds more defects then functional automated testing
4. Record-playback is automated testing
5. Any person with IT background can be auto tester
6. It's easy to combine both manual and automate dtesting in a single role person
7. Test automation ROI is impossible to predict and calculate
8. Test automation estimation is not real activity
9. Test automation completely differs from s/w development
10. Bad developer can be good enough for automation
11. Developers can effectively develop functional auto tests
12. Auto testing is not about QA
13. Auto testing should not be a part of overall QA and QC
14. Manual testing coverage is preferable on any project/functionality and any project phase
15. Nobody should rely on automated testing
16. It's impossible to create robust and efficient auto testing at all
17. Everything can be automated
18. Is auto testing a sort of machine intelligence? Though, I don't trust that
19. Test automation should be capable to predict any system and environmental outcomes. Whatever issue occur - auto test suite should remove a problem and/or think out a new workflow path, recovery scenario, invent workaround from a scratch
20. Test automation is about writing a few code lines which result in testing major functionality. It's easy!

I wish to hear more myths and strange points you ever met on test automation. Let's share yours here commenting to this post.

Looking for Contributors or Fellows for this Blog

Hey there!

If you have to tell something novice, interesting or maybe amazing - I'd like to see your content published here. Also, fell free to discuss oportunity of authorship on this blog too.

Have a nice holidays!

My post is featured in Automated Software Testing Magazine !

Hey!

it was nice to be informed my blog 20 Essential design patterns for automated testing (http://at4qa.blogspot.com/2010/01/20-essential-design-patterns-for.html) had been chosen as featured in Automated Software Testing Magazine, Issue March 2010. I got the news while being in vacation - especially pleasant.

Download the official e-version of this magazine here [http://www.automatedtestinginstitute.com/home/ASTMagazine/2010/AutomatedSoftwareTestingMagazine_March2010.pdf]. Then, flip to page 36 - there is my first column out there, cool!

WMI Code creator from MS

Just stumbled on a post about MS tool to ease WMI querying/executing/eventing here
http://qtpcodes.blogspot.com/2010/03/use-wmi-code-creator-tool-for-quick-wmi.html

The tool is downloadable from Microsoft site here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=2cc30a64-ea15-4661-8da4-55bbc145c30e&displaylang=en

While trying it out, came up with a thought - this is really handy for testers, especially for automation ones. Btw, being hand on this tool, you can develop system-level and network-communication apps and utils using Scripting only utilizing powerful resource management tool!

I'm going to deeply learn the supported WMI namespaces and classes and write some scripting hooks just for fun.

PS Author(s) had provided capability to convert generated code snippets among C#, VB .Net and VBS languages

JSystem Test Automation Framework

Introduction
JSystem is an open-source framework for writing and managing automated system tests. The aim of JSystem is to provide one simple central framework for functional and system testing of software and embedded systems.

JSystem is an open system that supports leading scripting languages (such as .Net, Python and Perl) and communication protocols. It also connects with leading testing tools, both open-source (Selenium, AutoIt, JUnit, etc.) and commercial (IXIA, Spirent, etc.).


The JSystem Architecture
The JSystem architecture comprises of the System under Test, the Automation Framework and Framework Services, outlined in the diagram below:



The System Under Test (SUT) is the device or software that is being tested, as well as the equipment and applications that participate in the test setup.

The Automation Framework does the testing. The framework divides the testing process into four layers that separate between the actual writing of the tests (done in Java), and the grouping and re-grouping of the tests into different test-scenarios, as well as executing and analyzing them – all of which don't require any knowledge of code and are done with an intuitive GUI interface (JRun):

The four Automation Framework layers are:
- System Object layer: written in Java, these are the basic building blocks of the test.
- Test-Case layer is used to create the tests.
- Test-Scenario Layer: groups the tests within a test-scenario.
- Management Layer: an application that executes analyzes and publishes the test results.

The Framework Services: provide the QA engineer with functionality that assists in testing.


What Makes JSystem Unique?
The JSystem architecture builds tests out of basic building blocks that can be used and reused to create new tests; the clear distinction between business logic and GUI enables the tester to change one without affecting the other – all this translates into much more efficient testing.

Because of its four-layer architecture, JSystem also enables to incorporate both QA people that write code, and QA people that don't. Although creating the System Objects is done in Java (albeit, very simple Java), creating the test-scenarios, executing and analyzing and then generating reports, is all done with the GUI interface (JRun). JRun enables creating test-scenarios with drag-and-drop. This is a screenshot of the JRun main menu:



JSystem also contains built-in drivers for all major testing tools, including freeware drivers for open-source tools (such as Selenium, JUnit and AutoIt) as well as commercial tools.

JSystem Deployments
JSystem is a mature product with deployments is large organizations, including telecommunications labs in such companies as Juniper, Nokia-Siemens, ECI, Avaya and Ericsson. JSystem is deployed in testing Web applications by companies such as Alcatel-Lucent and Taiana, and in testing semiconductors in companies such as Intel, Altair and Metalink.

For more information on JSystem, please go to their website: www.jsystemtest.org.

DISCLAIMER: Copyright (c) Tal Harel. This article is prepared by Tal Harel as material for this Blog (http://at4qa.blogspot.com)