Why We use “FAIL” for many reasons?
Really why we are so restricted by test automation tools such as Mercury QTP and AutomatedQA TestComplete and other popular ones give us just 2 options – either report Pass or report Fail. If a test log contains at least one Fail – the whole run is getting Fail. This is OK. However I definitely want to have different statuses for checkers belonging to test design and other things occurring from time to time in automation. I propose the following breakdown:
Failed = on unmatched Assertions, comparisons and checkpoints. If test automation is checking GUI consistency, explicit GUI checkpoints should be designed
Error = on any runtime errors, exceptions, unexpected failures, compilation errors, missed objects, improper interfaces/methods/properties applied to objects. Generally – all fails out of checkpoint and assertions
Unfortunate thing is that tools do not provide such statuses. Basically we have only Fail and Warning (but the last one does not affect overall status). The problem is rising when your team builds huge test automation suites and this team is responsible for test execution analysis. And not only for analysis and reporting to customers and management but that team should do it timely and efficiently with minimal effort. It means also that product issues should be reported ASAP and first to provide whole real picture on build quality as early as possible. The second thing is clearing up testing code, fix infrastructure problems, and discover reasons of unexpected failures (all that is what I indicated as Error). Well, actually the optimal work path under this pressure is essential. But how I can differentiate where is Fail in order to process and analyze first and where is that unexpected Error (the second matter), if tools vendors gave me generic Fail only??? I must to drill down to all failed test instances one by one.
Another not good practice is to continue test execution on any failure. I rather prefer to stop and cleanup execution at any failure (either Failed or Error)
Here we go and the time to workaround is come.
We will create custom status in high-level reporting.
Turn off logging at all at the very testing beginning:
QTP: Reporter.Filter = rfDisableAll
TestComplete: Log.Enabled = False
then we will need to write 2 wrapping functions which will report Failed and Warning
//Call this function on any test checkpoint/assertions failed
function LogFail (message, priority, options) {
Log.Enabled = True // QTP: Reporter.Filter = rfEnableAll
//Log Failure code
Log.Enabled = False // QTP: Reporter.Filter = rfDisableAll
CleanUp (True, "Fail")
}
//Call this function on any unexpected error, exception…
function LogError (message, priority, options) {
Log.Enabled = True // QTP: Reporter.Filter = rfEnableAll
//Log Failure code
Log.Enabled = False // QTP: Reporter.Filter = rfDisableAll
CleanUp (True, "Error")
}
After that we will learn high-level reporting (aka Dashboard) to understand and properly display if a test log contains at least one Warning – show up Error. Otherwise – overall run status should be matched with native status decided by test tool, i.e. Pass=Pass, Fail=Fail
In TestComplete/QTP test run should message to Dashboard (TestRunStatus variable) DB in such a way within Cleanup () method:
Function CleanUp (doTerminate, ErrLevel) { //doTerminate True/False
//Some operations to free execution
//…
//
If (ErrLevel = “Fail”) TestRunStatus = Fail
Else If (ErrLevel = “Error”) TestRunStatus = Error
//Calling termination whole test execution
If (doTerminate) TerminateExecution()
}
So what we’ve got? Localization project issues (including defects) is getting quicker, as Fail now means failure in functionality or behavior of tested application (AUT) – wrong result which is also known as deviation from Expected result; opposite status Error means something unexpected happened (test code problem, infrastructure, changes in AUT interfaces (e.g. changed Button caption results in impossibility to recognize the Button on screen).
Moving forward, someone can come up that Error and/or Fail could be split up on a few more. E.g. Error set can be considered as Exception | Environmental Error | 3rd Party exception | Object missed | Object improper method/property | Execution crash. We can think out any custom execution statuses but not sure if that really needed.
The proposed solution is good way to avoid mix up of different failures. Another approach I’d like to recommend (especially for those who use TestComplete) is to employ priority of logged messages (in our case Fail). TestComplete allows specifying priority of posted message to log. This gives opportunity to make more flexible model of failures reporting. Somehow it implies to make your framework remember maximum number of priority of posted Fail to the log. Finally maximum priority value will be Fail priority for test run overall. Then you can design Dashboard with usable coloring to represent priority and to give direction to analysis order from huge scope. In the example below, Fail (500) could be used as Fail in application test itself (assert/checkpoint fail), Fail (100) – unexpected exception (e.g. database connection exception), Fail (300) – Browser hang (who knows, maybe our tested application is too leaking and caused to browser hang)
bugpub.com
Posts
