Transform QA Philosophy with Test Automation and Crowd Testing

How to get my startup tested out quickly and cheap?

2012-01-26T20:43:00.000-08:00

Professional testers, hackers and super Users are very hard to find. Most of them are busy on full time basis or just hard to reach out them. Ultimately we end up with hiring full time tester to reduce business risk. Don't get me wrong but your relatives and friends as well as potential Users will never find serious defects and breaches in your awesome app. Testing is a skill, it is not just boring digging. Believe there are lots of testing techniques and design approaches which are unknown for users and natural for testers. Moreover, expert testers have feeling worked out by years of practice under real conditions with real tools and apps under control. It is time needed to become good tester.

Ok, now you are armed and the only thing to be done is just to go to http://bugpub.com and ask professional community to test out your app or site as better as possible in order to meet first Users which will leave positive feedback recommend and return back to you. Do not experiment with your Users.

The process as simple as sending request to estimate your project and setup ypour Jira. First bugs will appear very shortly!

8 TOP Bugs Digest: Gmail vs Yahoomail

2012-01-24T18:01:00.000-08:00

Please welcome top 8 bugs of Gmail and Yahoomail

Gmail 8 Top Bugs:

Yahoomail 8 Top Bugs:

PLEASE, if you like some of these bugs don't leave them alone – send comment, Like it, tweet it or Share it (all that can be done in Comment section for each and any bug report). We will keep posting about most interesting bugs of popular and widely used technology products.

Join BugPub

Unexpected disappointment by Google

2012-01-03T15:26:00.000-08:00

You are working, your Gmail is as usual is opened in background. You check inbox from time to time. Suddenly you get new "gray" theme of Gmail (same as classical blue but just very unwonted color schema (grey, no separator in inbox grid, just as list of paper). Google nicely tells you that they have changed the color scheme and that is it. You should find out yourself in settings how to switch to classical view.Found it, switched back. Happy. Google Labs or Google Experiments don't touch my settings!

Ok, went back to work, left Gmail window as is, did not log off/on. Exactly in 10 minutes went to Gmail to check email. Oh My Google, they force me to switch to yet another brand new look and feel. Force because that dude on your window does not allow you to disregard. He says you have an option to find out yourself how to temporarily switch back to classical mode! Besides once you dig out that setting (try yourself), Google ask to provide a reason why I want back my look and feel which is my 5 years habit. What is next

Pitfalls of wrong usage or non-professional crowd testing by James Whittaker

2011-12-22T17:21:00.000-08:00

James Whittaker has been blaming the crowd testing vendor of their choice for hundreds and hundreds duplicated bugs which are already known by Google team.

"This dog food process is insane"

"Why we do this crowdsourcing it is expensive and it does not do value"

I like this:

"common... do something useful we pay you lots of money"

Though, in one year and one month after that meetup, James admits that matured crowdtesting has a big future and top crowdtesters will be doing a quarter million of dollars per year or even more. Sure they will make these amounts but only passionate, proficient and dedicated individuals. So the bottom line for both Testers and Crowdtesting clients is: go with professional crowd testing.

Most Interesting Facebook bugs Digest

2011-12-20T19:58:00.000-08:00

Hi All,

Today our excitement comes from idea of sharing with you several Facebook.com bugs which appear outstanding and interesting for us. In addition we followed up these findings and all other bugs with Facebook tech support.
PLEASE, if you like some of these bugs don't leave them alone – send comment, Like it, tweet it or Share it (all that can be done in Comment section for each and any bug report). Based on your opinion we will sum up impressions and ultimately will determine "best Facebook bug(s)". It will be driven by people! We will keep posting about most interesting bugs of popular and widely used technology products.

Join BugPub > the CrowdTesting by real professionals with LinkedIn profiles from around the world

SOPA & PIPA Heat up to Boiling Point

2011-12-16T15:18:00.000-08:00

While we hear and see tons of ads and commercials by TV saying "Stop Online Piracy", there is steady flowing and overflowing work of lobbying Stop Online Piracy Act and also Preventing Real Online Threats to Economic Creativity and Theft of Intellectual Property Act of 2011. I would say we are witnesses of big battle of Piracy vs Openness which is in essence wrong because openness and piracy have almost nothing in common, these are just different notions. They have same in common as having glass door of your house and people which can observe your private life through that transparent door.

Just yesterday 83 Internet Inventors have signed and published a letter against SOPA and PIPA initiatives. Most impressive part for me:

"...

If enacted, either of these bills will create an environment of tremendous fear and uncertainty for technological innovation, and seriously harm the credibility of the United States in its role as a steward of key Internet infrastructure. Regardless of recent amendments to SOPA, both bills will risk fragmenting the Internet's global domain name system (DNS) and have other capricious technical consequences. In exchange for this, such legislation would engender censorship that will simultaneously be circumvented by deliberate infringers while hampering innocent parties' right and ability to communicate and express themselves online.

All censorship schemes impact speech beyond the category they were intended to restrict, but these bills are particularly egregious in that regard because they cause entire domains to vanish from the Web, not just infringing pages or files. Worse, an incredible range of useful, law-abiding sites can be blacklisted under these proposals. In fact, it seems that this has already begun to happen under the nascent DHS/ICE seizures program.

Censorship of Internet infrastructure will inevitably cause network errors and security problems. This is true in China, Iran and other countries that censor the network today; it will be just as true of American censorship. It is also true regardless of whether censorship is implemented via the DNS, proxies, firewalls, or any other method. Types of network errors and insecurity that we wrestle with today will become more widespread, and will affect sites other than those blacklisted by the American government.

The current bills -- SOPA explicitly and PIPA implicitly -- also threaten engineers who build Internet systems or offer services that are not readily and automatically compliant with censorship actions by the U.S. government. When we designed the Internet the first time, our priorities were reliability, robustness and minimizing central points of failure or control. We are alarmed that Congress is so close to mandating censorship-compliance as a design requirement for new Internet innovations. This can only damage the security of the network, and give authoritarian governments more power over what their citizens can read and publish.

..."

From https://www.eff.org/deeplinks/2011/12/internet-inventors-warn-against-sopa-and-pipa

Opinions?

Automate Your Automation

2011-11-28T14:50:00.001-08:00

I think if one claims that they have test automation, I mean full-fledged test automation, it must imply not only automated tests per say but it should include process automation. Automated tests (the code) are useful in same extent as manual tests which were never executed. It is just having no value unless it is executed at least once. When we execute tests, we kind a returning back these expenses invested in test design, same for automated tests. However as automation specialists, we should go beyond of this point. We should leverage automated tests by setting up proper automation process which will command all aspects and flows in test automation.

What to be process-wise automated?

Test execution implies triggered executions - kicked of by Continuous integration server, on demand or scheduled. In state of the art project, it could be smart test execution - when only subset of test are executed which belong to a affected with new build pieces of software
Test Reporting and test Results management. Keeping historical data in one place, in mid and long term provide you very valuable statistics, metrics and lessons learnt. Reporting is essential for managers and stakeholders, it helps them to make decision. So our job is to give them a tool which will build report for them and provide valuable information in one screen.
Results analysis and collaboration. Let your team mark results, exchange, share and tag specific test run results with one click; avoid effort duplication; keep tests stable and up to date by acting immediately; make it transparent
Test debugging. Helps you to avoid tedious and hardly manageable test code fixes and improvements, given tool should ease all internal collaboration and accelerate decision making process
Version control is irreplaceable in test automation. Do versioning, branching, tagging, etc!
Auto Tests deployment. Something should run in background to make sure that right and latest version of tests and framework is being executed. In advanced cases, it should be able to deploy test code which is compatible with specific software under test version. Sounds cool!
Test Lab, infrastructure deployment and roll up (e.g. virtual machines stand by and suspend - especially in case of dealing with virtual data centers, on demand cloud centers - basically where run time is run money off)
Environment and SUT availability monitoring (all data that can be precondition for whether start actual test execution or report fatal error and quit)

Gredy addresses most of these process automation aspects. Gredy automates test automation

Would you add more to the list of automation automation?

Everything changes irrevocably and hardly... Just surf's up!

2011-11-18T21:56:00.001-08:00

It is ongoing and no way to stopthese changes - I talk about big changes in s/w development practices andmethodologies, approaches and trends. I don't think everybody realize how Agile hadcaptured whole market, it made it softly and steady, you know, kind a "killme softly". Killers finally will be killed by other gangsters but it isdifferent story. Now we play slightly different game - nimble, quick,lightweight, not thinking about long term - just make it happen today, fewsprint and release. For the time being, I worked for 3 crazy_about_Agilecompanies. And since I'm test automation guy, it was never problem for me; I'mlike a fish in sea. Agile brings new concept of QA, at some extend it breakstraditional QA, it removes user advocates, it kills idea of independent assessmentand verification.

However it brings new opportunities, here are some of these:

- test automation is integral and nobody can escapefrom that responsibility. No more manual testers! What's the heck?

- much more exploratory, ad-hoc, session-based,security, whatever-you -call-challenging or truly breaking testing. Awesometechnical challenge instead of boring step by step testing

- quick bug reporting with tech insight. Do meaningfulheadline, attach logs and some piece of code maybe - you are set. No morerudementary 12 detailed steps to reproduce

- User evolvement - crowdtesting, dog food, beta,alpha demo, blah. Community rulez. The benefits - UA, A/B, functional,compatibility, distributed testing in one in one bottle. BugPub is a gate forreal crowdtesting with win-win model

- No more kind a tools which cover all your needs. This expensive stuff is now called crap. So tools should be either free or cheap and right, I mean perfectly fit your needs

- No more bunch of managers which just double staff

It is ongoing, it is right over you, just surf's up!

Market fragmentation trend - best in class tools for specific domain and no longer solutions that can do everything and nothing

2011-11-16T09:24:00.001-08:00

I really amazed by this great trend which really flips and flops giant software enterprises. This is why they hurry to purchase small start ups and little companies with one but awesome product or service. These companies understands that their heavy solutions are not serving consumers so good and cheap as those little but flexible players. It is understandable that big companies desire to surround their customers by "one stop" solution, closing intentionally integration with other vendors and framing proprietary closed ecosystem which is eventually very costly caused by huge R&D investments to build this one-stop.
So now we as consumers use a lot of apps on our desktops, on web and smartphones (especially!)

This trend is sweeping software companies and IT departments themselves - we want only having deal with best tools solving specific problem and helping out to be productive + enjoy their look and feel. And it does not really matter whether this or that tool is open source or commercial - the most important is User experience, perception and passion to use that tool every day.

A quick example HP ALM (or any other ALM at this point on market) - a tool which can "do everything"... This tool attempts to fit all possible software entities (bugs, requirements, test cases, runs, test automation) into one sort of table view dogma. In essence this is boring, not domain specific - it is just like reflecting database structure on Web, bunch of fields and folder which can at least confuse and even scary. Sorry for negative example given first but it is really ~~pain~~, costly pain.

The time for positive. Atlassian. This company create tools, specific ones, flexible, evolving. People love these tools. You can buy bundle, you can buy one tool. These tools are based on open standards, APIs, etc, so you free to build your ecosystem with tools and technologies of your choice. Let's go back to HP ALM, regarding integration, do they provide integration with CI, which test automation I can use with them apart from QTP, can I integrate with another nice bug tracker, do they support other requirements management systems? NO. Mercury tools gonna follow same mistake as Rational did after acquiring by IBM.

The problem of this kind a system/solution is not implementation, the problem is not even its cost, the problem is just idea itself, the idea of one-stop solution.

There are a lot and a lot great tools on market - a universe of them are commercial and bunch of them open source (especially technologies and frameworks)

Lastly, can you call Wallmart or Amazon one-stop store for you? Software market fragmentation is ongoing

Automation Metrics and KPIs - how to beautify and not sink

2011-11-15T21:00:00.001-08:00

Post-mortem, yearly reports, metrics, KPIs, predictive analysis, graphs, %%%%. Again big reporting for big managers!

My 5 cents which hopefully could help to run off all that nightmare inventing that stuff.

Key metrics for showing test automation contribution and value:

1. Coverage - higher %, bigger scope, more influence

against manual test cases
against functionality/use cases
against source code
against API

2. Test automation ROI (notorious ROI - I wanna present you my approach for its calculation little bit later). Showing ohw automation saves money/time. Btw, at some point you can show a trend of your ROI instead of current or projecting ROI. For me, seeing dynamics is way better and clean than static numbers. You can have achieved only 90% ROI though if you will show me a steady progress over time, I can value it higher of current 150% ROI which was finally achieved in 5 years. More advanced way to show your dynamics/trend is to lay on 2 graphs on one plot: projected ROI dynamics vs. real ROI dynamics

3. Defecs Discovery Rate (number of found issues vs number of auto tests). I found this metric more representative if you will correlate this metric to the same metric but applied to manual testing. In this case you expose productivity of manual team against manual testing in terms of revealed defects.

In the example below, you can find more project-wise metrics and KPIs as taken from Gredy

68.60%	27	778	99.23%		2011-11-21 # 2.1234.23 - 50.00% 2011-11-22 # 2.1234.23 - 47.62% 2011-11-23 # 2.1234.23 - 46.15% 2011-11-07 # 2.1234.23 - 41.38% 2011-11-18 # 2.1234.23 - 40.00%	2011-11-04 # 2.1234.23 - 11.11% 2011-11-27 # 2.1234.23 - 16.67% 2011-11-09 # 2.1234.23 - 17.86% 2011-11-30 # 2.1234.23 - 20.00% 2011-11-25 # 2.1234.23 - 22.73%	TST_ID_876.004: 16 TST_ID_235.962: 15 TST_ID_511.669: 12 TST_ID_677.207: 12 TST_ID_173.008: 11
Avg. daily success rate	Number of test cases in suite	Number of all test execution	Stable ("Ready") executions vs. All		5 Best builds/days (by success %)	5 Worst builds/days (by success %)	5 most frequent failing tests (total failed times)

The first Nokia Windows Mobile smartphone is here to beat others

2011-10-26T15:47:00.000-07:00

What's up?
Today, London, Stephen Elop, Nokia president, presents a very first Nokia - Microsoft Windows phone. It is really exciting, especially being part of this truly innovative mobile Company. Hands on videos are below - enjoy!
What I really like - facebook seamlessly integration, skype is part of MS ecosys after acquiring first one, MS office and finally - the big hope is this phone gonna stay synched up with Windows 8 as you work on one same machine - if this illusion will come true - Windows across all devices will be very user friendly so ppl will be tapped to one big solution without need to manually synchronize, manage, convert and worry about platforms incompatibilities. Another big market for MS appears to be enterprise which is totally under Windows. + it would be awesome if next version will come with Kinect. Will see if Win Mobile can crack the code.

Fun ad:

SOAP UI is hot

2011-10-21T17:32:00.000-07:00

Hey there, what's up with API nowadays?
They are hot topics wherever you go. No API = no product for now. Well, Soap UI is on a wave of this trendy tech flop. Why? Let's see:

Easy to get into it, small learning curve
Perfectly, thoughtfully built to test SOAP and Restful web services
Supports proprietary and specific protocols
No coding required
If coding is needed - use lightweight non-typed but still OOP Groovy
Functional scripts can be used for load testing
Nice approach for assertions and verification
Through data/variables transferring
Documentation is pretty good

Now how to make happen logging in Gredy after SOAP UI test execution?

Easy! Basically there are 2 ways to enable it but everything starts from nice built-in Teardown script (there is Setup too). So you can write some Groovy in Teardown script on level of test suite or/and test case. This script will automatically kicked off once test suite (case) will be finished (all iterations including iterative, e.g. when you have multiple ones driving data through external data source)

Ok, now two ways to send data to your gredy server:

1. Sending POST request in accordance with given documentation. The data structure is given here:

Array
(
[product] => YOUR_PRODUCT_FULL_NAME
[acronym] => YOUR_PRODUCT_ACRONYM
[owner] => YOUR_PRODUCT_OWNER
[descr] => YOUR_PRODUCT_DESCRIPTION
[ProjectName] => TEST_PROJECT_NAME
[DesignSteps] => TEST_PROJECT_TEST_DESIGN
[Resolution] => TEST_PROJECT_ASSOCIATED_COMMENTS
[ToDOs] => TEST_PROJECT_ASSOCIATED_TODO
[Defects] => TEST_PROJECT_ASSOCIATED_DEFECTS
[SuiteName] => SUITE_BELONGED_THIS_TEST_RUN
[Tier] => TIER_TESTED_ON
[Date] => DATE_OF_RUN_YYYY-MM-DD
[ErrorsCnt] => NUMBER_OF_ERRORS_OCCURED_ON_TEST_PROJECT_RUN
[WarnCnt] => NUMBER_OF_WARNINGS_OCCURED_ON_TEST_PROJECT_RUN
[Path_to_log] => DIRECT_PATH_TO_TEST_RUN_LOG
[Path_to_Report] => DIRECT_PATH_TO_TEST_REPORT_OR_WHATEVER
[ReportErrors] => NUMBER_OF_ERRORS_OCCURED_ON_Path_to_Report
[Version] => TESTED_VERSION_OR_BUILD
[StandName] => MACHINE_ON_TEST_RUN
[Locale] => TESTED_LOCALIZATION
[Language] => TESTED_LANGUAGE
)

2. Sending data via database directly as given below. This script is one-stop script which will do everything for you. The only thing you should care is database connectivity [sql = Sql.newInstance("jdbc:mysql://MySQL_SERVER", "db_user","db_password", "com.mysql.jdbc.Driver");]

Another thing which can be customized is the way how you name test steps (def testname = i + " - " + testRunner.testCase.name + " - " + r.testStep.name;)

Finally, you can send more data to database (see Gredy's data model)

the Groovy script:

import static org.junit.Assert.*;

import groovy.sql.Sql

com.eviware.soapui.support.GroovyUtils.registerJdbcDriver( "com.mysql.jdbc.Driver" );

def i=0;

//def r=testRunner.results;

//r in testRunner.results

for( r in testRunner.results )

{

i++;

def product = testRunner.testCase.testSuite.project.name;

def suite = testRunner.testCase.testSuite.name;

def testname = i + " - " + testRunner.testCase.name + " - " + r.testStep.name;

def err=0;

sql = Sql.newInstance("jdbc:mysql://MySQL_SERVER", "db_user","db_password", "com.mysql.jdbc.Driver");

try{

log.info("Status: " + r.status);

log.info("TestStep Name: " +r.testStep.name);

def rows = sql.firstRow("select * from products where name=${product}");

if (rows==null )

sql.execute("INSERT INTO products (Name) VALUES (${product})");

rows = sql.firstRow("select * from suites where SuiteName=? and product_Id IN (SELECT id FROM products WHERE Name=?)", [suite,product]);

if (rows==null )

sql.execute('INSERT INTO suites (SuiteName, product_Id) VALUES (?, (SELECT id FROM products WHERE Name=?))', [suite,product]);

rows = sql.firstRow("select Name from projects where Name=? and testLevel IN (SELECT Id FROM suites WHERE SuiteName=? and product_id IN (SELECT id FROM products WHERE Name=?))", [testname,suite,product]);

if (rows==null )

sql.execute('INSERT INTO projects (Name, testLevel, Defects) VALUES (?, (SELECT Id FROM suites WHERE SuiteName=? and product_id IN (SELECT id FROM products WHERE Name=?)), "")', [testname,suite,product]);

if (r.status.toString()=="FAILED")

err=1;

else err=0;

sql.execute('INSERT INTO logs (PrjID, Date, ErrorsCnt, tier, Path_to_Log) VALUES ((SELECT Id FROM projects WHERE Name=? and testLevel IN (SELECT Id FROM suites WHERE SuiteName=? and product_id IN (SELECT id FROM products WHERE Name=?))), CURRENT_DATE, ?, ?, ?)', [testname,suite,product,err,context.expand('${#Project#tier}'),"\\\\"+InetAddress.localHost.hostName]);

}

catch(Error e){}

if (sql != null) {

sql.close();

}

log.info("Status: " + r.status);

}

Here are links to get more familiar with advanced features of SOAP UI and Groovy:

Test Automation Market Share - Links!

2011-09-15T22:09:00.000-07:00

Current annual market value of testing and QA is 12b $. And we now that test automation is very trending so I would say most of employers are interested in having QA with test automation background with some specific skills/tools set profficiency. These few links just give a clue how market of automated testing was looking like in the beginning of 200x and for this moment the structure and size of market are very different and fragmented.

See these presentations:

Forester analysis: www.forrester.com/Events/Content/0,5180,-1403,00.ppt

ISQT Step-Auto conference'07: http://isqtinternational.com/defect/images/Bhaskar_Mphasis.pdf

Test automation tools evaluation as of 2002: http://www.vcaa.com/tools/wsipc-automatedtestingtoolevaluation.pdf

Another independent observations: http://www.google.com/url?sa=t&source=web&cd=19&ved=0CGsQFjAIOAo&url=http%3A%2F%2Fconnectedtesting.com%2FPresentations%2FTest%2520Automation%2520-%2520Business%2520v2.ppt&ei=WNZyTvetIunu0gHftKz3CQ&usg=AFQjCNHjLBjUwmd-t7A5FbWn9PmQw41DRQ

Test Automation Management: A Call For Better Tools

2011-06-23T15:05:00.000-07:00

My article is get published in June'11 issue of Automated Software Testing Magazine! That is cool news - check that out: http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=1276:ast-magazine&catid=105:ast-cover-description&Itemid=122

Direct link to download PDF of that magazine issue:
http://www.automatedtestinginstitute.com/home/ASTMagazine/2011/AutomatedSoftwareTestingMagazine_June2011.pdf

Also I wanna put the copy of the content as blog post right over here:

Test Automation Management: A Call For Better Tools
One of the cool things about beginning a test automation effort is that only a minimal level of technical expertise is required – basic skills in testing, some coding and knowledge of key technologies. The shockingly unobvious thing in test automation is the difficulty surrounding management of tests in mid-sized and large automation teams. The major difficulty comes into play when your team tries to analyze the results of automated test that have been run against your daily build and attempts to keep those tests up-to-date and stable for the next day’s build. Moreover, as a release’s delivery date draws closer there is often an increase in scope, instability, expected turnaround and visibility. In order to effectively manage this chaos some mechanism must be employed to facilitate and assist in organizing the team’s work in an efficient way. The output from automated test scripts is typically a low level log and stats that are useful for debugging and internal analysis by automators, but are less useful for higher level status and issue reporting. Not to mention, they can be monotonous and just plain boring! So this alone is not effective for managing the test effort.
To truly understand the need for an effective automated test management solution think about issue tracking, source control, and task management systems. These tools are all about easing our collaboration by making inter- and intra-team communication more efficient. The same standard should be placed on a test automation management system in ways similar to the following:

• Intra-team Business Requirements – Test automation engineers should have a collaborative environment that provides reporting, tasking, and a transparent view of the entire project. Ideally this environment enables clear traceability with other testing artifacts such as defect reports (product and internal automation), test cases and software specifications.

• Inter-team Business Requirements - Project stakeholders should have access to a flexible, transparent and multilevel reporting tool to see what is going on with automation and the tested product. It should offer a high level picture of the product’s evolution and trends, as well as aid in evaluating the maturity and progress of the test automation effort.

The Need for Something New?
There are a crop of available test management systems that exist, but the question that must be answered is “do they meet our needs?” While you could theoretically use many of these tools to some effect, many of them have serious disadvantages in the context of heavy test automation.

The basic features that exist in test management systems include:
- Test case and test suite management
- Test cycle and test release management
- Planning, group tasking and scheduling
- Defect tracking
- Reporting and trends analysis (usually defects trends, execution trends)
- Traceability

Let’s take an honest look at how these features are often used on real projects:
Test case and test suite management – Once created, test cases are rarely kept up to date. Some of them may be used for a subsequent release, some will be junk. Also, most of the tools I’ve seen tend to complicate test case writing – multiple tabs and double clicks to navigate back and forth, page response, etc. I recognize the advantage of fielded data but should it come at the expense of features such as auto formatting, formulas, very simple copying and cloning? Many testers think not, and desire this type of simplicity, evident in the fact that they often create cases in Excel spreadsheets and then import them into the test management system for managers.

Test cycle and release management – Basically, this nice feature is often implemented as a folder that inherits test cases from the test plan. Since the test management tool is often disconnected from the Continuous Integration (CI) server, this feature is not automatically tied to the creation of a new release on the CI server, which creates a disconnect that can make it tedious to keep the CI server and test management tool in sync.

Planning – As a manager, you can assign any test or test suite for execution for a particular day. But who wants to repeatedly perform this boring activity? Typically, testing tasks are based on the functionality to which a tester has been assigned.

Defect tracking – Who knows, maybe some companies use this feature. Very often, however, this feature of a test management solution goes to waste, because the organization uses some separate defect tracking solution or has no defect tracking solution at all.

Reporting – I suppose this is the most important feature but only if other features are used accurately. There is no value in the reporting of incorrect or incomplete data.

Traceability – If you have granular requirements in a single Test or Application Lifecycle Management (ALM) tool, it is nice to be able to obtain automatic linkages between tests and requirements. If specifications live somewhere outside of the tool used for test management, there is minimal value from this feature because the linking process and the process of keeping the links up to date will be a manual task.
So, my conclusion is that existing test management tools are useful when their whole set of functionality is used; particularly when used on mid-to-large scaled projects, on distributed and/or remote teams and for cross project analysis. Otherwise, they may be more trouble than they are worth. So we must ask ourselves the following question: “If these tools are flawed for manual test case management, why we should we rely on them from a test automation perspective?”
What is expected?
From my experience I have drawn a few specific requirements for test automation management tools and I would like to share them here.

Useful Reporting – Reporting should be meaningful and should facilitate the continual learning of lessons over time. This implies the presence of trend analysis over historical data, calculating key performance indicators (KPIs), root cause analysis, and graphical data representation.

Compliance with development practices – A test management solution that seriously works for test automation should comply with basic software development practices. For example, offering access to standardized, xUnit formatted logs through a web services interface.

Traceability – It’s important to be able to trace other testing artifacts to the test automation process, thus making it integral within the project. It is very useful to have all relevant information or references in one place: i.e. references to defects that are covered by test, reference to test design, data and specifications, and references to the CI server (i.e. link to the build number).

Making test debugging less painful – Helping test developers organize their debugging work is often an essential task. Sometimes finding the cause of a test failure and applying fixes requires a lengthy effort, an effort that should be eased by the information provided by a test management tool.

Compliance with Agile lifecycle – The ability to obtain quick daily status information is important for getting an effective start on a new Scrum day. Just enough information to recognize changes is often all that’s necessary. The changes are usually marked by differences in errors found in the test logs between day A and day A+1. For example, the difference in the number of failures (log errors) signals probable changes either in the AUT or in test code (broken test code base, instabilities) or at times in both.

Interfaces for triggered and manual test execution – Ideally test automation possesses a facility to run a bunch of tests remotely against several machine instances. This includes hardware management (physical and virtual), tasks queue management and tasks prioritization to rule the order of execution if concurrent use of resources takes place. The interface should not be limited to UI elements, but should provide a simple API that may be used by a CI server to schedule and manage tasks. An example of a good practice is when the CI server triggers test execution through that API upon the deployment of a new build. The API might be a simple REST service in a POST or GET request implementation. The most challenging part of this component is the queuing mechanism: prioritization, watching, management and coordination.

Tool agnostic interface – All capabilities should be agnostic to the utilized test automation tool. You can use multiple tools or migrate from one to another but your test automation management is done via a single unified interface. Interfaces with CI should be decoupled in order to get rid of preferences. It is the responsibility of the CI server to schedule tasks by means of calling the API, while test automation can return reporting in an xUnit format.

Transparency of deliverables and internal team collaboration – The idea is to make the tool clear and lightweight for any stakeholder. It should be flexible enough to provide information with an appropriate level of detail.

Keep it simple – The look and feel of the end user interfaces as well as automatic interfaces (i.e. results submission interface, integration with CI severs) should be intuitive enough to facilitate a quick startup.

Reducing test automation costs – Ultimately the test automation management tool should facilitate making test automation profitable, reactive and valuable.

Current state of the art

There are a few tools providing test automation management capabilities at some extent of described expected functionality. This article is not about comparing them. Here are the tools I’m aware about:

Quality Center – does not segregate test automation from manual. Though there is scheduling capability through Web UI and through API, which is actually complicated and unstable COM object. HP QC has straightforward interfaces with QTP only. Yes you can hack it to run something else but why then buy the tool?

QAComplete – ALM product similar to HP QC but looks like more flexible in terms of compatibility with automation tools (e.g. they provide connector for QTP test runner). In same way as QC, the tool mixes ups manual and automated effort and does not provide specific tools to debug and to have collaborative environment specific for test automation engineers.

Gredy – lightweight Web tool that explicitly serves as test automation management system and does not have preferences to automation tools. Has advanced capabilities to manage and analyze test execution results, assists in debugging, internal automation team collaboration and provides flexible reporting.

Tapper - open source solution from AMD written on Ruby. It has ready to go interfaces for test scheduling and viewing test execution reports.

Bromine – provides capability to manage test cases and test lab. The tool is tightly tied to Selenium automation tool. Tests can be run on remote machines with installed Selenium IDE/RC.

Litmus – open source from Mozilla and currently used by Mozilla to run and manage test automation suites. Has nice look and feel and thoughtful approach to manage tests and analyze test execution results.

Presentation from Meetup: Why Test automation guys should have hacker mindset?!

2011-05-31T16:25:00.000-07:00

The slides from meetup "Why Test automation guys should have hacker mindset?!" are now available for everyone:

Why software bugs are not public?

2011-05-13T11:48:00.000-07:00

Weird question, isn’t? I think the situation is horrible because we as Users mostly have no way to share, keep track and eventually have influence on the software or services they produce. When you or your friend submits a support request, submits an issues – it just goes out for internal company’s judgment. It is totally up to them and their business decision whether to fix that problem one day or talk to you back “We apologize for inconvenience…”. We as user have no idea and tool how to drive our expectations.

Some companies integrate on their web sites user’s voice plug-ins that allows sharing some ideas how to improve something. Some remarkable companies and their products have public bug tracking – most of them running open source projects (Linux distributives, Mozilla, some of Google’s projects, MySQL, Java, Apache, PHP). Looks like generally these are geeky-centric software.

What about Consumer and proprietary software and services? I don’t wanna to mention companies, but the majority of their major projects are not providing public bug and change request tracker for end Users.

Yes you as User can post request but they will decide what, where and how to fix that and all this happens because your voice is not visible for others, the whole Community of Users don’t share the issue in organized single place. Though, we still may find some information here and there in internet as gathering grain of sand, it is not enough to be influential. To have an influence as Users of software, all we need is pretty organized, social and easily accessible place for all our findings and wishes. Btw, BugPub is such a place.

Testability influence on software design - super new concept from Google or old habitual TDD?

2011-01-19T21:18:00.000-08:00

Hey,

just found out these slides from GTAC '10 that generally represent how flexible s/w design impact on testability. Then authors draw conclusions that design should be impacted (or even driven) by testability - see latest slides. In this presentation you will find good examples, explanations and correlation with OO practices.
Now I'm thinking - nothing new - it is all about Test driven development (TDD). So you write and after that you have a hint on how to design s/w code itself as you will support that test further. If so, there are less chances to make mistakes "not see a tree for the forest" kind of. I'd like to experience such a care about testability by evolving in code design analysis

Hey mid and grand SW companies, look inside Mozila's testing business. Like It!

2011-01-19T20:34:00.000-08:00

Steve Balmer exclaimed "Devs, Devs, Devs..." (hopefully now it is not turned to 'Sales, Sales, Sales'). Mozila's testing sounds like Tools, Crowd, Community.

I like their Litmus for testing management and as well as lots of free add-ons. Honestly talking about browser, my pref have shifted to Chrome but I'm still with FF.

Essentials for Test Automation - survey results here

2010-12-03T08:56:00.000-08:00

AT4QA is the winner as best blog of this year by ATI and other's awards

2010-11-25T06:17:00.000-08:00

Hey the test automation world!

The blog At4QA is recognized as the best one in this year by Automated Testing Institute.
I’m proud and excited to be a part of this honor and being standing in a row with rocks of the industry – tools and individuals ( I suggest to look through the whole magazine issue)
Thank you all a lot, we will Keep this way!
Here is a downloadable version of the magazine issue

If you can't download - see how it looks right here below

Survey: Essentials for Test Automation

2010-09-29T07:48:00.000-07:00

View Survey

Fortunate Test Automation

2010-09-05T02:32:00.000-07:00

Just imagine you are going to sell your test automation project to Project/Test/QA management in a way as sales managers/engineers present, propose and account a solution. Under solution we can mean outsourcing project development, COTS, open-source assuming paid hosting, integration and support or whatever else. Right automated testing is product itself too. As told James Bach "Useful test automation is a major software project". I’m not trying to escape from original topic 

Really how you will sell that unclear automation? You need to prepare the best proposal, solution, surveys, fact sheets, comparisons and of course happy case studies. Once your presentation is polished out - you are ready to present it to your customer, stakeholders or, in general, decision-makers.

In fact, people like and want real cases, they want to hear that you or whoever else did the same for some of "Fortune 100" companies. Moreover they want to know how, why and for how much they had got that. Your potential Clients want to know risks, issues, lessons learnt and benefits of your proposal from other guys who had got the experience. They would like to ask them but usually cannot. And my target as seller is to forestall them, I'm trying to show them case studies.

Case studies could be personalized or anonymized. The last one appears as pattern (proper or anti) and still serves as message reflecting a real-life situations and experience. Someone may say it's tricky to show anonymous reviews/surveys. I'd rather agree in most cases. But what if your market is so limited and closed for sharing information? Answer - nothing but tricky hinting. What if your study covers a huge group of customers/users? In this case you can extract some representative groups with the same obtained result/capabilities/behavior/model or generally segment a target audience groups. I apologize for my passion, but statistical analysis is what I like as well as test automation.

The fortunate case studies from me are cheerfully given but please consider them as patterns of proper or happy software automated testing.

Case #1 Various environments support, compatibility with other software, components...
Test automation framework is designed as platform-independent. So that test engineers easily run test sets against various environments and combinations of installed software in order to test compatibility and fault tolerance under different real-life usage scenarios. For Web systems it could be running on various browsers and versions, with different combinations of installed plug-ins and add-ons on different set of OSs

Case #2 Multiple Localization testing
Automated tests are not tied to GUI visible captions, it means recognition is implemented by checking invariant attributes (ID, location binding in hierarchy, appearance index...). However localization validation is not nightmare. For that, localization database to be designed which feeds data for checking localization attributes in GUI. Basically you can run a verification of GUI either at runtime or as batch-state-checkpoint. The last one is better as it brings a separate level of testing called GUI localization testing. Other words, whole automation scope is split out on functional testing (the same code without parameterization will test product behavior for all locales) and GUI testing (will just verify appearance using dynamic parameterization from localization DB).

Case #3 Unstable GUI but stable API
Some companies made and forced a decision of testing by pulling API. It means end-User UI is not touched upon testing. The drawback is that the approach will not cover real user interface but test coverage and stability will be incomparable to functional testing through GUI. Yet other advantages are less expensive overall automation, quicker execution and integration with codebase of AUT.
However project decided to have GUI testing anyway. For that, test automation guys came up with GUI testing as additional verification layer which was inlined to API tests. It's really simple and clever - call or change something in application by API, then verify result again with API and additionally and independently in GUI.

Case #4 Business needs to support software As-Is-It-Now for a few years but fixing bugs will be applied by patches, CFs and so forth
This case is a usual for COTS and enterprise systems. Support requires testing and main effort is regression. Business should make sure that fix does not affect existing functionality. This is very risky to ship critical fix without a round of regression testing. Here is automation can come as cheap or almost free solution (someone just have to run and review results)

Case #5 System does not evaluate significantly between releases. New release is just old one + some fixes + additional features
It's again nice point to leave regression for test automation. The rest effort is spending on new features and bugs. From this point automation becomes very effective as returning investements on automation is continious process. It's like planting - initially you just spend after a while you gather in the crops but a little effort on support still have place.

Case #6 Often commits, often builds
Making test automation as part of continuous integration in aggressive project environment for providing ASAP and often feedback about build quality. Some tests could be run as sanity checking against each code pre and/or post commit. The rest suite is run upon each new build arrival

Case #7 Mission critical project shipping
In this case stakeholders may agree to have automated tests as part of delivery package. For instance a new build is not accepted if test automation does not yet ready as planned for a build/release and if whole test suite does not result a Green (passed) result. This is really challengeable. Just imagine test automation is a inseparable part from a product. I like this approach especially for self-tests and embedded automated tests. Take a look at the example

Case #8 Our system is a legacy back-end data processing and some unstable (under continuous development) to changes UI. Almost no money on test automation
under this case goes well a claim "We just need to support that unknown legacy system with small money". Well, if small effort, consider automation of system logs parsing, checking datatbase consistency, reliability and data integrity by running queries periodically, automatic memory dumps processing (learn in depth), build in a memory leaks checkers, continous health monitoring of environment

Case #9 Test automation is looks like chaotic but it's smart distributed one indeed
Imagine you have a cloud of various environment and you have a smart dispatcher which decides which tests, where and how to run based on its acknowledges about the cloud. This is amazing abstraction that presents accelerated test running relying on wide spread test stands and acknowledged by stands load/usage. System completely controls execution, engineer just to check results out in real-time. IMO it's state of art!

Case #10 Integration with 3-rd party systems
Test automation daemon may sit as listener somewhere in integration point. Automated test should understand native communication language, so listening traffic and verifying result is just a deal. Let say we can create Web services testing (SOAP/WSDL) which utilizes the same data schema as it incorporated in web service itself. Test can proxy, interpret, stub or listen data.

Case #11 Security is a priority number one on Agile (shipping often) project
It’s not a secret security testing can be perfectly modeled and automated. The core of security tests will work against most of web projects (XSS, SQL injections, session hijacking, tampering, sniffing and so forth). If a project runs short iterations and releasing frequency is high, security tests could be run over and over again against pages and contained. For instance test engine walks through all web pages and then feeds basic dangerous scripts (JS, SQL, VBS, sh) to all fields, then triggering form submission. You can even simulate DOS attacks and brute force password hacking. Although there are already many tools which will do it for you, thus our job is just to automate execution and reporting.

Is it enough? It does not but I need to stop this story in a fear the story will never finish. Actually any project can produce different cases and practices since inviting a best fit to project solution demands creativity and out of the ordinary thinking. Patterns themselves don't require using them as is, they are rather reusable good practices which could be ported and adopted if it make sense in your project. Using combination of patterns may bear new patterns or a single pattern of level higher than reused ones.

It would be very nice if we will share specific cases, patterns and lessons learnt of fortunate test automation. I just doubt why bloggers running test automation exchange their successes and achievements too rarely for the industry; instead we can found risks, problems, obstacles and why-dont-run-low-ROI-automation mostly in the internet.

TestNG unit testing framework for Java: annotations, listeners, interceptors

2010-09-04T02:27:00.000-07:00

I'm running a short research devoted to synthesize various techniques, framework and practices in automated testing including Unit testing. Why am I doing that? Don't know, and I'm not going to explain my intentions as can't find certain ones myself. Some frameworks presented in this blog appear as just highlight, some provide detailed overview.

TestNG is the next one in a series of Unit testing FW overviews. Here are a documentation http://testng.org/doc/documentation-main.html#testng-xml and JavaDoc http://testng.org/javadocs/index.html

As not a real user of that framework, in fact I can't run and report in-depth analysis. IMO the core axis and feature of TestNG is wide use of annotations. The whole list of annotations and attributes in TestNG is here. The set of @BeforeXXX and @AfterXXX annotations serve as inherited implementation of setup() / teardown() in xUnit. testing.xml specifies how test suites are built, how execution flow is supposed to be and like sequencing, shuffling.

Authors took care about compatibility with Junit Reporter XML schema, though they claim that their own reporter contains more specific information so that TestNG's native reporting schema is slightly different. But compatibility with jUnit reporter looks weak as developer should care about reporting himself using org.testing.Reporter

Advanced things like annotations customization, parallelism, listeners, interceptors are available at abovementioned referenced API and documentation.

See also:
- MS's Framework overview
- Google's Framework overview

Blog Day is Today - let's open up for guest blogging

2010-08-31T11:48:00.000-07:00

In accordance with BlogDay.org

one long moment on August 31st, bloggers from all over the world will post recommendations of 5 new Blogs, preferably Blogs that are different from their own culture, point of view and attitude. On this day, blog readers will find themselves leaping around and discovering new, unknown Blogs, celebrating the discovery of new people and new bloggers.

I would like to expand this by the proposal - Let's open our blogs for guest blogging (sure content should fit with blog theme and pass validation by original author(s)). The back links will track back to guest author's blog. Thus we will expand our net. What do you think? My blog is really open for your greate content in a field (you know - test automation, QA, QC and technologies). At the same time I have some interesting articles to be published as guest author. Let's follow up!

Meanwhile, I follow up the Blog Day goal - 5 blog post:

Prepare to interview with Geekinterview.com and Glassdoor.com

2010-08-31T02:10:00.000-07:00

The http://www.geekinterview.com/Interview-Questions goes certainly to my list of Favorites.

It really contains huge database of interesting and original questions which everyone may be questioned upon IT interview. I was surprised - there are a lot of shared real-life questions on Testing (see there is break sown by different sections including automation). The direct links on test automation section, testing tools, separately for QTP and TestComplete users - VBS questions, JS questions

You guys as creator of this resource - are rocks!

Want to get hired by a particular Company? Be prepared by crawling questions from real interviews on Glassdoor.com (inside look at Salaries, Reviews and Interviews provided by real Users). This is a direct link to Interview Tab

If you are lucky one and passed along several tech interviews at Your-Dream-Company, you are rolling over package negotiation with the employer Glassdoor.com again comes to play a role of naviagtor in that tricky moment "Neither play over nor loose offer".

Other useful resources on this topic? All are Welcomed to suggest!

Links: Unit Test Patterns and Framework overview by Marc Clifton

2010-08-30T02:02:00.000-07:00

I don't use "Advanced Unit Test" but it looks pretty and beateful solution.

First, read a view of Unit testing patterns. I found it interesting
http://www.codeproject.com/KB/architecture/autp5.aspx

This is really interesting vision on unit patterns? Running automation over UI and aka grey-box automatoin is perfectly modeled and presented in terms of MVC pattern (Model-View-Controller) -shortcut within the page
The rest content (framework itself, shortcuts:

The power of right motivation by RSA.org - awesome animate!

2010-08-25T08:31:00.000-07:00

Must to see!

Other videos of thir public researchs can be found here (http://comment.rsablogs.org.uk/videos/)

Working with objects in JS

2010-08-24T11:41:00.000-07:00

The starting point to get confidence over JavaScript OO (object-oriented approach) full support take a look at this post (http://nefariousdesigns.co.uk/archive/2006/05/object-oriented-javascript/). Emulation? Right - these are not C++ or Java techniques with deidcated keywords (protected, private, abstract, extends, template...) and operators (::, ->, : <>...). But - it (JS) works and fits pretty well to develop GUI, test automation harness, tools and utilities. Keep it simple, right?

Hovewer the primary intention of the post is to give Ready-to-Go utilities to work with JS's objects as data structures for TestComplete (JS) users. The excerpt is here:

 //USEUNIT TypeTraits  
 function Clone ( obj )  
 {  
      if (!TypeTraits.IsReference ( obj ))  
      {  
           return obj;  
      }  
      var newObj = new Object ( );  
      for ( var key in obj )  
      {  
           newObj [ key ] = Clone ( obj [ key ] );  
      }  
      return newObj;  
 }  
 function toString( obj )  
 {  
      var _collectionToString = function( obj, indent ){  
           var newIndent = indent || "";  
           newIndent += "  ";  
           var pairs = [];  
           for (var key in obj)  
           {                 
                if ( obj[key] === null )  
       {  
         var value = "null";  
       }  
       else if ( typeof obj[key] == "object" )  
       {  
                     var value = _collectionToString( obj[key], newIndent )  
       }    
                else  
       {  
                     var value = obj[key];  
       }  
                pairs.push( "\n"+newIndent+key+" : "+value );  
           }  
           return ( "{"+pairs+"\n"+indent+"}" ).replace( /\{[\n\s]+\}/mg,"{}"); //remove all spaces between empty {}  
      }  
      return _collectionToString( obj, "" )  
 }  
 function merge( /*Object*/ base, /*Object*/ another )  
 {  
      /*  
           PURPOSE:  
                Merge base object with another object  
           RETURNS  
                Merged object.   
                Fields of "base" replaced with those of "another"  
      */  
      //Clone base into resulting object to avoid damage to base object       
      var result = new Object();  
      if ( base )  
      {  
           for ( property in base )  
           {  
                if ( base.hasOwnProperty(property) )  
                     result[property] = base[property];  
           }  
      }  
   // merge  
      if ( another )  
      {  
           for ( property in another )  
           {  
                if ( another.hasOwnProperty(property) && ( another[property] !== null) )  
                     result[property] = another[property];  
           }  
      }  
      return result;  
 }

Control recognition problem

2010-08-23T05:12:00.000-07:00

TestComplete has lots of built-in features to operate with properties and methods of tested object. As was mentioned in a previous section, all wrappers over controls of AUT are implemented in components. This part of codebase is much often subject of changes from release to release and usually those changes are unpredictable. One day, our team was not ready to run in-time full regression tests against fresh build because of recent changes in DOM tree for many objects. We were need to invite something robust, not silent and not verbose; this mechanism should report something into logs when a control was not found directly but it should attempt to recognize that control using set of provided properties. We developed robust pattern that posses obvious advantages:

Easy code maintenance: to call an object located on a page, just write something like this:

var conext = Common.GetAppTop();

var res = recognizeControl ( w.Panel(0).Panel(0).Panel(0).TextNode(0).Link(0), ["ObjectType","innerHTML"], ["Link","Log in"], context);

Robust change management: if object direct path was changed on a page, test log will report an error but the control will be found indirectly either within a particular html element (e.g. cell, row) or within whole web page.
Automatic checking of correspondence between properties values of direct object path and specified object properties for indirect search. In this way, we achieve discipline of test development process and may reveal hidden mismatches. For instance, this code can reveal that “Log In” link has changed innerText property to “Log Innn” but test will continue work with this control because direct object path is correct.

In conclusion, the proposed approach of object recognition provides checking relevant properties and expected object location simultaneously. The code snippet of the proposed pattern is presented below.

Figure. Test automation framework

Code snippet of pattern for object recognition and verification on the fly in JavaScript:

function recognizeControl( directPath, _propCollection, _propValues, topParentPath )

{

try

{

var res;

if ( typeof ( _propCollection ) == 'string' )

{

if ( Init.GetCurBrowser() == "firefox" )

_propCollection = _propCollection.replace( /innerText/g, "innerHTML" );

propCollection = _propCollection;

}

else

propCollection = Arrays.ConvertArray ( _propCollection );

if ( typeof ( _propValues ) == 'string' )

propValues = _propValues;

else

propValues = Arrays.ConvertArray( _propValues );

if ( TypeTraits.IsFit( directPath ) && directPath.Exists )

{

if ( TypeTraits.IsFit( _propCollection ) && TypeTraits.IsFit( _propValues ))

PropMatch ( directPath, _propCollection, _propValues );

return directPath; //specified direct link is returned

}

var res;

if ( TypeTraits.IsFit( topParentPath )&& topParentPath.Exists )

res = topParentPath.FindChild( propCollection, propValues, 200 );

if ( TypeTraits.IsFit( res )&& res.Exists )

return res; //the object was found located within the Parent object

else {

if ( TypeTraits.IsFit( topParentPath ) && topParentPath.Exists )

Log.Warning( 'Can`t find child with property name [ ' + propCollection.toString() + ' ]; and property value [ ' + _propValues.toString() + ' ]; on parent object;',' Parent:\r\n' + topParentPath.FullName + '\r\n Page:\r\n' + GetPage().URL, 300, f );

res = GetPage().FindChild ( propCollection, propValues, 200 );

if ( TypeTraits.IsFit( res ) && res.Exists )

return res; //the object was found located within the Page

else

Log.Error( 'Cant find child with property name [ ' + _propCollection.toString() + ' ]; and property value [ ' + _propValues.toString() + ' ]'+'');

}

return false; //default returning if nothing was found

}

catch ( ex ) {

Log.Error ( 'Exception occurs in Splitter.recognizeControl', ex.description, 300, f );

Init.exceptionHandler( arguments );

return false;

}

// PropMatch validates if object having direct path has predefined properties as alternatives;

function PropMatch ( directPath, _propCollection, _propValues ){

var str;

try {

if (( typeof _propCollection == 'string' ) || ( typeof _propCollection == 'number' ))

_propCollection = new Array( _propCollection );

if (( typeof _propValues == 'string' ) || ( typeof _propValues == 'number' ))

_propValues = new Array( _propValues );

for ( var i = 0; i < _propCollection.length; i++ ){

var iProp = _propCollection[i].toString();

var iProp = eval ( directPath.FullName + "." + iProp + ";" );

if ( typeof iProp != 'undefined' ){

var hasText; // Boolean flag for text existing

if ( _propValues[i].indexOf ( '*' ) == -1 ){

hasText = ( iProp == _propValues[i]);

}

else

hasText = true;

if ( !hasText )

Log.Warning ("The direct path to object [" + directPath.FullName + "] has property [" + _propCollection[i] + "] value [" + iProp + "] but expected value is [" + _propValues[i] + "]");

}

else

Log.Warning ( "Object [" + directPath.FullName + "] has no property [" + _propCollection[i] + "]");

}

catch ( ex ){

Log.Error( 'Exception occurs in Splitter.PropMatch', "The direct path to object [" + directPath.FullName+"] has property [" + _propCollection.toString() + "] value [" + iProp + "] but expected value is [" + _propValues.toString() + "]", 300, f );

return false;

}

Automated testing of JS apps - embedded overview

2010-08-18T13:12:00.000-07:00

I found tasty presentation on JS (Java Script) code test automation (Unit) in LinkedIn. Well, as fan of JS, cant's skip it to be shared. Enjoy this overview with frameworks comparision and various code snippets!

Why We use “FAIL” for many reasons?

2010-08-17T01:57:00.000-07:00

Why We use “FAIL” for many reasons?

Really why we are so restricted by test automation tools such as Mercury QTP and AutomatedQA TestComplete and other popular ones give us just 2 options – either report Pass or report Fail. If a test log contains at least one Fail – the whole run is getting Fail. This is OK. However I definitely want to have different statuses for checkers belonging to test design and other things occurring from time to time in automation. I propose the following breakdown:

Failed = on unmatched Assertions, comparisons and checkpoints. If test automation is checking GUI consistency, explicit GUI checkpoints should be designed

Error = on any runtime errors, exceptions, unexpected failures, compilation errors, missed objects, improper interfaces/methods/properties applied to objects. Generally – all fails out of checkpoint and assertions

Unfortunate thing is that tools do not provide such statuses. Basically we have only Fail and Warning (but the last one does not affect overall status). The problem is rising when your team builds huge test automation suites and this team is responsible for test execution analysis. And not only for analysis and reporting to customers and management but that team should do it timely and efficiently with minimal effort. It means also that product issues should be reported ASAP and first to provide whole real picture on build quality as early as possible. The second thing is clearing up testing code, fix infrastructure problems, and discover reasons of unexpected failures (all that is what I indicated as Error). Well, actually the optimal work path under this pressure is essential. But how I can differentiate where is Fail in order to process and analyze first and where is that unexpected Error (the second matter), if tools vendors gave me generic Fail only??? I must to drill down to all failed test instances one by one.

Another not good practice is to continue test execution on any failure. I rather prefer to stop and cleanup execution at any failure (either Failed or Error)

Here we go and the time to workaround is come.

We will create custom status in high-level reporting.

Turn off logging at all at the very testing beginning:

QTP: Reporter.Filter = rfDisableAll

TestComplete: Log.Enabled = False

then we will need to write 2 wrapping functions which will report Failed and Warning

//Call this function on any test checkpoint/assertions failed

function LogFail (message, priority, options) {

Log.Enabled = True // QTP: Reporter.Filter = rfEnableAll

//Log Failure code

Log.Enabled = False // QTP: Reporter.Filter = rfDisableAll

CleanUp (True, "Fail")

}

//Call this function on any unexpected error, exception…

function LogError (message, priority, options) {

Log.Enabled = True // QTP: Reporter.Filter = rfEnableAll

//Log Failure code

Log.Enabled = False // QTP: Reporter.Filter = rfDisableAll

CleanUp (True, "Error")

}

After that we will learn high-level reporting (aka Dashboard) to understand and properly display if a test log contains at least one Warning – show up Error. Otherwise – overall run status should be matched with native status decided by test tool, i.e. Pass=Pass, Fail=Fail

In TestComplete/QTP test run should message to Dashboard (TestRunStatus variable) DB in such a way within Cleanup () method:

Function CleanUp (doTerminate, ErrLevel) { //doTerminate True/False

//Some operations to free execution

//…

If (ErrLevel = “Fail”) TestRunStatus = Fail

Else If (ErrLevel = “Error”) TestRunStatus = Error

//Calling termination whole test execution

If (doTerminate) TerminateExecution()

}

So what we’ve got? Localization project issues (including defects) is getting quicker, as Fail now means failure in functionality or behavior of tested application (AUT) – wrong result which is also known as deviation from Expected result; opposite status Error means something unexpected happened (test code problem, infrastructure, changes in AUT interfaces (e.g. changed Button caption results in impossibility to recognize the Button on screen).

Moving forward, someone can come up that Error and/or Fail could be split up on a few more. E.g. Error set can be considered as Exception | Environmental Error | 3^rd Party exception | Object missed | Object improper method/property | Execution crash. We can think out any custom execution statuses but not sure if that really needed.

The proposed solution is good way to avoid mix up of different failures. Another approach I’d like to recommend (especially for those who use TestComplete) is to employ priority of logged messages (in our case Fail). TestComplete allows specifying priority of posted message to log. This gives opportunity to make more flexible model of failures reporting. Somehow it implies to make your framework remember maximum number of priority of posted Fail to the log. Finally maximum priority value will be Fail priority for test run overall. Then you can design Dashboard with usable coloring to represent priority and to give direction to analysis order from huge scope. In the example below, Fail (500) could be used as Fail in application test itself (assert/checkpoint fail), Fail (100) – unexpected exception (e.g. database connection exception), Fail (300) – Browser hang (who knows, maybe our tested application is too leaking and caused to browser hang)

Customizable random strings generator in JavaScript (TestComplete)

2010-08-12T13:28:00.000-07:00

What this guy can give you? Answer: the class allowing to generate a custom (predefined by pattern or type) string. The code is fully compatible with TestComplete - just add it as library to your Project. Another application - to use as routine code for JS scripts in web. In this case you have to hack some line, like logging, other TestComplete's function calls.

 //USEUNIT TypeTraits   
  var _String;  
 function RandomStringFactory ( minLen, maxLen, type, pattern )  
 {  
  new GeneratedStringObject ( minLen, maxLen, type, pattern );  
  return _String;   
 }  
 function GeneratedStringObject ( minLen, maxLen, type, pattern )   
 {  
  this.minLen      = minLen;  
  this.maxLen      = maxLen;  
  this.type           = type;  
  this.pattern      = pattern;  
  _String = GenerateStringWrapper ( this );  
 }  
 function GenerateStringWrapper ( obj )   
 {                    
  return GenerateString ( obj.minLen, obj.maxLen, obj.type, obj. pattern );  
 }                                      
 function GenerateString ( minLen, maxLen, type, pattern ) //minLen, maxLen, type, pattern  
 {                             
      //var minLen=11, maxLen=1150, type="free", pattern="dddcccsss$dddsssccc";   
       var resString = "";  
       var len;  
  if ( ValidateParameters ( minLen, maxLen, type ))  
  {   
   //Randomize();  
   len = minLen+Math.floor ( Math.random ( ) * ( maxLen - minLen ));  
   switch ( type )   
       {  
       case ("alpha")          :     resString = ProduceAlphaStr ( len );  
            break;  
       case ("num")               : resString = ProduceNumStr ( len );   
            break;   
       case ("alphanum")     : resString = ProduceAlphaNumStr ( len );   
            break;   
       case ("email")          : resString = ProduceEmailStr ( len );   
            break;  
    case ("validEmail"): resString = ProduceValidEmailStr ( len );   
            break;   
       case ("free")               : resString = ProduceFreeString ( len );   
            break;  
       case ("custom"):   
            if ( pattern != "" && pattern != null )  
                 resString = ProduceCustomString ( pattern );  
            else   
                     Log.Error ( "No pattern for randomly generated field with type Custom string" )   
            break;  
       default: resString = ProduceFreeString ( );  
            break;   
       }  
  }                        
  return resString;  
 }  
 function ValidateParameters ( minLen, maxLen, type )  
 {  
      if ( !isNaN ( minLen ) && !isNaN ( maxLen ) && minLen <= maxLen )  
       if ( TypeTraits.IsFit ( type ))   
        return true;  
   else   
           return false;  
 }  
 function ProduceCustomString(pattern)  
 {  
  var res                              = "";  
  var AlphaSet                    = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";  
  var NumSet                         = "0123456789";  
  var SpecialCharSet     = "!#$%&')(*+,-./:;<=>?@[\]^_`{|}~";  
  //Randomize();  
  for ( var i = 0; i < pattern.length; i++)   
      {   
       var curChar=pattern.charAt (i);  
            switch (curChar)   
       {  
       case ("c") : ch = AlphaSet.charAt (Math.floor ( Math.random () * ( AlphaSet.length )));  
            break;  
       case ("d") : ch = NumSet.charAt ( Math.floor ( Math.random () * ( NumSet.length )));    
                 break;   
       case ("s") : ch = SpecialCharSet.charAt ( Math.floor ( Math.random () * ( SpecialCharSet.length )));   
            break;  
       default: ch = curChar;   //any const char, e.g. @ - apply as is  
       }  
       res += ch;  
  }  
  return res;  
 }  
 function ProduceAlphaStr( len )   
 {  
      var pattern = "";  
      for ( var i = 0; i < len; i++ )  
      pattern += "c"   
      return ProduceCustomString ( pattern );  
 }  
 function ProduceNumStr ( len )   
 {  
      var pattern = "";  
      for ( var i = 0; i < len; i++ )  
      pattern += "d"   
      return ProduceCustomString ( pattern );  
 }  
 function ProduceAlphaNumStr ( len )   
 {  
      //Randomize();  
      var pattern="";  
      var fl;  
      for ( var i = 0; i < len; i++ )  
       {   
           fl = Math.floor ( Math.random () * (2));  
           var ch = (fl == 0) ? "d":(( fl==1 ) ? "c" : "s");   
           pattern += ch;   
      }  
       return ProduceCustomString ( pattern );  
 }  
 function ProduceFreeString ( len )  
 {                   
      //Randomize();  
      var pattern = "";  
      var fl;  
      for ( var i = 0; i < len; i++ )  
      {  
           fl = Math.floor ( Math.random () * (3));  
           var ch = (fl==0) ? "d":((fl==1) ? "c" : "s");   
           pattern+=ch;   
      }  
      return ProduceCustomString ( pattern );  
 }  
 function ProduceEmailStr ( len )   
 {   
      //Randomize();  
      var pattern = "";  
      var fl;  
      var ret = true;  
      var ch = "";  
      for ( var i = 0; i < len; i++ )  
      {                 
           var remainedCh = len - i;  
           if ( remainedCh > 9 )  
           {  
                fl = Math.floor ( Math.random () * (2));  
                ch = (fl == 0) ? "d":((fl == 1) ? "c" : "s");   
           }  
           else   
           {  
                if ( ret == true )   
                {  
                     ch = "@";   
                     ret = false;  
        }  
                else   
                {  
                 if ( remainedCh == 4 )   
                          ch= ".";  
                 else   
                          ch= "c";    
                }   
        }  
           pattern += ch;   
      }  
      return ProduceCustomString ( pattern );  
 }   
 function ProduceValidEmailStr ( len )   
 {   
      //Randomize();  
      var pattern = "";  
      var fl;  
      var ret = true;  
      var ch = "";  
      for ( var i = 0; i < len; i++ )  
      {                 
           var remainedCh = len - i;  
           if ( remainedCh > 9 )  
           {  
                fl = Math.floor ( Math.random () * (2));  
                ch = (fl == 1) ? "d":"c";   
           }  
           else   
           {  
                if ( ret == true )   
                {  
                     ch = "@";   
                     ret = false;  
        }  
                else   
                {  
                 if ( remainedCh == 4 )   
                          ch= ".";  
                 else   
                          ch= "c";    
                }   
        }  
           pattern += ch;   
      }  
      return ProduceCustomString ( pattern );  
 }  
 function GetRandomElementFrom( /*Array*/ itemSet )  
 {  
      /*  
      PURPOSE  
           Select and return random element from given array of items  
      ARGUMENTS  
           itemSet: array of elements of any type;  
      RETURNS  
           random element form given set  
      */  
      if ( !itemSet || (itemSet.length == 0 )){  
           return null  
      }  
      else {  
           return itemSet[ Math.floor( Math.random() * itemSet.length ) ]  
      }  
 }  
 function test ()   
 {  
      var w;  
      for ( i = 0; i < 20; i++ )  
      {  
      w = GenerateString (12, 20, 'validEmail', null );  
      w = w.toLowerCase ();  
      Log.Message (w);  
      }  
 }

Technically efficeint and clearly explained Framework over Selenium

2010-08-06T06:41:00.000-07:00

Guys did their best to highlight clearly in detail the approach and organization of framework over Selenium IDE. Actually I worked with 3 frameworks relying on XML. The difference is only that their framework connect data with UI elements explicitly in XML. My frameworks usually bind data and UI in scripting code. So that test data, steps (test case instructions) and UI map are loosely coupled.
Also their framework could be improved with advantages of applying XML schema and other XML techniques (http://at4qa.blogspot.com/2010/01/tools-xml-orm-for-efficient.html)

GTAC_SeleniumFramework

JSON generic converters

2010-08-05T03:58:00.000-07:00

For those who AJAXing and work closely with JS this intercahnge format is essential. For those who things it's tricky - check out first the origianl site devoted to JSON - json.org and then - wikipedia page (http://en.wikipedia.org/wiki/JSON) to realize how it's easy. Built-in JS's methods stringify and parse are basic for processing JSON back/forth strings

The advanced scenarios are smart Stringifiers and converters forth/back XML. The last one is especially applied task for making data compatible between 2 technologies (one may not know about JSON) and to improve performance of data transition (JSON is ~30% less and faster in remote data excahnging then XML)

Just to share with you guys ready-to-go Java Script libraries:

Stringifier (with smart logic) find here implementation (http://www.thomasfrank.se/downloadableJS/jsonStringify.js) and notes from author
XML<->JSON conversion in JS: first implementation and second implementation

Those who relies on frameworks and platforms - JSON's classes with all needed features are included in Java, .Net and some databases (JSOn serializers on a fly)

P.S. jsut for fun - the same problem the same solution - Win Registry serializer on VC++ http://www.codeguru.com/cpp/w-p/system/registry/article.php/c2859

Microsoft Testing framework (Microsoft.VisualStudio.TestTools namespace) overview

2010-08-03T08:51:00.000-07:00

If you are working within .Net and other MS's platforms, you probably should be happy having Visual Studio Team System as part of Visual Studio IDE. This is getting more interesting if your ALM is Team Foundation Server as Visual Studio with TFS perfectly fit and compatible(can't be different?)

What I like - is detailed useful and thought-out Team Test API. Yes API! This is presented within 5 namespaces (references are here):
Microsoft.VisualStudio.TestTools.LoadTesting
Microsoft.VisualStudio.TestTools.UnitTesting
Microsoft.VisualStudio.TestTools.UnitTesting.Web
Microsoft.VisualStudio.TestTools.WebTesting
Microsoft.VisualStudio.TestTools.WebTesting.Rules

Besides of API, VS Team System provides GUI User-friendly tools for managing test sets, code edition within VS IDE (with all native capabilities) and record-playback feture. Supporting several languages running over CLR is also benefit of testing framework from Microsoft.

One more important thing is Load testing pure functionality as part of Test Team API (performance/load modeling, profiling, watching performance counters, monitoring DB productivity and so fort) and all that goes with VS license!

The Getting Started page of Unit Testing framework can be found on this MSDN's page. As ususal in unit testing, API from MS gives various assertions, implemented within few classes. I found interesting and most useful from my standpoint, the following classes and theirs polymorphic assertions methods:
Assert, StringAssert, CollectionAssert, AssertFailedException
The polymorphic approach of assertions afford the abstract interface which ease programming by datatype independency. It means proper implemntation of a particular data type is an internal thing and yuo should not care about that, as well as type trating, code and workflow safety, error and exception handling, reporting. Check out the benefit, for example from overloading list of Assert.AreEqual

Data-driven testing (DDT) is mostly implemented as data native agnostic model and I found that DDT is provided in 2 classes: DataSourceElement and
DataSourceAttribute. If you need to process details of a particular test, just instantinate an object of TestContext

To understand a standart unit test workflow, check out the excerpts (from MSDN):

the original calss implementation

using System;

namespace SampleClassLib
{
public class DivideClass
{
public static int DivideMethod(int denominator)
{
  return (2 / denominator);
}
}
}

the testing code is

using Microsoft.VisualStudio.TestTools.UnitTesting;
using SampleClassLib;
using System;
using System.IO;
using System.Windows.Forms;

namespace TestNamespace
{
[TestClass()]
public class DivideClassTest
{
[AssemblyInitialize()]
public static void AssemblyInit(TestContext context)
{
MessageBox.Show("Assembly Init");
}

[ClassInitialize()]
public static void ClassInit(TestContext context)
{
MessageBox.Show("ClassInit");
}

[TestInitialize()]
public void Initialize()
{
MessageBox.Show("TestMethodInit");
}

[TestCleanup()]
public void Cleanup()
{
MessageBox.Show("TestMethodCleanup");
}

[ClassCleanup()]
public static void ClassCleanup()
{
MessageBox.Show("ClassCleanup");
}

[AssemblyCleanup()]
public static void AssemblyCleanup()
{
MessageBox.Show("AssemblyCleanup");
}

[TestMethod()]
[ExpectedException(typeof(System.DivideByZeroException))]
public void DivideMethodTest()
{
DivideClass target = new DivideClass();
int a = 0;
int actual;
actual = target.DivideMethod(a);
}
}
}

If the shown snippets are not clear why and how, study the Structure of Unit Tests page

Looks like everything in the box.
AND one more interesting feature in .Net Reflection - thus testers and developers see internal staff of implementation, such as all properties and methods of UI control and so forth. Btw, using that test code basic verification testing (aka baseline) can be generated on fly

This simple VBS script can be used to health monitor your infrastracture

2010-08-03T00:58:00.000-07:00

This simple VBS script can be used to health monitor your infrastracture. Each monitored machine should have turn on RPC service.

The provided code should be saved as .vbs, e.g. HostsHealthMon.vbs

To call the script, run in CL this (or make cmd file to wrap this call):
cscript.exe HostsHealthMonitor.vbs>>\\YouRStorage\HostsHealtMon_Log.txt



Option Explicit

Dim Hosts, RootUser, RootPwd, forceAlarmCPU, forceAlarmHD, forceAlarmRAM, forceAlarmConnection
Dim ptr, partitionRedLimit, CPUUsageRedLimit, RAMFreeRedLimit, IterationDelay, resetCount
Dim lastCacheLogPath
Dim httpServerUrl

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    The Sub is responsible for initialization and reseting (on beginning of each cycle) settings
' Populate this set of variables With your data as descibed in comments for each block to configure run
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Sub InitVars ()
  
   'Machines for checking details: host name or IP, User Id and password
    Hosts = Array("host1", "10.90.23.255", "host02", "host03", "Server01")
    RootUser = Array("Admin", "User", "Admin", "User01", "TestUser")
    RootPwd = Array("123west", "123west", "123west", "123west", "123west")
   
    'Flags reseting
    forceAlarmConnection = Array (True, True, True, True, True)
    forceAlarmCPU = Array (True, True, True, True, True)
    forceAlarmHD = Array (True, True, True, True, True)
    forceAlarmRAM = Array (True, True, True, True, True)
   
    'Threasholds
    partitionRedLimit = "5"
    CPUUsageRedLimit = "95"
    RAMFreeRedLimit  = "50"
   
    'Settings for defining frequency of the reporting
    IterationDelay = 100
    resetCount = 5
   
    'Only latest results will be stored in this file (no history). E.g. a host had a problem with CPU load in previous full iteration and now it soes not, then the log file will contain nothing about that
    lastCacheLogPath = "\\hostStorage\HostsHealtMon_Cache.txt"
    Call Delete_File ( lastCacheLogPath )

End Sub

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    removes specified file
' Parameter       sFile - name of folder to be removed
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Public Function Delete_File (ByVal sFile)
On Error Resume Next
    Const c_Proc = "Delete_File"
    Const c_DeleteReadOnly = True
   
    Dim iResult, oFSO

    ' Define default error code
    iResult = True

    Set oFSO = CreateObject("Scripting.FileSystemObject")

    ' Check if specified folder exists
    If Not oFSO.FileExists(sFile) Then
        iResult = True

    ' The file found
    Else
       
        ' Remove specified folder
        Call oFSO.DeleteFile (sFile, c_DeleteReadOnly)

        ' Check operation result
        If Err.Number <> 0 Then
            Wscript.Echo "Can't remove the file '" &amp; sFile
            iResult = False
        Else
            iResult = True
        End If

    End If ' The folder found
    Set oFSO = Nothing
    Delete_File = iResult

End Function

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description:    Writes in file
' Parameters:    objFile - file object
'                strText - text to write
' Result:        True / False
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Public Function FileWrite(ByRef sFile, ByVal sText)
    Dim iResult
    Dim objFSO, objFile
    iResult = True
    Const ForAppending = 8
   
    Set objFSO = CreateObject("Scripting.FileSystemObject")
       
        Err.Clear
        Set objFile = objFSO.OpenTextFile (sFile, ForAppending, True)

        If Err.Number <> 0 Then
            Wscript.Echo "Error appending to file " &amp; sFile
            iResult = False
            Err.Clear
            Else
                objFile.WriteLine sText
                objFile.Close
           
        End If
               
    Set objFSO = Nothing
    Set objFile = Nothing
   
    FileWrite = iResult
End Function


Sub Main()

    InitVars ()
   
    Do While True
        On Error Resume Next
        ptr=0
        resetCount = resetCount - 1
        For Each Host In Hosts
            Set objSWbemLocator = CreateObject("WbemScripting.SWbemLocator")
           
            Dim alarmStr, tmpStr
            alarmStr = ""
            tmpStr = ""
           
            Err.Clear
            Set objWMIService = objSWbemLocator.ConnectServer(Host, "root\CIMV2", RootUser (ptr), RootPwd (ptr),  "MS_409")
           
            If Err.Number <> 0 Then
                'Wscript.Echo "Warning establishing RPC connection with " &amp; Host &amp; ": "&amp; Err.description
                tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "Error: Unable to establish RPC connection with " &amp; Host &amp; ": "&amp; Err.description
                If CBool(forceAlarmConnection(ptr)) = True Then
                    Wscript.Echo tmpStr
                    alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                    forceAlarmConnection(ptr) = False
                End If   
            End If
           
            'Wscript.Echo Date &amp; " " &amp; Time &amp; " " &amp; Host
           
            'Show free space on logical disk
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_LogicalDisk",,48)
            For Each objItem In colItems
                'Wscript.Echo "FreeSpace: " &amp; objItem.FreeSpace
                If objItem.FreeSpace <>"" Then
                    If CInt(Left(objItem.FreeSpace, Len(objItem.FreeSpace)- 9)) < CInt(partitionRedLimit) Then
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - Free space on a partition at : " &amp; Host &amp; " = "  &amp; Left(objItem.FreeSpace, Len(objItem.FreeSpace)- 9) &amp; " GBs"
                        If CBool(forceAlarmHD(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmHD(ptr) = False
                        End If
                    End If
                   
                'Else
                '    forceAlarmHD(ptr) = True       
                End If
            Next
           
            'Show CPU load percentage
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_Processor",,48)
            For Each objItem In colItems
                'Wscript.Echo "CPU load on: " &amp; Host &amp; " = "  &amp; objItem.LoadPercentage
                If CInt(objItem.LoadPercentage) > CInt(CPUUsageRedLimit) Then
                    WScript.Sleep(3000)
                    If CInt(objItem.LoadPercentage) > CInt(CPUUsageRedLimit) Then 'Double check after 5 sec
                   
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - CPU usage on: " &amp; Host &amp; " = "  &amp; objItem.LoadPercentage &amp; " %"
                       
                        If CBool(forceAlarmCPU(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmCPU(ptr) = False
                        End If
                       
                    End If
                   
                'Else
                '    forceAlarmCPU(ptr) = True       
                End If
            Next
           
            'Show Memory MB free space
            Set colItems = objWMIService.ExecQuery( _
                "SELECT * FROM Win32_PerfFormattedData_PerfOS_Memory",,48)
            For Each objItem In colItems

                If CInt(objItem.AvailableMBytes) < CInt(RAMFreeRedLimit) Then
                    WScript.Sleep(1000)
                    If CInt(objItem.AvailableMBytes) < CInt(RAMFreeRedLimit) Then 'Double check after 5 sec
                   
                        tmpStr = Date &amp; " " &amp; Time &amp; " " &amp; "ALARM event - RAM free space: " &amp; Host &amp; " = "  &amp; objItem.AvailableMBytes &amp; " MB"
                       
                        If CBool(forceAlarmRAM(ptr)) = True Then
                            Wscript.Echo tmpStr
                            alarmStr = alarmStr &amp; " <br />" &amp; tmpStr
                            forceAlarmRAM(ptr) = False
                        End If                       
                       
                    End If   
                   
                End If
            Next
       
        'Wscript.Echo alarmStr
        If alarmStr<>"" And alarmStr<>" <br />" Then
            Call FileWrite(lastCacheLogPath, alarmStr)
        End If
       
        Set objSWbemLocator = Nothing
        ptr = ptr + 1
       
        Next
       
        WScript.Sleep(IterationDelay)
       
        If resetCount <= 0 Then
            Call InitVars()
        End If
       
    Loop
   
End Sub

Call Main

Google Testing and Mocking frameworks overview

2010-07-31T03:54:00.000-07:00

Google's guys that their Testing (take a look into the link to see the purpose) and Mocking (take a look into the link to see the purpose) frameworks can provide ligtweight and rich solution for unit testing over xUnit. Sounds interesting for me but not as the solution user so far. Eventually I want walk through the source codes and documentations in order to grab potential patterns, practices and maybe one day will come up with reusing those fixtures.

The detailed How to (not for dummies) are here (Testing) and here (Mocking)
Super! The use and syntax are really easy as both give you high-level macro instructions and capability to manipulate various (sometimes tricky) parameters. Basically the parameters are designed in source as flag resolver. The Testing FW (framework) advanced things like parametrized and typed tests are implemented in polymorphic template style with some ugly code geration (gtes-param-test.h). It's interesting to note that Testing and Mocking FWs attempts to give you a tool to model behaviour and workflows, various assertions just can ease and reduce testing code but the set of verifiers will not make revolution in testing. Here is a mocking style example:

using ::testing::InSequence;
using ::testing::Return;
...
{
InSequence s;

for (int i = 1; i <= n; i++) {
EXPECT_CALL(turtle, GetX())
.WillOnce(Return(10*i))
.RetiresOnSaturation();
}
}

the Testing excercises:

// Tests that the Foo::Bar() method does Abc.
TEST_F(FooTest, MethodBarDoesAbc) {
const string input_filepath = "this/package/testdata/myinputfile.dat";
const string output_filepath = "this/package/testdata/myoutputfile.dat";
Foo f;
EXPECT_EQ(0, f.Bar(input_filepath, output_filepath));
}

// Tests that Foo does Xyz.
TEST_F(FooTest, DoesXyz) {
// Exercises the Xyz feature of Foo.
}

} // namespace

int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

You can easily inline your tests and mocks into source code (as well as make mocking as part of tests), just need to take care about proper headers inclusions.
The implementation looks like standart xUnit's
namespace {

// The fixture for testing class Foo.
class FooTest : public ::testing::Test {
protected:
// You can remove any or all of the following functions if its body
// is empty.

FooTest() {
// You can do set-up work for each test here.
}

virtual ~FooTest() {
// You can do clean-up work that doesn't throw exceptions here.
}

// If the constructor and destructor are not enough for setting up
// and cleaning up each test, you can define the following methods:

virtual void SetUp() {
// Code here will be called immediately after the constructor (right
// before each test).
}

virtual void TearDown() {
// Code here will be called immediately after each test (right
// before the destructor).
}

// Objects declared here can be used by all tests in the test case for Foo.
};

The testing FW provides inheritance from Environment down to test case down to tests so that generic methods and properties will override parent's implementation.

The Testing FW clasess self-explaining ones by the names - listed below (taken from internal namespace):
class AssertHelper;
class DefaultGlobalTestPartResultReporter;
class ExecDeathTest;
class NoExecDeathTest;
class FinalSuccessChecker;
class GTestFlagSaver;
class TestInfoImpl;
class TestResultAccessor;
class TestEventListenersAccessor;
class TestEventRepeater;
class WindowsDeathTest;
class UnitTestImpl* GetUnitTestImpl();
void ReportFailureInUnknownLocation(TestPartResult::Type result_type,
const String& message);
class PrettyUnitTestResultPrinter;
class XmlUnitTestResultPrinter;

F-1 round ride Kremlin in Moscow today

2010-07-18T03:59:00.000-07:00

Today I have watched Formula-1 bolides nearby Kremlin in Moscow. It's warth to first time observe such road-jets (can't say cars) just in few steps from myself!

These are very reactive and noisy. Actually the noise is felt by skin - FANTASTICALLY!
The next time I hope I will see real race.

Unfortunate thing - I did not take a camera, trying to catch on mobile phone was failed cause F-1 is very fast :)

Code Refactoring - once you meet it...

2010-07-05T12:12:00.000-07:00

The world changes continously, just so software code should pass through changes too. Basically lifecycle is following:

Prototype or simple model
Design as skeleton
Design refactoring
Some stuff is implemented
Refactoring should be done over new stuff
Code and design are ready
But code and design fit badly (inefficient memory management, resource leaks, security flows, unstable run, weak code support, improper interfaces and technologies, code smells...)
Run new refactoring. Dangerous and nightmare as hard!

* This imagination from standpoint of refactoring only. Change requests, new features and bugs and emergency wishes can appear between any of listed steps! Some projects recursively call Step #1 along any Step #X. Better case if you enjoy a few iterations 4-5-4, 6-7-6 or 7-3-7

The tool for refactoring is refactoring :)
The standard simple approach is to collect all code issues in defect tracker, assign to a proper player, come up with refactoring method, define expected results and benefits, estimate cost of fix and plan it. Once a refactoring item is ready - it should be tested inside and outside (Integration), unit tests (if exists) should give green (or rework them to meet with code changes). The last step is integration to production, QA testing and verification that "Refactoring is took place and results in expected out"

Usually refactoring is a step after code review or informal design/code inspection. The motivation to run refactoring should be as inside as outside, i.e. inside - when lead developer or architect initiate code review process and outside - when QA or customers or management have complains on product quality caused by bad design or coding.

Why refactoring is painful? Simply, more works done behind - more risks and potential affecting points on each refactoring item. And higher cost of refactoring. The same as for defects discovering and fixing - rate is continiously rising to production. Do you remeber this curve?

Consequently, frequent code review - less painful refactoring and less cost totally on refactoring. The general motivation of refactoring should be improve quality attributes of software, don't be driven by this task as something fancy or trendy.

Now, it's time for technical matter - some refactoring methods most used in industry. The good list of refactorings is listed here (http://www.refactoring.com/catalog/). The list is clearly complicated and easy to understand for those who are familiar with refactoring and coding practices, almost all links show up briefly meaning and how to apply with code excerpts and or UML diagrams. Most of the items can be fit and applied to automated testing either your code in scripting languages or real object oriented languages.

Basically we can break down refactoring approached to 2 groups: running top-down and running down-top. The first presumes review and improvements from highest levels (architecture and design) lowering to lines of code. The second method - is vice versa. Refactoring is not only finding and fixing issues in code pieces, it's also about designing proper patterns up to level of general architecture. Some time ago I made a post on patterns applied for automated testing 20 Essential design patterns for automated testing, however we can use it to revise the current design.

And the Winner is (poll results)...

2010-06-27T23:48:00.000-07:00

Hey guys,
the voting of this poll http://at4qa.blogspot.com/2010/05/poll-best-test-automation-ide.html is closed. Thank you all who participated and responded!!!

Shortly - facts:

there were 105 responds
Mercury/HP Quick Test Pro and Selenium share the first position as 24:24. I could outline the results as: Selenium - the best choice for Web (only), QTP - is the best choice for supporting various platforms and technologies; another observation - QTP is the best as proprietary and paid solution, Selenium - is the best as open solution. Sure Selenium can't replace QTP completely
... but TestComplete (2-nd) or MS' Tester edition / Test Automation FX (3-rd) - probably could
SilkTest is loosing market completely?
Full layout - on the image below:

James Whittaker's interview for uTest - thoughts of leadership

2010-06-25T12:22:00.000-07:00

Today just have got email newspapper from uTest. I took an interest in recent interview with James Whittaker (Google's chief technologist and formely Microsoft's employee). See the original version here Part I and here Part II

Listed below are excerpts and trending things by James Whittaker, I found interesting:

- Test Automation is highly in demand. Google invests bunch of money and effort on
automation

- Right test automation is a part of innovation

- Cloud testing and TesterNet (both automation and manual). I rather express it as raising abstraction and independency from technologies. Generally - simplification

- Embedded test automation framework (e.g. Google's Web Test Framework for self-testing Google Chrome). It is not novice. Watchers and embedded tracers, debuggers were invented and used years ago. Embedding unit tests into User's experience with app is good idea but does not seem too innovative

- Methodologies should not screw up real work effort and achivements. Using best practices which fit to project is right way, adoption and flexibility are must

- Shortly we will get a book How we test in Google from James Whittaker (as inspiration by Alan Page's book, Microsoft)

- James Whittaker looks like a fan of exploratory testing!

Test Automation Infrastracture Architecture and Automation Practices Generalization

2010-06-14T05:23:00.000-07:00

Hey friends,

it looks like my time on blogging decreases day by day. I have less and less time to think up and then write something novice and interesting on test automation. Is it writer's block? Or Summer? :))))

Inspiration will come but not today and may not in this month, so far I imbibe, learn and think. This post is just a bunch of technical and organizational things as conclusions and generalization on accumulated posts.

While I'm preparing a presentation for high-management, want to share some pieces of that. The below diagram represents test automation infrastracture as whole picture to highlight scope of works and to inspire feel of confidence that we provide really professional test services and use advanced techniques and practices upon building the entire and completed automated system.

Another image represents our secured approach of code developement - nothing new - just following proper development practices.

Wherever you automates testing, it's suggested to invest in the best development practices that will return the investements down the road depending on project duration and long-term purposes.
Some of that truly should be applied for serious test automation (formerly the posts of this blog):

ROI on automation testing department establishment

On stabilization test suites. No fuel but destination is too far

Naming and coding conventions: are they important for test automation?

Source control and change management in test automation

Test Automation Dashboard in just One day!

20 Essential design patterns for automated testing

Adopting TDD practices into white-box test automation

Measuring test (manual/automation) effectiveness

Lastly some things on organizational approaches and practices

Automated Testing for QA http://at4qa.blogspot.com/

View more presentations from Peter Ka.

Automated Testing in Java - Presentation with reach code excerpts

2010-06-14T04:23:00.000-07:00

Very technical oriented presentation that i'd like to share with you. The embed slides are right here:

Automated Testing In Java

View more presentations from franzsee.

Poll: The best Functional Test Automation IDE for YOU is

2010-05-28T09:13:00.000-07:00

The best Functional Test Automation IDE for YOU is:customer surveys

10 Objects to be Hand On!

2010-05-25T11:31:00.000-07:00

This post is just short manual or reminder on generic system-available objects in Windows environments.
10 So let's go and begin for instantination. In order to unify and isolate external objects management, I prefer either Singleton or Factory simple patterns (http://at4qa.blogspot.com/2010/01/20-essential-design-patterns-for.html). They will take care about objects lifecycle management:
- object creation if not exist in current context or reference returning if an instance lives (singleton)
- new instance creation (if factory)
- dynamic object update when needed
- object life finalization destroy and freeding memory as it actual for languages w/o garbage collection or with with weak one
- protected/private use
- logging and exception handling

In VBScript new instance of object can be created as:
Set Obj= CreateObject(ObjStrSemantic),
e.g Set Obj= CreateObject("MSXML.DOMDocument")
I will address objects by mean of string variable ObjStrSemantic down the road:

"Scripting.FileSystemObject" - all operations over files and folders
"Shell.Application" - Standard shell Win object for running basic system operations
"WScript.Shell" - This is more prefearable object then listed above
"VBScript.RegExp" - Regular expressions object
"Scripting.Dictionary" - Dictionary object type. Basically, this structure is used in VBS or to create arguments of this type for subsequent calls
"WScript.Network" - Implements basic interfaces for networking
"InternetExplorer.Application" - can be used to manipulate on IE and can be candidate to replace testing tool functionality, clicking and so on
"ADODB.Connection" and "ADODB.Recordset" - basic object for obtaining connection with source through ADODB connector and object of Recordset type serving to store retrieved data from ADODB in memory
"MSXML.DOMDocument"("MSXML2.DOMDocument.3.0") - essential object to process XML through standard DOM methods, XPath. Why you may need it? You can develop Web testing code omitting testing vendors tools which make the same but giving own custom interfaces. Desing data (DDT) or keyword-driven(KDT) testing framework leveraging power of XML. Check out my post about XML possible use (http://at4qa.blogspot.com/2010/01/tools-xml-orm-for-efficient.html)
"WbemScripting.SWbemLocator" - Windows Management Instrumentation (WMI) is really powerful tool to manipulate over various system operations (hardware and software). I think WMI covers all system capabilities including all related to administration, providing remote connection and querying using SQL-like language WQL and unified API for administration automation. The link http://www.microsoft.com/downloads/details.aspx?FamilyID=2cc30a64-ea15-4661-8da4-55bbc145c30e&displaylang=en leads on download page of WMI code editor tool from MS - indeed, this tool is simple and handy.

Simple WMI usage for local querying:
Set oWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & HostName & "\root\cimv2")
Set colOperatingSystems = oWMIService.ExecQuery ("Select * from Win32_OperatingSystem") 'Querying OS info

Simple WMI usage for remoting:
Set objSWbemLocator = CreateObject("WbemScripting.SWbemLocator")
Set objWMIService = objSWbemLocator.ConnectServer(Host, "root\CIMV2", UID, Pwd, "MS_409")
Set colItems = objWMIService.ExecQuery("SELECT * FROM Win32_LogicalDisk",,48)

Working on test automation or infrastracture framework, it's better to design wrappers over methods, properties and events of appropriate objects and put these wrappers into appropriate custom Registry classes which will represent appropriate object interfaces. What you will get from that? At least:

Unified and safe access to objects, methods and properties (Registry class is better to implement using Singleton)
Your IDE/TDE will see instructions and Intellisense will work!
You will follow proper coding standards, you will avoid duplications and improper use. As result, you get back quality and code consistency
More long-term advantages...

Let's perfect scripting skills, MSDN will help in that - don't forget to utilize Scripting Technet space where you can enjoy examples and practices, follow at http://technet.microsoft.com/en-us/library/cc498722.aspx

Model-Based test automation with randomization

2010-05-24T02:14:00.000-07:00

There is a pure work articulating on Model-based testing and it looks like the page http://www.goldpractices.com/practices/mbt/index.php widely covers all about that. Though it's theoretical at most, this is good start point to acquire knowledge and apply on practice. Also I liked the case studies showing effectiveness of model-based testing.

From test automation standpoint, this is really powerful thing which is not replacement of traditional static test design for BVT, extended tests. The automatic test suite modeling allow to run regression and exploratory tests which can cover much more s/w paths. It's true that covering all possible combinations is not efficient way even for automation as everything has limits. To run your test optimally, suggested use is applying:
- limitations
- preconditions
- coverage control points (if previous test has visited that destination, why we should repeat that over?)
- some randomization for both control flow operations (steps composition) and test data feeding on

To build models, I'd suggest to use XML technologies as flexible approaches for relational data storage, querying and transformation. Look at some tips here http://at4qa.blogspot.com/2010/01/tools-xml-orm-for-efficient.html

Importance of Basing on Testability in Automation

2010-05-20T23:49:00.000-07:00

Greetings!

I'm back after very hard work on Kaspersky Antivirus new Release. We successfully passed technical Release. Our automated testing is proven and valuable weapon for daily build testing on a lot of test stands simulataneously. The automation is really indispensable on this project since it provides significant coverage and feel of confidence about each build quality.

The official Beta release of Kaspersky Internet Security/Antivirus 2011 is available of Company's forum here:
http://forum.kaspersky.com/index.php?s=22c83ab5a899d1bdafd4afc822a3ead6&showforum=16

The first thing i'd like to share with you is my thoughts about testability. This term can have relation to any software artifact: code, documentation, architecture, design, interface, structure, product and so on. In fact, the testability metric can be applied to any level of software, downside to code piece for unit testing. Let say if code piece is completely for protected usage, you can't pull interfaces of that class, so that testability is difficult and you need to invite workarounds like fake objects having rights to treat that class.

For routine functional automated testing, product testability plays key role for defining whether s/w suitable for automation or not. Formally I suggest to come up and register the requirements to product testability within Project book and Testing Strategy documents. Believe me or not but this is a ground and shield for you in developed project especially in strongly formalized projects. If you provide external test automation services these requirements are must as early as possible on a project. In essence, testability requirements is a basis of constructive proposals, negotiations and argumentations during project progress.

What kind of information should be listed out as requirements? Not sure I'm able to cover all things together but some general fitting to most projects are:

Changes in software external GUI interfaces to be addressed as part of appropriate change management procedures

Changes in software external non GUI interfaces to be addressed as part of appropriate change management procedures
Changes in software internal tested interfaces to be addressed as part of appropriate change management procedures
All changes to be planned, discussed and notified when it takes place
All interfaces with which testing automated, should be frozen when the code freeze will take place
Changes in software interfaces should not reduce capabilities of approved testing tools or significantly affect test automation codebase.
All GUI objects should be recognizable by the use of approved testing tool
All GUI objects should be uniquely identifiable by the use of approved testing tool
All standard GUI objects' methods and properties to be accessible by the use of approved testing tool
All intermediate and low-level software interface used for automated testing should be stable
On All planned changes, appropriate persons should be notified prior the date of changes
The data structures processed by test automation should provide access to each granular element using conventional methods. For instance, DB querying, object deserialization, XPath, Registry, CSV, INI parsing and so on.
Access to GUI objects should be unified for identical object types. All custom objects should have minimal and similar access methods
Custom controls not supported by approved test automation tools should be either enriched by typical access methods or special tool should be provided capable to recognize those objects
Preferably, all typical interfaces should be identical between related projects, i.e. development should try to come up with cross-project solution of interfaces design
Preferably, changes should be shown to automated testers before new deployment in order to adapt testing code in advance of deployment or releasing

Some projects may have individual requirements, some - not. You are free to adopt the list above and propose on review and approve requirements. Anyway, I hope my list from past experience could help one day for someone.

PS subscribe to dev codebase changelist to be prepared for changes in unstable project.

My Presentation is Featured of the Day on SlideShare.net

2010-05-20T23:42:00.000-07:00

Nice to be informed by SlideShare about my presentation "Effective QA as driver towards innovations..." was chosen as featured of the day. I'm really surprised as I did not expect whatever and did not prepare - just wrote down some thoughts and put some illustrations.

11000+ views, shares and favorites. Looks like huge traffic!
Link on the uploading:
http://www.slideshare.net/PeterBiz/automated-testing-for-qa-httpat4qablogspotcom

Interesting Pages from MicroSoft Tester Center

2010-04-24T05:47:00.000-07:00

I've dug out a few really interesting pages from Microsoft Tester center [http://msdn.microsoft.com/en-us/testing/default.aspx] space hosting on MSDN.

The couple technical-oriented links talking about some Code/Design review aspects and practices at Microsoft:

http://msdn.microsoft.com/en-us/library/ff519670%28v=MSDN.10%29.aspx

http://msdn.microsoft.com/en-us/library/ff521647%28v=MSDN.10%29.aspx

The following 3 links from on management and improving processes from a personal blog of Microsoft Test manager
http://blogs.msdn.com/anitag/archive/2009/03/22/business-focused-vs-people-focused-managers.aspx

http://blogs.msdn.com/anitag/archive/2009/03/29/automation-maturity-what-should-a-test-manager-focus-on.aspx

http://blogs.msdn.com/anitag/archive/2009/08/19/should-bvts-pass-100.aspx

I don't think that MS Tester center is super resource with always fresh and novice content. I believe it's a bit static and archaic but for someone it can be good start point in testing journey. More attractive and concrete Microsoft!

Hybrid-level test automation is the most effective

2010-04-23T09:26:00.000-07:00

Hey test automation gurus and those who just in the beginning of this challengeable road!

After reading http://googletesting.blogspot.com/2010/04/googles-innovation-factory-and-how.html I’ve got some thoughts about test automation effectiveness from test-level point of view. Most of developed s/w products have various interfaces lying within different layers. Pulling these interfaces in tests we get some testing level coverage. Just for example, working through GUI, test likely to be called full System E2E level (aka Integration), going down we can call objects of GUI shell representing native methods and properties of GUI entities (calling Get/Post requests, working through DOM, COM objects). The last approach is still integration testing or system level but bits farther from User’s use case – it means we don’t test GUI itself.

Component (module)-level functional testing is deeper one as tests here operate over a particular component using its native external interfaces. Let’s call it API-level testing. This approach requires being subscriber and maintainer of module API. Surely this testing is quicker and more robust at most cases due to absences of intermediate layers and integration containments.

Components integration potentially can utilize different levels of interfaces but this is drawback of supporting different technologies. Here is example: run system job through Component A API, validate Component B response through GUI. The better case: run system job through Component A API, validate Component B in database. The most valid case for this testing level: run system job through Component A API, validate Component B behavior using Component B API. The ideal case: Components A and B have single unified API or have higher layer inheriting and unifying all components API’s and externalizing secured API.

The Unit-test level is when you test product’s code itself (Classes, methods and structures). This is important level which must be used for producing good apps. Unit testing to be relied on development resources.

Hybrid-level model in essence is a mix of all (though probably but Unit-level). But the formula of proportions and efficacy is different from project to project. I just can show some examples of reasonable scenarios of combining different test levels:

API level testing is using to manipulate system components, transmit data and trigger events. Verification is running on API-level, then in GUI as double checkpoint.

Integration between 2 components is verifying in module integration level (API). Integration with GUI layer is checking in GUI as checkpoints.

Change data in database (SQL scripting); checking results in API as checkpoint. And vice versa – change data using UI, validate on database side (SQL querying). Let’s complicate these cases by adding validation in intermediate layer (e.g. check Web service I/O, check queue object, check POST/GET Header and Body)

Google Testing Blog: Google's Innovation Factory (and how testing adapts)

2010-04-20T11:58:00.000-07:00

This is amazing that GoooGle reveal their some internals about testing, continious inegration, ROIs and savings of different approaches and digits. Really appreciated! See theirs presentation here:

Weekly Fun Relax on FR : QABurger

2010-04-15T12:29:00.000-07:00

What are differences between professional and matured QA and so-called QA as some organizations like to use this term but don't follow proper pracices? The answer is simple - they differ like different fast food and restaraunt meal cooked by skilled cooker.

So to cook QABurger, you need to mix:
- developers response for testing - so they combine development and testing;
- practice Ad-hoc testing only without Test plans and schedules;
- gather all issues and defects somewhere in any format - just write something to revisit one day;
- don't prioritize and plan fixes and new features - let's to begin from easy task, then go down by fix/implementation complexity;
- use very simple and clear metric of you quality attribute like number of defects - that's enough to make some judgements and to define scope of works;
- simplify all processes - chaos is a pat of creativity;
- deploy new features and fixes on production as soon as you can after checking on development machine - more frequent commits simplify and accelerate work
- give your system for testing for potential Users - testing engineers will yield the same result but for money

Data Migration/Transformation testing tips

2010-04-15T06:50:00.000-07:00

Data migration/transformation projects are actual in s/w industry especially because systems mature, systems integration, needs in transforming data from legacy systems, for data improvement, consolidation, separation and purifying tasks. Some data migration activities may be so complex that require involvemnet of significant resources, skilled professionals and special tools. It can be truly noticed that data transformation itself in modern projects is sophisticated and challengable solution requiring individual approach and process modeling.

Data migration testing runs as grey or even white-box testing at most cases. Simple verification against on presentatoin layer can't gurantee good coverage, discovering hidden defects and this testing itself is not effective as requiring intensive data verification without deep inside research.

Here are my tips for testers who run or are going to test data migration:

- learn purpose and business model of project and Company to realize migration target

- study transformation schemas and scripts(XSD, SQL scripts)

- study development algorithms which operate over original data to produce transformed one. Notice you have to correspond the algorithms with specifications and requirements on migration process

- study database schemas (all from original to final through intermidiate states)

- develop and check your own scripts (SQL, XML, XLS, XSLT) to retrieve data for testing

- design sufficient and efficient data sets for testing data validity and integrity

- store your scripts and data sets under version control and link with versions of AUT

- run tests and store actual outputs in test logs.

- Also consider saving test logs as database records in testing DB with linking on defect entries. This approach will allow to revisit test run results as soon as you need in structured view

- Keep your knowledge about everything in product changes up to date

- Check data transformation in all points where it's being transformed in a whole product (DB, presentation, services, batch jobs, intermidiate mediums, etc)

- Make sure and double check correspondence and compatibility of original data types and transformed types for each entity, field and macro-data. This is a part of getting familiar with database architecture and schemas

- Run test frequenly against various data sets representing various combinations and tricky moments (like using special characters, sontrol symbols, code snippets, too long strings, invalid types and mixed content; strings that potentially can be truncated, encoded or echoed)

- More CREATIVITY and Hacking! You should BREAK IT!

For those who need to continue learning this topic, take a look at Wikipedia's related pages and this article about Data warehouse testing

http://www.developeriq.in/articles/2010/mar/10/data-warehouse-testing/

Normal distribution in action: defect distribution modeling and prediction

2010-03-29T12:35:00.000-07:00

Preface: to be on the same page, it's recommended to review the following WP articles:
http://en.wikipedia.org/wiki/Normal_distribution

http://en.wikipedia.org/wiki/Theory_of_errors

http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics

Though, the notion normal (Gaussian) distribution appears potentially helpful for s/w project planning and analysis, but it's strongly recommended to play with this tool carefully and what more important - with statistically sufficient and significant data set. Yes there is a constraint - the project should be big enough. However, does it really important to forecast new defects and persistent ones on a project having 5000 SLOC? Don't think so.

Obviously this theory may help to:
- estimate volume of non-covered/discovered defects. By this - rebuild test plan to achieve proper test coverage.
- predict volume and distribution of newly revealed product defects. By this - come up and/or numerically adjust with project sign off date. Somehow metrics could shift project release or may play start point of more resources negotiation.
- model efficient test automation (functional and unit) coverage. By this - achieve high ROI on automation.

All you need is to:
- select a sampling variable. In our case likely it is number of valid reported defects, e.g. weekly/daily. It depends periods you are going to operate on.
- calculate mean over this variable
- calculate variance of this sampling variable
- Then build and graph Probability density function (PDF)

This graph as well as mean and variance calculations are easy task using Excel formulas and graphs. The resulting graph should be build together with original sample in time series (histogram). The view of the curve itself shows "normality» of this sample. Overlapping of two graphs shows divergence of normality with real state. But don't hurry to make judgments on this curve it's just tip for you to feel confidence of project control. E.g. the curve with open tail (end in time series) may signal that testing should be prolonged as there are undiscovered defects as expected statistically.

The next advanced application is prediction. To make it you need to build either ideal normal distribution or use existing one. Then the restore function will give you remaining sample sub-set (show future). So that, you may say how many defects will be found for example week by week. Or what functionality needs to be tested with more effort

Finally to calculate precision over your calculations you have to come up with confidence intervals of your observations.

Myths and silly Viewpoints about Test Automation

2010-03-23T06:31:00.000-07:00

Test automation is just spending money without return
Automated testing is always profitable. Test automation fits to any kind of project
Unit testing is better and finds more defects then functional automated testing
Record-playback is automated testing
Any person with IT background can be auto tester
It's easy to combine both manual and automate dtesting in a single role person
Test automation ROI is impossible to predict and calculate
Test automation estimation is not real activity
Test automation completely differs from s/w development
Bad developer can be good enough for automation
Developers can effectively develop functional auto tests
Auto testing is not about QA
Auto testing should not be a part of overall QA and QC
Manual testing coverage is preferable on any project/functionality and any project phase
Nobody should rely on automated testing
It's impossible to create robust and efficient auto testing at all
Everything can be automated
Is auto testing a sort of machine intelligence? Though, I don't trust that
Test automation should be capable to predict any system and environmental outcomes. Whatever issue occur - auto test suite should remove a problem and/or think out a new workflow path, recovery scenario, invent workaround from a scratch
Test automation is about writing a few code lines which result in testing major functionality. It's easy!

I wish to hear more myths and strange points you ever met on test automation. Let's share yours here commenting to this post.

Looking for Contributors or Fellows for this Blog

2010-03-20T04:05:00.000-07:00

Hey there!

If you have to tell something novice, interesting or maybe amazing - I'd like to see your content published here. Also, fell free to discuss oportunity of authorship on this blog too.

Have a nice holidays!

My post is featured in Automated Software Testing Magazine !

2010-03-16T08:56:00.000-07:00

Hey!

it was nice to be informed my blog 20 Essential design patterns for automated testing (http://at4qa.blogspot.com/2010/01/20-essential-design-patterns-for.html) had been chosen as featured in Automated Software Testing Magazine, Issue March 2010. I got the news while being in vacation - especially pleasant.

Download the official e-version of this magazine here [http://www.automatedtestinginstitute.com/home/ASTMagazine/2010/AutomatedSoftwareTestingMagazine_March2010.pdf]. Then, flip to page 36 - there is my first column out there, cool!

WMI Code creator from MS

2010-03-16T08:40:00.000-07:00

Just stumbled on a post about MS tool to ease WMI querying/executing/eventing here
http://qtpcodes.blogspot.com/2010/03/use-wmi-code-creator-tool-for-quick-wmi.html

The tool is downloadable from Microsoft site here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=2cc30a64-ea15-4661-8da4-55bbc145c30e&displaylang=en

While trying it out, came up with a thought - this is really handy for testers, especially for automation ones. Btw, being hand on this tool, you can develop system-level and network-communication apps and utils using Scripting only utilizing powerful resource management tool!

I'm going to deeply learn the supported WMI namespaces and classes and write some scripting hooks just for fun.

PS Author(s) had provided capability to convert generated code snippets among C#, VB .Net and VBS languages

JSystem Test Automation Framework

2010-03-04T07:08:00.000-08:00

Introduction
JSystem is an open-source framework for writing and managing automated system tests. The aim of JSystem is to provide one simple central framework for functional and system testing of software and embedded systems.

JSystem is an open system that supports leading scripting languages (such as .Net, Python and Perl) and communication protocols. It also connects with leading testing tools, both open-source (Selenium, AutoIt, JUnit, etc.) and commercial (IXIA, Spirent, etc.).

The JSystem Architecture
The JSystem architecture comprises of the System under Test, the Automation Framework and Framework Services, outlined in the diagram below:

The System Under Test (SUT) is the device or software that is being tested, as well as the equipment and applications that participate in the test setup.

The Automation Framework does the testing. The framework divides the testing process into four layers that separate between the actual writing of the tests (done in Java), and the grouping and re-grouping of the tests into different test-scenarios, as well as executing and analyzing them – all of which don't require any knowledge of code and are done with an intuitive GUI interface (JRun):

The four Automation Framework layers are:
- System Object layer: written in Java, these are the basic building blocks of the test.
- Test-Case layer is used to create the tests.
- Test-Scenario Layer: groups the tests within a test-scenario.
- Management Layer: an application that executes analyzes and publishes the test results.

The Framework Services: provide the QA engineer with functionality that assists in testing.

What Makes JSystem Unique?
The JSystem architecture builds tests out of basic building blocks that can be used and reused to create new tests; the clear distinction between business logic and GUI enables the tester to change one without affecting the other – all this translates into much more efficient testing.

Because of its four-layer architecture, JSystem also enables to incorporate both QA people that write code, and QA people that don't. Although creating the System Objects is done in Java (albeit, very simple Java), creating the test-scenarios, executing and analyzing and then generating reports, is all done with the GUI interface (JRun). JRun enables creating test-scenarios with drag-and-drop. This is a screenshot of the JRun main menu:

JSystem also contains built-in drivers for all major testing tools, including freeware drivers for open-source tools (such as Selenium, JUnit and AutoIt) as well as commercial tools.

JSystem Deployments
JSystem is a mature product with deployments is large organizations, including telecommunications labs in such companies as Juniper, Nokia-Siemens, ECI, Avaya and Ericsson. JSystem is deployed in testing Web applications by companies such as Alcatel-Lucent and Taiana, and in testing semiconductors in companies such as Intel, Altair and Metalink.

For more information on JSystem, please go to their website: www.jsystemtest.org.

DISCLAIMER: Copyright (c) Tal Harel. This article is prepared by Tal Harel as material for this Blog (http://at4qa.blogspot.com)

"I'm doing what i'm being paid for..."

2010-02-24T23:57:00.000-08:00

That stupid thinking of workers in IT is not acceptable since skilled IT guys should continuously move ahead, learn more things (in adjacent fields too) and grow value and contribution day by day by joint effort.

If you hear from a guy such words as in the post title, probably there are 2 alternatives behind the sentence:
- either this guy is really making great effort but is remained unmotivated by management, project progress, company culture/processes or anything else
- or this guy is pushover and they whine is just attempt to defend themselves. Likely this person is not skilled, not experienced or does not correspond to requirements on that position.
I like to meet guys that don't give in difficulties and don't afraid to face challenges even if s/he never run assigned duties. It's really cool to see how novice manages well something unfamiliar with.

I wish good luck everyone in career path and keep faith - good managers keep watch on your effort!

GUI Test automation using PowerShell

2010-02-11T05:33:00.000-08:00

I dug MSDN's articles describing how to utilize PowerShell for test automation
Win forms applications [http://msdn.microsoft.com/en-us/magazine/cc163301.aspx]
Web applications [http://msdn.microsoft.com/en-us/magazine/cc337896.aspx]

Also here is their blog [http://testfirst.spaces.live.com/blog/]. PowerGUI is IDE for code development. Check out the highlighting video as demo http://powergui.org/shares/powergui/sbin/docs/AutomatedTesting/PowerShell-Automated-Software-Testing.html

My critical review is below...

As author of those articles Dr. James McCaffrey himself states that PowerShell can give lightweight solution for non-complex test automation. I rather agree with this point as PowerShell does not poses to be test automation environment but you can utilize his capabilities for automation too. It may be deal to use PowerShell scripting approach for supplementary action items in your overall test automation framework - that's like using OS objects (FileSystem, MMC, XMLDom, etc). Besides that you may consider to build own test framework basing on PowerShell, Visual Studio environment and moreover using .Net capabilities. This is a big deal that is requiring big effort and addressing to figure out this project reasonability.

Restriction of supporting Microsoft's technologies only may be considered as improper solution of product automation. As I understood, for Win forms PowerShell can process COM and .Net controls, while for Web-based it can work in IE through DOM.

So putting all together, appears for me that PowerShell is not "power" for test automation but it still can assist as part of whole test automation project; another way is to utilize PowerShell for small test projects and only over Microsoft's environments.

PS cmdlet sounds cool and smells Microsoft.

TOOLS: Diff with Sorting and RegExp filtering

2010-02-03T07:28:00.000-08:00

Data comparison is one of essential parts of testing. There are varieties of assertion approaches which fulfill automatic checking of actual output against expected result (or etalon). I'd like to describe one completed recipe which consists of the following steps:

Select a proper diff tool (supporting CL). Basically this tool should be capable to run diff against files (one-to-one) and against folders (recursively over all files) and should have variety of options (filtering, exclusions, omitting, grouping, file names/extensions filtering, escaping and supplementary rules and options)
Specify filtering options using regular expressions: -excluding lines and - excluding substrings
Employ supplementary steps like lines ordering (e.g. sort command in Win CL), merging

#1-2 are supported in ExamDiffPro - I've used this tool with no problems. Another advantage of ExanDiffPro is capability to import/export settings. It's especially valuable if you need to run comparision under differenet options in CL. If you need to run comparison against huge data sets, try to look at comparing using checksum MD5

That's all I have for this post. Your experience?

Cloud testing as SaaS is to kill performance and reliability testing vendor’s tools? {Generalized: ~Cloud~ for development}

2010-02-02T11:30:00.000-08:00

Not sure I have enough time to run deep investigation whether this question answers Yes or No. But at first sight this approach is profitable against purchasing very costly tools like from HP or Borland. Hopefully the dynamic of expanding SaaS for testing will grow down the road. It seems the biggest player on the cloud testing is Amazon. If they really provide honest and cost-efficient plan with capability and flexibility of load testing model, so performance professionals should take a sharp look on this trending technology. Indeed, why should I pay thousands $$ for load testing especially for a single application/task?

Eventually the SaaS will come to narrow segments like load testing, hosting specific environments like test stands, build and continuous integration servers. I’m also interested if SaaS providers will place such plans like temporary/limited access to dedicated environments running various OSs, apps and services and as premium services they could prepare and let on lease required virtual machine configurations. I think this a great gate for vendors of professional toolkits, let say HP could provide such plans for their QC, QTP and LoadRunner. Here is a scheme: ready environment + preinstalled and configured vendor’s tool + supplementary package (like open source popular tools, e.g. Subversion, notepad++, XAMPP, etc). That would be perfect for automated testing particularly on short and middle projects duration. Resulting in significant reduce of cost on environmental needs will cause many advantages such as test automation ROI improvement and removing overheads on administration and supporting environments.

Such huge players like HP, Microsoft (Win Oss, Visual Studio, MS), Oracle&Sun made steps in this direction, the rest guys will move up too sooner. But I’d like to have their services more affordable, as well as with hosting specific dev/QA software and finally, I wish they propose flexible tariffs with “pay just for how long you use service”. Also there is a field to host, maintain and administrate Open source only, for instance – providing variations of preconfigured Linux distributives jointly with services – please don’t mix with hosting – I want various environments for development and testing with temporary ownership.

Is anyone who has success story of using SaaS for development and testing?

XML for building high-end test frameworks

2010-02-02T05:10:00.000-08:00

The pasted slides below provides real technical-oriented review. I like such kind of content that does not wate time for nothing like guru's philospphy. Take a look at this:

Automated Testing Of Web Applications Using XML

View more presentations from diongillard.

iPad launch : antagonisms

2010-01-28T04:50:00.000-08:00

There were a lot of rumors and guesses about look and feel of being announced super new gadget with screen touch and multitouch revolutionary technologies and lots surprises for consumers. Yesterday they showed this as usually gracefully exposing mystery. Actually I did not pay attention on guesses how it will look like even from "trusted" sources. Why? Because of Apple's strategy is to run a few fake projects so that employment does not know what will be the real product on store shelf.

By this tactic Apple:
• stirs up public
• minimizes risks of thefts
So they launched iPad but results don't impress, moreover - mostly I see articles and posts with blaming containment. This one article seems better yet represents real thinks
http://mashable.com/2010/01/28/what-we-learned-about-apple-yesterday/

Will see...

An interesting post from Google's QA about patterns to testing

2010-01-27T08:45:00.000-08:00

http://googletesting.blogspot.com/2010/01/interviewing-insights-and-test.html

I think this is good point and reflection but IMO professional QA guys combine all the best from this classification and care this suite in daily activities. I don't like one-sided approach as it risky to limit own vision. Also flexibility should be a part of career path since different projects expect different approach as well as thinking.

Tools: XML ORM for efficient parameterization

2010-01-25T09:32:00.000-08:00

Test automation parameterization is a part of architecture, thus Object-relational mapping can significantly simplify your routine works and reduce risks of unprotected and unauthorized data access. Starting from scratch will require to learn and utilize some worked out technologies such as

- XPATH for retrieving/modification/inserting data into XML
- XQUERY as unified SQL-like language
- DOM model. This is like low-level or direct access to elements of DOM documents
- New technology from Microsoft LINQ

I rather suggest to use XML 1.1 namespace (see difference 1.0 with 1.1 here). Additionally, using schema binding to test data XML storages will bring you long-term benefits:
- automatic validation (opening within browser or loading XML through Microsoft XMLDOM:
oXML = CreateObject("microsoft.xmldom") oXML.load(sFile)
- unified designed data structure with advantages of default values, flexible implication of entities, values lookup and restrictions and many more pros

Basically you have to select a single approach: either your schemas defined in DTD or in XSD.
Certainly big test and long test automation projects with intensive and heterogeneous data (and possibly tested objects mapping) parameterization will profit from careful data/objects designing through XML schema.

You can build a hybrid model which will utilize several technologies in a single package encapsulating internal staff.

One more good practice is to split up logical pieces of huge XML into several ones in order to simplify concurrent usage and change control, also think about keeping schema(s) itself as separate document and include properly, so that final model will have 3 layers:

1. Schema definition (or multiple schemas)
2. Common or master XML (which binds schema(s), describes common data and includes particular modules):

3. Specific test data description (test cases, tested objects specs... per component/module/test level)

Test Automation Dashboard in just One day

2010-01-24T12:38:00.000-08:00

The value of automated testing is determined by reporting framework definetly at most. Your team must provide clear, in-time and customizable reporting for stakeholders. Nowadays data presentation plays vital role in assesssing whether your product is useful and useable or not.

The test reporting should be able to show various views of accumulated data. Someone will like to see graphs over all runs, someone will check daily runs carefully drilling down grids and logging lines, someone will want to see something like traffic lights (red, yellow, green).

Development such system is the big deal as actually full-grown project. I developed this one with my team 2 years ago. That was Dashboard which was utilized over all test automation projects. We built the Dashboard using XAMPP (Apache, MySQL), plus some extra JS code and jQuery. To plot graphs we used jGraph open source library. I really like this app as it is flexible, has pretty presentation, navigation, filtering. Moreover we implemented test running and test stands dispatching through the web dashboard interface. Though this infrastructure project had been running in background, we spend on that good portion of our budget and tehnical skills.

This challenge may frustrate if you have limited resources which are dedicated to developing tests only. There is a recipe! Likely your project uses defect-tracking system - so your task is simple utilization this tool for your daily runs. Atlassian Jira is perfect candidate to be reused for test automation as this very flexible and cusomizable system supporting a lot of technologies (just check 3-rd parties integration - http://www.atlassian.com/software/jira/tour/plugins.jsp). Nevertheless if you use Trac or BugZilla or ClearQuest - I believe (since I worked with them), you can apply the same approach.

I see 2 options how to deploy this integration:

1. Integration into project(s) structure with new ticket type (to isolate from regular project tickets)

2. Create new project wth required break down by components (test project or test level), releases, builds (e.g. each new run is a new build); fields customization and so on.

How to tune up:
1. Customize ticket fields
2. Identify rules on new build (e.g. each run) and ticket submission (each test case failure)
* Preferably your reporting should be granular with proper rules of counting failured/not_run/passed steps. The perfect approach is to trace auto test against manual test case (failure in one or more step(s) mean test case fail)
3. Create preliminary views in tracking system: spreadsheets, graphs using built-in filters
4. Think out roles, assignements, priorities and other useful fields
5. Teach your test automation framework to communicate with defect tracker:
- come up with interface: GUI, API, DB, Web service...
- new run submission
- new ticket submission (form filling)
- supplementary data retrieving/update/insert
6. Create a single-view Dashboard page as container of customized gadgets through filtering (or querying)

The final dashboard look and feel is limited by your defect tracker. I like Jira's filtering options and graphs:

The last concern, what if your management will not like this idea to submit issue reports automatically? And here is your option - deploy your own bug tracker instance, establish project and required break down (components, releases, builds...) and integrate with test automation suite. Enjoy with your new reporting tool and don't forget to show your solution to everyone potentially interested.

FunRelax on FR: reflect Dr.House's "diagnose the undiagnosable" on Testing

2010-01-22T08:24:00.000-08:00

Hey everybody who visit me on Friday!
It's friday, you know - so relax at the end of business day!

Did you ever try to draw an analogy testing with diagnosis in medicine. Actually we work over similar hard problems occuring in our practices. Sure, usual common scenario is simple cases which we treat in routine scope. But sometimes we meet with challenge in our discovery - without that our job is boring. Wanna more? We both work on finding out defects and their root cause - medicine reveals them in health, s/w testers run research against software products which are also complex system. Not so complex as human organism but anyway...

House uses investigatory method is to eliminate diagnoses logically as they are proved impossible like did Sherlock Holmes. Or in proper terms inductive reasoning. Actually both characters are very similar.
I think Software tester should be able to use this approach too, though it's limited by personal mentality. Testing is continious research resulting in discovery - move ahead in your mastery.
But today - enjoy upcoming weekends!

Effective Quality Assurance as Driver towards Innovations or Building Сonsumer-Oriented Software Products

2010-01-20T02:31:00.000-08:00

Effective Quality Assurance as Sriver towards Innovations or Building Сonsumer-Oriented Software Products

View more presentations from PeterBiz.

Non-verbose list of testing tools

2010-01-19T01:28:00.000-08:00

Recently found the resource trying to list all Functional testing tools
http://www.opensourcetesting.org/functional.php

and all Unit testing tools
http://www.opensourcetesting.org/unit_ada.php

20 Essential design patterns for automated testing

2010-01-18T02:10:00.000-08:00

I’m follower of building automated testing by analogy with s/w development in most of practices. Some practices are being grown initially from testing or properly improved in testing as legacy development practice. Actually, I don’t like to separate professional automated testing and development. We produce the same code but having opposite purpose – to check and verify the code of developed software – it does not matter what kind of interfaces we use upon testing. Anyway we test application and by this we test s/w code itself either implicitly or explicitly.
Design patterns have high importance for building cost-efficient products with high-degree of reusability and based on component oriented models. Design patterns provide abstraction and robust platform for reusable s/w design. My intention is to think over what existing design patterns fit to automated testing. Let’s do this!

As a fan of Web 2.0 and Wikipedia, I’ll refer on definitions and articles by linking to Wikipedia pages. See full list of design patterns on dedicated page. If you need to check details on a particular pattern, just click on appropriate link in Table listing all patterns within section “Classification and list”.

Creational patterns

Abstract factory. This is a proper mechanism of building abstraction layer for derived object types. For instance, it may be useful for implementation a framework layer responsible for working with different control types (e.g. MFC controls, pure Java controls, .Net controls,…) but operating as a single interface.

Builder pattern can be employed on test data construction and as a helper in many cases. Also you can use this approach for test suite initialization, stand preparation

Factory pattern can be used in any context for simplification object instantiation. There are a lot of places where you can fit this pattern for auto testing (dynamic and static test data instantiation; logging, reporting, test stand objects, etc)

Lazy initialization can be used to reduce expenses of keeping objects in memory. The best way is to combine this approach with object clean up once it no longer is being used.

Prototype as creational pattern simplifies object instantiation and avoids sub classing like in abstraction factory model

Singleton is a protected method of keeping only one object instance in memory. Where can be used: all objects which should follow through whole test cycle or round, for example, logging and stack trace, high-level test data, environmental data, 3rd party objects initialization to keep resources. If an object is dynamic one, you have to provide special interfaces like update(), rollback(). Also this pattern is proper way for developing high-performance models, optimization resource allocation and expenses and in environments with weak or absenting garbage collection and memory clean up.

Structural patterns

Adapter or wrapper can be used anywhere for resolving data compatibility and interface simplification. Simple and basic pattern

Composite patterns can be used for representation complex object like current test data set in a tree model (one to many relationships). I prefer to rely this approach on XML test data presentation as structured object model

Decorator is proper way to extending functionality without affecting original code. I suggest to use it as helper or quick adding workaround s and backdoors

Proxy as intermediate level for access control. I’m not sure about usage context in test automation but as this is essential one, I include this to the list. Who knows, maybe this guy can be used in some specific tested applications.

Facade pattern. I like to use this pattern for managing test data through uniform interface so that I don’t take care about used test data native. All specific code of data native processors placed somewhere inside, while decorator simply returns an object. Very handy.

Behavioral patterns

Chain of responsibility is a proper pattern for designing own logging and stack trace
Command pattern. It is extremely valuable for building keyword-driven test automation frameworks where provided interface operates with macro languages. Macro is a composite state which implements completed and composite User step which is actually intermediate level between granular steps and completed workflow (like Use case). The whole workflow (e.g. Use case) is being implemented in test scenario designed from a certain macros (steps).
Another usage of this pattern is testing transactional logic by mirroring some part of transactional model of application itself on test code.

In joint use with Abstract factory, you can build a layer for typed control of mixed technologies. In my practice, I build object loader and abstracted invoking object properties and methods for supporting different COM object versions at fly. You can go ahead and implement similar abstraction for different technologies, plus empower your design by invention composite commands for batch controls processing. For example, wizards processing by invoking meta-commands (like “set up default settings”)

Interpreter can be used to simplify working with custom string patterns. It’s really useful for substituting using predefined templates and for running code pieces using Eval() method
Iterator is essential for handling corresponding controls, like grids, tables, custom lists. Basic ally, iterator should have first(), last(), next(), add(), remove(), isLast() methods.
Null object is a basic and helpful approach of default value implementation. I can state – this simplifies routine test development.

Observer pattern. I advice to be hand on with this guy especially for testing AJAX applications and other ones which are asynchronous in nature. This pattern can be used in designing event-based test auto9mation frameworks.

State pattern. I did not use this one, though I believe it’s a basis of behavior-driven test frameworks

Strategy pattern can serve for switching algorithms using polymorphism or reflection (btw, JS implementation is really fun). The Strategy perfectly correspond with Open/Closed principle

Template method is a generic for designing reusable code (not only object-oriented) with overriding. I’d like to try this template to implement advanced test data engine.

I intentionally left TDD patterns out of this scope, since they are really specific ones, moreover I tried to highlight them (stub, fixture, fake, mock, object mother, object registry, assertions) in a separate post. Finally, most of the mentioned patterns are implementable in all modern programming languages (both OO and scripting, like JS, VBS) .

Well, enjoy designing patterns and inventing your own!

Brainstorming or flip your mind

2010-01-15T09:32:00.000-08:00

It’s well known about brainstorming as a powerful method to generate ideas, practices, grow team spirit and as result reinforce project timeline by sharing thoughts among team members. Sure it’s amazing way to motivate people, calm down some guys and optimize people effort. Indeed regular brainstorming sessions allows mentoring team indirectly, reveal talents and identify weak players for further decision making and training. What are the key success factors of brainstorming session? Here is my vision on that:

1. Be prepared and relaxed – it’s not reporting meeting with your boss. No bosses in brainstorming – all are partners.

2. Every participant should feel freedom to share and voice thoughts and ideas. Don’t make brainstorming as a necessary event – people don’t like when they are constrained. Say NO bureaucracy!

3. Everyone should feel s/he is wanted to be listened. Interruption and switching to other speaker makes everyone unmotivated.

4. Relax audience by jokes, fun plans and stories; imagine you are child and plays how you want

5. Flip your mind and try to walk outside your habitual outlook. People often see things as in tunnel missing lots of things beyond. So be observer 360 degree.

Someone should take about collecting of brainstorming outputs as all this matter may be useless today and come in handy tomorrow. Who knows? Things happen. And yes, even if you turn brainstorming to holiday or team building, you will see how the sessions appear more and more productive.

Good luck in generating ideas!

Advanced slides on QA and automation

2010-01-15T01:45:00.000-08:00

As a big fan of Web 2.0 and social media, I like to find out some sort of interesting artifacts, people and trends. Recently I walked through SlideShare.com and found there some interesting presentations about QA, test automation and fitting QA to business. Once again I receive evidence that Internet is a heap of junk where everyone may find what they need and sometimes very very usable - the essence is to use mediators skilfully (search engines, forums, catalogs, reviews, social nets, etc.)
So let's take a look at some interested selected presentations, perhaps there are some amusing details for you?

Automated web application testing with Selenium

View more documents from techdude.

Role Of QA And Testing In Agile

View more presentations from Naresh Jain.

Automated Testing vs Manual Testing

View more presentations from didev.

Software Testing

View more presentations from Ecaterina Valica.

Software Testing: Application And Script Independent Automation Framework: The Need For Data Normalization

View more presentations from guest0efb5e.

Software Testing Life Cycle

View more presentations from gueste730d5.

Helping in forming QA and testing KB on Wikipedia

2010-01-14T08:55:00.000-08:00

I'm working from time to time on contributing to Software testing portal on Wikipedia (http://en.wikipedia.org/wiki/Portal:Software_Testing)

If anyone can help to expand this KB - please don't hesitate - let's do it. Wikipedia is free space but high-quality content is a must.

"Quality is like health: you can't buy it. Hovewer you can bring it up and then keep it up diligently" (from myself)

Fun but useful Testers categorization

2010-01-14T08:48:00.000-08:00

Check this paper out! IMHO this break down may fit to likely any profession.

Where you are in your path? may be you jumb from one category to other?

End-to-End (E2E) testing with white-box technique

2010-01-13T07:23:00.000-08:00

The importance of system testing also known as E2E testing is very high especially on Product/Release delivery. This type of test basically requires running a continuous cycle against full path of system(s) under tests. It involves integration testing with 3rd party components, internal systems and interfaces as well. For example, test is running on User presentation then acquired results are used for verification in back-end system. I prefer and suggest continuing this effort by checking collected data using grey-box technique on DB side, or checking XML responses from Web services, queuing system or any other sort of object. Surely it’s challenge for those who are not hand on with specific technologies but learning in action is the best practice than anything else like books reading.

Another interesting thing is to automate E2E testing as complete cycle which follows data from birth to its death (CRUD: create-read, update, delete). In my previous project we had developed such monster. The basic idea behind the code is to collect all actual data from front-end tests. After that this data is picking up with new test project from SVN for feeding all the bunch of XMLs for verification in back-end system, checking DB views and pinging Web services. We used XML format as perfect data object which supports many-to-many associations and structured well for querying via XPath.

Measuring test (manual/automation) effectiveness

2010-01-12T03:33:00.000-08:00

The main metrics of testing quality are coverage and number of rejections. In terms of software testing, we have to move toward minimization of invalid found defects, minimization of non-found valid defects and maximization testing coverage. The same is valid for automated testing but basically measurements being performed in relation to manual effort, while manual testing is benchmarking against UAT, defects found in production by users and stakeholders.

Scientifics interpret the same notions in terms of Errors:
Error type I – false positive (false alarm) – when reported defect is invalid.
Error type II – false negative (missing) – when defect was not revealed in testing.
See more details on this theory on Wikipedia [http://en.wikipedia.org/wiki/Type_I_and_type_II_errors]
There are special statistical measures which operate with those notions (see below an excerpt from Wikipedia.org)

Full details on that see here: http://en.wikipedia.org/wiki/Sensitivity_(tests)

So one-two metrics can be employed for both manual and automated testing analysis in time (for instance showing up in QA reports as curves on a graph). I would select Accuracy (ACC), Precision (http://en.wikipedia.org/wiki/Accuracy) and False discovery rate (FDR).

For automated testing, I would suggest to measure against manual testing considering test automation coverage but not overall (manual+UAT+Production), so that false positive (false alarm) is when test automation reveals wrong defect, false negative – when automated test did not find defect when expected due to auto test covers that functionality.
To test your auto test :), you may run pilot analysis using so called defect seeding technique.

Couple excerpts from memories (JS, TestComplete)

2010-01-12T02:59:00.000-08:00

I like to have facade for abstraction data retrieving from any supported source. Generally I used either CSV or XML as data natives. XML data source might contain instruction to build a randomized string by substitutable pattern or have instruction to run a DB query or finally the value might be simple string. This is a simple decision-maker excerpt (sorry this blog does not allow to insert well-looked code snippets):

Another useful code which may help in tracing and problem localization is a custom exception handling wich also can be used as call chain mehanism in order to use as stack trace (Log4j is much better for this). The snippet below is serving as exception stack trace but that method can be easily transformed for use in any context.
Btw, TestComplete 7 has cool built-in stack trace so in that way problem localization becomes much quicker.

Free web rendering services: for testing, SEO and just for fun

2010-01-12T02:31:00.000-08:00

Good time to share some Web-based tools which are very useful and handy to have in your bag!

1. Have you ever run compatibility testing against different browsers/OSs? It’s very time-consuming and routine operation. http://browsershots.org/ makes all that automatically in distributed way. Just feed needed page and wait for awhile. Very handy!

2. Would you like to take a look at how any site was presented in Internet? WayBackmachine (http://www.archive.org/web/web.php) is a powerful crawler of web pages, so that you can see site history in time. Also it may be useful to anyone who needs to check a history of a particular site for fun or for example, to check if you can trust this site or not (like Company history and survey). One limitation: if a site prohibits to be scanned (settings in Robot.txt), Waybackmachine could not return you result

3. Essential tool for SEO is http://www.seo-browser.com/ (SEO-browser). Check how a site is seen for search machines.
4. Site performance optimization is very important to make it usable and reliable for visitors. So the following services can make analysis and advices on that:
http://www.websiteoptimization.com/
http://tools.pingdom.com

5. Want to know Internet speed and check your ISP quality of services? Run this guy http://www.speedtest.net/

Adopting TDD practices into white-box test automation (part #1)

2010-01-11T20:18:00.000-08:00

I will base on Lasse Koskela's book and Gerard Meszaros book plus some my personal knowledges.

So the first part will be analysis of “Practical TDD and Acceptance TDD for Java developers”.
As usual I will try to be short and straightforward. I will paste some excerpts which seems are fit to black-box auto testing or can be adopted down the road, e.g. on new framework development. Let’s take a look on interesting excerpts below.
“A good test is atomic.
A good test is isolated.”
This sentence sounds reasonable for all types of tests – test should be granular, isolated (for user as test runner). My addition – a completed test round should leave the world in the same state as it starts.

“That’s what I call the power of imagination! And it has a name, too. It’s called programming by intention. Programming by intention, the concept of writing code as if another piece of code exists—even if it doesn’t—is a technique that makes you focus on what we could have instead of working around what we have.”
Good engineer should be able to follow this rule – be ready to use imagination (write test before actual code).

“Take a moment to think about what we could improve in the test code. Duplication. Semantic redundancy. Anything that jumps out. Anything we perhaps should clean up.”
Refactoring is important step on improving work entirely.

“Happy path vs. error situations
As a rule of thumb, I drive for the happy path first before proceeding to handle exceptional scenarios, such as exceptions thrown by third-party APIs, malformed input, and so forth. The reason for doing so is mostly about value. Even the most robust system capable of handling any conceivable error situation is practically useless when it isn’t able to perform its basic function. On the other hand, even a system that, for example, is prone to crash if the network is temporarily down still provides value as long as the network isn’t down.”
Agree, first to implement a logic of happy path then all the rest branches.

“To summarize the essence of fixtures, they’re the whole shebang—the state of the whole runtime environment—rather than just the instance variables of a test class instance and the internal state of those objects.
…
Experienced TDD’ers are often good at creating focused tests. A key to being able to have a focused test is to have a suitable fixture that brings the system and the objects involved to a state in which the test doesn’t need to do much more than a single assertion call, possibly after first invoking a one-liner on the object being tested.”
Fixture is a essential pattern for robust and efficient test frameworks.

“Stubs are essentially the simplest possible implementation of a given interface you can think of. For example, stubs’ methods typically return hardcoded, meaningless values.
Fakes are a degree more sophisticated than stubs in that they can be considered an alternative implementation of the interface. In other words, a fake looks like a duck and walks like a duck even though it isn’t a real duck. In contrast, a stub only looks like a duck.
Mocks can be considered even more sophisticated in terms of their implementation, because they incorporate assertions for verifying expected collaboration with other objects during a test. Depending on the implementation of a mock, it can be set up either to return hardcoded values or to provide a fake implementation of the logic. Mocks are typically generated dynamically with frameworks and libraries, such as EasyMock, but they can also be implemented by hand.”
Those are again good patterns for employing in routine tasks. You may even design them unknowing that you use mock. At least the definitions are helpful for communicating on the same language of terms.

“Aiming at helping you avoid creating testability challenges for yourself, here are a few simple design guidelines that we should pay attention to:
- Choose composition over inheritance.
- Avoid static and the Singleton.
- Isolate dependencies.
- Inject dependencies.”
I agree with all these recommendations but second one - sometimes static and singletons may pay good result for quick and temporary implementation. Also singleton is self-protection mechanism for reloading, instantiating more objects than one.

Also Koskela provides some basic unit testing patterns that may be used upon building custom test automation frameworks. They are:
• Assertions as exercises metods
• Parametrized method creation (this guy easily implementing in JS :)
• One more interesting pattern is Object mother: "an object or a set of objects that does the following:
- Provides a fully-formed business object along with all of its required attribute objects
- Returns the requested object at any point in its lifecycle
- Facilitates customization of the delivered object
- Allows for updating of the object during the testing process
- When required, terminates the object and all its related objects at the end of the test process"
• Finally I like provided by author combination of Fixture setup, Teardown and Test object registry as a dedicated classes. This approach simplifies instantiation, automatic object cleanup by triggering, protected access and all these together truly correspond with single responsibility principle.
Regarding techniques on testing legacy code, there are highlighted just some common aspects. So I would recommend reading some special books like "Working Effectively with Legacy Code" by Michael Feathers, and books on code refactoring such as “Refactoring: Improving the Design of Existing Code”, MartinFowler et. all.

Robot Framework

2010-01-11T08:55:00.000-08:00

I found that interesting but not yet tried this framework.

See details here
http://code.google.com/p/robotframework/

documentation:
http://robotframework.googlecode.com/svn/tags/robotframework-2.1.2/doc/userguide/RobotFrameworkUserGuide.html

Btw, I did not find out there whether it works with XML and schemas or not.

Someone tried that guy?

Who is ideal test automation engineer?

2010-01-11T07:10:00.000-08:00

Ideally highly-professional test automation is hybrid of manual QA/testing and tests developers. I’m sure this is a rule for both project contribution and personalaqXC career path. If you are skilled QA analyst, test designer plus has broad knowledge and experience working with test automation professional environments, frameworks and besides all that may combine manual and automated effort – you are valuable for talent hunters!

As QA analyst the person should have real experience in comprehensive testing, test design, planning, and risks assessment. Also this role requires being great team player with good communication skills, creativeness and non-stop drill down against AUT – your desire to break the system should be as instinct. So that testers have some freedom to break software using any tool and technique and this makes the role very interesting as you become investigator and subject matter over a particular application and business domain.

Talking about automated testing, the core skills per me are:
- Knowledge of internal mechanisms of interacted objects (e.g. DOM model, COM object, MFC, Pure Java). Basically you have to be able to invent efficient object recognition model with high robustness on intermediate changes and proper control identification (avoiding concurrent identification)
- Knowledge of test automation tool(s) itself
- Familiar with programming language on which you are going to write tests
- Knowledge of development practices, principles (like single responsibility Open/Closed, reusability, OOP, procedural approach…) and common design patterns (singleton, decorator, factory, wrapper, etc)
- Regular expressions. That’s a very powerful tool for us. Must to know concepts and how to use. Ideally you are ready to write any expression at fly.
- Knowledge on performance testing, modeling and identification of bottlenecks (for load testing)
- Knowledge of theory and practices on relational databases.
- Good knowledge of one on-demand RDBMS, querying and performance analysis
- XML –based technologies and service model (XML, XSLT, XSD, DTD, WSDL)
- Experienced in version control systems
- Experienced in tuning application/web/DB servers
- Broad knowledge of operation systems (server and consumers solutions) as advanced user or as administrator.

Yep, I forgot to mention that QA should be in-time and clear reporter (defects, issues, test logs). It appears the list of skills and responsibilities is long and ideal. Surely project needs defines certain set of skills and responsibilities.

Design patterns for automated testing

2010-01-07T01:01:00.000-08:00

This post will be filled in soon with design patterns implementation that perfectly fit to our duties. As for now just announcing as reminders for myself.

Wwwidgets: socialize and share

2010-01-06T10:09:00.000-08:00

Widgets are stedy becoming more and more popular and valuable tools for web developers and for social markeing.
Here are my findings of most useful and easy to deploy widgets (the list is based on my opinion):
Facebook group:
I found useful Share , Live Stream and Fan-box

LinkedIn
Just Profile widget looks interesting for now. Also for corporate sites may be it's good point to add Company insider special widget

Twitter and TweetMeme
I think a pair of Profile widget and ReTweetbutton are enough for increasing conversation

Digg
I found only one their widget http://digg.com/add-digg

Special buttons:
my favorite button which gathered all stuff together (<200 sites!) is AddThis - here is example http://www.addthis.com/web-button?type=bm&where=website&url=&bm=tb3&analytics=0

Whatever you will use, my suggestion - don't overload your app by these plugins. The golden rule - 3 on a page are readbale (my blog is overloaded :) Yeah it goes against good practices.

I will continue this conversation about widgets (broken down by destinations) in the subsequent posts some time later. Please share all your knowledges about social and sharing widgets here. (I tried to cover most of popular systems but MySpace - i found API only)
Disclaimer: this post tells about light-weight widgets only and does not consider serious APIs.

Automated security testing

2010-01-04T11:55:00.000-08:00

Mostly testing relies on automated suites for functional testing according to provided test-cases (that can be used for manual replication too). I think it's good point to thinking about porting automation concept over other types of testing. First coming to the mind are: localization, compatibility, all types of performance, and all types of security and safety testing.

The last two guys - security and safety - i believe are proper candidates to be automated. Why? Because of security flows can be introduced while any development and maintenance phase. You might polish out your s/w from security flows but new change request can bring a defect in design which may subsequently make your s/w unsecured (e.g. improper handling cookies or turning off fields validation common mechanism)

Such tests like SQL injections, XSS, session hijacking, cookies substitutions, overflows (as negative tests) and unvalidated I/O, access and permissions control can be easily simulated by test automation and can be run repetitively.
Btw, denial of services can be modeled with load, stress and volume testing.

The resource owasp.org provides tons of info about security and hacking on Web. You can find there lots of helpful data.
They have a comprehensive list of potential vulnerabilities in s/w:
http://www.owasp.org/index.php/Category:Vulnerability
and a category of attacks:
http://www.owasp.org/index.php/Category:Attack
I strongly appreciate this society for their job!

Handy add-ons to Firefox can simplify your process of developing and running hacks. The simple list of essential tools:
Tamper Data
SQL InjectMe
XSSMe
View Cookies
CookieSwap
HackBar

Test automation strategy as part of QA strategy

2010-01-04T11:24:00.000-08:00

Happy new year all!

Right at the start of this year I have a challengeable task - detail test automation strategy within overall QA strategy on anti-virus software (new Release) in our Company. Since we have taught schedule and intentions to improve processes by thought out documentation, formalization and model optimization, this is really important to plan and reflect all things together in a single artifact - test strategy (calling also test approach). Another reason to combine both manual and automated efforts - is to ease traceability between risks, dependencies and schedule over whole department. Applying mutual practices prepares manageable and unified processes.

I'm going to highlight the following items in that test automation strategy:
- Test objectives
- Assumptions
- Types of testing on automation
- Test tools
- Resources (money, time, skills)
- Application under test specific details (technologies, quality attributes...)
- Test environment (test labs, stands and infrastructure)
- Test requirements (requirements to test cases on automation)
- Entry/Exit criteria
- Project plan (high-level and/or detailed as reference on appropriate artifact, deliverables and milestones)
- Risks (impacts, likelihood, mitigation plans)
- Review strategy

Well done! Is there something more to add or making this list redundant? Your advices are always welcomed!

On stabilization test suites. No fuel but destination is too far

2010-01-02T10:28:00.000-08:00

This crazy activity is not only about debugging. More sophisticated your test automation framework and infrastructure – harder test suites stabilization. Under test suite I understand here a set of test cases, ready_to_go code, prepared environments, service utilities, auto running and so on. So that is a completed end-to-end package.
Right now our brave team works on stabilization of test suite developed on overdesigned framework. And this seems eats about 60% of time on full task. As most of just have started to understand this framework, we hope this time will be significantly cut off on subsequent test suites but IMHO we will stabilize and debug 40% at least of total time on implementation. Will see.
This activity requires strong concentration and tries your nerves on tuff. Especially debugging is time consuming because of auto tests runs workflow with long chain of calls and usually debugging runs altogether with real data. I rather think debugging in test automation is much difficult than in development (if test framework is really framework) due to:
- Mostly automated tests are developed on scripting languages and unmanaged code
- Code is not compiling and running inline (so first error you may find somewhere deep)
- Weak garbage collection
- Weak test development environments and debugging mechanisms
- I did not yet meet good stack trace for black-box automation (TestComplete works on that and it becomes better but it’s not yet cool as in Log4j)
- Unstable test engines itself and communication with OS resources fails sometimes
- Some scripting languages have poor automatic type casting (e.g. VBS)
- Difficult or impossible to integrate new technologies to get higher abstraction (ORMs, patterns, logging, self-testing, reporting, external APIs with intellisense and ohers)
Also test automation suites should have lots failover and preparation mechanisms to be more robust, for instance, initialization, clean up, terminations, checkers, workarounds, turn over, synchronizations, waiters and alternative recognizers, behavioral logic, logs lockers/unlockers, wake uppers, exit gates… Huge list and finally most of code should we wrapped in try-catch-finally (it’s not about VBS, it has another pattern). If you are novice in test automation read on this. Professional automated test is a product as we know about software.

Statistical and numerical analysis for QA and testing

2010-01-02T02:45:00.000-08:00

Stat analysis appears very powerful tool in a hands of analyst if processing data set has significant number of entries, also the set should have normal distribution otherwise the processing will output wrong results. Let say if you processed all data from a segment A, then processed only 10% of data from B and 1% from segment C, resulted joint product of these observations will be uncertain. The bottom line: all your deals with statistics should be careful since it involves wide spread of theory and practice.
Statistic and numerical analysis are very useful for the following QA practices:
- density defects measurements
- predicting defects quantity in developed s/w
- DRE and other metrics
- ROI calculations
- planning
- probabilities and likelihoods
- design test data and steps (tools like pairwise testing, finite state machine, boundary testing, matrix of transitions and so on)

For performance tests:
- avg response time, std, min, max
- measurements from uncertanity and independent sources

For automated functional testing:
- ROI (basing on accumulated data and assesement)
- failure probabilities calculations
- for building effective test automation suites - design parameters permutations for achieving best coverage and to reveal hidden defects
- random data and variations
- modeling system with feedback(s)

Can you add some more?