Automate Your Automation

I think if one claims that they have test automation, I mean full-fledged test automation, it must imply not only automated tests per say but it should include process automation. Automated tests (the code) are useful in same extent as manual tests which were never executed. It is just having no value unless it is executed at least once.  When we execute tests, we kind a returning back these expenses invested in test design, same for automated tests. However as automation specialists, we should go beyond of this point. We should leverage automated tests by setting up proper automation process which will command all aspects and flows in test automation.

What to be process-wise automated?

1. Test execution implies triggered executions - kicked of by Continuous integration server, on demand or scheduled. In state of the art project, it could be smart test execution - when only subset of test are executed which belong to a affected with new build pieces of software 
2. Test Reporting and test Results management. Keeping historical data in one place, in mid and long term provide you very valuable statistics, metrics and lessons learnt. Reporting is essential for managers and stakeholders, it helps them to make decision. So our job is to give them a tool which will build report for them and provide valuable information in one screen.
3. Results analysis and collaboration. Let your team mark results, exchange, share and tag specific test run results with one click; avoid effort duplication; keep tests stable and up to date by acting immediately; make it transparent
4. Test debugging. Helps you to avoid tedious and hardly manageable test code fixes and improvements, given tool should ease all internal collaboration and accelerate decision making process
5. Version control is irreplaceable in test automation. Do versioning, branching, tagging, etc!
6. Auto Tests deployment. Something should run in background to make sure that right and latest version of tests and framework is being executed. In advanced cases, it should be able to deploy test code which is compatible with specific software under test version. Sounds cool!
7. Test Lab, infrastructure deployment and roll up (e.g. virtual machines stand by and suspend - especially in case of dealing with virtual data centers, on demand cloud centers - basically where run time is run money off)
8. Environment and SUT availability monitoring (all data that can be precondition for whether start actual test execution or report fatal error and quit)

Gredy addresses most of these process automation aspects. Gredy automates test automation

Would you add more to the list of automation automation?

Everything changes irrevocably and hardly... Just surf's up!

It is ongoing and no way to stop these changes - I talk about big changes in s/w development practices and methodologies, approaches and trends. I don't think everybody realize how Agile had captured whole market, it made it softly and steady, you know, kind a "kill me softly". Killers finally will be killed by other gangsters but it is different story. Now we play slightly different game - nimble, quick, lightweight, not thinking about long term - just make it happen today, few sprint and release. For the time being, I worked for 3 crazy_about_Agile companies. And since I'm test automation guy, it was never problem for me; I'm like a fish in sea. Agile brings new concept of QA, at some extend it breaks traditional QA, it removes user advocates, it kills idea of independent assessment and verification.

However it brings new opportunities, here are some of these:

- test automation is integral and nobody can escape from that responsibility. No more manual testers! What's the heck?

- much more exploratory, ad-hoc, session-based, security, whatever-you -call-challenging or truly breaking testing. Awesome technical challenge instead of boring step by step testing

- quick bug reporting with tech insight. Do meaningful headline, attach logs and some piece of code maybe - you are set. No more rudementary 12 detailed steps to reproduce

- User evolvement - crowdtesting, dog food, beta, alpha demo, blah. Community rulez. The benefits - UA, A/B, functional, compatibility, distributed testing in one in one bottle. BugPub is a gate for real crowdtesting with win-win model

- No more kind a tools which cover all your needs. This expensive stuff is now called crap. So tools should be either free or cheap and right, I mean perfectly fit your needs

- No more bunch of managers which just double staff 

It is ongoing, it is right over you, just surf's up!

Market fragmentation trend - best in class tools for specific domain and no longer solutions that can do everything and nothing

I really amazed by this great trend which really flips and flops giant software enterprises. This is why they hurry to purchase small start ups and little companies with one but awesome product or service. These companies understands that their heavy solutions are not serving consumers so good and cheap as those little but flexible players. It is understandable that big companies desire to surround their customers by "one stop" solution, closing intentionally integration with other vendors and framing proprietary closed ecosystem which is eventually very costly caused by huge R&D investments to build this one-stop.
So now we as consumers use a lot of apps on our desktops, on web and smartphones (especially!)

This trend is sweeping software companies and IT departments themselves - we want only having deal with best tools solving specific problem and helping out to be productive + enjoy their look and feel. And it does not really matter whether this or that tool is open source or commercial - the most important is User experience, perception and passion to use that tool every day.

A quick example HP ALM (or any other ALM at this point on market) - a tool which can "do everything"... This tool attempts to fit all possible software entities (bugs, requirements, test cases, runs, test automation) into one sort of table view dogma. In essence this is boring, not domain specific - it is just like reflecting database structure on Web, bunch of fields and folder which can at least confuse and even scary. Sorry for negative example given first but it is really pain, costly pain.

The time for positive. Atlassian. This company create tools, specific ones, flexible, evolving. People love these tools. You can buy bundle, you can buy one tool. These tools are based on open standards, APIs, etc, so you free to build your ecosystem with tools and technologies of your choice. Let's go back to HP ALM, regarding integration, do they provide integration with CI, which test automation I can use with them apart from QTP, can I integrate with another nice bug tracker, do they support other requirements management systems? NO. Mercury tools gonna follow same mistake as Rational did after acquiring by IBM.

The problem of this kind a system/solution is not implementation, the problem is not even its cost, the problem is just idea itself, the idea of one-stop solution.

There are a lot and a lot great tools on market - a universe of them are commercial and bunch of them open source (especially technologies and frameworks)

Lastly, can you call Wallmart or Amazon one-stop store for you? Software market fragmentation is ongoing

Automation Metrics and KPIs - how to beautify and not sink

Post-mortem, yearly reports, metrics, KPIs, predictive analysis, graphs, %%%%. Again big reporting for big managers!

My 5 cents which hopefully could help to run off all that nightmare inventing that stuff.

Key metrics for showing test automation contribution and value:

1. Coverage - higher %, bigger scope, more influence

• against manual test cases
• against functionality/use cases
• against source code
• against API

2. Test automation ROI (notorious ROI - I wanna present you my approach for its calculation little bit later). Showing ohw automation saves money/time. Btw, at some point you can show a trend of your ROI instead of current or projecting ROI. For me, seeing dynamics is way better and clean than static numbers. You can have achieved only 90% ROI though if you will show me a steady progress over time, I can value it higher of current 150% ROI which was finally achieved in 5 years. More advanced way to show your dynamics/trend is to lay on 2 graphs on one plot: projected ROI dynamics vs. real ROI dynamics

3. Defecs Discovery Rate (number of found issues vs number of auto tests). I found this metric more representative if you will correlate this metric to the same metric but applied to manual testing. In this case you expose productivity of manual team against manual testing in terms of revealed defects.

In the example below, you can find more project-wise metrics and KPIs as taken from Gredy

68.60%	27	778	99.23%		2011-11-21 # 2.1234.23 - 50.00% 2011-11-22 # 2.1234.23 - 47.62% 2011-11-23 # 2.1234.23 - 46.15% 2011-11-07 # 2.1234.23 - 41.38% 2011-11-18 # 2.1234.23 - 40.00%	2011-11-04 # 2.1234.23 - 11.11% 2011-11-27 # 2.1234.23 - 16.67% 2011-11-09 # 2.1234.23 - 17.86% 2011-11-30 # 2.1234.23 - 20.00% 2011-11-25 # 2.1234.23 - 22.73%	TST_ID_876.004: 16 TST_ID_235.962: 15 TST_ID_511.669: 12 TST_ID_677.207: 12 TST_ID_173.008: 11
Avg. daily success rate	Number of  test cases in suite	Number of all test execution	Stable ("Ready") executions vs. All		5 Best builds/days  (by success %)	5 Worst builds/days (by success %)	5 most frequent  failing  tests  (total failed times)