Tuned up Agile in test automation

Once again very taught schedule and unreal project plan? I will add something more pessimistic - Not enough resources and limited budget. At the same time we have the same classic project triangle (“choose any 2 from 3”). I believe you ever face those challenges and mitigating plan can’t help you I think. I met these embarrassments over again right at these days, before New Year. And I don’t want to leave for holidays with this unresolved stuff. How I can force against? Is there a sword which will help me to keep faith and win? I need it, you know. I can’t fit to schedule!
First coming to the mind is to begin planning more pessimistically and make cycles longer with rare milestones. Fantastic but customer wish to trace each our commits and don’t want to hear about long-term technical promises and issues. That’s fine since he dictate a model.
And this seems normal approach to test automation entirely because nobody wants to wait for completion whole bunch of test cases and only after that to get first chance to run them. As usual, they want to run almost each newly created test case right within current scope. So my suggestion if your situation is similar of mine think about incorporating agile. I practice this as it’s perfectly fits to common targets and intention of automation testing groups.
Let’s see on some of key Agile(Scrum) moments and figure out if they can be used or not.
• Short iterations and often deliverables. Since we have to run right after implementation, this approach works by default
• Fuzzy responsibility and Scrum master. I think it fits even better than for s/w development.
• Everyone may contribute to everything – no roles. We are all test automation engineers with different skills, so one may perfectly communicate, one may design data storage, one may grow infrastructure tool, etc. And everyone can change other one.
• Scrum meetings and communication simplification (desks, video, verbal, sticks, calendar…). All this speed up overall implementation process.
• Clever reporting and coverage. If test automation does not have that on bard, its value looks not high.
Anything else? Of course, I just showed which I recalled while writing this post and if there are some missed points, don’t consider my faults seriously. Feel free to share your vision here.

Make your contribution visible () or Die() [php]

Hi!
Some time ago I had a necessity to think over how to make my management/Customer happy with provided test automation services. I should say – it was really challenge, especially in my case when QA manager does not know product technical details and not going to study them, also he never worked on test automation. Once he was in a bad mood or did not understand something from reporting and test framework, he escalated to higher management about his complains. Thus he built a shield for himself to avoid mess-ups of his weak knowledge of test automation.

If I never developed special tool for clear and effective reporting and share among higher management (we developed Web app with database in backend), I believe he might easily terminate test automation in organization or maybe initiate restructure. So bottom line: contribute to success and make your contribution visible as many persons as possible in your company, otherwise someone will control over your team simply by voicing concerns. See my article highlighting some sort of test automation reporting:
And by the way, test automation team should not be tool for manual testers!
Have a fruitful bugs hunting!

Pure and reach article about JavaScript from my frined (in Rus)

My friend Evgeny is a fan of JS and I found his article very reach and professional. I also fun of JS and related libraries.
Check that article out here:
http://ru.wikipedia.org/wiki/JavaScript

Data natives as parameterization and core of framework

Before starting invention of test automation framework, one of very important question is what data natives you will use. Writing down these natives, consequently will simplify building framework on lower levels. My preferences listed below:
1. Test data – XML (schema is optional)
2. Big test data collections with the same structure (spreadsheets –CSV, XSL…)
3. Test logs – XML + HTML
4. High-level Test reports (Database + web GUI)
5. Configurations and settings - XML
After that appropriate routines (data drivers) should be developed as part of framework essential libraries. For instance, wrappers for working with XML through Xpath, methods for connecting to DB, retrieving data and closing connection, parse CSV files and so forth. Good luck!

Test automation ROI: developing calculations (tips)

Here are my tips on ROI calculations for a test case (more likely you will count it for test cases set):
1. Identify estimated number of runs for this item
2. Identify reusability on the developed code and take it into account
3. Identify code complexity, your lacks
4. Identify possible risks
5. Estimate how long it will take to deliver this item as automated test
6. Estimate running time on this test case manually for a single run
7. Put all results from 1-6 into your formula – get resulted ROI.
Ooops I forgot to mention that before initiating this difficult process you have to have appropriate coefficients which will be used in formula. These coefficients are statistical data which can be gathered in post project analysis or can be taken from any other source (e.g. your concurrent). Don’t’ forget that ROI is a useful tool which may simplify you making decision whether you will develop test case or not and prove your vision to management with certain calculations.

Naming and coding conventions: are they important for test automation?

It does not matter how many lines of code you predict to be written and how many people involved in automation, the approach of writing code should be single and right. For this purpose I recommend to invent naming/coding convention which highlights all common principles and patterns as well as very specific to your environment and AUT. Once you get it right, distribute and follow this artifact, you will see how your team will advance and work efficiently. Actually naming and coding convention is something like communication plan inside of your team, thus your players will talk on the same language, minimizing overheads on misunderstanding among people and organic speech (bla-bla-bla).

Ideas?

Source control and change management in test automation

Since test automation is a combined activity of test code development and testing itself, all appropriate practices and methodologies of s/w development can be well adjusted to automated testing, excepting some of very specific things. Pretty sure, your test automation team can and mostly should to use source control and predefined change management; branching, merging, versioning, traceability, measuring coverage, regular committing and commenting are all about highly professional test automation.

Btw, as you move forward - codebase growth and accumulate defects in source code, your team can consider to use defect tracking system and follow all proper activities related to this approach.

So version control requires to use special tools (SVN, CVS, new distributed systems, Perforce), training, invention repository structure and convention. Basically for test automation you have to come up with proper automatic traceability and integration with other artifacts (requirements, test cases, test plans). Also in right way, you have to think out repository structure to achieve high rate of code reusability and suitable logging/reporting. All these things together build your source control versioning and management. Change management should be based on agreed policies to be followed by everyone in your group.

Test automation ROI: don’t formalize everything

As a guy who has to sell test automation to my Customers, I have to design some measurable data for showing out ROI of requested feature as a candidate to be automated. But this activity is required from time to time. I believe that formal measuring ROI (using formulas and statistics) on each piece of functionality is redundant steps in QA. Mostly, management and technical experts are familiar which feature set must be automated because of: this is a core of application, these tests will be run tens or maybe hundred times, these tests will reduce cost on overall testing and QA, these tests are easily expanded by parameterization… This list maybe long but I tried to list top priority items.
Another set of functionality stays versus “must be automated”. If it does not make sense automating some feature, why we need to ask formula? Sounds good. The remaining features probably should be passed through your ROI formulas. That’s my vision. What do you think?
PS How to calculate ROI for test automation candidates, I will try to highlight in separate topic.

Choosing right test automation tools

Many Companies run special activity of investigation on choosing proper test automation development environment. And this big deal inquires lots of evaluations, reading reviews and surveys, asking gurus’ advices, modeling integration with other tools and trying to figure out how to fit a particular tool to overall test automation strategy.

As a first step I suggest to put all your objectives and priorities on a list. Then, try to segment all considered tolls from top to down. For instance, let’s begin breaking down by all tools on open source and proprietary, free and requiring to be purchased. After that, try to go deeper, i.e. segment tools by features, supported technologies (tip: don’t assume that if a tool support a lot of technologies is the best choice, you may be glad tool which supports technology on which your AUT is being developed), supported environments (OS, browsers and so on – this matter is especially important), supported test automation frameworks, parameterization, object recognition, logging and reporting, technical support and documentation. Also study these tools on supported development languages. I strongly advice to sharp your focus on those tools, which support modern and powerful languages (Java, JavaScript, Perl, Ruby). Surely, there are some tricky moments more, like tool stability, integration with other tools (bug tracking, test management, reporting frameworks), and possibility to run simultaneously and remotely. So that’s enough, I think you will have some more specific reflecting your particular needs.
Once we put all things together, we can rate the considered tools and as result making decision likely will be selection between 2 or 3 good tools.

And a last sentence, fit test automation tool with your team skill set, if your team is skilled OO developers, don’t spend money - select open source, your team will take care about developing good test automation framework without needs to use clickable and drag-and-dropable interface. And vice versa, if your team just analysts w/o development background select something easy to use w/o needs to code, but in this case test flexibility rate is almost zero.

ROI on automation testing department establishment

Before stating to your management that test automation is on demand and may help development process by reducing cost on testing and improving product quality by continuous testing with short cycles, try to predict that all your benefits may not make sense until you show some numbers.
The most important number is ROI; the next one is a graph of these returns in time(schedule of progress). Both metrics will reinforce all your words and as result your presentation will have positive impression.
Assuming your product has 1000manual test cases and thought-out flexible and parameterized test automation framework. Firstly, count people needs, and then design a budget. Secondly, based on a staff count, schedule test coverage and make graph on this schedule (X- data, Y- number of covered test cases). After that estimate approximately number of runs and estimate predicted number of discovered defects. Finally calculate ROI based on all these calculations and metrics and then make a graph of investment returns in time.
Repeat all steps described above - -revisit people needs, recalculate your budget and so on. Finally, show your management 2-4 cases of budget/ROI. I like to give management alternatives – psychologically they are feeling that they have to choice something anyway [grin]. Actually, this is not tricky step, but is a good practice on making organizational/technical proposal.

Why test automation?

Indeed, why someone decides to have automated testing and someone thinks that automated testing is losing money? Both sides have their own motivation on that question. Please share your opinion and ideas in this blog post.

As for me, it depends on many factors; some of them: Company targets, approaches and methodologies to testing and QA, staff skills set, long-term and short-term objectives, company/department process model, budget, management priorities, AUT specifics, speed of AUT evolution, phases of AUT development. The last two are especially important – let’s try to figure out how to establish efficient test automation while AUT is under active and continuous development; but if some pieces of functionality are stable enough, those become likely right candidates on test automation by porting out regression test suite on automation framework.

From the past

That was my origianl paper to international conferenece (for which i forgot to pay but it was accepted). Nvertheless - it was my vision couple years ago!

Check it out

Have you ever work with heuristic frameworks?

Me – never, just heard about those and read some concepts. Actually, it seems to me these frameworks may be proper to some specific AUT and/or specific kinds of testing (monkey tester?)
Did you ever have exp with something similar? Share yours here.

Negotiate your plans with Customers or Customer will negotiate you

I strongly believe that keeping on track and up to date your customer is vital and works in the same way as in regular s/w development. If you have project plan but current situation does not allow you to fit into these frames, you have to update your customer and negotiate to revisit original plan. This is a proper practice and sliding schedule is a normal in s/w development Test automation is not different, this is also development but in test so most of s/w development and management practices are perfectly fit to automated testing.
The coordination and negotiation should be assigned for appropriate point person who will be responsible for connecting Customer with team, representing and defending both sides.

TestComplete – the best choice as environment for that money (as proprietary software)

I’m not going to list all TestComplete’s pros and cons and compare against others. Moreover I think many tools should not be compared at all due to TestComplete is too far better, e.g. modern TestComplete for 2000$ (with http load testing) and old-fashioned HP QTP for 15000$.
My opinion TestComplete is the best for now for black-box testing in Windows, particularly considering its very competitive price. If your needs are very limited and you will develop own framework, maybe choosing free and/or open-source tool is better decision.
If you need some more details on TestComplete or just to discuss about this blog, please post here. If you need professional consultancy, please contact me directly.

Introducing my blog...

Hey everyone!

I decided to move all stuff from different sources (tweets, forums, articles) into this new blog. It makes sense because of I can't control distributed things - no free time :)

I hope you will find here interesting portions of my thoughs. Also I will be glad to see your opinions and discussions here. Thanks in advance!