Thursday, January 28, 2010

iPad launch : antagonisms


There were a lot of rumors and guesses about look and feel of being announced super new gadget with screen touch and multitouch revolutionary technologies and lots surprises for consumers. Yesterday they showed this as usually gracefully exposing mystery. Actually I did not pay attention on guesses how it will look like even from "trusted" sources. Why? Because of Apple's strategy is to run a few fake projects so that employment does not know what will be the real product on store shelf.


By this tactic Apple:
• stirs up public
• minimizes risks of thefts
So they launched iPad but results don't impress, moreover - mostly I see articles and posts with blaming containment. This one article seems better yet represents real thinks
http://mashable.com/2010/01/28/what-we-learned-about-apple-yesterday/

Will see...

Wednesday, January 27, 2010

An interesting post from Google's QA about patterns to testing

http://googletesting.blogspot.com/2010/01/interviewing-insights-and-test.html

I think this is good point and reflection but IMO professional QA guys combine all the best from this classification and care this suite in daily activities. I don't like one-sided approach as it risky to limit own vision. Also flexibility should be a part of career path since different projects expect different approach as well as thinking.

Monday, January 25, 2010

Tools: XML ORM for efficient parameterization

Test automation parameterization is a part of architecture, thus Object-relational mapping can significantly simplify your routine works and reduce risks of unprotected and unauthorized data access. Starting from scratch will require to learn and utilize some worked out technologies such as

- XPATH for retrieving/modification/inserting data into XML
- XQUERY as unified SQL-like language
- DOM model. This is like low-level or direct access to elements of DOM documents
- New technology from Microsoft LINQ

I rather suggest to use XML 1.1 namespace (see difference 1.0 with 1.1 here). Additionally, using schema binding to test data XML storages will bring you long-term benefits:
- automatic validation (opening within browser or loading XML through Microsoft XMLDOM:
oXML = CreateObject("microsoft.xmldom") oXML.load(sFile)
- unified designed data structure with advantages of default values, flexible implication of entities, values lookup and restrictions and many more pros

Basically you have to select a single approach: either your schemas defined in DTD or in XSD.
Certainly big test and long test automation projects with intensive and heterogeneous data (and possibly tested objects mapping) parameterization will profit from careful data/objects designing through XML schema.

You can build a hybrid model which will utilize several technologies in a single package encapsulating internal staff.

One more good practice is to split up logical pieces of huge XML into several ones in order to simplify concurrent usage and change control, also think about keeping schema(s) itself as separate document and include properly, so that final model will have 3 layers:

1. Schema definition (or multiple schemas)
2. Common or master XML (which binds schema(s), describes common data and includes particular modules):

3. Specific test data description (test cases, tested objects specs... per component/module/test level)

Sunday, January 24, 2010

Test Automation Dashboard in just One day


The value of automated testing is determined by reporting framework definetly at most. Your team must provide clear, in-time and customizable reporting for stakeholders. Nowadays data presentation plays vital role in assesssing whether your product is useful and useable or not.
The test reporting should be able to show various views of accumulated data. Someone will like to see graphs over all runs, someone will check daily runs carefully drilling down grids and logging lines, someone will want to see something like traffic lights (red, yellow, green).

Development such system is the big deal as actually full-grown project. I developed this one with my team 2 years ago. That was Dashboard which was utilized over all test automation projects. We built the Dashboard using XAMPP (Apache, MySQL), plus some extra JS code and jQuery. To plot graphs we used jGraph open source library. I really like this app as it is flexible, has pretty presentation, navigation, filtering. Moreover we implemented test running and test stands dispatching through the web dashboard interface. Though this infrastructure project had been running in background, we spend on that good portion of our budget and tehnical skills.

This challenge may frustrate if you have limited resources which are dedicated to developing tests only. There is a recipe! Likely your project uses defect-tracking system - so your task is simple utilization this tool for your daily runs. Atlassian Jira is perfect candidate to be reused for test automation as this very flexible and cusomizable system supporting a lot of technologies (just check 3-rd parties integration - http://www.atlassian.com/software/jira/tour/plugins.jsp). Nevertheless if you use Trac or BugZilla or ClearQuest - I believe (since I worked with them), you can apply the same approach.

I see 2 options how to deploy this integration:

1. Integration into project(s) structure with new ticket type (to isolate from regular project tickets)

2. Create new project wth required break down by components (test project or test level), releases, builds (e.g. each new run is a new build); fields customization and so on.

How to tune up:
1. Customize ticket fields
2. Identify rules on new build (e.g. each run) and ticket submission (each test case failure)
* Preferably your reporting should be granular with proper rules of counting failured/not_run/passed steps. The perfect approach is to trace auto test against manual test case (failure in one or more step(s) mean test case fail)
3. Create preliminary views in tracking system: spreadsheets, graphs using built-in filters
4. Think out roles, assignements, priorities and other useful fields
5. Teach your test automation framework to communicate with defect tracker:
- come up with interface: GUI, API, DB, Web service...
- new run submission
- new ticket submission (form filling)
- supplementary data retrieving/update/insert
6. Create a single-view Dashboard page as container of customized gadgets through filtering (or querying)

The final dashboard look and feel is limited by your defect tracker. I like Jira's filtering options and graphs:


The last concern, what if your management will not like this idea to submit issue reports automatically? And here is your option - deploy your own bug tracker instance, establish project and required break down (components, releases, builds...) and integrate with test automation suite. Enjoy with your new reporting tool and don't forget to show your solution to everyone potentially interested.

Friday, January 22, 2010

FunRelax on FR: reflect Dr.House's "diagnose the undiagnosable" on Testing


Hey everybody who visit me on Friday!
It's friday, you know - so relax at the end of business day!

Did you ever try to draw an analogy testing with diagnosis in medicine. Actually we work over similar hard problems occuring in our practices. Sure, usual common scenario is simple cases which we treat in routine scope. But sometimes we meet with challenge in our discovery - without that our job is boring. Wanna more? We both work on finding out defects and their root cause - medicine reveals them in health, s/w testers run research against software products which are also complex system. Not so complex as human organism but anyway...

House uses investigatory method is to eliminate diagnoses logically as they are proved impossible like did Sherlock Holmes. Or in proper terms inductive reasoning. Actually both characters are very similar.
I think Software tester should be able to use this approach too, though it's limited by personal mentality. Testing is continious research resulting in discovery - move ahead in your mastery.
But today - enjoy upcoming weekends!

Tuesday, January 19, 2010

Non-verbose list of testing tools

Recently found the resource trying to list all Functional testing tools
http://www.opensourcetesting.org/functional.php

and all Unit testing tools
http://www.opensourcetesting.org/unit_ada.php

Monday, January 18, 2010

20 Essential design patterns for automated testing

I’m follower of building automated testing by analogy with s/w development in most of practices. Some practices are being grown initially from testing or properly improved in testing as legacy development practice. Actually, I don’t like to separate professional automated testing and development. We produce the same code but having opposite purpose – to check and verify the code of developed software – it does not matter what kind of interfaces we use upon testing. Anyway we test application and by this we test s/w code itself either implicitly or explicitly.
Design patterns have high importance for building cost-efficient products with high-degree of reusability and based on component oriented models. Design patterns provide abstraction and robust platform for reusable s/w design. My intention is to think over what existing design patterns fit to automated testing. Let’s do this!

As a fan of Web 2.0 and Wikipedia, I’ll refer on definitions and articles by linking to Wikipedia pages. See full list of design patterns on dedicated page. If you need to check details on a particular pattern, just click on appropriate link in Table listing all patterns within section “Classification and list”.

Creational patterns
Abstract factory. This is a proper mechanism of building abstraction layer for derived object types. For instance, it may be useful for implementation a framework layer responsible for working with different control types (e.g. MFC controls, pure Java controls, .Net controls,…) but operating as a single interface.

Builder pattern can be employed on test data construction and as a helper in many cases. Also you can use this approach for test suite initialization, stand preparation

Factory pattern can be used in any context for simplification object instantiation. There are a lot of places where you can fit this pattern for auto testing (dynamic and static test data instantiation; logging, reporting, test stand objects, etc)

Lazy initialization can be used to reduce expenses of keeping objects in memory. The best way is to combine this approach with object clean up once it no longer is being used.

Prototype as creational pattern simplifies object instantiation and avoids sub classing like in abstraction factory model

Singleton is a protected method of keeping only one object instance in memory. Where can be used: all objects which should follow through whole test cycle or round, for example, logging and stack trace, high-level test data, environmental data, 3rd party objects initialization to keep resources. If an object is dynamic one, you have to provide special interfaces like update(), rollback(). Also this pattern is proper way for developing high-performance models, optimization resource allocation and expenses and in environments with weak or absenting garbage collection and memory clean up.

Structural patterns
Adapter or wrapper can be used anywhere for resolving data compatibility and interface simplification. Simple and basic pattern

Composite patterns can be used for representation complex object like current test data set in a tree model (one to many relationships). I prefer to rely this approach on XML test data presentation as structured object model

Decorator is proper way to extending functionality without affecting original code. I suggest to use it as helper or quick adding workaround s and backdoors

Proxy as intermediate level for access control. I’m not sure about usage context in test automation but as this is essential one, I include this to the list. Who knows, maybe this guy can be used in some specific tested applications.

Facade pattern. I like to use this pattern for managing test data through uniform interface so that I don’t take care about used test data native. All specific code of data native processors placed somewhere inside, while decorator simply returns an object. Very handy.


Behavioral patterns
Chain of responsibility is a proper pattern for designing own logging and stack trace
Command pattern. It is extremely valuable for building keyword-driven test automation frameworks where provided interface operates with macro languages. Macro is a composite state which implements completed and composite User step which is actually intermediate level between granular steps and completed workflow (like Use case). The whole workflow (e.g. Use case) is being implemented in test scenario designed from a certain macros (steps).
Another usage of this pattern is testing transactional logic by mirroring some part of transactional model of application itself on test code.

In joint use with Abstract factory, you can build a layer for typed control of mixed technologies. In my practice, I build object loader and abstracted invoking object properties and methods for supporting different COM object versions at fly. You can go ahead and implement similar abstraction for different technologies, plus empower your design by invention composite commands for batch controls processing. For example, wizards processing by invoking meta-commands (like “set up default settings”)

Interpreter can be used to simplify working with custom string patterns. It’s really useful for substituting using predefined templates and for running code pieces using Eval() method
Iterator is essential for handling corresponding controls, like grids, tables, custom lists. Basic ally, iterator should have first(), last(), next(), add(), remove(), isLast() methods.
Null object is a basic and helpful approach of default value implementation. I can state – this simplifies routine test development.

Observer pattern. I advice to be hand on with this guy especially for testing AJAX applications and other ones which are asynchronous in nature. This pattern can be used in designing event-based test auto9mation frameworks.

State pattern. I did not use this one, though I believe it’s a basis of behavior-driven test frameworks

Strategy pattern can serve for switching algorithms using polymorphism or reflection (btw, JS implementation is really fun). The Strategy perfectly correspond with Open/Closed principle

Template method
is a generic for designing reusable code (not only object-oriented) with overriding. I’d like to try this template to implement advanced test data engine.

I intentionally left TDD patterns out of this scope, since they are really specific ones, moreover I tried to highlight them (stub, fixture, fake, mock, object mother, object registry, assertions) in a separate post. Finally, most of the mentioned patterns are implementable in all modern programming languages (both OO and scripting, like JS, VBS) .

Well, enjoy designing patterns and inventing your own!

Friday, January 15, 2010

Brainstorming or flip your mind

It’s well known about brainstorming as a powerful method to generate ideas, practices, grow team spirit and as result reinforce project timeline by sharing thoughts among team members. Sure it’s amazing way to motivate people, calm down some guys and optimize people effort. Indeed regular brainstorming sessions allows mentoring team indirectly, reveal talents and identify weak players for further decision making and training. What are the key success factors of brainstorming session? Here is my vision on that:

1. Be prepared and relaxed – it’s not reporting meeting with your boss. No bosses in brainstorming – all are partners.

2. Every participant should feel freedom to share and voice thoughts and ideas. Don’t make brainstorming as a necessary event – people don’t like when they are constrained. Say NO bureaucracy!

3. Everyone should feel s/he is wanted to be listened. Interruption and switching to other speaker makes everyone unmotivated.

4. Relax audience by jokes, fun plans and stories; imagine you are child and plays how you want

5. Flip your mind and try to walk outside your habitual outlook. People often see things as in tunnel missing lots of things beyond. So be observer 360 degree.

Someone should take about collecting of brainstorming outputs as all this matter may be useless today and come in handy tomorrow. Who knows? Things happen. And yes, even if you turn brainstorming to holiday or team building, you will see how the sessions appear more and more productive.

Good luck in generating ideas!

Advanced slides on QA and automation

As a big fan of Web 2.0 and social media, I like to find out some sort of interesting artifacts, people and trends. Recently I walked through SlideShare.com and found there some interesting presentations about QA, test automation and fitting QA to business. Once again I receive evidence that Internet is a heap of junk where everyone may find what they need and sometimes very very usable - the essence is to use mediators skilfully (search engines, forums, catalogs, reviews, social nets, etc.)
So let's take a look at some interested selected presentations, perhaps there are some amusing details for you?







Thursday, January 14, 2010

Helping in forming QA and testing KB on Wikipedia

I'm working from time to time on contributing to Software testing portal on Wikipedia (http://en.wikipedia.org/wiki/Portal:Software_Testing)

If anyone can help to expand this KB - please don't hesitate - let's do it. Wikipedia is free space but high-quality content is a must.


"Quality is like health: you can't buy it. Hovewer you can bring it up and then keep it up diligently" (from myself)

Fun but useful Testers categorization

Check this paper out! IMHO this break down may fit to likely any profession.

Where you are in your path? may be you jumb from one category to other?

Wednesday, January 13, 2010

End-to-End (E2E) testing with white-box technique

The importance of system testing also known as E2E testing is very high especially on Product/Release delivery. This type of test basically requires running a continuous cycle against full path of system(s) under tests. It involves integration testing with 3rd party components, internal systems and interfaces as well. For example, test is running on User presentation then acquired results are used for verification in back-end system. I prefer and suggest continuing this effort by checking collected data using grey-box technique on DB side, or checking XML responses from Web services, queuing system or any other sort of object. Surely it’s challenge for those who are not hand on with specific technologies but learning in action is the best practice than anything else like books reading.

Another interesting thing is to automate E2E testing as complete cycle which follows data from birth to its death (CRUD: create-read, update, delete). In my previous project we had developed such monster. The basic idea behind the code is to collect all actual data from front-end tests. After that this data is picking up with new test project from SVN for feeding all the bunch of XMLs for verification in back-end system, checking DB views and pinging Web services. We used XML format as perfect data object which supports many-to-many associations and structured well for querying via XPath.

Tuesday, January 12, 2010

Measuring test (manual/automation) effectiveness

The main metrics of testing quality are coverage and number of rejections. In terms of software testing, we have to move toward minimization of invalid found defects, minimization of non-found valid defects and maximization testing coverage. The same is valid for automated testing but basically measurements being performed in relation to manual effort, while manual testing is benchmarking against UAT, defects found in production by users and stakeholders.

Scientifics interpret the same notions in terms of Errors:
Error type I – false positive (false alarm) – when reported defect is invalid.
Error type II – false negative (missing) – when defect was not revealed in testing.
See more details on this theory on Wikipedia [http://en.wikipedia.org/wiki/Type_I_and_type_II_errors]
There are special statistical measures which operate with those notions (see below an excerpt from Wikipedia.org)



Full details on that see here: http://en.wikipedia.org/wiki/Sensitivity_(tests)

So one-two metrics can be employed for both manual and automated testing analysis in time (for instance showing up in QA reports as curves on a graph). I would select Accuracy (ACC), Precision (http://en.wikipedia.org/wiki/Accuracy) and False discovery rate (FDR).

For automated testing, I would suggest to measure against manual testing considering test automation coverage but not overall (manual+UAT+Production), so that false positive (false alarm) is when test automation reveals wrong defect, false negative – when automated test did not find defect when expected due to auto test covers that functionality.
To test your auto test :), you may run pilot analysis using so called defect seeding technique.

Couple excerpts from memories (JS, TestComplete)

I like to have facade for abstraction data retrieving from any supported source. Generally I used either CSV or XML as data natives. XML data source might contain instruction to build a randomized string by substitutable pattern or have instruction to run a DB query or finally the value might be simple string. This is a simple decision-maker excerpt (sorry this blog does not allow to insert well-looked code snippets):



Another useful code which may help in tracing and problem localization is a custom exception handling wich also can be used as call chain mehanism in order to use as stack trace (Log4j is much better for this). The snippet below is serving as exception stack trace but that method can be easily transformed for use in any context.
Btw, TestComplete 7 has cool built-in stack trace so in that way problem localization becomes much quicker.

Free web rendering services: for testing, SEO and just for fun



Good time to share some Web-based tools which are very useful and handy to have in your bag!

1. Have you ever run compatibility testing against different browsers/OSs? It’s very time-consuming and routine operation. http://browsershots.org/ makes all that automatically in distributed way. Just feed needed page and wait for awhile. Very handy!

2. Would you like to take a look at how any site was presented in Internet? WayBackmachine (http://www.archive.org/web/web.php) is a powerful crawler of web pages, so that you can see site history in time. Also it may be useful to anyone who needs to check a history of a particular site for fun or for example, to check if you can trust this site or not (like Company history and survey). One limitation: if a site prohibits to be scanned (settings in Robot.txt), Waybackmachine could not return you result

3. Essential tool for SEO is http://www.seo-browser.com/ (SEO-browser). Check how a site is seen for search machines.
4. Site performance optimization is very important to make it usable and reliable for visitors. So the following services can make analysis and advices on that:
http://www.websiteoptimization.com/
http://tools.pingdom.com

5. Want to know Internet speed and check your ISP quality of services? Run this guy http://www.speedtest.net/

Monday, January 11, 2010

Adopting TDD practices into white-box test automation (part #1)

I will base on Lasse Koskela's book and Gerard Meszaros book plus some my personal knowledges.

So the first part will be analysis of “Practical TDD and Acceptance TDD for Java developers”.
As usual I will try to be short and straightforward. I will paste some excerpts which seems are fit to black-box auto testing or can be adopted down the road, e.g. on new framework development. Let’s take a look on interesting excerpts below.
“A good test is atomic.
A good test is isolated.”

This sentence sounds reasonable for all types of tests – test should be granular, isolated (for user as test runner). My addition – a completed test round should leave the world in the same state as it starts.

“That’s what I call the power of imagination! And it has a name, too. It’s called programming by intention. Programming by intention, the concept of writing code as if another piece of code exists—even if it doesn’t—is a technique that makes you focus on what we could have instead of working around what we have.”
Good engineer should be able to follow this rule – be ready to use imagination (write test before actual code).

“Take a moment to think about what we could improve in the test code. Duplication. Semantic redundancy. Anything that jumps out. Anything we perhaps should clean up.”
Refactoring is important step on improving work entirely.

“Happy path vs. error situations
As a rule of thumb, I drive for the happy path first before proceeding to handle exceptional scenarios, such as exceptions thrown by third-party APIs, malformed input, and so forth. The reason for doing so is mostly about value. Even the most robust system capable of handling any conceivable error situation is practically useless when it isn’t able to perform its basic function. On the other hand, even a system that, for example, is prone to crash if the network is temporarily down still provides value as long as the network isn’t down.”

Agree, first to implement a logic of happy path then all the rest branches.

“To summarize the essence of fixtures, they’re the whole shebang—the state of the whole runtime environment—rather than just the instance variables of a test class instance and the internal state of those objects.

Experienced TDD’ers are often good at creating focused tests. A key to being able to have a focused test is to have a suitable fixture that brings the system and the objects involved to a state in which the test doesn’t need to do much more than a single assertion call, possibly after first invoking a one-liner on the object being tested.”

Fixture is a essential pattern for robust and efficient test frameworks.

“Stubs are essentially the simplest possible implementation of a given interface you can think of. For example, stubs’ methods typically return hardcoded, meaningless values.
Fakes are a degree more sophisticated than stubs in that they can be considered an alternative implementation of the interface. In other words, a fake looks like a duck and walks like a duck even though it isn’t a real duck. In contrast, a stub only looks like a duck.
Mocks can be considered even more sophisticated in terms of their implementation, because they incorporate assertions for verifying expected collaboration with other objects during a test. Depending on the implementation of a mock, it can be set up either to return hardcoded values or to provide a fake implementation of the logic. Mocks are typically generated dynamically with frameworks and libraries, such as EasyMock, but they can also be implemented by hand.”

Those are again good patterns for employing in routine tasks. You may even design them unknowing that you use mock. At least the definitions are helpful for communicating on the same language of terms.

“Aiming at helping you avoid creating testability challenges for yourself, here are a few simple design guidelines that we should pay attention to:
- Choose composition over inheritance.
- Avoid static and the Singleton.
- Isolate dependencies.
- Inject dependencies.”

I agree with all these recommendations but second one - sometimes static and singletons may pay good result for quick and temporary implementation. Also singleton is self-protection mechanism for reloading, instantiating more objects than one.

Also Koskela provides some basic unit testing patterns that may be used upon building custom test automation frameworks. They are:
• Assertions as exercises metods
• Parametrized method creation (this guy easily implementing in JS :)
• One more interesting pattern is Object mother: "an object or a set of objects that does the following:
- Provides a fully-formed business object along with all of its required attribute objects
- Returns the requested object at any point in its lifecycle
- Facilitates customization of the delivered object
- Allows for updating of the object during the testing process
- When required, terminates the object and all its related objects at the end of the test process"

• Finally I like provided by author combination of Fixture setup, Teardown and Test object registry as a dedicated classes. This approach simplifies instantiation, automatic object cleanup by triggering, protected access and all these together truly correspond with single responsibility principle.
Regarding techniques on testing legacy code, there are highlighted just some common aspects. So I would recommend reading some special books like "Working Effectively with Legacy Code" by Michael Feathers, and books on code refactoring such as “Refactoring: Improving the Design of Existing Code”, MartinFowler et. all.

Robot Framework

I found that interesting but not yet tried this framework.

See details here
http://code.google.com/p/robotframework/

documentation:
http://robotframework.googlecode.com/svn/tags/robotframework-2.1.2/doc/userguide/RobotFrameworkUserGuide.html

Btw, I did not find out there whether it works with XML and schemas or not.

Someone tried that guy?

Who is ideal test automation engineer?

Ideally highly-professional test automation is hybrid of manual QA/testing and tests developers. I’m sure this is a rule for both project contribution and personalaqXC career path. If you are skilled QA analyst, test designer plus has broad knowledge and experience working with test automation professional environments, frameworks and besides all that may combine manual and automated effort – you are valuable for talent hunters!

As QA analyst the person should have real experience in comprehensive testing, test design, planning, and risks assessment. Also this role requires being great team player with good communication skills, creativeness and non-stop drill down against AUT – your desire to break the system should be as instinct. So that testers have some freedom to break software using any tool and technique and this makes the role very interesting as you become investigator and subject matter over a particular application and business domain.

Talking about automated testing, the core skills per me are:
- Knowledge of internal mechanisms of interacted objects (e.g. DOM model, COM object, MFC, Pure Java). Basically you have to be able to invent efficient object recognition model with high robustness on intermediate changes and proper control identification (avoiding concurrent identification)
- Knowledge of test automation tool(s) itself
- Familiar with programming language on which you are going to write tests
- Knowledge of development practices, principles (like single responsibility Open/Closed, reusability, OOP, procedural approach…) and common design patterns (singleton, decorator, factory, wrapper, etc)
- Regular expressions. That’s a very powerful tool for us. Must to know concepts and how to use. Ideally you are ready to write any expression at fly.
- Knowledge on performance testing, modeling and identification of bottlenecks (for load testing)
- Knowledge of theory and practices on relational databases.
- Good knowledge of one on-demand RDBMS, querying and performance analysis
- XML –based technologies and service model (XML, XSLT, XSD, DTD, WSDL)
- Experienced in version control systems
- Experienced in tuning application/web/DB servers
- Broad knowledge of operation systems (server and consumers solutions) as advanced user or as administrator.

Yep, I forgot to mention that QA should be in-time and clear reporter (defects, issues, test logs). It appears the list of skills and responsibilities is long and ideal. Surely project needs defines certain set of skills and responsibilities.

Thursday, January 7, 2010

Design patterns for automated testing

This post will be filled in soon with design patterns implementation that perfectly fit to our duties. As for now just announcing as reminders for myself.

Wednesday, January 6, 2010

Wwwidgets: socialize and share

Widgets are stedy becoming more and more popular and valuable tools for web developers and for social markeing.
Here are my findings of most useful and easy to deploy widgets (the list is based on my opinion):
Facebook group:
I found useful Share , Live Stream and Fan-box

LinkedIn
Just Profile widget looks interesting for now. Also for corporate sites may be it's good point to add Company insider special widget

Twitter and TweetMeme
I think a pair of Profile widget and ReTweetbutton are enough for increasing conversation

Digg
I found only one their widget http://digg.com/add-digg

Special buttons:
my favorite button which gathered all stuff together (<200 sites!) is AddThis - here is example http://www.addthis.com/web-button?type=bm&where=website&url=&bm=tb3&analytics=0

Whatever you will use, my suggestion - don't overload your app by these plugins. The golden rule - 3 on a page are readbale (my blog is overloaded :) Yeah it goes against good practices.


I will continue this conversation about widgets (broken down by destinations) in the subsequent posts some time later. Please share all your knowledges about social and sharing widgets here. (I tried to cover most of popular systems but MySpace - i found API only)
Disclaimer: this post tells about light-weight widgets only and does not consider serious APIs.

Monday, January 4, 2010

Automated security testing


Mostly testing relies on automated suites for functional testing according to provided test-cases (that can be used for manual replication too). I think it's good point to thinking about porting automation concept over other types of testing. First coming to the mind are: localization, compatibility, all types of performance, and all types of security and safety testing.

The last two guys - security and safety - i believe are proper candidates to be automated. Why? Because of security flows can be introduced while any development and maintenance phase. You might polish out your s/w from security flows but new change request can bring a defect in design which may subsequently make your s/w unsecured (e.g. improper handling cookies or turning off fields validation common mechanism)

Such tests like SQL injections, XSS, session hijacking, cookies substitutions, overflows (as negative tests) and unvalidated I/O, access and permissions control can be easily simulated by test automation and can be run repetitively.
Btw, denial of services can be modeled with load, stress and volume testing.



The resource owasp.org provides tons of info about security and hacking on Web. You can find there lots of helpful data.
They have a comprehensive list of potential vulnerabilities in s/w:
http://www.owasp.org/index.php/Category:Vulnerability
and a category of attacks:
http://www.owasp.org/index.php/Category:Attack
I strongly appreciate this society for their job!

Handy add-ons to Firefox can simplify your process of developing and running hacks. The simple list of essential tools:
Tamper Data
SQL InjectMe
XSSMe
View Cookies
CookieSwap
HackBar

Test automation strategy as part of QA strategy



Happy new year all!

Right at the start of this year I have a challengeable task - detail test automation strategy within overall QA strategy on anti-virus software (new Release) in our Company. Since we have taught schedule and intentions to improve processes by thought out documentation, formalization and model optimization, this is really important to plan and reflect all things together in a single artifact - test strategy (calling also test approach). Another reason to combine both manual and automated efforts - is to ease traceability between risks, dependencies and schedule over whole department. Applying mutual practices prepares manageable and unified processes.

I'm going to highlight the following items in that test automation strategy:
- Test objectives
- Assumptions
- Types of testing on automation
- Test tools
- Resources (money, time, skills)
- Application under test specific details (technologies, quality attributes...)
- Test environment (test labs, stands and infrastructure)
- Test requirements (requirements to test cases on automation)
- Entry/Exit criteria
- Project plan (high-level and/or detailed as reference on appropriate artifact, deliverables and milestones)
- Risks (impacts, likelihood, mitigation plans)
- Review strategy

Well done! Is there something more to add or making this list redundant? Your advices are always welcomed!

Saturday, January 2, 2010

On stabilization test suites. No fuel but destination is too far

This crazy activity is not only about debugging. More sophisticated your test automation framework and infrastructure – harder test suites stabilization. Under test suite I understand here a set of test cases, ready_to_go code, prepared environments, service utilities, auto running and so on. So that is a completed end-to-end package.
Right now our brave team works on stabilization of test suite developed on overdesigned framework. And this seems eats about 60% of time on full task. As most of just have started to understand this framework, we hope this time will be significantly cut off on subsequent test suites but IMHO we will stabilize and debug 40% at least of total time on implementation. Will see.
This activity requires strong concentration and tries your nerves on tuff. Especially debugging is time consuming because of auto tests runs workflow with long chain of calls and usually debugging runs altogether with real data. I rather think debugging in test automation is much difficult than in development (if test framework is really framework) due to:
- Mostly automated tests are developed on scripting languages and unmanaged code
- Code is not compiling and running inline (so first error you may find somewhere deep)
- Weak garbage collection
- Weak test development environments and debugging mechanisms
- I did not yet meet good stack trace for black-box automation (TestComplete works on that and it becomes better but it’s not yet cool as in Log4j)
- Unstable test engines itself and communication with OS resources fails sometimes
- Some scripting languages have poor automatic type casting (e.g. VBS)
- Difficult or impossible to integrate new technologies to get higher abstraction (ORMs, patterns, logging, self-testing, reporting, external APIs with intellisense and ohers)
Also test automation suites should have lots failover and preparation mechanisms to be more robust, for instance, initialization, clean up, terminations, checkers, workarounds, turn over, synchronizations, waiters and alternative recognizers, behavioral logic, logs lockers/unlockers, wake uppers, exit gates… Huge list and finally most of code should we wrapped in try-catch-finally (it’s not about VBS, it has another pattern). If you are novice in test automation read on this. Professional automated test is a product as we know about software.


Statistical and numerical analysis for QA and testing

Stat analysis appears very powerful tool in a hands of analyst if processing data set has significant number of entries, also the set should have normal distribution otherwise the processing will output wrong results. Let say if you processed all data from a segment A, then processed only 10% of data from B and 1% from segment C, resulted joint product of these observations will be uncertain. The bottom line: all your deals with statistics should be careful since it involves wide spread of theory and practice.
Statistic and numerical analysis are very useful for the following QA practices:
- density defects measurements
- predicting defects quantity in developed s/w
- DRE and other metrics
- ROI calculations
- planning
- probabilities and likelihoods
- design test data and steps (tools like pairwise testing, finite state machine, boundary testing, matrix of transitions and so on)

For performance tests:
- avg response time, std, min, max
- measurements from uncertanity and independent sources

For automated functional testing:
- ROI (basing on accumulated data and assesement)
- failure probabilities calculations
- for building effective test automation suites - design parameters permutations for achieving best coverage and to reveal hidden defects
- random data and variations
- modeling system with feedback(s)

Can you add some more?