The Cost and Utility of Types of Automated Testing
Automated Testing is not a new capability in the software development world. That said despite the most earnest promises of large and well established vendors there are clear limits and specific uses which determine the effectiveness of a given implentation. The reality is that even today the capabilities of computers and IT is such that there are no low cost machines that can apply the nuanced judgement that is required for varied sets of complex problem solving that is required by testing IT and software deployments. What current automated testing is suited to do well is repetitive regression testing. For what is called GUI (graphical user interface) testing the two largest vendors supplying tools to do this are HP/Mercury and IBM/Rational. Notably the actual names of the tools that have done this have changed over the years but for now it looks like both venders have settled on calling their product some variation of the name Functional Test. What these tools do is identify application components of an application under test and allow a tester to script a test that will be exercised on initiation of script execution. By simply using the push of a button the script will click on a button or enter text into a field or apply a menu pull down or scroll down in a scroll box. What this means is that the tester can deploy a script routine to perform predetermined actions that execute along a prespecified path that will only vary if specific inputs are provided to tell it to how to vary its path. They also can automatically capture errors that occur. What it cannot do that a human can do is dig down and investigate and troubleshoot the error once it occurs. The human can have the capability to go one step further in helping to provide resolution to the issue.
What does it take then to setup this kind of test automation deployment. Well, if you go with IBM/Rational or HP/Mercury then it can rapidly get very costly. $6000 or more a seat / user license is the starting point and that does not include additional plugins. That is just to have the raw capability… this also does not include the time and expense related to hiring someone for creating the scripts and the added support costs that these two vendors charge. Large corporations are often most certainly willing to pay this. Less large companies find it cost prohibative. There are other open source options such as Selenium and Ruby/Watir which are entirely free and have vibrant and open communities willing to share workarounds. That said even given this these tools have the same constraints as the vendor supported tools specifically the inability to do diagnosis and troubleshooting that remain well within the exclusive domain of human capability. They are a better choice in that the base cost is eliminated and the ability of getting support is more likely however the actual development costs remain. And let there be no confusion automating a test effort is a true development effort.
The test automation effort is in fact a parallel development effort. Test automation excels and is best suited for regression testing over all other types of testing. It takes time to configure and rescript a test as slight as changes might be that have occured for the software being tested. That is the reality. Though there can be some anticipation there is little automation of prediction the actual changes. This means the automation script must be changed by hand somehow whether or not by parameters in a spreadsheet or actual lines of code it must be changed. What this means is that GUI test automation is best suited for applications whose interface does not actually change and is stable from update to update. That is where the best ROI will come. Otherwise with a constantly changing interface the only limited benefit can be derived by being able to execute a script in such a way that it saves time and effort for the tester.