Treat Automated Testing as Real Software Development
Over the last few weeks I have been working in a project that has invested heavily in automated regression testing. To that extent the project even developed a dedicated interface to allow accessing and controlling the user interface from testing programs. As a result testing is done via custom programs instead of using commercial record & playback tools. This gives the project much more flexibility and the ability to use open source testing frameworks that over the last years have matured considerably.
Back to the topic: working in this project has reminded me of what a fundamental lesson it is and how often it is ignored. All too often I see software professionals who are used to build large enterprise style applications to blithely treat automated testing as something where you just set up the tools and then testers write test scripts. Granted that the automated testing will result in much less code and even less sophisticated code than the application itself, but it is still non-trivial.
Here are some reasons why you need to address this like a real software development project:
Build a Testing Framework
I am not saying re-invent the wheel. There are by now excellent tools in the commercial and the open source area to support automated testing. My preference is for the latter as they give much more flexibility. Testing frameworks that once were used for Unit Testing have grown up and are quite capable of being used in full system level automated testing. Still you will need to develop an overall execution framework that is robust enough to withstand the demand of continuous testing over long periods of time. That means you need to select the right tools - think proof of concept evaluation of candidate tools. Then you need to tie everything together using a combination of configuration and glue code and build additional components which you may not be able to find in the commercial nor the Open Source domain.
Architect the Testing Framework
Building robust test frameworks is non-trivial. Application failures of the SUT (System Under Test) are expected so the framework needs to anticipate this and be able to recover from them in a robust fashion. Test Setups need to be robust to avoid them introducing failures that would obscure the actual test cases. Poor resource management by the testing framework may cause inexplicable failures after continuous execution over days. Integration to other tools (build environments, configuration control, reporting databases etc) means potential failure points that a right architecture can make more manageable.
Build a Test Support Library
When test scripts are being developed it quickly will become apparent that there is a lot of code repetition. Setup for tests often involves the same actions over and over again. So does validation of test results. If the test involves the UI then certain navigation paths repeat over and over again. All this needs to be made available through utility libraries to support the faster development of test scripts. Experience shows while some can be designed before regression testing starts - most will emerge during the development of test scripts. So a process and a supporting infrastructure to maintain support libraries and disseminating the knowledge of them to the test authors is critical.
Build a good Analytical Infrastructure
Automated testing is about running many tests fast. The cost of building an automated test is much higher than executing it manually – so to get you investment back you need to have a situation where you can run it often – such as right after each build, automated, regularly. That results in lots and lots of test results. When done well you can expect a few thousand test runs each day, day in day out. Once you do that it is apparent that you need to be able to slice and dice your test results. You want to be able to look for trends, look for patterns. Because tests are unattended you need to keep logs and configuration information for each test run that you may have to be able to access and scrutinize in a quick fashion.
Build a good Logging Capability
In comparison to a human doing manual testing automated tests are extremely dumb. The human tester has the ability to think "Mhh – that's strange" and then investigate. So when an automated test case fails the test log may not tell the whole story. Once you investigate a test failure there is probably a lot of valuable information that was not recorded because the test developer didn't realise it might be useful. There is the noise factor, test cases can fail not because the application has a defect but the environment does: time outs, defects in the test case script, defects in the test interface, the system it runs on – all these can cause failures and the test script may not be reporting sufficient information to discern this right away.
So you need to ensure enough information is logged – some of it automatically by the test execution infrastructure some by making it easy for the test writer to log information in a structured way. The log results need to be structured to be easily accessible – which must be integrated into the analytics infrastructure.
Build a Reporting Infrastructure
Apart from being able to analyze the test results you need to have a simple overall reporting infrastructure. After investing all that money (which often is more than anticipated) you need to demonstrate the value of the testing. You should be able to report the performance of the automated testing. Since the testing is automated, the reporting should well be automated as well. It needs to show how many test cases are being run, how many fail and trends in testing.
Build a Process to Maintain Automated Testing
Once the framework is in place you need to feed it with test scripts. Each test script is a comparatively simple program. But it still is subject to software development rules. You must have a process that is effective. You need to understand the status of each test case: is it under development, or is it in production (i.e. is it actively used in regression testing). Defects can be raised against test scripts. Test scripts must be written so that they are maintainable – that implies coding standards and code reviews.
Conclusion
I have just but scratched the surface of building an automated testing facility. But I hope it demonstrates that the project planning needs to recognize the effort and respond to it by providing it the right resources – otherwise you are setting yourself up for a waste of money. This is demonstrated in many projects where automated testing is either abandoned or remains outright anemic in nature.