Blog Barista: Mary Pawloski | June 5, 2019 | Testing & QA | Brew time: 5 min

As the software industry moves away from the Waterfall Software Development Life Cycle (SDLC) to an Agile methodology, some sort of automated testing is necessary for survival. Continuous integration, continuous development, and DevOps (the production and delivery of products and services at a faster than traditional pace) are being used to make the SDLC fit the Agile mold.

In this evolving and fast paced industry, how can a company not have some sort of automated testing processes in place? What one factor would possibly prevent a company from implementing automated testing? Cost.

There are myriad tools out there which claim to be able to make the testing world of any type better, stronger, and faster. Some have been around for a long time. Some have evolved to keep up with the pace of technology. Some are relatively new. And then, some are tailored for a specific type of software or industry application. All, including those that are “freeware,” have a substantial cost associated with them.

You may be asking yourself, what are the real costs associated with implementing automated testing? Well, let’s take a look at the process as a whole.

Well first of all, the process of finding and implementing an automated testing tool can be both difficult and exhausting. Your company will have to find someone who knows how to use an automated testing tool and make it work with the application being developed. If that person is you, then they’ll need to factor in the cost of not having you do your regular job. And if there isn’t someone with these qualifications on staff, they’ll need to take the time and expense to hire someone who has automated testing experience. Regardless, time will be needed to identify and try the applications that claim to work with the programming language you’re using. Keep in mind, the one that appears to be the best fit isn’t always the right choice.  

In addition to any licensing costs, every software application has an upfront cost of training. Hopefully, you or whoever is selected has enough experience to immediately start using the application and train others on its use. But, your company still may need to purchase a package that includes formal training (again, there is a loss of billable hours). Or, you might need to take the time to complete the online tutorials so you are competent enough to use the software, and able to show others how to use it.

After you’ve selected the tool you want to use and completed any necessary training, you can start building your automated test cases. At this point you might think, “Well, it should be smooth sailing from here. The cost of all that setup stuff wasn’t such a big deal.” Wrong!

Just because it appears that the carefully researched choice of an automated testing tool will work with your application’s programming language, doesn’t mean that there won’t be any hiccups. For instance, say you’re trying to implement something such as a circular rule (OK, maybe that’s a bad example, but it’s just an example) which can’t be handled by the application, or maybe it requires a plug-in. You are now a week or two into adding your automated test scripts and you’ve run into a hiccup, you may say, “I’ll just call technical support. There must be a plug-in or some work around for this issue.” Well, not necessarily.

Oh, by the way, did you add in the cost of support services in case you don’t know why something isn’t working?

Sometimes it feels like you just got out of the starting gate, only to find that your thoroughbred application has a broken leg. You have a gut feeling that you’re not going to be able to finish the race on this horse. How long are you willing to continue running down the track with a bandaged leg before you know that it’s time to call it and move on to horse number two? Trust me, it happens.

Let’s say that you’ve finally found an application that best fits the programming language your application is being developed in. Your service contract is in place and you are happily creating those beautiful automated tests that will make your software quality shine like gold. But, don’t forget you still have to run these tests in a production test environment. That production test environment needs to have data to run the scripts you create. If the data doesn’t exist, more time needs to be added for it to be created.

You then need to identify the order in which you will run the scripts. Do any scripts need to be run in a specific sequence? Once the scripts have ran, the results need to be reviewed to identify if there was a failure and the cause of that failure.

There can be many reasons why a script failed. Is something missing in the script? Is the environment set up correctly? Is data missing? Is this really a bug? Is the world really round?

Therefore time needs to be factored into a release to deal with issues that pop up while the scripts are running. Just because a script ran successfully in the last release and there hasn’t been any changes around that script’s code, doesn’t mean it will run successfully in the current release.

Once you’ve established your library of automated scripts and set up the automated script schedule, you have to account for the time needed to update all scripts that were either impacted in the last release, or that will be impacted in this release. You should also include a risk assessment of the impact these changes will have. Scripts that pose a greater risk will need to be run as early in the automation sequence as possible. There is a good chance this will impact the script run schedule. All of this requires time.

One final note, up-to-date documentation is always a necessity. This is especially true if you decide to implement an automated testing process on your project. Having consistent, complete and understandable system documentation that clearly defines the automated process flow will ensure that the right thing is done at the right time. Making it worth the time and cost of updating it.

So after looking at the process as a whole, it’s clear that the cost of automated testing includes a lot more than just the cost of the tool you plan to use (corporate or individual licenses). You should add the cost of customer support to help with those unknown and unexpected tweaks and/or failures. There is also the cost of training on how to use the tool (loss of project hours and any fees for training not included). If your company needs to hire someone, there’s the cost of recruiting, a potentially larger salary that comes with experience, and project ramp-up time.  

Let’s not forget the cost associated with the time needed to initially write and maintain the automated scripts, set up the environment, and create any necessary data. As well as the time needed to update and maintain the scripts before and during the current testing Sprint. And finally, there’s the cost associated with the documentation for the automated process which needs to be kept up to date for any and all changes and special situations. After all, if you suddenly won the lottery and moved to France, someone else will have to come in and pick up where you left off.

Other recent posts: