Blog Barista: Mary Pawloski | June 5, 2019 | Testing & QA | Brew time: 5 min

As the software industry moves away from the Waterfall Software Development Life Cycle (SDLC) to an Agile methodology, some sort of automated testing is necessary for survival. Continuous integration, continuous development, and DevOps (the production and delivery of products and services at a faster than traditional pace) are being used to make the SDLC fit the Agile mold.

In this evolving and fast paced industry, how can a company not have some sort of automated testing processes in place? What one factor would possibly prevent a company from implementing automated testing? Cost.

There are myriad tools out there which claim to be able to make the testing world of any type better, stronger, and faster. Some have been around for a long time. Some have evolved to keep up with the pace of technology. Some are relatively new. And then, some are tailored for a specific type of software or industry application. All, including those that are “freeware,” have a substantial cost associated with them.

You may be asking yourself, what are the real costs associated with implementing automated testing? Well, let’s take a look at the process as a whole.

Well first of all, the process of finding and implementing an automated testing tool can be both difficult and exhausting. Your company will have to find someone who knows how to use an automated testing tool and make it work with the application being developed. If that person is you, then they’ll need to factor in the cost of not having you do your regular job. And if there isn’t someone with these qualifications on staff, they’ll need to take the time and expense to hire someone who has automated testing experience. Regardless, time will be needed to identify and try the applications that claim to work with the programming language you’re using. Keep in mind, the one that appears to be the best fit isn’t always the right choice.  

In addition to any licensing costs, every software application has an upfront cost of training. Hopefully, you or whoever is selected has enough experience to immediately start using the application and train others on its use. But, your company still may need to purchase a package that includes formal training (again, there is a loss of billable hours). Or, you might need to take the time to complete the online tutorials so you are competent enough to use the software, and able to show others how to use it.

After you’ve selected the tool you want to use and completed any necessary training, you can start building your automated test cases. At this point you might think, “Well, it should be smooth sailing from here. The cost of all that setup stuff wasn’t such a big deal.” Wrong!

Just because it appears that the carefully researched choice of an automated testing tool will work with your application’s programming language, doesn’t mean that there won’t be any hiccups. For instance, say you’re trying to implement something such as a circular rule (OK, maybe that’s a bad example, but it’s just an example) which can’t be handled by the application, or maybe it requires a plug-in. You are now a week or two into adding your automated test scripts and you’ve run into a hiccup, you may say, “I’ll just call technical support. There must be a plug-in or some work around for this issue.” Well, not necessarily.

Oh, by the way, did you add in the cost of support services in case you don’t know why something isn’t working?

Sometimes it feels like you just got out of the starting gate, only to find that your thoroughbred application has a broken leg. You have a gut feeling that you’re not going to be able to finish the race on this horse. How long are you willing to continue running down the track with a bandaged leg before you know that it’s time to call it and move on to horse number two? Trust me, it happens.

Let’s say that you’ve finally found an application that best fits the programming language your application is being developed in. Your service contract is in place and you are happily creating those beautiful automated tests that will make your software quality shine like gold. But, don’t forget you still have to run these tests in a production test environment. That production test environment needs to have data to run the scripts you create. If the data doesn’t exist, more time needs to be added for it to be created.

You then need to identify the order in which you will run the scripts. Do any scripts need to be run in a specific sequence? Once the scripts have ran, the results need to be reviewed to identify if there was a failure and the cause of that failure.

There can be many reasons why a script failed. Is something missing in the script? Is the environment set up correctly? Is data missing? Is this really a bug? Is the world really round?

Therefore time needs to be factored into a release to deal with issues that pop up while the scripts are running. Just because a script ran successfully in the last release and there hasn’t been any changes around that script’s code, doesn’t mean it will run successfully in the current release.

Once you’ve established your library of automated scripts and set up the automated script schedule, you have to account for the time needed to update all scripts that were either impacted in the last release, or that will be impacted in this release. You should also include a risk assessment of the impact these changes will have. Scripts that pose a greater risk will need to be run as early in the automation sequence as possible. There is a good chance this will impact the script run schedule. All of this requires time.

One final note, up-to-date documentation is always a necessity. This is especially true if you decide to implement an automated testing process on your project. Having consistent, complete and understandable system documentation that clearly defines the automated process flow will ensure that the right thing is done at the right time. Making it worth the time and cost of updating it.

So after looking at the process as a whole, it’s clear that the cost of automated testing includes a lot more than just the cost of the tool you plan to use (corporate or individual licenses). You should add the cost of customer support to help with those unknown and unexpected tweaks and/or failures. There is also the cost of training on how to use the tool (loss of project hours and any fees for training not included). If your company needs to hire someone, there’s the cost of recruiting, a potentially larger salary that comes with experience, and project ramp-up time.  

Let’s not forget the cost associated with the time needed to initially write and maintain the automated scripts, set up the environment, and create any necessary data. As well as the time needed to update and maintain the scripts before and during the current testing Sprint. And finally, there’s the cost associated with the documentation for the automated process which needs to be kept up to date for any and all changes and special situations. After all, if you suddenly won the lottery and moved to France, someone else will have to come in and pick up where you left off.

4 Comments

  1. Haha, almost like testing is a full-time job :). I really like this candid observation of the true costs of automated testing. There have been many attempts to create a single person that has both the business knowledge about edge cases and process of a BA and the technical knowledge about implementation of a developer. This is a very niche role. Many automated testing tools claim to offer help, but all seem to be missing that at some point the knowledge of both implementation and business really is necessary.

    In my opinion, testing is best done by the developer as a means to help development. Business, of course, has the very important job of relaying all necessary acceptance criteria and edge cases so the developer can turn those into automated tests. However, the inevitable technical hurdles seem more insurmountable to overcome than the business knowledge hurdles.

    Reply
    • I agree that automated testing, for more than just confirming the build is good and that the code isn’t out of sync, should be done by development, but development doesn’t necessarily have the time, resources, and business knowledge needed to create the automation test scripts needed to accurately test all that needs to be tested (i.e., not all developers make good testers and/or BAs).

      I would say that a combined team of developers and testers paired to perform automated testing would be the best approach. But then again, that would add more to the cost!

      Reply
  2. Once the organization commits to automated testing tools and processes, there is a point at which the benefits outweigh the cost. Would you say it is worthwhile to build the team, from the very beginning, with a least one member experienced in both the technical and procedural aspects of automated testing?

    Reply
    • I whole-heartedly agree with you. It is best to start a project with an automated test team in place. In my personal opinion, I think the best approach (especially if you have a limited budget), is to start with a more technically inclined tester (possibly a former developer or someone with development skills) and a tester who understands the business end.

      While there may be some head butting initially, I believe these two skill-sets would counter balance each other so that the tests are created with the correct information and run in the right order. Your automated testing team needs to solve more problems than it causes.

      Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

Other recent posts:

Creating A Successful Interview Process

Creating A Successful Interview Process

Interviewing with Company Culture in Mind Is Better for Business
Blog Barista: Jessica Carnacchi | June 19, 2019 | Business Practices | Brew time: 5 min
Company culture can make or break a business; it has the power to attract and retain top talent or the power to drive it away. (No pressure.) While this isn’t necessarily breaking news, building and maintaining a healthy company culture…

read more
Creating a Slack Notifier using Jenkins Pipeline

Creating a Slack Notifier using Jenkins Pipeline

Blog Barista: Jim Rasche | June 12, 2019 | Developer Tools | Brew time: 9 min
Jenkins Pipeline is a powerful, expressive tool to describe your Continuous Integration process, unshackling you from the limitations of the “freestyle” GUI. This post will walk you through creating a simple Jenkins Pipeline that messages you through Slack…

read more

Pin It on Pinterest