Skip to content

Destination Velocity with Quality, Be Aware Potholes Ahead

Kobiton Inc. recently published their STATE OF TEST AUTOMATION 2020-2021 EDITION report where they surveyed more than 350 software testers and mobile developers to learn how organizations have (or haven’t) adapted Test Automation.  Comparing company approaches to test automation including a more granular perspective broken down by the following categories:

  • Company Size
  • Release Frequency
  • Company Revenue
  • QA Team Size
  • Budget

Kobiton’s Findings:  Regardless of company size, all organizations agree that the top 5 challenges to starting Test Automation are:

  1. The biggest struggle to starting test automation is Evaluating and Choosing the Right Tools, especially with new frameworks and tools popping up overnight, organizations are in a constant state of “analysis-paralysis”.
  2. The second biggest automation pain point is Training/Acquiring skilled automation engineers.
  3. The third biggest challenge is Setting an Approach: Deciding what to automate.
  4. Reporting and Metrics was the 4th biggest automation challenge:
    • While Speed of Delivery and the Number of Bugs Found were the top 2 metrics indicating successful Test Automation, Hours Saved and Percentage (%) of tests automated were close behind and minimal % points different.    (27.9% – 21.9%)
  5. The budget was 5th:  Convincing the organization and leadership of the need to invest in Test Automation.

Automation does not come cheap and presents a paradox:  Organizations want to release on a weekly or daily basis, yet it takes 1-3 days to initially code test cases, and then anywhere from 1 day to 2 weeks to update automation scripts with each release.  This makes daily or weekly releases incredibly challenging.  Despite this complexity, the ROI behind automation is compelling. Almost 40% of respondents spend between 30-49% of their entire QA budget on automation, and surprisingly, almost 20% spend 50-70% of their budget on automation.

The message is clear: Automation isn’t cheap or easy, but it is necessary for innovation and a modern release frequency. If you’re spending less than 10% of your budget on test automation, you’re significantly lagging behind your peers.

I would like to add some color:

Evaluating and Choosing the Right Tools:

I agree that choosing the right tools is one of the top 3 challenges that companies face.  I would add that without solving the 2nd & 3rd biggest struggles, the #1 issue will remain the biggest pain point to starting Test Automation.
A proactive approach is to hire experts, like Software Velocity Partners (SVP), who can quickly assess your product, company, industry, infrastructure, people, culture, and most of all your budget; to determine the best approach, most suitable & appropriate tools, given the existing skills, training, and budget your company is prepared to invest.

Underestimation of the skills, knowledge, and capabilities in product, business, and technical acumen to do a good job developing the “appropriate” test automation.

Here is a sample list of skills required:

  • Architecture and development design skills
  • Understanding the Product and Business objectives, especially the Customer’s SLAs (direct or inferred)
  • Knowledge of Test Tools/Frameworks and Platforms to develop, use or integrate with for your specific product needs
  • Best practices, methods, and rules of engagement for test automation development, execution, measurement & upkeep.

Do any of these look familiar? If you are a development leader, the answer is “yes”.  The amount of work and discipline required to develop test automation is highly underestimated. Additionally, there is an overestimation of the expected outcomes and velocity of test automation development.

Let Software Velocity partners bridge this gap and assist in training or finding talent that fits your needs!
Setting an Approach:
  • This is also not surprising.  Companies do not look at Test Automation / Quality Engineering with the same lens as developing their revenue-generating products, even though the same engineering practices are required.
  • Additionally, many companies start writing test scripts without a plan.  Before too long, there is a mountain of redundant methods and test cases, causing either flaky results, unstable environments, or opaqueness in test coverage.
  • I have lived in your shoes. Our mission is to help you avoid these kinds of struggles, so you start or recover with the “right” approach.
Reporting and Metrics was the 4th biggest automation challenge:
  • Speed of Delivery was rightfully rated as the top success metric of test automation, especially given the widespread need for agility and increased release frequency in our DevOps / Modern Delivery world.
  • Oddly, TEST COVERAGE was not mentioned as a test automation metric.
    • Test Coverage definition: The measurable amount of the product’s features, purpose, and SLAs (promised & expected) that has been validated by testing.
    • CX & Ease of USE is included in the SLAs.
  • I believe in Measuring What Matters Most!  Providing visibility and transparency is the key to operational success.
BUDGET:

Convincing the organization and leadership of the need to invest in Test Automation.  This is very challenging especially if the product does not have many visible defects exposed or if leadership has never been burned by the lack of test automation.

    • For ROI Analyses:
      • Manual to Automated is easy – time saved & increased coverage.
      • New or untested areas, especially NON-Functional Areas – are not so easy.
      • I found the Opportunity Cost of “what-if something bad happens” as the best method to use for ROI analysis.