Chat with us, powered by LiveChat

Test Automation vs. No Test Automation

by Jeffery Thompson August 29, 2019

Should you incorporate test automation into your software development approach? In our opinion, this should not be the question. We believe you must incorporate test automation. The only question is when you should do it and where. Should you implement test automation on both frontend and backend?

In this article, we’ll share our 17+ years of experience with both frontend and backend test automation and where we feel it’s best to use it and why.

Nevertheless, automated software testing is only beneficial if it’s done right. Learn more about how to plan a successful testing strategy and find out how to prepare the budget for the best software testing.

What is Test Automation?

TA (testing automation) is using software tools to test the software you’re building. While manual testers are just people emulating the user behavior and testing various software units, automated testing tools enter data, compare actual and expected results, and write reports without human interaction.

Automated testing is mostly used within large projects where testers need to test the same code over and over again. It is vastly used for high-risk projects, such as bank payment services which we can’t possibly imagine testing in real time. On top of that, test automation is excellent for repetitive testing tasks and large projects that can be tested overnight to save time and money.

Test Automation vs. No Test Automation

Most companies follow something called the testing pyramid. Testing pyramid refers to three different testing layers:

  • Unit tests – testing of the low level application units like models, controllers or UI components, performed in isolation from other application parts.
  • Integration tests – testing of multiple components working as a whole. Usually performed through emulating API calls.
  • UI tests – simulation of the user’s behaviour within the application (filling up the forms, following links, clicking buttons etc)

As you can see in the pyramid above, there are different testing layers and different rules about how to execute them. The term “testing pyramid” was first introduced by Mike Cohn, the founder of Scrum. Although you can rely on it 100%, the pyramid serves as a good rule of thumb to help you determine how much, when, and why to automate your testing process.

Unit Tests

Unit testing is the foundation of software testing. First, you divide the application into smaller units. Then you test each unit or function to validate that it performs the function that it was designed for. Each component of the program is tested independently.

Unit testing checks whether individual components work properly with all expected input types, in accordance with the component design.

Unit testing is performed in a controlled environment so that the testing team can give a predefined set of data to the testing component and observe output actions and results. Along with that, the testing team has to check the internal data structure and logic for input and output data.

When a unit test is completed, another check follows to determine whether the interface between components was adequately defined and implemented.

Test Automation vs. No Test Automation

How Does Unit Testing Work?

To do unit testing a developer writes a section of code in the app to test the function. After the test is done, a developer comments and removes the test code when the app is deployed.

Unit testing can also be done in an isolated environment to reveal any unnecessary actions between the code being tested and other units in the app.

One of the most rigorous unit testing is the TDD – Test Driven Development. TDD requires full test coverage and has better organic results, but is very time consuming and costly. In TDD, all tests are written as functions that determine if the value you got matches the desired value you expected from the function which you had in mind when you wrote it.

TDD is often called red-green-refactoring due to its specific stages of development. In the red phase, you focus on the tests you want to build, while in the green stage, you work on how to implement the code, so your test works. In the third stage – refactoring, you work on improvements and look for new ways to increase efficiency and to write better code.

Pros & Cons of Automated Unit Testing

Pros:

  • Chance to catch problems early in the development stage
  • Possibility of making changes accordingly without affecting other units
  • Simple design and documentation process
  • Saving time
  • The cost of finding a bug and fixing it in the unit testing phase is much less than doing all that when the app is live

Cons:

  • You might end up with writing wrong tests
  • You might get lost with naming them
  • The overall development budget increases depending on the coverage rate. The 100% coverage usually increases the development efforts for up to 50% to 150% depending on the task/application complexity.

Integration Tests

Integration testing is a phase in software testing where individual software system units are combined and tested as a group to detect possible errors. This is the middle part of the testing pyramid, and it always happens before the final UI testing. The goal of the integration testing is to integrate units, to find errors, fix them, and to build up the entire structure of the system as foreseen by design.

Unit testing ensures that all individual modules of one software system are tested and that each of them individually works as planned. Unit testing does not guarantee that these modules will work fine if they integrate into the system. Some errors occur only when two units connect.

Integration

How Does Integration Testing Work?

When we perform integration testing, we are emulating API requests and checking their responses against expected behavior. For example, when you are submitting a login form and enter an incorrect password, you should get an error code saying that the password is wrong. If you enter the right password, an expected result is to get an authentication token with a success status code.

API tests are performed to test the entire service functionality – all the system components integrated and working together. API (Application Programming Interfaces) testing is one of the essential parts of software testing as it reveals bugs and inconsistencies and checks the core logic of the application.

Apps mostly have three layers – data layer, API layer (service or integration layer), and user interface layer. The API layer defines the logic of an app. It determines how users will interact with the app, how they will input data, and what functions will the app perform. While UI testing is more about how an app looks and feels, API testing is about validating business logic, communication with servers, and data responses.

Pros & Cons of automated integration testing

Using automation testing tools is a much better solution than writing your code to test API since that process is daunting and time-consuming. Even though there are some downsides to automated API testing, TA is highly recommended, especially in agile development where things change and update all the time.

Pros:

  • Time efficient
  • No need for user interface
  • High code coverage
  • When done in the early stages of development it gives a chance to catch errors and avoid big issues
  • Easy to integrate with other tests
  • Gives developers more confidence and a chance to write better code
  • Automated integration testing tests all the system layers and test how all the components work together

Cons:

  • Validating and verifying the output in a different system is difficult for testers
  • The entire testing operation depends on a previously defined testing strategy

User Interface Testing

UI testing is usually performed as a simulation of the usual user actions, like filling in the different application forms, performing clicks, taps, scrolls etc in order to test how the application behaves against its expected behaviour. They usually takes much more time to be performed because they require UI rendering and acting to be emulated which is usually pretty slow.

Also UI tests are pretty fragile because of their nature. When test developer describes some system behaviour, he relies on the UI structure to define how the specific elements should look, behave, which values they should contain etc. If the structure is changed, which happens pretty often during active development, the tests become failed (false negative) and must be fixed.

This is why UI tests are still very important but take the least place in the piramide. It’s too expensive to keep them up to date if you’re trying to cover everything by them, it’s too long to wait until they are being performed if there is too many of them. But in most cases this is the only way to catch some bugs caused by broken front-end / back-end interaction.

UI-testing

How does Automated UI Testing Work?

UI test developer defines the scenarios which cover most of the functionality which is supposed to be covered. The goal is to cover as much as possible without repeating actions too much. It’s very different from unit tests where you’re covering each case by a separate unit test.

Why? Each of unit tests is very cheap in terms of running time. But it’s much more important to catch specific broken cases without researching what specifically is broken within the huge test scenario. So keeping them simple but repetitive is pretty fine for unit tests.
Unlike unit tests, UI tests are much more expensive in terms of running time, so making UI test scenarios more complicated but covering as much as possible without repeating actions is worth it.

We mostly make apps for the end users who are, mostly, not familiar with the source code and the business logic in the software architecture. They decide to use an app solely on how it looks and feels like. For them, we need to go to the top of the pyramid, reserved for UI testing, that always comes last and requires more manual work and less test automation tools.

UI testing is mostly performed by a manual tester who tests and uses an app as a real user would do. Test planning includes creating a strategy that will cover all possible interactions a user could have. Test execution includes a set of operations that help him manually check that the app is behaving like it should, i.e., that the features provide the result they were designed for. The biggest downside of the manual UI testing is that it’s time-consuming. To save time, you can turn to automated UI testing, but it will only help you if you do it properly.

How does Automated User Interface Testing Work?

Automated testing tools are designed specifically for User Interface testing. They run the app and emulate the interactions a real user would have. It records the entire process, including the mouse movements and keyboard inputs. This is called record and replay testing as testers repeat this process over and over until they test all interactions one user could have with the app.

Another automated UI testing tool is model-based. First, you build a model – a graphical description of the system behavior that will enable you to get more efficient test cases. Before calculating outputs and running the tests, a tester needs to determine the inputs. After the test is run the QA testing team compares the results with what was expected to happen in the first place.

Pros & Cons of Automated UI Testing

Pros:

  • Since manual tests are prone to human error, with automated testing, you will get consistent results
  • Automated UI testing saves a lot of time and budget
  • It gives you a chance to simulate a large number of users, which you could never do manually

Cons:

  • Automated tests can give you a misleading sense of quality. Because these tools test only what they are programmed to test, they give the best results when combined with manual testing.
  • Finding too many irrelevant bugs. Automated tests can fail due to a number of reasons, and even the smallest UI change can break them.

our team

Can I Fire My Testing Team Now that I Use Test Automation?

The short answer is no. Read below how automation makes things much better but it can’t fully replace manual testing.

If you have automated unit and integration tests, and automated UI tests, some think that you don’t need manual testers anymore. The big truth about software testing is that you can’t have two things – 100% bug-free software and 100% testing automation. The reason is simple. Manual testers understand the users, they know how human psychology works, and they can make important decisions when it comes to software releases. Also, testers communicate back and forth with the development team and even with the stakeholders when needed, so they make sure that the business requirements are met and that the release is good enough for the end user.

So,no, you shouldn’t fire your manual testing team, but you should automate most of the testing processes. That will leave your QA team more time to focus on issues tools can’t recognize or do exploratory testing, which computers can’t do. Automated tests are predesigned to precisely do what you tell them to do, which leaves them no room to “think” outside of the box and explore different parts of the application.

How Much does Test Automation Cost?

For budget planning, you need to go back to the testing pyramid model. Since unit testing is the base of the testing strategy and it’s fast to run, it’s safe to spend from 50% to 150% of the development budget, based on coverage, on unit testing. Unit tests should be performed along with the coding, and since they don’t require too much integration, they can be run locally.

I know, you’re yelling at me right now saying why would you ever consider spending 50% to 150% of the development budget on unit tess! The answer is yes, I’ve seen systems that require over 5,000 hours to completely test manually. This can kill a product, innovation, budget, etc.

The middle part of the pyramid is reserved for the integration (API) testing. API testing is an essential part of the testing process as it provides the fully integrated backend testing. You can spend from 10% to 50% backend development budget on API testing.

The top of the pyramid is the final testing stage. Since UI testing is time-consuming, we recommend spending less time there. You still need to check the integration between backend and frontend but don’t spend more than 15-30% of your development budget on UI testing.

Testing Automation is Worth it When Done Right

With the right budget planning and a proper testing automation tools, you can increase speed to market and improve your product. TA helps you find more bugs, increases efficiency, and allow you to record the entire process. This means that you can use and reuse the same testing operations without spending hours at the computer doing the same daunting tasks.

With the right amount of automation, the entire team will benefit from increased productivity, and they will have accurate testing results which they can use to improve the product.