Configure automated integration and functional test automation

 

The first thing that comes to mind when talking about Continuous Delivery is that everything needs to be automated.

Otherwise, you can't deploy multiple times a day. But how to deal with testing then?

Many companies still have a broad suite of manual tests to be run before delivering to production. Somehow these tests need to run every time a new release is created.

Instead of automating all your manual tests into automated UI tests, you need to rethink your testing strategy.

As Lisa Crispin describes in her book Agile Testing, you can divide your tests into multiple categories.

Source: https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants

We can make four quadrants where each side of the square defines our targets with our tests.

• Business facing - the tests are more functional and often executed by end users of the system or by specialized testers that know the problem domain well.
• Supporting the Team - it helps a development team get constant feedback on the product to find bugs fast and deliver a product with quality build-in.
• Technology facing - the tests are rather technical and non-meaningful to business people. They're typical tests written and executed by the developers in a development team.
• Critique Product - tests that are there to validate the workings of a product on its functional and non-functional requirements.

Now we can place different test types we see in the other quadrants. For example, we can put Unit tests, Component tests, and System or integration tests in the first quadrant.

In quadrant two, we can place functional tests, Story tests, prototypes, and simulations. These tests are there to support the team in delivering the correct functionality and are business-facing since they're more functional.

In quadrant three, we can place tests like exploratory, usability, acceptance, etc.

We place performance, load, security, and other non-functional requirements tests in quadrant four.

Looking at these quadrants, specific tests are easy to automate or automated by nature. These tests are in quadrants 1 and 4. Tests that are automatable but most of the time not automated by nature are the tests in quadrant 2. Tests that are the hardest to automate are in quadrant 3.

We also see that the tests that can't be automated or are hard to automate are tests that can be executed in an earlier phase and not after release.

We call shift-left, where we move the testing process towards the development cycle.

We need to automate as many tests as possible and test them.

A few of the principles we can use are:

• Tests should be written at the lowest level possible.
• Write once, run anywhere, including the production system.
• The product is designed for testability.
• Test code is product code; only reliable tests survive.
• Test ownership follows product ownership.

By testing at the lowest level possible, you'll find many tests that don't require infrastructure or applications to be deployed.

We can use the pipeline to execute the tests that need an app or infrastructure. To perform tests within the pipeline, we can run scripts or use specific tests tools.

On many occasions, these are external tools that you execute from the pipeline, like Owasp ZAP, SpecFlow, or Selenium.

You can use test functionality from a platform like Azure on other occasions. For example, Availability or Load Tests executed from within the cloud platform.

When you want to write your automated tests, choose the language that resembles the language from your code.

In most cases, the application developers should also write the test, so it makes sense to use the same language. For example, write tests for your .NET application in .NET, and write tests for your Angular application in Angular.

The build and release agent can handle it to execute Unit Tests or other low-level tests that don't need a deployed application or infrastructure.

When you need to do tests with a UI or other specialized functionality, you need a Test agent to run the test and report the results. Installation of the test agent then needs to be done upfront or as part of the execution of your pipeline.