Test strategy

A test strategy is an outline that describes the testing approach of the software development cycle. The purpose of a test strategy is to provide a rational deduction from organizational, high-level objectives to actual test activities to meet those objectives from a quality assurance perspective. The creation and documentation of a test strategy should be done in a systematic way to ensure that all objectives are fully covered and understood by all stakeholders. It should also frequently be reviewed, challenged and updated as the organization and the product evolve over time. Furthermore, a test strategy should also aim to align different stakeholders of quality assurance in terms of terminology, test and integration levels, roles and responsibilities, traceability, planning of resources, etc.

Test strategies describe how the product risks of the stakeholders are mitigated at the test-level, which types of testing are to be performed, and which entry and exit criteria apply. They are created based on development design documents. System design documents are primarily used, and occasionally conceptual design documents may be referred to. Design documents describe the functionality of the software to be enabled in the upcoming release. For every stage of development design, a corresponding test strategy should be created to test the new feature sets.

Test levels

Test levels define the scope and focus of testing activities within the software development lifecycle.^{[citation needed]} Test levels are typically organized hierarchically, with each level targeting different aspects of the system under test.

Common test levels include unit testing, integration testing, system testing, and system integration testing.^{[citation needed]}

Unit testing focuses on individual components or modules in isolation.

Integration testing verifies the interfaces and data exchange between integrated components. End-to-end integration testing spans the complete technology stack, using API calls, database interactions, and selective user interface testing to verify system-wide functionality.^{[citation needed]}

System testing evaluates a complete integrated system against specified requirements.

System integration testing tests interactions between multiple integrated systems.

Responsibility for different test levels varies by organization and project structure. Developers commonly perform unit testing, while dedicated testing teams often handle system-level testing activities.^{[citation needed]}

Roles and responsibilities

The roles and responsibilities of the test leader, individual testers, and project manager are to be clearly defined at a project level in this section. This may not have names associated, but the role must be very clearly defined.

Testing strategies should be reviewed by the developers. They should also be reviewed by leads for all levels of testing to make sure the coverage is complete, yet not overlapping. Both the testing manager and the development managers should approve the test strategy before testing can begin.

Environment requirements

Environment requirements are an important part of the test strategy. It describes what operating systems are used for testing. It also clearly informs the necessary OS patch levels and security updates required. For example: a certain test plan may require Windows 8.1 to be installed as a prerequisite at testing.

Testing tools

There are two methods used in executing test cases: manual and automated. Depending on the nature of the testing, it is usually the case that a combination of manual and automated testing is the best testing method.

Risks and mitigation

Any risks that will affect the testing process must be listed along with the mitigation. By documenting a risk, its occurrence can be anticipated well ahead of time. Proactive action may be taken to prevent it from occurring, or to mitigate its damage. Sample risks are dependency of completion of coding done by sub-contractors, or capability of testing tools.

Test schedule

A test plan should make an estimation of how long it will take to complete the testing phase. There are many requirements to complete testing phases. First, testers have to execute all test cases at least once. Furthermore, if a defect was found, the developers will need to fix the problem. The testers should then re-test the failed test case until it is functioning correctly. Last but not the least, the testers need to conduct regression testing towards the end of the cycle to make sure the developers did not accidentally break parts of the software while fixing another part. This can occur on test cases that were previously functioning properly.

The test schedule should also document the number of testers available for testing. If possible, assign test cases to each tester.

It is often difficult to make an accurate estimate of the test schedule since the testing phase involves many uncertainties. Planners should take into account the extra time needed to accommodate contingent issues. One way to make this approximation is to look at the time needed by the previous releases of the software. If the software is new, multiplying the initial testing schedule approximation by two is a good way to start.

Regression test approach

When a particular problem is identified, the programs are debugged and the fix is applied to the program. To make sure that the fix works, the program will be tested again for that criterion. Regression tests will reduce the likelihood that one fix creates some other problems in that program or in any other interface. So, a set of related test cases may have to be repeated, to test whether anything else is affected by a particular fix. How this is going to be carried out must be elaborated in this section.

Consider different testing levels when selecting regression test cases. Unit-, integration- and system test cases are good candidates. Select cases that have direct relationship with the fix and also include few business critical cases that prove basic business scenarios still work. Remember also that non-functional testing (security, performance, usability) plays an important role in proving business continuation.

In some companies, whenever there is a fix in one unit, all unit test cases for that unit will be repeated.

Test groups

From the list of requirements, we can identify related areas, whose functionality is similar. These areas are the test groups. For example, in a railway reservation system, anything related to ticket booking is a functional group; anything related with report generation is a functional group. In the same way, we have to identify the test groups based on the functionality aspect.

Test priorities

Among test cases, we need to establish priorities. While testing software projects, certain test cases will be treated as the most important ones, and if they fail the product cannot be released. Some other test cases may be of lesser functional importance, or even cosmetic and if they fail, we can release the product without much compromise on the functionality. These priority levels must be clearly stated. These may be mapped to the test groups also.

Test status collections and reporting

When test cases are executed, the test leader and the project manager must know, where exactly the project stands in terms of testing activities. To know where the project stands, the inputs from the individual testers must come to the test leader. This will include, what test cases are executed, how long it took, how many test cases passed, how many failed, and how many are not executable. Also, how often the project collects the status is to be clearly stated. Some projects will have a practice of collecting the status on a daily basis or weekly basis.

Test records maintenance

When the test cases are executed, it is important to keep track of the execution details such as when it is executed, who did it, how long it took, what is the result etc. This data must be available to the test leader and the project manager, along with all the team members, in a central location. This may be stored in a specific directory in a central server and the document must say clearly about the locations and the directories. The naming convention for the documents and files must also be mentioned.

Requirements traceability matrix

Ideally, the software must completely satisfy the set of requirements. From design, each requirement must be addressed in every single document in the software process. The documents include the HLD, LLD, source codes, unit test cases, integration test cases and the system test cases. In a requirements traceability matrix, the rows will have the requirements. The columns represent each document. Intersecting cells are marked when a document addresses a particular requirement with information related to the requirement ID in the document. Ideally, if every requirement is addressed in every single document, all the individual cells have valid section ids or names filled in, then we know that every requirement is addressed. If any cells are empty, it represents that a requirement has not been correctly addressed.

Test summary

The senior management may like to have test summary on a weekly or monthly basis. If the project is very critical, they may need it even on daily basis. This section must address what kind of test summary reports will be produced for the senior management along with the frequency.

The test strategy must give a clear vision of what the testing team will do for the whole project for the entire duration. This document can be presented to the client, if needed. The person who prepares this document must be functionally strong in the product domain, with very good experience, as this is the document that is going to drive the entire team for the testing activities. Test strategy must be clearly explained to the testing team members right at the beginning of the project.

References

Ammann, Paul and Offutt, Jeff. Introduction to software testing, New York: Cambridge University Press, 2008
Bach, James (1999). "Test Strategy" (PDF). Retrieved October 31, 2011.
Dasso, Aristides. Verification, validation and testing in software engineering, Hershey, PA: Idea Group Pub., 2007