Why We Need a Test Strategy
So today I was asked, by a major project, to review their test plan and test cases. I asked ‘can I see the test strategy’. There were lots of blank faces around the virtual room and, after a long pause, the Project manager suggested that a strategy was not needed.
So it began. The project was delivering a policy based data loss security service that would sit in line and therefore had the potential to be massively disruptive. There would be in excess of 50 rules and each rule had multiple permutations of keywords, data types and regular expressions to match upon.
My first two follow up questions were ‘are you testing every rule and every permutation of every rule’. Answers from around the project team ranged from no we are sampling, through we will take a risk-based approach, to yes we are testing everything.
My next question was ‘what is the scope of the the test all functional areas, non-functional requirements, all intergetrations, unit, system, integration etc’. Again there was no consensus in the room.
I labour the point but without a test strategy, it would have been easy to make a case to either approve or reject the test plan.
Parts of the Test Strategy
Feeling helpful and wanting my project colleagues to success, I agreed to sit with them and help to create the test approach. We went through each part of the test strategy to define:
For the scope, we agree the extent of the functional and non-fucntional testing to execute and, as importantly, what would not be tested. We agreed that each policy would be tested but not every permutation of the rules, in effect adopting a risk-based approach (more on that in a later post).
We agreed the roles and responsibilities for each party in the creation of the test data and test cases. Who would create, approve and execute the test plan. How evidence would be captured and defects recorded. We agreed how the test cases would be linked back to the strategy to ensure coverage.
We agreed the environment in which the tests would be executed. Unusually we were having to test in production as there were no lower environments for the security tooling. To avoid production disruption we were able to scope the tools to only apply to a small number of test users.
As well as the tools used to execute and automate testing, we also discussed where the test plan would be stored, how each test case was evidenced and the platform used for defect recording and tracking to closure.
Finally we recorded what would be deemed a successful test and allow the project to move onto the next phase in the SDLC.
Test Strategy Begets the Test Plan
With the test strategy agreed it was an easy ask to review the plan and individual test cases to ensure that we had complete coverage of the test strategy objectives. On first pass we identified gaps that might well have otherwise been missed.