A RiskBased Test Strategy.docx

1、A RiskBased Test StrategyA Risk-Based Test StrategyDr. Ingrid B. OttevangerIQUIP Informatica B.V.PO Box 263, 1110 AG Diemen, The NetherlandsTel: +31 20 660 6600Fax: +31 20 695 3298E-mail: i.b.ottevangeriquip.nl1. IntroductionThe development of a test strategy is a means of communication with the cus

2、tomer commissioning the test on such matters as the organization of testing and the strategic choices that go with it. The test strategy indicates how testing is to be carried out. In order to make the best possible use of resources and time, it is decided on which parts and aspects of the system th

3、e emphasis should fall. The test strategy forms an important basis for a structured approach to testing and makes a major contribution to a manageable test process.The customer who commissions the test will expect specific qualities of the system when in production, and wants to know whether the rel

4、eased system will meet these requirements. If the system qualitatively does not meet the requirements or only to a limited extent, this implies high damage for the organization, for instance since high rework costs will be needed or clients/users will be unsatisfied. Therefore, this situation forms

5、a risk for the organization. Risk in this paper is defined as:A risk is the chance of an error occurring (chance of failure) related to the damage expected when this error does occurTesting covers such risks by giving insight into the extent to which the system meets the quality demands. When qualit

6、y turns out to be insufficient timely measures can be taken, e.g. rework by developers. If the shipping of the system implies many risks for the organization, better testing is obvious as a solution. And the reverse also holds:No risk, no test Although in the above we refer to quality and risks in a

7、 general sense, there may be large differences depending on the situation. It is of great importance to discuss this with the customer, and to translate the customers wishes in this respect into the way testing will be performed. Thus, the test strategy is directed towards finding the optimal balanc

8、e between the test effort to be exerted and the coverage required for the risks. To this purpose the risks are specified up to the level of quality characteristics and separate subsystem. In doing so it becomes possible to find a suitable test coverage for the assessed risks. Here a higher test cove

9、rage usually results in more test effort. In order to reach at the variation in test coverage needed, the use of more than one test specification technique (test design technique), each offering a specified test coverage, is crucial.An analogy with insurances may clarify this matter a bit more. A pe

10、rson wants to cover a relevant risk and takes an insurance with a coverage fitting this risk as best as possible. This insurance takes a certain premium. If the person wants to pay less, an insurance with a lower coverage is bought. The consequence is that there will be no payment if the uncovered r

11、isk occurs. On the other hand, if coverage were to large, then too much premium is paid, since a situation has been insured which is unlikely to occur for this person.The balance between budget and risk coverage2. Risk AssessmentTest strategy is based on risk assessment. This means assessing the dam

12、age of the consequences of defects, both undetected prior to operation and occurring during operation.Risk assessment takes place on the basis of quality characteristics and subsystems. For instance, if the system is insufficiently user-friendly, what will be the negative consequences. And what will

13、 be the damage when the salary calculation module in a payroll system does not work correctly.In order to be able to perform this assessment well, the separate aspects of a risk are considered: Risk = chance of failure x damage, where chance of failure is related to aspects including frequency of us

14、e and the chance of an error occurring.These aspects are listed below: Frequency of useIn a function which is used dozens of times each day the chance of an error demonstrating itself is much bigger than with a function used once a year. Chance of errorFor the assessment of the chance of errors the

15、following list can be helpful. It presents the locations where errors tend to cluster. It is partly based on H. Schaefer, 1996 (Surviving under time and budget pressure, in: Conference Proceeding EuroSTAR1996, Amsterdam, the Netherlands): Complex functions; Completely new functions; (Especially freq

16、uently) adjusted functions; Functions for which certain tools or techniques were employed for the first time; Functions which were transferred from one developer to another during development; Functions that were realized under extreme time pressure; Functions which had to be optimized more frequent

17、ly than on average; Functions in which many defects were found earlier (e.g. in previous releases or during earlier reviews); Functions with many interfaces; Inexperienced developers; Insufficient involvement of users; Insufficient quality assurance during development; Insufficient quality of low-le

18、vel tests; New development tools and development environment; Large development teams; Development teams with sub-optimal communication (e.g. owing to geographical spread or personal causes); DamageIf and when the error manifests itself, what will be the damage for the organization. Aspects are cost

19、s of repair (both of the system and of the consequences), forgone income and loss of clients or of confidence. Usually the damage increases if the error has its impact on other functions or systems. In the case of errors occurring in batch processes there may be a possibility to prevent them from ha

20、mpering users, so that the eventual damage will be smaller than with similar on-line processes. Of course, this only holds if errors are detected on time.Because of the complexity of the matter, it is impossible to assess risks with complete objectivity and in detail: it is a global assessment. It i

21、s therefore important for the risk assessment not to be carried out by the test manager alone. A large number of people involved in the scheme should contribute: customer, users, development team, accountants, IT auditors and so on. This not only increases the quality of the strategy, but it also ha

22、s the advantage that the different parties are more aware of the risks and the extent to which testing contributes to making these risks manageable in a better way.The developer of the test strategy should realize that users are the best people to assess the damage and the frequency of use when valu

23、ing the risks (end-users, system managers and application managers, line management), whereas project team members are best to assess the chance of error (project managers, designers, programmers, project quality staff, test manager).The focus in risk assessment is on product risks, or, in other wor

24、ds, what is the risk for the organization if the product does not demonstrate the expected quality. In addition to this, there are also (test) project risks. If the system must be in production on January 1st, if functional specifications are produced too late, if no experienced testers are availabl

25、e, or if the test infrastructure is not ready on time, then we speak of (test) project risks. These are not taken into account in determining the test strategy; they do play a role in the test plan. In developing a test strategy the aim is to see to it that the test will be organized in such a way t

26、hat with a certain extent of reliability the most important problems will be found; the problems will be found in an early stage; the problems that require the most rework time will be found first: efficient use is made of resources; and eventually an accurate quality advice can be given.This can be

27、 summarized as: Test strategy aims at finding the most important errors as early as possible against the lowest costsIn practice, the development of a test strategy is often planned to coincide with preparing the budget, for example with the help of test point analysis. The advantage is that the con

28、sequences of the adopted strategy are immediately translated into time required for testing, and consequently the cost of testing, which makes the strategic choices manageable. If the time available for testing is more or less fixed, it is also possible to use test strategy combined with test point

29、analysis to determine what is achievable within the time limits. It is probably even more important to make it clear at this time which parts cannot be tested, or cannot be fully tested, and what risks will therefore be incurred.3. Quality CharacteristicsThe quality characteristics we distinguish ca

30、n be divided into dynamic and static quality characteristics. The dynamic quality characteristics deal with features of the information system in use; examples are security, usability, continuity, traceability, functionality, userfriendliness, suitability, efficiency, performance. The static are con

31、cerned with intrinsic characteristics of the information system and the documentation, as considered from the standpoint of developers and future system managers. Examples are manageability, maintainability, connectivity, reusability, portability, testability.4. ProcedureIn developing a test strateg

32、y we distinguish between master test planning and a test plan for a specific test level, e.g. acceptance test or system test. The procedure can be followed both for development of new systems and for maintenance situations. For the latter, however, it is best to make a few adjustments in the basic procedure (cf 4.4).The development of a test strategy is not something that can be done purely methodically or formally. The below steps are aids and indicators. Experience and skills of the performer of this activity in the area of testi