Mutation Testing/Defect Seeding
Intentionally introducing defects in to the software to know the rate of its detection and its surrounding effects. The ratio of the number of seeded defects detected to the total number of defects seeded provides a rough idea of the total number of unseeded defects that have been detected.
Defects Found
Total Defects= 30 * 200 = 400
15
In the above example, 15 out of 30 seeded defects are found. It means test cases have missed 15 seeded defects. If you are doing adhoc testing (where you are not executing test cases) then you have missed 15 scenarios where the defects are unnoticed. It is the best Testers capability measurement technique in following ways:
Risk in defect seeding will be more when seeded defects are ignored for removal from the actual code.
Comments from the people who used this technique(Collected from few sites):
Comment #1:
I looked at mutation test some time ago as a method for checking the efficacy of my automated regession testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.
This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement.
Comment #2:
I use it, and consider it a success.
If you want to use mutation testing for Java, I highly recommend to use the new Javalanche tool by David Schuler rather than Jester and friends. Javalanche manipulates bytecode rather than sourcecode, and is thus orders of magnitudes faster.
Comment #3:
One problem encountered with mutation testing is that it's slow, because by default you do a full test run (either a test file, or a suite of test files) for each mutation generated.
One way to make mutation testing faster would be to stop the test run for a given mutant once a single failure has been encountered (but only during mutation testing). Even better would be for the mutation tester to remember what was the first test to kill the last mutant, and give that first to the next mutant. Is there anything in ruby that does either of these things, or is my best bet to start monkey patching?
Comment #4:
I'm certainly no expert, but I would think that mutation testing would be easier in functional languages. Mutations can done by redefining operators in functional languages as easily as functions are defined in other languages
Comment #5:
Mutation testing is used to test the quality of your test suite. This is done by mutating certain statements in your source code and checking if your test code is able to find the errors. However, mutation testing is very expensive to run, especially on very large applications. There is a mutation testing tool, Jester, which can be used to run mutation tests on Java code. Jester looks at specific areas of your source code, for example: forcing a path through an if statement, changing constant values, and changing Boolean values
Example:
A group intentionally seeded the program with 30 errors, which is seeded widely over the application functionality. When the testing complete, If You find 15 seeded defects and 200 unseeded defects then total number of defects can be derived as follows:
Total Defects = Defects Introduced * Unseeded Defects FoundDefects Found
Total Defects= 30 * 200 = 400
15
In the above example, 15 out of 30 seeded defects are found. It means test cases have missed 15 seeded defects. If you are doing adhoc testing (where you are not executing test cases) then you have missed 15 scenarios where the defects are unnoticed. It is the best Testers capability measurement technique in following ways:
- Determines the functional areas missed while designing the test cases.
- Detemines the number of Scenarios missed while adhoc testing.
Risk in defect seeding will be more when seeded defects are ignored for removal from the actual code.
Comments from the people who used this technique(Collected from few sites):
Comment #1:
I looked at mutation test some time ago as a method for checking the efficacy of my automated regession testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.
This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement.
Comment #2:
I use it, and consider it a success.
If you want to use mutation testing for Java, I highly recommend to use the new Javalanche tool by David Schuler rather than Jester and friends. Javalanche manipulates bytecode rather than sourcecode, and is thus orders of magnitudes faster.
Comment #3:
One problem encountered with mutation testing is that it's slow, because by default you do a full test run (either a test file, or a suite of test files) for each mutation generated.
One way to make mutation testing faster would be to stop the test run for a given mutant once a single failure has been encountered (but only during mutation testing). Even better would be for the mutation tester to remember what was the first test to kill the last mutant, and give that first to the next mutant. Is there anything in ruby that does either of these things, or is my best bet to start monkey patching?
Comment #4:
I'm certainly no expert, but I would think that mutation testing would be easier in functional languages. Mutations can done by redefining operators in functional languages as easily as functions are defined in other languages
Comment #5:
Mutation testing is used to test the quality of your test suite. This is done by mutating certain statements in your source code and checking if your test code is able to find the errors. However, mutation testing is very expensive to run, especially on very large applications. There is a mutation testing tool, Jester, which can be used to run mutation tests on Java code. Jester looks at specific areas of your source code, for example: forcing a path through an if statement, changing constant values, and changing Boolean values
Comments
I would not say I successed, I would not say it was completely failed :)
You can check out some results here:
http://abeletsky.blogspot.com/2010/07/using-of-mutation-testing-in-real.html