"Repro testing" - Do any automated test suites do this?

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

User avatar
Pesto
Posts: 737
Joined: Wed Sep 05, 2007 5:33 pm UTC
Location: Berkeley, CA

"Repro testing" - Do any automated test suites do this?

Postby Pesto » Tue Mar 08, 2011 10:09 pm UTC

I thought up a new feature for an automated testing framework, and am wondering if any existing frameworks do this.

Normally, tests are used to define the desired behavior of a system. When you change something, you can run the tests to make sure that your change does not have any unintended effects on another part of the system.

Let's say you are tasked with going through the bug database and fixing bugs. Being the good engineer that you are, you write a test to express the bug, the test fails, you fix the bug, the test passes and you commit your changes. The existing tests can tell you that you haven't inadvertently broken any existing behavior. However, they don't tell you whether you inadvertently fixed another bug.

This is entirely plausible. If bug X and bug Y are both tracked in the bug database and have the same root cause, fixing X will fix Y, too. The guy the fixes X will mark X as fixed. A couple weeks later when someone else comes along and tries to fix Y, he'll have to spend a lot of time trying to reproduce the bug, eventually closing the item as not reproducible.

Here's what I propose. Instead of, or in addition to, entering something into the bug database, you write a test that reproduces the bug. This way, there is a test that reproduces bug X and a test that reproduces bug Y. When you run the tests, the framework will give you a message "All bugs reproduced." When you fix bug X, then run the tests again, you will get a message "Failure to reproduce bug Y." This way you can immediately mark both bug X and bug Y as fixed.

Does this feature exist in existing testing frameworks?

Carnildo
Posts: 2023
Joined: Fri Jul 18, 2008 8:43 am UTC

Re: "Repro testing" - Do any automated test suites do this?

Postby Carnildo » Wed Mar 09, 2011 9:37 am UTC

Where I work, the effort to reproduce Y will be done off the codebase Y was reported in (eg. the most recent release) before testing against the current development codebase. This makes your situation a non-issue: can reproduce with the reported codebase, cannot reproduce with the development codebase, therefore it was fixed in between.

User avatar
Pesto
Posts: 737
Joined: Wed Sep 05, 2007 5:33 pm UTC
Location: Berkeley, CA

Re: "Repro testing" - Do any automated test suites do this?

Postby Pesto » Wed Mar 09, 2011 4:43 pm UTC

But even that work could be avoided with an automated test to reproduce the bug.

Moose Hole
Posts: 398
Joined: Fri Jul 09, 2010 1:34 pm UTC

Re: "Repro testing" - Do any automated test suites do this?

Postby Moose Hole » Wed Mar 09, 2011 6:01 pm UTC

Sounds logical. Where I work, the bug database has a "steps to reproduce" field that can be followed to reproduce the bug. If bug frequency is "always," then when the engineer goes to fix it and it doesn't fail, he can reasonably assume the bug has already been fixed (with a little bit of due diligence of course).

However, some bug frequencies are "sometimes" or "rarely." In these cases, it is difficult to know if your bug has been fixed, even if you are actively fixing it.

User avatar
Pesto
Posts: 737
Joined: Wed Sep 05, 2007 5:33 pm UTC
Location: Berkeley, CA

Re: "Repro testing" - Do any automated test suites do this?

Postby Pesto » Wed Mar 09, 2011 6:57 pm UTC

Yeah, a system to track intermittent bugs would be more difficult.

You could use this type of thing to gather statistics. Perhaps something like out of the last 100 or 1000 runs the bug was detected X number of times, along with the most recent detection date. If an intermittent bug hasn't been repro'd in six months you could consider it fixed.

User avatar
OmenPigeon
Peddler of Gossamer Lies
Posts: 673
Joined: Mon Sep 25, 2006 6:08 am UTC
Contact:

Re: "Repro testing" - Do any automated test suites do this?

Postby OmenPigeon » Thu Mar 10, 2011 2:40 am UTC

It seems like a lot of work to solve a problem that I, at least, don't actually have. I can't recall a single time that I or anyone I've worked with has accidentally fixed a bug while working on a different one. Occasionally an issue will get reported against a released build that's been fixed in a later unreleased build, but it's relatively rare. And the cost for marking an issue as 'fixed in build x.x.x.x' or 'can't reproduce' is pretty low. So is the cost for searching any decent bug database to make sure there aren't any obvious conflicts.

The costs of your system are going to be somewhat larger, I think. You're going to have to run a whole new slew of tests on every build, which you already know will all fail. After working for some time with a test suite that can take upwards of 90 minutes (and dealing indirectly with test suites that take a full day) I'm leery of anything that's going to add more time to my tests. You're also going to have to make sure that every bug has an automatable reproduction, which imposes a non-trivial cost on your testers, and limits the kinds of people you can have doing testing. You'll also need a tight integration between your bug tracking system, your source control and your test harness. I've not seen that anywhere. This is also going to make it much harder to change your bug tracker, test tools and source control. Granted, many shops change those rarely if ever, but I've seen some pretty big gains from occasionally upgrading core systems as more modern products come out.

In short, I see a pretty marginal gain here with some significant costs. I'm skeptical that it would be worth it.
As long as I am alive and well I will continue to feel strongly about prose style, to love the surface of the earth, and to take pleasure in scraps of useless information.
~ George Orwell

User avatar
Pesto
Posts: 737
Joined: Wed Sep 05, 2007 5:33 pm UTC
Location: Berkeley, CA

Re: "Repro testing" - Do any automated test suites do this?

Postby Pesto » Sun Mar 13, 2011 1:38 am UTC

Yeah, if you've got a decently structured application and good test coverage already, it probably wouldn't be worth it. Unfortunately that's not the case where I work.

The application I work on is a giant pile of PHP spaghetti with no automated tests to speak of. We've started writing automated tests, but at this point any automated tests will be an improvement. And the code is in such a state that changing one thing one place could potentially have ripples all across the application. Plus I expect we have plenty of QA and support people who could probably handle writing test cases. I know the QA on my team writes a lot of tests.


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 8 guests