Blog posts on software testing

Rss logo

Posts selected fromManagement Blog - Engineering Blog - Investing Blog and other blogs

  • Software Testers Are Test Pilots

    Software testers should be test pilots. Too many people think software testing is the pre-flight checklist an airline pilot uses.

    ...

    Checklists are extremely useful in software development. And using checklist-type automated tests is a valuable part of maintaining and developing software. But those pass-fail tests are equivalent to checklists - they provide a standardized way to check that planned checks pass. They are not equivalent to thoughtful testing by a software testing professional.

    continue reading: Software Testers Are Test Pilots

  • The Edge-case Excuse

    I have found “edge cases” to actually mean we don’t want to fix it. Often the issue isn’t needing some special code to deal with an “edge case” it is the coding was done poorly and breaks in many different “edge cases.” It isn’t that those edge cases need to be coded for. It is that the code should have been written in a robust way that didn’t break for lots of “edge cases” but the excuse given for not fixing the fundamental coding fragility is the bugs found are just “edge cases.”

    continue reading: The Edge-case Excuse

  • Software Testing and the Impact on Quality

    The aim for software testing is not just to fix the bugs software testers catch but to function as part of a system to figure out the reasons those bugs were created and improve processes so fewer bugs are created in the future.

    continue reading: Software Testing and the Impact on Quality

  • Hexawise Buys the Beers

    George Box was an amazing person, scientist and statistician. One of the traditions George started in Madison, Wisconsin was the Monday Night Beer Sessions.

    An excerpt of Mac Berthouex’s introduction to An Accidental Statistician: The Life and Memories of George E. P. Box:

    I met George Box in 1968 at the long-running hit show that he called “The Monday Night Beer Session,” an informal discussion group that met in the basement of his house. I was taking Bill Hunter’s course in nonlinear model building. Bill suggested that I should go and talk about some research we were doing.

    Hexawise has decided to bring this tradition to software testing...

    continue reading: Hexawise Buys the Beers

  • 10 Questions to Ask When Designing Software Tests

    Too often software testing fails to emphasis the importance of experienced software testers using their brains, insight, experience and knowledge to make judgements about what is the best course of action. Hexawise is a wonderful tool to help software testers but the tool needs a smart and thoughtful person to wield it and produce the results we all want to see.

    continue reading: 10 Questions to Ask When Designing Software Tests

  • 6 Questions to Ask Your Software Testers

    Making sure your software testing process is staying current with the best ideas in software testing is an important factor in creating great software solutions that your customers love. Often companies understand the need to stay current on software coding practices that are successful but fewer organizations pay attention to good practices in software testing. This often means there is a good deal to be gained by spending some time to examine and improve your software testing practices.

    continue reading: 6 Questions to Ask Your Software Testers

  • 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing

    Studies show that 84% of defects found in production could have been found by testing every pair of parameter values.  Those bugs were the result of interactions between 2 parameters.  The complexity of software means there are many interactions and as the data shows often the bugs seen in production are the result of such interactions.

    continue reading: 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing

  • Examine the Results of Your Testing Practices and Continually Improve Your Methods

    The idea of delibrately examining your software development and testing practices will be familar to those using agile retrospectives. The power of continually improving the development practices used withing the organization is hard to appreciate but it is immense. The gains compound over time so the initial benefits are only a glimpse of what can be achieve by continuing to iterate and improve.

    continue reading: Examine the Results of Your Testing Practices and Continually Improve Your Methods

  • Add Constraints to Processes Carefully

    Frequently I see unnecessary constraints creating the edge case excuse. By burdening your process with unnecessary constraints you create edge cases that fail and then use the excuse that each of the edge cases is rare and therefore you can’t justify the expense of fixing them.

    But if you designed the process sensibly in the first place the edge case never would have failed and you wouldn’t need special work arounds for such “edge cases.” A simple example of this is unnecessarily complex web page code that fails if to submit a button without javascript. Yes, a small number of users won’t have enabled all javascript to run (today anyway) so it is an “edge case” to deal with if you don’t have the form work without javascript. But there is no decent reason to have it fail in most cases.

    continue reading: Add Constraints to Processes Carefully

  • Don’t Assume All Web Users Have a Fiber-like Connection

    I have pointed out for more than a decade how poorly many web pages are coded. They often assume a very low latency connection and without it the user experience is poor.

    ...

    I can only assume they just test the pages on their fiber connection and if it works they are ok. This is a very bad idea for nearly every website.

    Learn to get a deep understanding of your customers and potential customers with focus on the customer’s “Jobs to be Done.”  Otherwise organizations find themelves creating solutions that work in test conditions but that don't work for their customers (or potential customers).

    continue reading: Don’t Assume All Web Users Have a Fiber-like Connection

  • Software Code Reviews from a Deming Perspective

    I think the “inspection” in code reviews is different enough that we can use code reviews as a valuable tool for managing software development. The waste of having processes that create defects and then use inspection to catch them is certainly something to avoid. A significant part of the effort in code reviews should be geared toward capturing learning that can be applied to current processes to improve them so fewer bugs are created in the future.

    In my experience this part of code reviews (using it to improve the existing processes) is not given the focus it should be. So I do believe that code reviews should focus more on why did we find something we decided to fix?

    continue reading: Software Code Reviews from a Deming Perspective

  • Testing Smarter with James Bach

    This is the first in a series of interviews by me aimed at highlighting insights and experiences by many of the software testing field’s leading thinkers.

    I test them using social science methods. That, after all, is how scientists attempt to test their theories about social life. That means an emphasis on qualitative analysis, but bringing in statistical methods whenever applicable.

    I agree that the medical world is a good example of where statistical methods and heuristic approaches are also needed. In testing complex things, some of what you need to do includes:

    • You must use time to your advantage-- observing systems over time the way primatologists observe chimps in the wild.
    • You must use Grounded Theory, beginning with immersion and observation, until patterns begin to reveal themselves.

    continue reading: Testing Smarter with James Bach

  • Testing Smarter with Dorothy Graham

    This interview with Dorothy Graham is part of my series of “Testing Smarter with…” interviews: gaining insights and experiences from many of the software testing field’s leading thinkers.

    The best way to have fewer bugs in the software is not to put them in in the first place. I would like to see testers become bug prevention advisors! And I think this is what happens on good teams.

    continue reading: Testing Smarter with Dorothy Graham

  • Testing Smarter with Alan Page

     This interview with Alan Page is part of my series of “Testing Smarter with…” interviews: gaining insights and experiences from many of the software testing field’s leading thinkers.

    The biggest thing I look for in testers is a passion and ability to learn. I've interviewed hundreds of testers... The testers who really impress me are those who love to learn - not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

    Alan spent over twenty years at Microsoft working on a variety of operating systems and applications in nearly every Microsoft division and now works at Unity.

    Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That's 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

    continue reading: Testing Smarter with Alan Page

  • Testing Smarter with Mike Bland

    This interview with Mike Bland is part of my series of “Testing Smarter with…” interviews: gaining insights and experiences from many of the software testing field’s leading thinkers.

    it’s not about defects; it’s about feedback and collaboration. If you arrange incentives to produce an adversarial relationship between team members, e.g. if developers are incentivized to minimize defects and testers are incentivized to report defects, then that’s a house divided against itself.

    Mike Bland aims to produce a culture of transparency, autonomy, and collaboration, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He's followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet.

    my advice to both developers and testers is to identify the priorities, the social structures and dynamics at play in the organization. How can you work with these structures and dynamics instead of against them—or do you need to create a culture of open communication and collaboration in parallel with (or even before) communicating the testing message?

    continue reading: Testing Smarter with Mike Bland