Test Automation vs. Mechanization

Wiki has a clear definition for both of those terms:

Mechanization provided human operators with machinery to assist them with the muscular requirements of work. Whereas automation is the use of control systems and information technologies reducing the need for human intervention. In the scope of industrialization, automation is a step beyond mechanization.

In testing, however, I see those two concepts often dangerously mixed: people tend to mechanize testing, thinking they are automating it. As a result, the estimated value of such “automation” is completely wrong: it will not take risks into account, it will not find important regressions, and it will give a tester a false sense of safety and completeness.  Why? Because of the way mechanization is created.

Both automation and mechanization have their value in testing. But they are different, and therefore should always be clearly distinguished. Whereas automation is a result of application risk analysis, based on knowledge of the application and understanding of testing needs (and thus finding tools and ways to cover those risks), mechanization  goes from the opposite direction: it looks for out of box tools and easiest ways to mechanize some testing or code review procedures, and makes an opportunistic usage of those methods. Mechanization, for instance, is great for evaluating a module/function/piece of code. It is, if you will, a quality control tool, but it does not eliminate a need for quality assurance testing, either manual or automated.

It’s like making a car: each detail of the car was inspected (probably in mechanized, or very standardized way) and has a “QC” stamp on it. But after all details were assembled, do they know if and how the car will drive? If car manufacturers did, they would not have to pay their test drivers. Yet, test drivers are the ones that provide the most valuable input, that is closest to actual consumer’s experience. In software, testers can play a role of both, quality control personnel, and test drivers. Taking away one of those roles, or thinking that test drive can be replaced with “QC” stamp is not the way to optimize testing. And bare existence of mechanical testing should not be the reason to consider application “risk free” or “regression proof”. Otherwise you may end up in a situation where your car does not drive, and you don’t even know.

Test Automation vs. Mechanization

First test your testers, then trust your testers

Quite frankly I am lucky: developers I worked with (well, most of them) were quite open to interactions with testers, didn’t take the “bad news” testers brought personally, and were able to learn to work with testers efficiently. In other words – they valued services testers provided, which I think is one of the most important factors in this relationship.
However, even those, who valued testers, often pointed that testing lacks some predictability, and the path tester follows looks almost accidental. And this is where I think the major difference between developers and testers is: while developers’ work is algorithmic by nature (define-design-do), and thus it’s possible to develop a fairly detailed plan ahead of time, same level of planning may not be possible when you try to break things. It doesn’t mean that testers always “shoot in the dark”: they have to have a master plan and initial evaluation of risks. But very often the most efficient next step tester takes depends to the large extend on the findings and outcomes of the previous one. And thus it can be difficult for a tester to specify precisely what and how he or she will be testing in several days from now.
Another aspect: developers have a clear finish line. Developer is done, when he or she implemented the required functionality, and it works (whatever this means for particular culture, project or a developer). For testers the finish line is always arbitrary (you can barely run out of test cases), and can only be defined as an abstract moment when tester ran out of important test cases. The term important is very well defined in the context of testing: important to stakeholders, company reputation, etc. But since it doesn’t have an associated precise mathematical formula, testers often have hard time explaining the others when they will be done, causing developers to feel uneasy. It takes some skill and experience (not to mention talent, which I think is the main factor of any work) for tester to choose the best point in time when tester can say “I’m done”. And it takes some skill and experience for developers to trust tester’s judgment. First test your testers, then trust your testers. The first part of this statement can be a subject of a separate post, and the second is described well in Product Bistro: Love Developers, and Trust QA by Mitchell Ashley.

First test your testers, then trust your testers

On regressions and excuses

Two of the most popular (and least pleasant) conversations between testers and developers, go like this:

    • Testers: “It turns they changed it, but they didn’t bother to tell us!”
      Dev: “Oh, we didn’t think it was important!”
    • Testers: “We told them about this issue long time ago, but they didn’t care”
      Dev: “From their explanation it didn’t sound like an important issue”

      However those conversations are easily eliminated, by following two simple rules:

      • Any change that can affect some portion of the code somewhere should cause a discussion with the goal to understand the risks. Change can be “accepted” with no further investigation/testing if it’s not considered to be risky by all sides, or it can cause the need for additional investigation and testing. I don’t call for discussion around each line of code: I’m sure every team can come up with a unit that makes sense for a particular product (e.g. component, module, or any other definition of functional area). And such discussion is especially important when no other changes are done, no extensive testing is planned in the same functional area, and thus regressions can slip undiscovered.
      • Similarly any issue or concern that is being raised, should not be discarded without understanding how it affects the product, which risks it involves, whether it requires further investigation and testing, etc. This especially applies to those “last day” findings, which often are discarded too fast, because everyone is in “Let’s ship it!” mood.
      On regressions and excuses