Daily Archives: December 28, 2017

A Software Tester’s journey from manual to political tester


I wrote this some years ago. I should simplify the context and incorporate what I reference from the OP and other responders, so that it stands alone. But this has  meaningful observations which took effort to reach, so I’m putting a copy up here to start with.

Wow, no exaggeration! I can see every event that befell poor Jim happening in the real world. HOWEVER, Jim’s a fortunate fellow, he has management attention at all! AND they look at results. AND there is a perception (no matter how shakily based) of overall product quality.

Jim was no worse than anyone else until he got automation started and mistook his personal satisfaction and enjoyment for the company’s obvious goal of shipping a stable or improving level of quality with fast turnaround on bugs and needed enhancements. This is engineering, not art. Its not self actualization, its a commercial business or a service enterprise which creates value.

All the way down this sad story, Jim accepts product failures, and testing failures. You Can Never Ignore Failures. Period. He should have turned political at that point and realized that Test, like anything, needs to be sold, shown to be valuable and productive, and needs allies. Therefore, tests need to actually be valuable and productive, and needs to make it easy for people to accept them, adopt them, and feel they are important support in their own success. Therefore he needed to measure success, as understood by his customers (developers, support, users) and maintain or improve its integrated value. Accepting failures leads to dead astronauts, wasted billions, wrongful convictions, Senate Select Committees, Frontline specials, sub-Redits, and worse.

Instead of seeing failures as a very, very, high priority, Jim turns into a man with a solution, wandering around, looking for a way to apply it. A tawdry tale, rendered no less tawdry by its oft retelling. Not insignificantly, Jim’s manager is clearly a weak and ineffective character who should have seen problems coming, or reacted when they appeared. Once Jim had made the case for automation, they might have hired someone who knew something about automation, or contracted with very carefully defined goals.

Jim might have split his team up front. He needed manual testers, who carried on the work that had been being done, with as much success as possible, and brain power applied to improve results and lower cost. A front line to hold success. Then a test automation group who focused on test automation with clear and obvious benefits

The automation environment needed to be something:

  • …anyone could run;
  • … which worked from a shippable product, as well as a release candidate or development build;
  • …which could be triggered directly from a product build, so the build group-and-release group ran it every time;
  • …which could be configured to run anything from a single, new, test to all existing tests
    • in a developer’s environment, before check-in, or
    • at any subsequent point, including on a previously shipped release with a support issue.

Setting up the test environment, creating a test to get the product to say “Hello world”, and recognizing that as a test pass ought to take no more than an hour longer than simply setting up the product. That assumption has to be proved every release or two with a calibrated innocent new-hire from somewhere.

Since all tests start by installing the product, license, etc, and starting it, the first thing to automate would be that. If there were changes in that functionality, over product history, the automation could start with the newest, but it had to support them all. Having this ‘smoke test’ be part of a full build would pay dividends to everyone, everywhere, and by designing with backward compatibility and future adaptability, thrash could be minimized, or at least recognized.

This would be a good time to look through the bugbase to determine where the most bugs were being found, where the most escapes were happening, and where the most critical bugs were coming from. Besides looking backward, a forward look at the product roadmap and with developers and management could highlight areas of future leverage

In parallel with automation, all of the above should be considered when selecting manual tests. Tests which never fail should be reduced relative to tests which find failures. Something that fails for diverse reasons may be a great canary in the coal mine, or might be a too fragile sponge that soaks up maintenance effort. In any event, continual improvement of the manual testing should run in parallel with introducing automation. After a small number of releases, the manual tests available should exceed the resources to run them. Selection of the ‘vital few’ should in intentional, not happenstance.

Most people can see the limitation of record and play back, so things should never stop there. The only tools worth looking at are tools that can be edited and rapidly expanded by iteration. Cut and paste is the lowest form of iteration and rapidly grows nightmares. Algorithmic test point generation is desirable, but data driven testing should get strong consideration. Algorithmic generation of literal tables which are then applied as tests separates the thing that needs to be done over and over, running the test, from the thing which is done less frequently, generating the test points.

In my life, I’ve seen a few of the failures in Jim’s story, but a lot of failures of usability by others, or by anyone, complete lack of testing in development plans. Test suites (with running tests) abandoned and no-longer run, until the next great hope get started. And far too little curiosity about which tests should be run, automatically or manually, to get the most bang for the buck.

Like I said, Jim is lucky!

View in discussion

Advertisements