We all know that there’s no such thing as a stupid question, but there are certainly stupid answers. All this recent talk of heuristics caused my brain to come up with what I’m calling The Sucky Answer Heuristic.
The Sucky Answer Heuristic is a specific instance of a more general emotional heuristic; if you feel an emotion, something is causing you to feel that emotion. If you’re testing something and have a response then – assuming you’re emotionally stable at the time – that response was likely triggered by something about the thing you are testing.
In this case, if you ask a question and the answer causes you to exclaim some version of “That sucks!”, then it’s worth examining further, and you should keep asking questions until the answers stop sucking, or they only suck. I’ll explain more about when to stop asking questions later.
What’s the point of focusing on Sucky Answers?
A sucky answer to a question is likely an indication of a lack of knowledge. If you’re asking a question of some oracle – which, as a tester, is quite likely – then a sucky response means your oracle isn’t a good oracle, or an oracle at all.
In either case, that’s a risk, and it’s our place to highlight all risks that threaten the value the product is meant to deliver.
Over time, as the product scope and architecture stabilises, the answers to the questions should improve as your oracles have a much better idea of what they’re producing. If the answers are getting worse, you’ve got problems.
In the worst case, really sucky answers could mean that you should stop asking questions, or perhaps even look for another role.
What does a sucky answer feel like?
How do you know what a sucky answer feels like? Well, it feels sucky. You get the answer and are left feeling unsatisfied, or more worried, or frustrated, or any number of negative emotions. It’s a subconscious, subjective judgement, but that’s fine.
Quality in all things is subjective, so the reaction to a poor quality response will also be subjective. However, here are some possible indicators of Sucky Answers, in ascending order of suckiness;
- An answer that contradicts what you already “know” (this is “good-sucky”; it suggests some clarification is required.)
- An answer that takes a while to arrive (though there could be reasons other than the suckiness of the answer for the delay)
- An answer that answers the question very vaguely (either in delivery, or content)
- An answer that doesn’t answer the question, or answers a different question
- An answer never even appears
- A response that doesn’t acknowledge the validity of the question; “That’s not a sensible question.”
- A response that doesn’t acknowledge the validity of the questioner; “Why are you asking that question?”
“The answer was sucky because it wasn’t the answer I wanted!”
If you get an answer that wasn’t what you wanted, it’s not necessarily sucky. If a decision is made that you don’t agree with, but the reasoning is explained and is sound and logical, then that’s not sucky. Those sorts of answers you have to accept.
Continuing to ask questions because you don’t like that answer will, in this situation, not improve quality and will just label you as a pain in the ass who always wants things their way.
OK, so how do I work this thing?
You already know how to work this thing. You do this already. It’s not a new idea, I’ve just given it a stupid name. It probably already has a sensible name, of which, in my ignorance, I am unaware.
That said, when I was thinking about this, a picture was appearing in my brain, so I drew it. It has some interesting bits on it that are worth explaining, that might help you decide how to use this feeling of suckiness more effectively.
Broadly, this is a graph of Answer Quality against Time. The Time axis can be a duration of your choosing. I chose the length of a release because, as I covered above, I expect answer quality to gradually increase over the course of a release until all questions are satisfactorily answered before release.
Each time a question is posed and an answer received, we rate the quality of the response, based on the criteria I listed above. The wobbly line is a rolling average of all answer quality scores. A poor answer drags the average down, a good one improves it.
Product Problems
Let’s focus above the X axis first. In this region, the answers are coming, but are delayed, or vague, or confused. I term this the Product Problem zone, because sucky answers point to likely issues with the product. It’s where people don’t yet know everything about the product – which is a risk – so their answers are a bit sucky.
Over time, as people learn more about the product and the scope is clarified, answers should suck less and the answer score average should improve.
When some major aspect of the release changes – the scope changes, a new platform is introduced, a feature is dropped / refactored – the level of uncertainty increases, answers get suckier, and the average falls.
Hopefully, at some point in the release, the answers you get will be readily available and pretty non-sucky. This is your personal “good enough” level. You want your answer to continue to be at least this good to feel confident about how things are going. Not that you should be asked if you feel the product is “good enough” to be released but, if you were, this would be one gut-check you could use.
If the answers are “good enough”, can I stop testing? As long as the product has stopped changing, you could. But as long as things are changing, and new things to be known are being created, you need to keep asking questions and sense-checking the answers. If you declare victory too early and take your eye off the ball, you could not ask a question that would have uncovered a major issue. So, regardless of how good the answers are, keep testing until you have to stop.
If you run out of questions, that’s a different problem. If you’ve run out of testing time, it’s probably not a problem. However, if you do have time left, you need to figure out how to ask more questions.
All this is well and good, and gives you a sense of how things are going, but’s that only useful if you communicate your findings to the right people. You are going to have to translate “My answer suckiness rating is 4” into something that those people understand. Risks are good. Managers understand risks.
Organisational Problems
Let’s now dip below the X axis. Down here are more fundamental problems. Down here your questions aren’t answered. The questions themselves are questioned. You are questioned for asking questions.
Down here, you are no longer able to add value to the product because elements of your organisation are preventing you from doing so. Your place as a valuable member of the organisation is in question.
Any questions you ask are seen as a negative and are interpreted as being destructive. Asking more only makes things worse.
You need to flag these sorts of responses up ASAP to anyone up to and including HR. If it’s only certain people / oracles who give these responses, you can possibly work around it, but if the problem is more systemic, then it’s probably time to look for a new company.
Summary
Pay attention to the responses you get when you ask questions. If you don’t like them, or how you got them, ask yourself why. Ask more questions, and flag up any risks that you identify.
Things will always suck a bit, but if you call attention to the suckiness you can use it as a way to help the team focus on areas of risk.