In the introduction to the Win/Loss Analysis in FinTech interview series, “It’s Not What You Think!”, we launched a discussion about topics including what is win/loss analysis, who it is for, why FinTech is different, and how to run an effective analysis program in that market. In this interview, Richard Case and I talk about blind interviews, including issues around confidentiality and anonymity, and how to keep human bias from sneaking into win/loss study results.
Dan: Richard, if you survey FinTech buyers about anything, you have to preserve their confidentiality. Otherwise, people won’t even take the call. Nobody wants to risk exposing their source of advantage to their competition.That trust is essential.
[see: Long/Short Asymmetries: Are Win & Loss Factors Different in FinTech Win/Loss Analysis?].
Richard: Right. And not only should the information they give be kept confidential, to take that a step further, survey responses should be anonymous - even to the sponsor of the study. Respondents must feel comfortable to speak openly about their reasons for buying or not buying something, even to criticize sellers and their offerings.
Dan: Sure. That anonymity protects respondents from having their words come back to haunt them. Sometimes, a respondent gets into a sensitive topic and pauses to confirm with me that the interview will be anonymous. After I reassure him that it is, he goes ahead and talks about things that I know he wouldn’t have discussed otherwise.
Richard: Exactly. But there’s a second, and bigger issue. And that is that win/loss analysis generally doesn’t work unless the interviews are not only anonymous, but are also blind, wherever that is possible.
Dan: And here you’re referring to the respondent being “blind” in the sense that he doesn’t know who has commissioned the research.
Richard: Right. Of course, the interviewer knows who he’s talking to, but the respondent doesn’t know who we are working for.
Dan: Absolutely. That’s critical for removing bias and not just for win/loss research. I was doing an interview-based study for a vendor who wanted to study how hedge funds use AI technology for front office trading applications, and mid-to-back office risk and compliance applications. It was a survey of use cases to guide the vendor’s product development. It was blind, as I didn’t tell the respondents who the vendor was. However, one interviewee correctly guessed the interview sponsor immediately.
Richard: Did you confirm the respondent’s guess?
Dan: No, I didn’t indicate at all whether he was right or wrong in his guess. So, the interviewee went on to say something to the effect that “if this is [vendor name] asking, then I say…” and finished his answer with a message aimed at the specific vendor. Other respondents guessed too, but they guessed wrong. Most had no idea. How might this have affected the results?
Richard: Well, did you detect a difference in the answers between the one that guessed right and those that guessed wrong?
Dan: Yes, possibly. But I think that was because the respondent who guessed correctly was using his own in-house-built technology, similar to what the vendor was able to supply, and knew the vendor space for that technology quite well. None of the other respondents who guessed knew the technology and space nearly as well.
Richard: Yes, this sometimes happens to me, too. People try to guess the interview sponsor. They've never guessed right, plenty guessed wrong. Some got some intel from their sales rep. The issue is that when you detect bias in a survey response, sometimes you need to throw away the results. I sometimes ask the sponsor a simple yes or no question - here's someone who guessed who's asking the question - do you want to keep the transcript? If they say “No,” you have to throw it away.
Dan: Such a waste!
Richard: Yeah, but you have to. Because the bias skews the results. When people guess (or are told, if the interview isn’t blind) who commissioned the report, the results focus on something narrow. Blind and anonymous, however, pays off in the quality of the interview results. When respondents don’t know who commissioned the research, they give you more reliable information. They talk about pricing, lock-in, and other critical issues. They give you the unvarnished truth only if they don’t know the sponsor.
[For more about anonymous and blind interviews, see: Win/Loss Best Practice #1: Confidential Interviews]
Dan: I think this larger truth plays out differently, depending on who you are talking to, at least in FinTech.
Richard: How so?
Dan: In FinTech, and I think in general, there are two types of buyers: In the back office, buyers in cost centers tend to want solutions that are tried and true, they want price efficiency, they want one throat to choke - possibly a single vendor, which is easier to manage. But in the front office, buyers in a function attached to revenue tend to want best-of-breed solutions. In FinTech, trading is where the money is made. These buyers want customization and performance more than just cost efficiency. These two groups want fundamentally different things.
Richard: Yeah, you could look at it that way, but how does this make a difference?
Dan: Well, in the back office, where buyers want tried and true, price efficiency, one neck to choke… if they get what they want, then they lose power to the vendor who controls their life. And they don't want to divulge that in an interview if they know who's asking.
Richard: Yes, a respondent who knows who will read their answers could be expected to tailor remarks. This is definitely important.
Dan: But in the front office, where buyers want best-of-breed, because they need the performance, the buyer will take a risk on something new. Because this is close to their secret sauce. So if they know who’s asking the survey questions, they just simply won’t open up about issues in the same way. And there’s nothing you’re going to be able to do, to get what is really needed from them. I think you have to make it clear from the outset that the interview is confidential or you will run into this. Once their book is closed, it’s closed for good.
Richard: Yes. We’re not magicians. We can’t pull information out of people that they won’t express. And we’re not “pontificators.” We need actionable data, we need the truth.
Dan: So, in either case, front or back office, confidentiality of the interview–both protecting the identity of the respondent and the client–is very important to getting that unbiased feedback you want. That gets to the truth. People are more willing to open up. Only when you get to the truth does the customer get the value they expect from the study.