The Subjective Nature of Data: The Danger of Narrow Queries

Every day we encounter statements, reports, and decisions based on data. The data is paraded as “fact” and the conclusions derived from that data as “indisputable”.

Data in its natural state may be fact. However, as soon as it is touched by human hands to be catalogued or filtered - data becomes subjective.

This is inescapable. Defining data categories, setting priorities on certain data points, and filtering collected data – all these activities that make data useful are unavoidably subjective.

Since subjectivity cannot be avoided, what strategies can balance that subjectivity?

Over the next few blog posts, I offer you some strategies that have worked well for me.

Here is the first:

Subjective Weakness: Using a narrow data query to prove a point.

Data Discipline: Widen the data query to expose other stories contained in the data.

 Maybe an example would help here.

A number of years ago, one of my customers ran a cardholder behaviour promotion. They provided a $5 credit to any cardholder setting up their first recurring payment. The qualifying recurring payment had to be with a cell phone company.

Data queries run during and immediately after the promotion showed a significant increase in cell phone recurring payments across their cardholder base.

About a year later, this customer ran the same data query to measure the “stickiness” of the promoted behaviour. Their analysis caused an uproar as it showed only 20% of those recurring payments still existed.

The uproar landed on my desk.

I looked at their query and the data it had returned. Their report had all the rewarded card numbers. In 80% of the cases, the qualifying recurring payment was no longer found in the cardholder’s transaction record.

So, I widened the data query to see if there was another story to be found.

I took the list of qualifying card numbers and ran a query of ALL transactions for each cardholder for the most recent 3 months. I then went through each card number looking for cell phone bill payments and for recurring payments.

Full disclosure: by the time I went through 100,000 of the 300,000 card numbers I had found a consistent pattern. The customer later validated my results against the remaining 200,000 card numbers.

What did I find*?

• 20% of the card numbers had no recurring payments in the past 3 months,

• 20% of the card numbers had cell phone payments (not recurring) that were flat dollar amounts (i.e. $50). As these payments were unscheduled, I felt safe assuming these cardholders had replaced their monthly cell phone plan with a prepaid plan (a very popular option at that time).

• 40% of the card numbers had recurring charges at cell phone companies that matched monthly fees on that company’s website – though these were different cell phone companies than the cardholder had used during the promotion. It appeared these cardholders had switched cell phone providers but had set up recurring payments with their new provider.

• 20% had no cell phone bills on their statement.

* The percentages are representative but reflect the proportions found in my research.

That made this promotion 60% successful after 1 year instead of 20% successful – much better results.

There was one other surprise hiding in the data. Though the promotion focused on cell phone bills, one caveat was that this would be the cardholder’s FIRST recurring payment setup – independent of merchant category code. I culled through the “failing 40%” of the cardholders. I found that half of these cardholders had active recurring payments with other merchants.

If the original query had verified that each qualifying cardholder had started with no recurring payments – this promotion now had a success rate of 80%!

The issue changed from “uproar” to “celebration” – all because of a wider data query that looked for alternate narratives contained in the same data.

Not every wider data query ends in celebration. Some debunk celebrated triumphs. However, in each case, a wider query and deeper analysis is key to making data more accountable to reality.