• Hristo Piyankov

Analytics should focus on the questions, not the answers

Updated: Nov 23, 2020


A good analyst delivers useful insights. While a great analyst will first ask all the right questions. One of the frustrations in analytics is wasting a lot of time on analysis just to redo it all over again. Tell me if this sounds familiar:

“I like it! But I needed something else.”

“That’s interesting! But we needed this data two days ago.”

“I know I asked for this. However, it doesn’t really solve my problem.”


This usually occurs when someone needs urgent results and no one spends the time to clarify the goal. Like it or not, the responsibility here lies on the analyst and not the requester. While analytics needs to speak “business”, you can’t expect business to speak “analytics”. So before you rush into preparing your data, spend the time to make sure you are asking the correct questions to the right person.


Can we agree on a one-sentence problem definition?


Before going into details of the problem – do the exact opposite. Agree on a one-sentence problem definition. However, this does not mean that the person requesting the analysis has to offer it to you. Usually, you will need to listen to their detailed story and synthesize the definition yourself. After that make sure that everyone present agrees on it. The reasons for this are quite simple:

  1. Since it’s not defined simple enough, there will be different problem interpretations.

  2. In case you cannot define it in one sentence, there is probably more than one problem.

  3. Moreover, if no one can define it simply, no one really understands it well enough.

As a result, good problem definitions include some basic components. Namely what is needed, to take care of which problem, occurring where and when.


Example of good one-sentence definitions:

  1. Analysis explaining the decrease in the number of online customer logins during this month, compared to the last one.

  2. Forecasting model to estimate average products per customer (cross-sell rate) over the net 3 years, month by month.

  3. Churn prevention predictive model, targeting customers who are likely to close their credit cards within the next 30 days.

  4. Interactive dashboard for regular monitoring of the call centre calling activities on a daily and weekly basis.

  5. A report showing the number of e-mail messages sent last year, week by week.

How urgent is it?


Ironically in the world of business analytics, all is urgent. As a result, if you have not expected the problem in advance and already started working on it – you are already late. Hence this is the most important question of all. The only reason it comes second and not first is that people would feel uneasy if you open with this question even before the problem is even discussed. The answer will define the approach from here on.


More often than not the deadline is “yesterday“. In this case, the best thing you can produce is a report over pre-defined dimensions. For this reason, there would be no time for data deep-dive & research. Seeing that, you are really relying on your business counterpart’s ability to name the problem areas up front. State this and set the right expectations for the time of delivery. Use a split of 60-20-20 for preparing the data, verifying the result, creating the visuals/output formatting.


A general rule of thumb for the development time needed. Nevertheless, those WILL vary greatly based on the situation. Consider double the time for anything you have never done before. And half the time for updating something already existing.

  1. Very basic report – 1 day

  2. Business scenario simulation – 2-3 days

  3. Fully automated & formatted report – 1 week

  4. Business case – 1 week

  5. Interactive dashboard – 2 weeks

  6. In-depth problem analysis across multiple dimensions – 2 weeks

  7. Budgeting model – 2 weeks

  8. Predictive model – 1 month

What is the deadline?


In case you have 2-3 days to work with this is enough time to produce a small to medium-sized analysis. Namely, one problem explored across several dimensions. Prepare your data and do a few iterations of the dimensions as needed. If you have some pre-prepared data or even better a DataMart to work with you have enough time to go through the analytics pipeline (to the right). In case you are asked for a predictive model, the time might be enough for a simple one. For example Market Basket Analysis, Naive Bayes Classifier or a K-Nearest Neighbours. Granted you feel comfortable with those algorithms. Even more so – have the proper tools to perform them.


Any longer deadline and it really starts to depend on a lot of factors. For example, budgeting/forecasting models take weeks. On the other hand, predictive models can take months. How big is your data? Also, do you need to collect the data first? Or do you have a database warehouse? Do you have ready data marts? Have you explored this problem in the past? Are there any specifics to the analytics result which you need to produce?


A high-level analytics pipeline (with % of your deadline reached each point), should look something like this:

  1. (2%) Problem/task clarification with business stakeholder

  2. (25%) Data preparation

  3. (35%) Data validation/checkup

  4. (45%) A rapid prototype of the first solution

  5. (50%) Validate prototype with business stakeholder

  6. (60%) Second data iteration

  7. (80%) Final solution preparation

  8. (95%) Solution validation/checkup

  9. (100%) Finalization of visuals & automation if anъ

Do YOU have any insights on the topic?


You would be amazed how often analysts forget this question. Even straight-up ignore some important facts if they are mentioned “by the way” from their business counterparts. On the other hand, business counterparts quite often forget to state really relevant information.  Asking this is especially important when you are very short on time or have no idea where to start. Almost always, the person asking for the analysis has at least a faint idea where the problem is coming from. This can save you a lot of time. Other times it will keep you from doing a lot of excess work. For instance, it might turn out that the budgeting model you are going to prepare, does not need to start from estimating the client acquisition. But rather this will be a variable and you need to simply predict the cross-sell ratio.


Another use of this question is to identify business changes. Normally, this is not the first place you would look in the data. Moreover, some are not in the data at all. Following up on “Why are our sales decreasing?” with a question such as “Did you expect this?” sometimes yields insightful answers. For instance:  “Well, we actually changed the product pricing/structure”. Or “There were public holidays in a company where we are outsourcing”. This will give you a starting point. From there you can isolate those events from the data and see if there was any other cause on top of them.


Let's discuss the details, please


After you finish with the generic questions it’s time to get into the problem’s details. In my opinion, the skill to break down the problem into smaller issues, even before you start looking at the data will largely define your success as an analyst. Therefore, no straightforward algorithm or list of questions for doing so exists. Unfortunately, each class of problems is unique and each problem within the class has its specifics.  Below I have listed some very common questions for particular areas. However please bear in mind, they are by no means exhaustive.


Reporting

  1. Since similar data is already present in another report – what is missing?

  2. Are you looking for a snapshot of the data or for change over time?

  3. When you open the report, what is the first thing you need to see?

  4. Who else will receive the report? What level of aggregation do they need?

  5. What business actions should data in this report trigger?

Analysis

  1. Has this happened before? What was the cause then?

  2. Were there any significant business structure changes lately?

  3. Is anyone else experiencing this? Other business lines? How about competitors?

  4. Do you suspect the cause is internal or external?

  5. Can we collect feedback from someone who deals with this first hand?

Predictive modelling

  1. On what level is the prediction needed – contract, customer, product line, other?

  2. What output is needed- exact probabilities, set of rules, ranking?

  3. Which is more important: accuracy of the prediction or model transparency/simplicity?

  4. How will the model be put into practice exactly?

  5. What are your criteria if the model is performing well or not?

Budgeting

  1. What is the base – acquisition, existing customer base, running contracts?

  2. Which is the main driver of new sales? New customers or existing ones?

  3. Do you think the last few years are representative of our business? Or do you expect significant internal changes? What about external ones?

  4. Is there seasonality in the business? If yes, on what level: daily, weekly, monthly?

  5. Finally, who and how will the output of this model be used?


Did we forget anything?


As the last step, briefly, go over all the major points on which you have agreed. Put an emphasis on re-stating the one-sentence problem definition. Finally, close with the question “Did we forget anything”? Thus making sure all the important points are covered. Furthermore, you will have a common agreement on the area, timing, and goal of your deliverable. Agree on the next steps, in terms of the pipeline. Are you expected to give a final output? Or can you do a few iterations together? Should you talk with the requester directly? Or should somebody else be the contact point? In either case, who will approve the final result?


In conclusion –  good analysts ask questions. But not because they do not understand, rather they know the correct question has the potential to bring more value than the analysis itself. At the same time, a properly formulated question is all it takes to arrive at the correct conclusion even before you look at the data, saving a lot of time for everyone.

Ironically the most widely spread misconception, across junior analysts, is that questions make them look uninformed and unprepared. While this could happen, the chances for it are actually greater if they don’t ask any questions at all.


#Analysis #Fundamentals #Reporting

8 views