Please read two articles attached and discuss the key points.

Let Curiosity Drive: Fostering Innovation in Data Science

ERIC COLSON, Chief Algorithms Officer, Stitch Fix

January 18, 2019 – San Francisco, CA

Source:

https://multithreaded.stitchfix.com/blog/2019/01/18/fostering-innovation-in-data-science/

The real value of data science lies not in making existing processes incrementally more efficient but rather in the creation of new algorithmic capabilities that enable step-function changes in value. However, such capabilities are rarely asked for in a top-down fashion. Instead, they are discovered and revealed through curiosity-driven tinkering by data scientists. For companies ready to jump on the data science bandwagon I offer this advice: think less about how data science will support and execute your plans and think more about how to create an environment to empower your data scientists to come up with ideas you’ve never dreamed of.

At Stitch Fix, we have more than 100 data scientists who have created several dozens of algorithmic capabilities, generating 100s of millions of dollars in benefits. We have algorithms for recommender systems, merchandise buying, inventory management, client relationship management, logistics, operations—we even have algorithms for designing clothes! Each provides material and measurable returns, enabling us to better serve our clients, while providing a protective barrier against competition. Yet, virtually none of these capabilities were asked for—not by executives, product managers, or domain experts, and not even by a data science manager. Instead, they were born out of curiosity and extracurricular tinkering by data scientists.

Data scientists are a curious bunch, especially the talented ones. They work towards stated goals, and they are focused on and accountable to achieving certain performance metrics. But they are also easily distracted—in a good way. In the course of doing their work they stumble on various patterns, phenomena, and anomalies that are unearthed during their data sleuthing. This goads the data scientist’s curiosity: “Is there a latent dimension that can characterize a client’s style?” “If we modeled clothing fit as a distance measure could we improve client feedback?” “Can successful features from existing styles be systematically re-combined to create better ones?” Such curiosity questions can be insatiable and the data scientists knows the answers lie hidden in the reams of historical data. Tinkering ensues. They don’t ask permission (eafp). In some cases, explanations can be found quickly, in only a few hours or so. Other times, it takes longer because each answer evokes new questions and hypotheses, leading to more tinkering. But the work is contained to non-sanctioned side-work, at least for now. They’ll tinker on their own time if they need to—evening and weekend if they must. Yet, no one asked them to; curiosity is a powerful force.

Are they wasting their time? No! Data science tinkering is typically accompanied by evidence for the merits of the exploration. Statistical measures like AUC, RMSE, and R-squared quantify the amount of predictive power the data scientist’s exploration is adding. They are also equipped with the business context to allow them to assess viability and potential impact of a solution that leverages their new insights. If there is no “there” there, they stop. But, when compelling evidence is found and coupled with big potential, the data scientist is emboldened. The exploration flips from being curiosity-driven to impact-driven. “If we incorporate this latent style space into our styling algorithms we can better recommend products.” “This fit feature will materially increase client satisfaction.” “These new designs will do very well with this client segment.” Note the difference in tone. Much of the uncertainty has been allayed and replaced with potential impact. No longer satisfied with mere historical data, the data scientist is driven to more rigorous methods—randomized controlled trials or “AB Testing,” which can provide true causal impact. She wants to see how her insights perform in real life. She cobbles together a new algorithm based on the newly revealed insights and exposes it to a sample of clients in an experiment. She’s already confident it will improve the client experience and business metrics, but she needs to know by how much. If the experiment yields a big enough win, she’ll roll it out to all clients. In some cases, it may require additional work to build a robust capability around her new insights. This will almost surely go beyond what can be considered “side work” and she’ll need to collaborate with others for engineering and process changes. But she will have already validated her hypothesis and quantified the impact, giving her a clear case for its prioritization within the business.

The essential thing to note here is that no one asked the data scientists to explore. Managers, PMs, domain experts—none of them saw the unexplained phenomenon that the data scientist stumbled upon. This is what tipped her off to start tinkering. And, the data scientist didn’t have to ask permission to explore because it’s low-cost enough that it just happens fluidly in the course of their work, or they are compelled by curiosity to flesh it out on their own time. In fact, if they had asked permission to explore their initial itch, managers and stakeholders probably would have said “no.” The insights and resulting capabilities are often so unintuitive and/or esoteric that, without the evidence to support it, they don’t seem like a good use of time or resources.

These two things—low cost exploration and empirical evidence—set data science apart from other business functions. Sure, other departments are curious too: “I wonder if clients would respond better to this this type of creative?” a marketer might ponder. “Would a new user interface be more intuitive?” a product manager inquires, etc. But those questions can’t be answered with historical data. Exploring those ideas requires actually building something, which is costly. And justifying the cost is often difficult since there’s no evidence that suggests the ideas will work. But with data science’s low-cost exploration and risk-reducing evidence, more ideas are explored which, in turn, leads to more innovation.

Sounds great, right? It is! But this doesn’t happen by will alone. You can’t just declare as an organization that “we’ll do this too.” This is a very different way of doing things. Many established organizations are set up to resist change. Such a new approach can create so much friction with the existing processes that the organization rejects it in the same way antibodies attack a foreign substance entering the body. It’s going to require fundamental changes to the organization that extend beyond the addition of a data science team. You need to create an environment in which it can thrive.

First, you have to position data science as its own entity. Don’t bury it under another department like marketing, product, engineering, etc. Instead, make it its own department, reporting to the CEO. In some cases the data science team can be completely autonomous in producing value for the company. In other cases, it will need to collaborate with other departments to provide solutions. Yet, it will do so as equal partners—not as a support staff that merely executes on what is asked of them. Recall that most algorithmic capabilities won’t be asked for; they are discovered through exploration. So, instead of positioning data science as a supportive team in service to other departments, make it responsible for business goals. Then, hold it accountable to hitting those goals—but enable the data scientists to come up with the solutions.

Next, you need to equip the data scientists with all the technical resources they need to be autonomous. They’ll need full access to data as well as the compute resources to process their explorations. Requiring them to ask permission or request resources will impose a cost and less exploration will occur. My recommendation is to leverage a cloud architecture where the compute resources are elastic and nearly infinite.

The data scientists will also need to have the skills to provision their own processors and conduct their own exploration. They will have to be great generalists. Most companies divide their data scientists into teams of functional specialists—say, Modelers, Machine Learning Engineers, Data Engineers, Causal Inference Analysts, etc. While this may provide greater focus, it also necessitates coordination among many specialists to pursue any exploration. This increases costs and fewer explorations will be conducted. Instead, leverage “full-stack data scientists” that possess varied skills to do all the specialty functions. Of course, data scientists can’t be experts in everything. Providing a robust data platform can help abstract them from the intricacies of distributed processing, auto-scaling, graceful degradation, etc. This way the data scientist focuses more on driving business value through testing and learning, and less on technical specialty. The cost of exploration is lowered and therefore more things are tried, leading to more innovation.

Finally, you need a culture that will support a steady process of learning and experimentation. This means the entire company must have common values for things like learning by doing, being comfortable with ambiguity, and balancing long- and short-term returns. These values need to be shared across the entire organization as they cannot survive in isolation.

Before you jump in and implement this at your company, be aware that it will be hard if not impossible to implement at an older, more established company. I’m not sure it could have worked, even at Stitch Fix, if we hadn’t enabled data science to be successful from the very beginning. Data Science was not “inserted” into the organization. Rather, data science was native to us even in the formative years, and hence, the necessary ways-of-working are more natural.

This is not to say data science is necessarily destined for failure at older, more mature companies, although it is certainly harder than starting from scratch. Some companies have been able to pull off miraculous changes. It’s too important not to try. The benefits of this model are substantial, and for companies that have the data assets to create a sustaining competitive advantage through algorithmic capabilities, it’s worth considering whether this approach can work for you.

Postscript

People often ask me, “Why not provide the time for data scientists to be creative like Google’s 20 percent time?” We’ve considered this several times after seeing many successful innovations emerge from data science tinkering. In spirit, it’s a great idea. Yet, we have concerns that a structured program for innovation may have unintended consequences.

Such programs may be too open-ended and lead to research where there is no actual problem to solve. The real examples I depicted above all stemmed out of observation—patterns, anomalies, an unexplained phenomenon, etc. They were observed first and then researched. It’s less likely to lead to impact the other way around.

Structured programs may also set expectations too high. I suspect there would be a tendency to think of the creative time as a PhD dissertation, requiring novelty and a material contribution to the community (e.g., “I’d better consult with my manager on what to spend my 20 percent time on”). I’d prefer a more organic process that is driven from observation. The data scientists should feel no shame in switching topics often or modifying their hypotheses. They may even find that their stated priorities are the most important thing they can be doing.

So, instead of a structured program, let curiosity drive. By providing ownership of business goals and generalized roles, tinkering and exploration become a natural and fluid part of the role. In fact, it’s hard to quell curiosity. Even if one were to explicitly ban curiosity projects, data scientists would simply conduct their explorations surreptitiously. Itches must be scratched!

Reference

Colson, E. (2019). Let Curiosity Drive: Fostering Innovation in Data Science, retrieved from

https://multithreaded.stitchfix.com/blog/2019/01/18/fostering-innovation-in-data-science/

Why Google defined a new discipline to help humans make decisions

Machine-learning systems are only as smart as their training data. So Google formalized the marshaling of hard and soft sciences that go into its decisions.

By Ciara Byrne

Cassie Kozyrkov is Google’s first-ever chief decision officer. She has already trained 17,000 Googlers to make better decisions by augmenting data science with psychology, neuroscience, economics, and managerial science. Now Google wants to share this new discipline–which it calls Decision Intelligence Engineering–with the world.

At Google, the need for someone like Kozyrkov stemmed from the company’s adoption of machine-learning technology across an array of products and services to make decisions reliably and at massive scale. A Machine Learning model which decides if a photo of an animal is a cat, can trigger actions accordingly: If it’s a cat, do A. If it’s not a cat, do B. And it can do it over and over without ongoing human involvement.

The problem is that an algorithm which learns from examples–in this case, photos of animals which are labeled as cats or not cats–is only as good as the examples it’s trained on. If the human being training the algorithm sometimes labels rabbits as cats, the algorithm will make bad decisions as efficiently as it does good ones. And the more sophisticated machine-learning applications get, the more opportunity there is for humans to introduce subtle problems into the final results.

Google needed a decision-making framework which enabled individual humans, groups of humans, and machines to make wise decisions. Such a process didn’t yet exist. So the company decided to build it.

DECISION INTELLIGENCE ENGINEERING

The well-established academic field of decision science covers the psychology, neuroscience, and economics of how human beings make decisions–but it doesn’t encompass the engineering perspective and the scale of automated decision-making. Likewise, data science doesn’t cover how humans think through a decision.

“A lot of the training that data scientists have assumes that the decision maker knows exactly what they need and the question and problem are framed perfectly,” says Kozyrkov. “The data scientist goes off and collects the data in service of that question, and answers it, or builds the machine learning system to implement it.”

That ideal scenario is all too rare in the real world. While working with Google’s data-science consulting arm, Kozyrkov often saw executives make decisions that were steered by unconscious bias rather than by the data itself.

Kozyrkov’s graduate training spans psychology, neuroscience, and statistics. Instead of just training decision-makers in data science, she set out to draw on the behavioral sciences to help them to make truly data-driven decisions. This means framing a decision effectively–often before looking at any data at all.

HOW TO DECIDE

The first step in Google’s framework asks decision-makers to determine how they will make the decision with no additional information. What would the default choice be? Let’s say you have to decide whether to stay at a hotel. You have photos of the hotel but no guest reviews. Based on the photos alone, would you stay there?

“We pretend that we don’t have a preference, but we’re really lying to ourselves,” says Kozyrkov. “We do have some kind of innate feeling about what seems to be the safer, better option under ignorance.”

The second step is to define how you would make the decision if you had access to any information you wanted. What would it take to convince you to stay at the hotel? Would you want to read every review or just see the average review score? If you use average review score only, does that number need to be 4.2 or 4.5 or something else to convince you to stay at the hotel?

This soul-searching exercise determines the metrics you need to make the decision and the cut-off point for each metric. In the hotel example, this could be end up being an average review score of more than 4.2 or a more complex, compound bottom line using the average rating weighted with one star reviews and the possibility of bedbugs.

“That step is actually skipped a lot in industry, where people will use fuzzy concepts, they’ll never make them concrete, and they won’t own up to the fact that they’re using them,” says Kozyrkov. “They’ll think that putting a bunch of mathematics near it fixes it.”

In the final step of this process, you should look at whether you can get access to all the data you ideally want to make the decision. If you’ve decided that you’ll stay at any hotel that has an average star rating of over 4.2, and you have access to hotel ratings, then you’re good to go. But if you’re factoring in bedbugs–but don’t have the full reviews which might mention them–then you have to make your decision under uncertainty. That introduces the potential for mistakes–and in automated decision-making, you might make mistakes many times.

As the decision maker, you must therefore consider which mistakes you can live with. Is it worse if you end up in a hotel with bedbugs or if you miss out on a hotel which would have best met all your criteria? How costly is one mistake versus the other? In the case of automated decision making, only when you have figured out which risks you are willing to accept can a data scientist gather relevant data and apply statistical analysis to help you to make a decision.

BEYOND DATA SCIENCE

Actually, Kozyrkov says, social scientists are often better equipped than data scientists to translate the intuitions and intentions of a decision-maker into concrete metrics. And ideally, data scientists and social scientists work together to define metrics, collect appropriate data, and apply it to automated decision-making.

“I think that we don’t realize how valuable social scientists are,” she says. “A data scientist might think that they’re qualified to create a survey (e.g. to measure user satisfaction), and then analyze data from it, but what happens if your users simply ignore your survey? We call that non-response bias. What if there’s some incentive for your users to lie to you? That’s not something that a data scientist is trained in handling, whereas if you worked in a social psychology lab, you deal with that day in and day out.”

Making decisions the Google way isn’t necessarily easy. Applying Decision Intelligence Engineering to an important, complex decision can take weeks, or, if multiple stakeholders are involved, even months. But with any luck, the result should be better, wiser decisions, especially at scale.

“We also figure out the right approach based on the importance of the decision,” says Kozyrkov. “A lot of the training prioritizes the most important decisions, but there are also approaches to taking decisions based on … no information at all, if the decision is not that important.”

Kozyrkov argues that Decision Intelligence Engineering is not just for experts like data scientists and social scientists. Everyone in a company can be involved in decision making using data. The trick is figuring out how each person can best contribute.

“Decision-making is something that our species does,” says Kozyrkov. “Everyone knows something about it, and I would say that everyone is an expert in at least some piece of the process.”

From the instructor: Google has a special website to teach AI. Check it out here:

https://ai.google/education/

Reference

Byrne, C. (2018). Why Google defined a new discipline to help humans make decisions, retrieved from

https://www.fastcompany.com/90203073/why-google-defined-a-new-discipline-to-help-humans-make-decisions