Key ways Netflix utilises AI and machine learning

Off the back of our previous post re-classifying how we think about the ecosystem of intelligent apps, I thought I’d share an example of how Netflix has used two of the functional AI components — Extraction and Recommendation — to create seamless, personalised “intelligent apps”.

Use Case #1 — the Netflix recommendation engine

Most of us are probably familiar with Netflix’s recommendation engine — it curates and presents a selection of content we might want to watch by using AI/ML models to recommend content based on our viewing history and the viewing histories of other users with similar tastes/characteristics.

Without Recommendation, the complexity of Netflix’s content library would make it near impossible for users to find content relevant to their specific interests. Which would lead to frustration, bad user experiences — and, eventually, churn.

So it’s critically important.

How it works:

This is an example of the Recommendation function we’ve talked about previously.

Every single thing you do inside Netflix is monitored. They track millions of micro-interactions each second. So they have a very complete and nuanced version of what they think you like.

The recommendation engine parses this against other users that like similar things and behave in similar ways. And then feeds back to you recommendations based on what this similar cohort did.

Here is a real use case of a family of three. This is what Netflix shows as “trending now”.

For Dad.

For Mum.

For Young Adult.

The real power of the module is the fact that it is constantly learning. It’s not just about whether someone like you watched something. It’s about how long they watched it. Did they binge-watch it? Did they go back to it? Did they start but then stop watching because they got bored? Etc.

Use case # 2 — thumbnail personalisation

You’d be aware of the recommendation use case, I’m sure — this one is a little less known.

Spoiler alert: you don’t see the same thumbnails as other people see.

Netflix conducted research into user behaviour and found that users spend less than two seconds considering each recommended title — and the single, most influential factor in whether they watched or not was the thumbnail presented to them.

So they created an intelligent app that creates and serves up a personalised thumbnail to every individual user on the platform.

How it works:

Using our functional classification — it is using both the Extraction and Recommendation functions.

Extraction:

One of the problems with traditional TV is that the showrunners typically provide standard advertising elements to everyone. And they’re not optimised for streaming — they’re for billboards or DVD covers.

Plus, they don’t change much across different providers — that’s how it has traditionally scaled. So there is very limited material provided for Netflix to use here.

So they have to create their own content.

First, they need to extract the information they need.

This involves capturing EVERY video frame, from EVERY show. And using a computer vision module (called aesthetic visual analysis [AVA]) to extract the information it needs. For reference, there are roughly 86K frames in an hour-long episode of Stranger Things

They then annotate each frame with relevant data they are going to use next.

These might be visual factors (brightness, colour), composition factors (principles like the rule of thirds, or is the character looking at the camera? etc), or contextual factors (which character is featured, what emotions are they showing).

This gives them the information to parse into the next process.

Recommendation:

As previously outlined. This process is about building a thumbnail just for you.

Netflix considers things like if you like thrillers or comedies to make decisions about the colour palette of the thumbnail. If you’ve watched more shows with a certain actor — and that actor is in this show — to make decisions about which characters to show. Which location you’re in, and if there might be certain cultural patterns that work better than others.

Source: Netflix Tech Blog.

All of this ends up showing up as individualised thumbnails. Which have been proven to increase engagement and reduce churn for Netflix.