Azure Video Indexer is one of my favorite services! It provides some amazing features such as:

  • Identify faces and celebrities
  • Extract text and objects (e.g. cat, table, car etc.)
  • Detect scenes and shots
  • Emotion and sentiment detection
  • Audio effects such as clapping, silence & speech

The full list of features can be viewed at https://vi.microsoft.com/en-us/

Get the credits rolling!

Recently, Video Indexer announced they’ve added the capability to detect rolling credits in the videos. One of the key scenarios enabled by this feature is the ability for video owners to identify a natural point in the video where they can recommend the next episode or show aka Netflix autoplay experience as shown below:

The Office - Netflix autoplay experience

Testing the rolling credits!

To test this out, I uploaded an open source video by the Blender foundation – Tears of Steel to Video Indexer. Once the video has completed indexing, I downloaded the insights JSON from the video details page like shown below:

Rolling credits JSON object can be accessed in the two places

  1. The labels object currently with id = "-1". The full path is jsonObj["summarizedInsights"]["labels"][4]["id"]
  2. The “framePatterns” object with id = "0". The full path is jsonObj["videos"][0]["insights"]["framePatterns"][0]["id"]

The Rolling Credits object for the Tears of Steel has the following time values according to Video Indexer

{
    "id": -1,
    "name": "RollingCredits",
    "appearances": [
        {
            "startTime": "0:10:31",
            "endTime": "0:11:50",
            "startSeconds": 631,
            "endSeconds": 710
        }
    ]
}

The results

When looking through the actual video, I noticed that there are a non-traditional set of credits that start at 9 min 49 second. This is surprising as Video Indexer thinks the credits should begin at 10 min 31 second mark.

When we look at the 10 min 31 second mark, I see the traditional “Rolling Credits” that move up slowly, start as expected. At first I was surprised but then I realized Video Indexer isn’t taking into account the different ending credit design. Maybe that’ll come in the future but this should be good enough to achieve our Netflix auto-play scenario! :)

Reach out if you have any questions! Feel free to follow me on