Microsoft’s Cortana hands on: The best of Google Now and Siri combined

Microsoft has been active in speech recognition research for decades, so it isn’t surprising that it has introduced a voice-based personal digital assistant — only that it took them so long to do so! Named Cortana, the application is a hybrid of local and cloud functionality that will be built into Windows Phone 8.1. I had a chance to try it out this week at Microsoft’s Build 2014 conference. For the most part I was quite impressed, although the code’s beta designation is certainly still appropriate.

Cortana combines elements of Siri and Google Now

The core of Cortana is a Siri-like speech recognition capability, coupled with enough intelligence to put some context around your commands and questions. Added to that is a card-based proactive display of what it thinks might be important to you based on your calendar, contacts, and interests — similar to the cards that Google Now pops up. Microsoft’s real claims to innovation in Cortana is its personalization and extensibility.

Cortana starts off by trying to get to know a little bit about you — what you like to do in your spare time, who is important in your life, what sports teams you follow, etc. It uses that information to begin preparing tidbits of information it can show you first thing in the morning, or when it thinks you might need them. It also learns about your habits and becomes more personal over time. Obviously that’s hard to test in a short demo session, so we’ll need to wait until it ships to see how powerful the learning capability is in real life. The other really exciting feature of Cortana is that it is completely extensible. Microsoft has documented how third parties can integrate with Cortana, and demonstrated integration with Flixster during the show.

Windows Phone 8.1: Cortana

Cortana can almost converse

Cortana has a reasonably sophisticated conversational model. You can follow-up a question with another related question, and it knows how to link them together. For example, you can ask it for nearby restaurants and then ask something like “which of them take reservations?” and it will know to filter the restaurant list. Unfortunately, it doesn’t know how to further link to sites like OpenTable to actually help you make them.

Cortana seems much better at complex commands than Siri or Google Now. For example, you can tell it, “the next time I talk to my wife, remind me to let her know that I spend too much time talking to my phone.” It will identify the contact entry for your wife, and in fact remind you the next time you call or text her contact that you have something you want to say. Like Google Now and Siri, Cortana leverages search technology — in this case Microsoft’s Bing. Besides relying on Bing for search and custom vocabularies (like lists of movies and restaurants), it also benefits from any personalization you’ve done in Bing.

Joe Belfiore introduces Cortana digital assistant

Several of the Microsoft demoers had trouble with Cortana recognizing some of their speech, but when I tried it out it was nearly perfect in recognizing everything I said — except for names of some foreign countries and cities. However, I did find that Cortana’s seemingly magical understanding of reminders and restaurants didn’t extend to other similar tasks you might delegate to an assistant. It had no idea how to find me a taxi, for example, or where the nearest bus stop was. Microsoft is quick to point out that Cortana is still in early beta (it won’t be publicly available until Windows Phone 8.1 ships later this spring), and it will be adding additional smarts to the system during that time of course.

The designers behind Cortana have worked really hard to have it work the way you’d like a personal digital assistant to — they interviewed human assistants to build a model of common tasks. For example, if you start speaking to it, it will reply with voice. But if you type at it, it will assume you don’t want to be using audio, and it will reply in text. It can also do some clever, if seemingly obvious, tasks like search your email (with your permission) for meetings and airline flights you might want to track or add to your calendar. All in all, Cortana is an exciting addition to the Windows Phone platform, and I suspect we’ll see more of it (her?) in future versions of Windows for the desktop and for tablets as well. It’ll be a while before Cortana is as intelligent as her Halo character namesake, though.