Interacting with Google Assist in Xamarin

Introduction

Voice interactions come in a few flavors on Google. First there is Google Now. It is very similar to Siri, in that it can answer voice queries and delegate to other applications if needed. Google Now is available on both iOS and Android however is a rather closed off solution that developers can’t interact with.

Moving on next is Voice Actions. Voice Actions are a way to continue on the voice interaction inside your app. You can ask your Android device to say take a picture and have your app respond to ask some questions such as which camera to use. Add a Conversation to your Android App with Voice Interactions gives a great walk through.

What we are moving on to is Google’s Assist API. This is new in Android 6.0 and we can think of it as an extension of Google Now that allows developers to provide visibility to their app and handle assistant requests. You may hear of this Google Assistant referred to as Now on Tap. Similar to Siri you hold down the home button and speak your intent.

Register Intent

First thing we want to do is register the Intent on the activity you want to launch when called from Google Assistant.

[IntentFilter(new []{Intent.ActionAssist}, Categories = new []{Intent.CategoryDefault})]

Provide Assistance

To enhance the Assist API we can give the Assist API some more information about the content in our app. This will help Assist give relevant information about your app to the user. You can either do this globally for the app by registering this.

using Android.App;
using Android.OS;
using static Android.App.Application;

namespace Demo.Droid
{
   public class AndroidListener : Java.Lang.Object, IOnProvideAssistDataListener
   { 
       public void OnProvideAssistData(Activity activity, Bundle data) {}
   }
}
public class MainActivity : global::Xamarin.Forms.Platform.Android.FormsApplicationActivity
{
    protected override void OnCreate(Bundle bundle)
    {
       base.OnCreate(bundle);

       // Application Level Assistance
       Application.RegisterOnProvideAssistDataListener(new AndroidListener());

       global::Xamarin.Forms.Forms.Init(this, bundle);
       LoadApplication(new App());
    }
}

Or you can do it per activity as shown below.

public override void OnProvideAssistContent(AssistContent outContent)
{
   base.OnProvideAssistContent(outContent);

   // Provide some JSON 
   string structuredJson = new JSONObject()
       .Put("@type", "MusicRecording")
       .Put("@id", "https://example.com/music/recording")
       .Put("name", "Album Title")
       .ToString();

    outContent.StructuredData = structuredJson;
 }

public override void OnProvideAssistData(Bundle data)
{
    base.OnProvideAssistData(data); 
}

Assist will open your activity when appropriate.

Continue Voice Interaction

If you want to go into some more advanced interaction and continue on the voice interaction you can use the VoiceInteractionSessionService and VoiceInteractionSession. Make sure you have the permission BIND_VOICE_INTERACTION checked. Then in your VoiceInteractionSessionService you can override the OnHandleAssist

 public override void OnHandleAssist(Bundle data, AssistStructure structure, AssistContent content)
 {
     base.OnHandleAssist(data, structure, content);

     // Here is some of the data you can get
     string structuredData = content.StructuredData; 
     Bundle extra = content.Extras;
     ClipData clipData = content.ClipData;
     Intent intent = content.Intent;
     Bundle intentExtra = intent.Extras;
     Uri uri = content.WebUri;
 
 }

Learn More

If you want to read the official Android Documentation have a look at Optimizing Content for the Assistant.

Microsoft MVP | Xamarin MVP | Xamarin Certified Developer |
Exrin MVVM Framework | Xamarin Forms Developer | Melbourne, Australia

Related Posts

Leave A Comment?