LUIS now understands people’s names

This one managed to slip past me, but recently Microsoft Language Understand Intelligent Service (LUIS) has been updated with a pre-built entity to match people’s names. Seems like an obvious requirement but is pretty tricky to implement. To match people’s names;

  1. Open your KB in the LUIS UI of your choice
  2. Go to the Entity menu
  3. select Add Prebuilt entity
  4. choose personName

Now you can create an Intent such as “Call” with “Call Jane” and “Jane” will be matched to personName. So far (and testing is difficult) I’ve thrown all my usual problematic name variants (Western, Nordic, Chinese, Indian) and it’s managed them all, even “Call peter phoxterd-hemmington-smythe” 🙂

Handling flow from Prompt Validators

Edit – warning this is about pre-release v4. Whilst it’s similar to the final release it’s not the same

Creating a basic flow with Botframework V4 is pretty straightforward. You define your steps and any other dialogs or prompts that you may wish to use. For example, consider this basic name capturing definition;

Add(Inputs.Text, new Microsoft.Bot.Builder.Dialogs.TextPrompt(TextValidator));
Add(Name, new WaterfallStep[]
{
    // Each step takes in a dialog context, arguments, and the next delegate.
    async (dc, args, next) =>
    {
        // Prompt for the user's first name.
        await dc.Prompt(Inputs.Text, "Hey, what's your first name?");
    },
    async (dc, args, next) =>
    {
        dc.ActiveDialog.State["1stName"] = args["Text"].ToString();

        // Prompt for the user's second name.
        await dc.Prompt(Inputs.Text, "what's your second name?");
    },
...
private Task TextValidator(ITurnContext context, TextResult toValidate)
{
    if (toValidate.Text == ".")
    {
        toValidate.Status = null;

    }

    return Task.CompletedTask;
}

The above code will reject the second name if the user types in a full-stop. But what will happen next needs to be considered. The current code will correctly reject the full-stop but it will restart the dialog and ask the user for their first name again. There are a couple of ways to handle this, and they can be implemented in unison.

Use a Retry Message

When a prompt has a Retry-Message associated with it then any failure from the Validator will keep the user on the same waterfall step.

...
 async (dc, args, next) =>
    {
        dc.ActiveDialog.State["1stName"] = args["Text"].ToString();
        // Prompt for the user's second name.
        await dc.Prompt(Inputs.Text, "what's your second name?",
            new PromptOptions
            {
                RetryPromptString = "what's your really your second name?"
            });
    },
...

The above code is probably all your need, but you may also want to equip your waterfall steps to be more…re-entrant.

Replay dialog

This is particularly useful if you provide a way to edit the user’s choices. The idea is that each step examines its arguments or state and if already fulfilled simply passes the flow onto the next step. In the final step you decide if everything has completed or if the flow should start again.
Example step;

async (dc, args, next) =>
    {
        if (args.ContainsKey("1stStep")
        {
          await next(args);
        }
        else
        {
          // Prompt for the user's first name.
          await dc.Prompt(Inputs.Text, "Hey, what's your first name?");
        }
    },
...

Final step;

async (dc, args, next) =>
    {
        if (AllConditionsMet(args))
        {
          // finish the dialog
          await dc.End(dc.ActiveDialog.State);
        }
        else
        {
          // replay the flow
          await dc.Replace("yourdialogName", dc.ActiveDialog.State);
        }
    },

Combining Retry and Replay

There is nothing stopping you using a combination of the methods. If you want to enable an editing scenario but want the framework to ensure a criteria is met for specific steps then just implement both.

Matching names in LUIS

I recently hit a thorny issue with allowing my Bot to understand names via LUIS. I tried to train the LUIS model with various utterances of, ‘how is ‘ but it really wasn’t getting anywhere. There are just so many variations of names that unless I retrieved a large name data set then this really wasn’t going to work. So I fell back to using a very specific match. Yes, that is not really in the spirit of LUIS but at least it keeps all the language models in one place. My trick is pretty simple;

  • Create your Intent; e.g. HowIs
  • Add your *specific* utterance; e.g. how is paul
  • Go to the Entities options and create a new Entity called ‘Name’…
  • Set the Entity Type to RegEx…
  • Add your RegEx, e.g. (?<=^.{7})(.*)
  • Hit train
  • Now when you visit your Intent you should see the name aspect labelled as ‘Name’. As I say, it’s pretty crude and frankly you don’t need LUIS to do such a match. But if you believe there is value in keeping all your language matching models in the same place then it’s the best solution I currently have.

    NB I tried using a training list and Pattern and neither seemed to help

    Unit Test Flow in Botframework V4

    In theory unit testing in Botframework V4 should easy enough as it’s based on .net core. But, as with many things, it might not be that obvious. So here is a quick snippet showing a test for a Waterfall flow;

    [TestMethod]
    public async Task Waterfall()
    {
        var convoState = new ConversationState<Dictionary>(new MemoryStorage());
    
        var adapter = new TestAdapter()
            .Use(convoState);
    
        //var dialogState = convoState.CreateProperty("dialogState");
        var dialogs = new DialogSet();
        dialogs.Add("textPrompt", new TextPrompt());
    
        dialogs.Add("test",
            new WaterfallStep[]
            {
                async (dc, args, next) =>
                {
                        await dc.Context.SendActivity("Welcome! We need to ask a few questions to get started.");
                        await dc.Prompt("textPrompt", "What's your name?");
                },
                async (dc, args, next) =>
                {
                    dc.ActiveDialog.State["name"] = args["Value"];
                    await dc.Context.SendActivity($"Thanks {args["Value"]}!");
                    await dc.End();
                }
            }
        );
    
        await new TestFlow(adapter, async (turnContext) =>
        {
            var state = turnContext.GetConversationState<Dictionary>();
            var dc = dialogs.CreateContext(turnContext, state);
            await dc.Continue();
            if (!turnContext.Responded)
            {
                await dc.Begin("test");
            }
        })
        .Send("Hello")
        .AssertReply("Welcome! We need to ask a few questions to get started.")
        .AssertReply("What's your name?")
        .Send("Paul")
        .AssertReply("Thanks Paul!")
        .StartTest();
    }
    

    A couple of points to note, you have to send a message to start the flow, and make sure you call turnContext.GetConversationState rather than using a local version of the state.

    Edit – you may want to also look at Automate Bot tests with saved transcript files , Testing using ConfigurationManager with .net core and How to write a unit test for Bot Framework v4

     

    Download Transcript option in Botframework v4

    This is a whistle stop tour for adding a method to allow the user to download their transcript.

    Automatically record the transcript

    You need to add the built-in transcript services. In this example I’ll use the In-Memory store, you’ll want to evaluate the others for production code. Add this to the startup.cs

    var memoryTranscriptStore = new MemoryTranscriptStore();
    …
    options.Middleware.Add(new TranscriptLoggerMiddleware(memoryTranscriptStore));
    

    Next we’ll create a component to expose the ITranscriptStore (memoryTranscriptStore) to the dialog context. We’ll do that via our own middleware;

    public class TranscriptProvider : IMiddleware
    {
        private readonly ITranscriptStore transcriptStore;
    
        public TranscriptProvider(ITranscriptStore transcriptStore)
        {
            this.transcriptStore = transcriptStore;
        }
    
        public async Task OnTurn(ITurnContext context, MiddlewareSet.NextDelegate next)
        {
            context.Services.Add(transcriptStore);
            await next();
        }
    }
    

    Now add this to the startup middleware;

    options.Middleware.Add(new TranscriptProvider(memoryTranscriptStore));
    

    Download the transcript

    The middleware will now capture all the activity traffic to and from the bot and user. We can add a simple mechanism to request the transcript file. In your bot’s OnTurn method we can hardcode a ‘Transcript’ message/command;

    if (context.Activity.Text.Equals("Transcript", StringComparison.InvariantCultureIgnoreCase))
    {
        var transcriptStore = context.Services.Get();
        var transcripts = await transcriptStore.GetTranscriptActivities(context.Activity.ChannelId, context.Activity.Conversation.Id);
    
        var transcriptContents = new StringBuilder();
        foreach (var transcript in transcripts.Items.Where(i => i.Type == ActivityTypes.Message))
        {
            transcriptContents.AppendLine((transcript.From.Name == "Bot" ? "\t\t" : "") + transcript.AsMessageActivity().Text);
        }
    
        byte[] bytes = StringToBytes(transcriptContents.ToString());
    
        var contentType = "text/plain";
        var attachment = new Attachment
        {
            Name = "Transcript.txt",
            ContentUrl = $"data:{contentType};base64,{Convert.ToBase64String(bytes)}",
            ContentType = contentType
        };
    
        var activity = MessageFactory.Attachment(attachment);
        await context.SendActivity(activity);
    
        return;
    }
    ...
    private byte[] StringToBytes(string transcriptToSend)
    {
        byte[] bytes = new byte[transcriptToSend.Length * sizeof(char)];
        System.Buffer.BlockCopy(transcriptToSend.ToCharArray(), 0, bytes, 0, bytes.Length);
        return bytes;
    }
    

    When your user types in ‘Transcript’ they’ll be provided with a download attachment called ‘Transcript.txt’.

    Production Ready

    The above code is great for early testing but you should probably consider using the Download Activity Type and providing a URL to the full transcript. The above code has a nasty weakness in that it must fit inside the maximum reply payload for the bot, ~94K. You could just truncate the body but I’ll leave that up to you. Note, as of writing the emulator has an issue where it will allow you to click on the download but then it gets into a pickle and launches Windows Store. If you try this on a webchat it works fine.

    Manipulating waterfall steps Botframework v4

    To be honest I’m not even sure that this is strictly supported or that it is even a good idea, but as a point of interest you can manipulate the waterfall steps in v4. E.g. The standard flow is; step 1 -> step 2 -> step 3 -> step n. If your code realises that the user should skip a step then it can invoke the ‘next’ function;

    async (dc, args, next) =>
    {
        if (someCondition)
        {
            await next(args);
            return;
        }
    }
    

    That’s pretty easy. The difficult question, and one that I’m not even sure is (or should be) a valid one, how do you go back a step? Well, it is possible but it’s messy and you can’t get back to the initial step (although that is just starting again). You can go back a step by manipulating the dialog state;

    // at step n
    async (dc, args, next) =>
    {
        if (someCondition)
        {
            var targetStep = 1;
            dc.ActiveDialog.Step = targetStep - 1;
            await Step1Prompt();
            return;
        }
    }
    

    I don’t recommend this approach, it’s ugly, but you know…possible.

    ChoicePrompt, how to always call the validator in Botframework v4

    BotFramework v4 has a number of helper prompts, TextPrompt, ChoicePrompt, etc. One common mechanism they share is a Validator delegate. When the user enters a value the Validator is invoked and you have an opportunity to check/change the outcome

    var obj = new ValidatingTextPrompt(async (context, toValidate) =>
        {
            if (toValidate.Text.Length < minimumLength)
            {
                toValidate.Status = null;
                await context.SendActivity(minimumLengthMessage);
            }
        }
        );
    

    The presence of the RetryPromptString means the ChoicePrompt will automatically retry of the user enters the incorrect value, such as 'frog'. However, what happens if the user enters the value '3'? Unfortunately this is considered as the 3rd choice and 'quit' will be selected. If your UI is really serving up numbers like this, that could be a real problem. Imagine if the list was 2,4,6 and you entered '3' or even worse '2'!? So I really want to add a Validator delegate that all prompts support;

    this.Dialogs.Add("choicePrompt", new ChoicePrompt(Culture.English, ValidateChoice));
    

    private async Task ValidateChoice(ITurnContext context, ChoiceResult toValidate)
    {
        var userMessage = context.Activity.Text;
        if (userMessage == "3")
        {
            toValidate.Status = null;
            await Task.Delay(1);
        }
    }
    

    Sorted right? Wrong. Unfortunately there are two problems with this solutions; a) this is only called when a value from the choices list is selected (really??) b) the resulting selected value is passed in and not the original, i.e. ‘quit’ is passed in rather than ‘3’. My solution is to derive a new ChoicePrompt that will always call the available Validator with the original values;

    public class ChoicePromptAlwaysVerify : Microsoft.Bot.Builder.Dialogs.ChoicePrompt
    {
        private readonly PromptValidatorEx.PromptValidator validator;
    
        public ChoicePromptAlwaysVerify(string culture, PromptValidatorEx.PromptValidator validator = null) : base(culture, validator)
        {
            this.validator = validator;
        }
    
        protected override async Task OnRecognize(DialogContext dc, PromptOptions options)
        {
            var recognize = await base.OnRecognize(dc, options);
            if (this.validator != null)
            {
                await this.validator.Invoke(dc.Context, recognize);
            }
    
            return recognize;
        }
    }
    

    The code works by forcing the recognize override to call the validator. The downside is that this code will be called twice when the user makes a good choice (sigh), but it’s a small sacrifice to regain some consistent control over the valid values. It also allows for more specialized messages as the RetryMessage is fixed and has no chance to give a contextual response.