Training a topic model to improve email routing
Archived
3 Tasks
20 mins
Scenario
U+ Bank uses Pega Customer Service™ to route incoming emails to the appropriate department based on the topic of the email. For several use cases (for example an address change), emails are routed based on keywords detected in the message. To improve the email routing, train the text prediction with a data set that contains classified messages.
Use the following credentials to log in to the exercise system:
|
Role |
User name |
Password |
|---|---|---|
|
Data scientist |
CSDataScientist |
rules |
|
Customer service representative |
cacsr |
rules |
Your assignment consists of the following tasks:
Task 1: Test the text prediction
As a data scientist, test the U+ Bank customer support text prediction with the two messages below and inspect the sentiment and topic classifications.
| Questions How would you estimate the sentiment of these two messages? Why is complaint detected as a topic in a message concerning an address change? |
Complaint
| I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services. |
Address change
| I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected] |
Task 2: Train the text prediction
Train the U+ Bank customer support text prediction with the CSTrainingData data set.
| Question – On how many topics will this CSV file train? |
Task 3: Test the trained prediction
Test the U+ Bank customer support text prediction after it is trained with the message concerning an address change from task 1.
Task 4: Confirm your work
As a customer, use the enablement email client to send an email to U+ Bank customer support.
| To: [email protected] Subject: Incorrect address Messsage: Hi U+ Bank, I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected]. Cheers, Sara Connors |
As a U+ Bank customer service representative, confirm that the incoming email is correctly routed.
Challenge Walkthrough
Detailed Tasks
1 Test the text prediction
As a data scientist, test the U+ Bank customer support text prediction with the two messages below and inspect the sentiment and topic classifications.
| Question – How would you estimate the sentiment of these two messages? |
Complaint
| I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services. |
Address change
| I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected] |
- Log in as a data scientist with user name CSDataScientist and password rules.
- On the Predictions landing page, click the U+ Bank customer support tile.
- In the upper right, click Test.
- In the Text field, enter the following message concerning a complaint.
| I am a very unhappy customer of yours. For the third time, I have noticed an invalid transaction. The amount is small, but could you please check? It is unfortunate because, in general, I like your services. |
- Click Test.
- Observe that a complaint is detected as the topic of the message with a confidence score of 1.
- In the Sentiment detection section, notice that the complaint has a negative overall sentiment score.
Tip: Check if your estimation of the sentiment of the message matches the sentiment score.
- Click Highlight in input text to identify the sentences with positive and negative sentiments.
Note: Although the email has a mixed positive and negative sentences, the overall sentiment is negative.
- In the Text field, enter the following message that concerns an address change.
| I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected] |
- Observe that both a complaint and an address change are detected as topics with a confidence score of one.
Tip: Check if your estimation of the sentiment of the message matches the sentiment score.
| Question – Why is complaint detected as a topic in a message concerning an address change? |
- Close the test panel.
- To the right of the complaint topic, click the Gear icon to view the keywords for this topic.
- On the Keywords tab, notice that wrong is one of Should words.
Tip: Because the word 'wrong' is present in the message, the complaint topic is detected.
- Click Cancel.
2 Train the text prediction
| Question - How many topics does this CSV file contain? |
- Download CSTrainingData and extract the CSV file.
- Open the CSV file in an editor and inspect the records.
- Filter the result column and check the possible values:
Tip: This file has training data for two topics: Account Address Change and Complaint.
- Switch back to the U+ Bank customer support text prediction in Prediction Studio.
- On the right, click Import.
- Click Choose file and select the CSTrainingData.csv file.
- Click Upload.
Note: that the two topics have over a hundred records pending training each.
- In the upper right, click Build.
- Select the U+ Bank customer support topic model.
- Click Build to begin training the text prediction.
- After the build is triggered, note that there is a training job in a pending state.
- In the upper left, click Actions > Refresh.
Tip: It may take up to a minute to build the model. Click Refresh until the build is complete.
- Once the prediction build is complete, click View report to see the result of the model building process.
- Close the Model training report.
Note: The training data consists of 70 percent of the records. The remaining 30 percent has been used for validation of the model.
3 Test the trained prediction
- In the upper left, click Test.
- In the Text field, enter the following message concerning an address change that you have tested in the first task.
| I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected] |
- Observe that after training the model, only an address change is detected as a topic with a high confidence score.
Note: The observed confidence score values may vary, as 30 percent of the training data is randomly used for validation.
- Verify that the email address, city, state, and zip code mentioned in the message are detected by the entity extraction models.
- In the lower-right corner, click the CA icon and log off.
Confirm your work
- On the Customer Engagement & Analytics landing page, click Email.
- In the navigation panel on the left, click Mail.
- In the top menu, click Write mail.
- Compose an email concerning an address change.
To: [email protected]
Subject: Incorrect address
Messsage:
Hi U+ Bank,
I have noticed, in my last account statement, you have used a wrong address. Please change my mailing address to read: 222 West Las Colinas Blvd., Irving, TX 75039, USA, effective immediately. And I'm happy to have a fresh email address: [email protected].
Cheers, Sara Connors - Click Send message.
- In the Pega CRM suite, log in as a customer service representative with user name cacsr and password rules.
- On the My workbaskets tab, select the Account Maintenance workbasket.
- If there is no incoming email, click the refresh button until the email is processed.
Note: Depending on the environment, it might take up to two minutes until the email is sent and processed in Customer Service.
- Click the ID number of the incoming email.
- Notice the neutral sentiment and the entities that are detected.
