Skip to main content

Monitoring predictions

6 Tasks

15 mins

 Visible to: All users
Beginner Pega Customer Decision Hub 8.8 English

Scenario

U+ Bank uses Pega Customer Decision Hub™ to display a personalized credit card offer to eligible customers on their website. The bank now wants a report on the boost in success rate that the artificial intelligence (AI) generates. As a data scientist, you inspect the Predict Web Propensity prediction that aims to optimize the click-through rate of the web banners that contain the credit card offers.

Use the following credentials to log in to the exercise system:

Role User name Password
Data scientist DataScientist rules

Your assignment consists of the following tasks:

Task 1: Inspect the Success rate graph

Inspect the success rate of the Control group, the NBA Prioritization group, and the Propensity group over time by answering the following questions:

  1. Is the success rate of the Control group stable over time? Is this expected?
  2. Is the success rate of the NBA Prioritization group higher or lower than the success rate of the Control group? Is this expected?
  3. Is the success rate of the Propensity group higher or lower than the success rate of the Control group? Is this expected?

Task 2: Inspect the Lift graph

Inspect the lift that the prediction generates by answering the following questions:

  1. What is the current boost in the success rate that the AI generates?
  2. What is the trend of the lift?

Task 3: Inspect the Performance graph

Inspect the performance of the prediction by answering the following questions:

  1. What is the current performance of the prediction?
  2. What is the current distribution of the prediction?

Task 4: Inspect the Propensity percentiles graph

Inspect the Propensity percentiles graph by answering the following questions:

  1. What is the trend for the 90th and 10th percentile for prediction output?
  2. Is there a reason for concern?

Task 5: Inspect the Propensity decile distribution graph

Inspect the propensity decile distribution of the prediction by answering the following questions:

  1. Where does the Propensity decile distribution graph peak?
  2. Is there a reason for concern?

Task 6: Inspect individual predictors

Inspect the numerical predictor Age and the categorical predictor CLV by answering the following question:

  1. Is there any reason to suspect data drift for these predictors?

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

Challenge Walkthrough

Detailed Tasks

1 Inspect the success rate graph

  1. On the exercise system landing page, click Pega CRM suite to open Customer Decision Hub.
  2. Log in as a data scientist:
    1. In the User name field, enter DataScientist.
    2. In the Password field, enter rules.
  3. In the navigation pane of Customer Decision Hub, click Intelligence > Prediction Studio to open Prediction Studio.
  4. On the Predict Web Propensity tile, click Open prediction to inspect the Analysis tab.
  5. Question: Is the success rate of the Control group stable over time? Is this expected?
    The Success rate graphs of the prediction
    Tip: The success rate of the Control group is dependent on customer behavior and engagement policies, so it may change over time.
  1. Question: Is the success rate of the NBA Prioritization group higher or lower than the success rate of the Control group? Is this expected?
    Tip: The success rate of the NBA Prioritization group is expected to be higher than the success rate of the Control group because the Next Best Action arbitration formula decides which credit card offer has the highest likelihood of a click-through. The credit card offer is random in the Control group.
  1. Question: Is the success rate of the Propensity group higher or lower than the success rate of the Control group? Is this expected?
    Tip: The success rate of the Propensity group is expected to be higher than the success rate of the Control group because the AI decides which credit card offer has the highest likelihood of a click-through. The credit card offer is random in the Control group.

2 Inspect the lift graph

  1. Question: What is the boost in success rate that the AI generates over time?
    1. Inspect the NBA Prioritization Lift graph.
      The Lift graphs
      Tip: Hover over a date to display the lift. Note that both the NBA Prioritization lift and the Propensity lift are positive.
  1. Question: What is the trend of the NBA Prioritization lift?
    Tip: Because U+ Bank implemented Pega Decision Management only recently, there is not yet enough data to detect a trend. Once sufficient data is collected, the system generates notifications when there is no lift, or when the lift drops significantly.

3 Inspect the performance graph

  1. Question: What is the current performance of the prediction?
    1. Inspect the performance graph.
      The performance graphs
      Tip: Hover over a date to display the performance. Notice that performance over time is increasing, which is expected. The more data that is accumulated, the higher the performance.
  1. Question: What is the current distribution of the prediction?
    1. Click Show distribution.
      The ROC graphs
      Tip: The distribution can be displayed as a ROC curve and a decile distribution for further inspection.
  1. Close the Show distribution dialog box.

4 Inspect the propensity percentiles graph

  1. What is the trend for the 90th and 10th percentile for prediction output?
    1. Inspect the Propensity percentiles graph.
      The Propensity percentiles graphs
      Tip: The lines do not show large fluctuations over time.
  1. Question: Is there a reason for concern?
    Tip: Large movements of the percentile values are a reason for concern. In this case, the lines do not show major fluctuations.

5 Inspect the propensity decile distribution graph

  1. Question: Where does the Propensity decile distribution graph peak?
    1. Inspect the propensity decile distribution graph.
      The Propensity decile distribution graphs
      Tip: The graph peaks on the low end of the propensity range.
  1. Question: Is there a reason for concern?
    Tip: The Propensity decile distribution graph peaks at the low end of the graph. This peak is considered good behavior. The graph also shows a long, decreasing tail towards the higher propensities, which means the model has received positive responses over time.

6 Inspect individual predictors

  1. On the Analysis > Predictors tab, in the Predictors field, enter Age and select the Customer.Age predictor.
  2. Question: Is there any reason to suspect data drift for this predictor?
    1. Inspect the Customer.Age percentiles graph.
      The percentiles graphs for Age
      Tip: The Median graph shows a stable value. This does not indicate data drift.
  1. On the Analysis > Predictors tab, in the Predictors field, enter CLV and select the Customer.CLV predictor.
    1. Inspect the Customer.CLV top values graph.
    2. The percentiles graphs for CLV
  2. Question: Is there any reason to suspect data drift for this predictor?
    Tip: The values show little change over time. This does not indicate data drift.

This Challenge is to practice what you learned in the following Module:

Available in the following mission:

Return to mission Continue mission
If you are having problems with your training, please review the Pega Academy Support FAQs.

Did you find this content helpful?

Yes
No

Want to help us improve this content?
Suggest an edit

We'd prefer it if you saw us at our best.

Pega Academy has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice