Monitoring an adaptive model
Archived
2 Tasks
25 mins
Scenario
BankCo, a retail bank, has recently launched two new credit card offers, BaseCard and YoungCard, to attract new customers. The bank has implemented a solution where adaptive models present the offer in which the customer is most likely to be interested.
It has been a few weeks since the start of the project, and the adaptive models are actively learning from customer responses and predicting the right offer for each customer. The bank wants to analyze the model reports and determine which predictors are better at predicting customer behavior and see how the models are performing. The bank also intends to analyze the acceptance rate of the offers.
Use the following credentials to log in to the exercise system:
| Role | User name | Password |
|---|---|---|
| Data Scientist | DataScientist | rules |
Caution: This challenge builds on top of the previous challenge. Hence when you click on Initialize Pega or Reset Instance in this challenge, you first need to complete the previous challenge: Configuring an adaptive model.
Your assignment consists of the following tasks:
Task 1: Simulate customer responses
Run the SimulateRealtimeInteractions data flow to simulate customer responses.
Task 2: Confirm your work
Test the decision strategy to verify that the propensity value has changed and examine the adaptive model reports.
Challenge Walkthrough
Detailed Tasks
1 Simulate customer responses
- Log in as a Data Scientist with user name DataScientist and password rules.
Note: If you are in Prediction Studio, in the lower left, click Back to Customer Decision Hub.
- In the navigation pane on the left, click Data > Data Flows.
- In the list of data flows, double-click SimulateRealtimeInteractions to edit the data flow.
- In the upper right, click Actions > Run to open the Data Flow test run page.
- When the Data Flow test run status changes from Progressing to Completed, close the Data Flow run test page.
- Close the SimulateRealtimeInteractions data flow.
- Close the list of data flows and return to the strategy canvas.
2 Confirm your work
- Run the strategy again and examine the Propensity values. The values have changed. They are no longer 0.5. Do you know why?
- On the strategy canvas, right-click the Card Response adaptive model component, and then select Open Adaptive Model to open the adaptive model form in Prediction Studio.
- In the adaptive model form, click Refresh data.
- On the Monitor tab, click Predictors.
- Examine the predictors report. Which predictors are used in all models? Which predictors are inactive in all models? Which is the highest performing predictor?
- Click the Models tab.
- To the right of the row, click Model report to view the YoungCard action report.
- Examine the list of predictors. Which are the active predictors? Which is the best performing predictor?
- Open the best performing predictor and check the bin distribution.
Note: As there is some randomization in the traning process, different training execution may produce different top performing predictors.
- Go back to the list of predictors. Which predictors are inactive?
Tip: Expand one of the grouped predictors to view the list of inactive predictors.