Skip to main content

Setting up data ingestion

3 Tasks

40 mins

 Visible to: All users
Beginner Pega Customer Decision Hub 8.8 Next Best Action Decision Management Data Model Data Integration English
Verify the version tags to ensure you are consuming the intended content or, complete the latest version.

Scenario

The project team at U+ Bank performed a data mapping workshop and mapped its existing data model to the analytical data model for the financial services customer. The U+ Bank data warehouse team prepared the customer and account data files and the manifest files. IT developed the technical infrastructure to upload the data daily to a repository that Pega Customer Decision Hub™ can access.

As a decisioning architect, your role is to prepare the data jobs to populate customer and account tables in Pega Customer Decision Hub™.

Use the following credential to log in to the exercise system:

Role User name Password
Decisioning architect DecisioningArchitect rules

Your assignment consists of the following tasks:

Task 1: Clear the Customer and Account tables

As a decisioning architect, use the Customer and Account data sets to clear any test data created in Customer Decision Hub.

Note: The exercise system contains customer and account data generated from a Monte Carlo data set in a previous exercise.

Task 2: Create a new import data job for the customer data

As a decisioning architect, create a new data job to ingest the customer data file by using the details from the following table:

Requirement Detail

Data source

Customer

Repository

defaultstore

File path

/IngestionData/CustomerData/

Manifest file name

CustomerDataIngestManifest.xml

Data file name

CustomerDataIngest.csv

Date field formats

Date time format: MM/dd/yyyy HH:mm
Date format: MM/dd/yyyy

Time format: HH:mm:ss

Import using

A schedule, once a day.

Failure policy

10 records per file.

Confirm that the system successfully ingests the data.

Task 3: Create a new import data job for the account data

As a decisioning architect, create a new data job to ingest the account data file by using the details from the following table:

Requirement Detail

Data source

Account

Repository

defaultstore

File path

/IngestionData/AccountData/

Manifest file name

AccountDataIngestManifest.xml

Data file name

AccountDataIngest.csv

Date field formats

Date time format: MM/dd/yyyy HH:mm
Date format: MM/dd/yyyy

Time format: HH:mm:ss

Import using

A schedule, once a day.

Failure policy

10 records per file.

Confirm that the system successfully ingests the data.

 

You must initiate your own Pega instance to complete this Challenge.

Initialization may take up to 5 minutes so please be patient.

Challenge Walkthrough

Detailed Tasks

1 Clear the Customer and Account tables

  1. On the exercise system landing page, click Pega CRM suite to log in to Customer Decision Hub.
  2. Log in as the decisioning architect:
    1. In the User name field, enter DecisioningArchitect.
    2. In the Password field, enter rules.
  3. In the header of Customer Decision Hub, in the search field, enter Customer, and then click the search icon.
    1. In the third filter list, select Exact Match.
    2. In the list of results, select the Customer data set with the Applies to class UBank-CDH-Data-Customer.
      Select the Customer Data set
  4. In the upper-right corner, click the Run to truncate the data set:
    1. In the Run Data Set: Customer window, in the Operation list, select Truncate.
    2. In the upper-right corner, click Run to truncate the customer table.
    3. Close the Status Page window, and then close the Run Data Set: Customer window.
  5. In the lower-left corner, click Back to Customer Decision Hub to return to Customer Decision Hub.
    Return back to the Customer Decision Hub
  6. Repeat steps 3–5 for the Account data set to truncate the account data.
    Select the Account Data set
  7. In the navigation pane of Customer Decision Hub, click Data > Profile Designer to view data sources.
    Launch Profile Designer
  8. On the Profile Designer landing page, click Customer to open the customer data source.
    Launch the Customer Data source
  9. On the Data Set: Customer landing page, click the Records tab to confirm that there are no items.
    Click the records tab in the Customer Data set
  10. Optional: To confirm that there are no items for the Account data set, repeat steps 8–10.

2 Create a new import data job for the customer data

  1. In the navigation pane of Customer Decision Hub, click Data > Data Jobs to create a new data job for ingesting customer data.
  2. On the Data Jobs landing page, click Create import data job.
  3. In the Create import data job: Name & target (1 of 9) window, select Customer, then click Next.
    Select the Customer data set for target
  4. In the Create import data job: Repository (2 of 9) window, in the Repository list, select defaultstore, then click Next.
    Choose the defaultstore repository
    Note: For the purposes of the exercise, all files use the defaultstore (a system-managed temporary file storage) repository. In a real-life scenario, all files are typically in a file repository such as AWS S3.
  1. In the Create import data job: Source location (3 of 9) window, in the IngestionData folder, navigate to the Customer manifest file:
    1. In the Name column, click the IngestionData folder.
      Choose the ingestiondata folder
    2. In the Name column, click the CustomerData folder.
      Choose the customerdata folder
    3. In the Name column, select CustomerDataIngestManifest.xml.
      Choose the manifest file
    4. Click Next.
  2. In the Create import data job: File configuration (4 of 9) window, configure the following settings:
    1. Select the First row contains fields (header) checkbox.
    2. In the Delimiter character list, select Comma(,).
    3. In the Date time format field, enter MM/dd/yyyy HH:mm.
    4. In the Date format field, enter MM/dd/yyyy.
    5. Click Next.
      Choose file configurations
  3. In the Create import data job: File preview (5 of 9) window, confirm the file contents:
    1. On the Manifest tab, confirm that the manifest file content is valid.
      Investigate the manifest file
    2. In the Data file tab, confirm that the data file content is valid.
      Investigate the data file
    3. Click Next.
  4. In the Create import data job: Field mapping (6 of 9) window, confirm that the field mappings are correct, then click Next.
    Complete the mapping of fields with columns
  5. In the Create import data job: Trigger (7 of 9) window, complete the following settings:
    1. In the Select the trigger that starts a new run section, select Schedule.
      Caution: In a real-life scenario, the trigger that you select depends on the data ingestion requirements. Use the File detection option to trigger the data ingestion process by a token file. File detection is the most used option, as the process initiates only when all files for ingestion are ready for processing. Use the Schedule option only when you know the exact time and guarantee that the system can ingest the file. The system does not process the file if the file is not in the repository at the scheduled time.
    2. In the Recurrence list, select Daily.
    3. In the Start time field, enter a time in the near future.
      Note: Add five to ten minutes to the current time in your time zone when the process can start.
      For example, if the current time is 13:30 PM EST, then set the start time to 13:35 PM EST.
    4. In the Time zone list, select your time zone.
    5. Click Next.
      Choose a schedule
  1. In the Create import data job: Failure policy (8 of 9) window, complete the following settings:
    1. In the Fail a run after more than field, enter 10.
    2. Click Next.
      Define the failure policy
  2. In the Create import data job: Review (9 of 9) window, review the configuration, then click Create.
    Review your configuration
  3. On the Data Jobs landing page, double-click the Import Customer row to see its details.
    Import Customer data job is now active
  4. On the Data Job: Import Customer landing page, in the Runs section, confirm that there are no scheduled jobs..
    The overview of the customer import data job
  5. Wait for the time that you configured for the schedule, and then, in the upper-right corner, click Refresh.
    Note: The time is always displayed in the time zone format (America/New_York) configured in the Localization settings in the OperatorID record.
  1. On the Data Job: Import Customer landing page, confirm that the system ingests the records.
    Import completed
  2. In the Details section, in the Target profile data source field, click the Customer data source.
  3. On the Data Set: Customer landing page, click the Records tab to confirm that the system ingests the customer data.
    Review the imported records

3 Create a new import data job for the account data

  1. In the navigation pane of Customer Decision Hub, click Data > Data Jobs to create a new data job for ingesting account data.
  2. On the Data Jobs landing page, in the upper-right corner, click Create import data job.
  3. In the Create import data job: Name & target (1 of 9) window, select Account, then click Next.
  4. In the Create import data job: Repository (2 of 9) window, in the Repository list, select defaultstore, and then click Next.
  5. In the Create import data job: Source location (3 of 9) window, in the IngestionData folder, navigate to the Account manifest file:
    1. In the Name column, select the IngestionData folder.
    2. In the Name column, select the AccountData folder.
    3. In the Name column, select AccountDataIngestManifest.xml.
    4. Click Next.
  6. In the Create import data job: File configuration (4 of 9) window, configure the following settings:
    1. Select the First row contains fields (header) checkbox.
    2. In the Delimiter character list, select Comma(,).
    3. In the Date time format field, enter MM/dd/yyyy HH:mm.
    4. In the Date format field, enter MM/dd/yyyy.
    5. Click Next.
  7. In the Create import data job: File preview (5 of 9) window, confirm the file contents:
    1. In the Manifest tab, confirm that the manifest file content is valid.
    2. In the Data file tab, confirm that the data file content is valid.
    3. Click Next.
  8. In the Create import data job: Field mapping (6 of 9) window, confirm that the field mappings are correct, then click Next.
  9. In the Create import data job: Trigger (7 of 9) window, complete the following settings:
    1. In the Select the trigger that starts a new run field, select Schedule.
    2. In the Recurrence list, select Daily.
    3. In the Start time field, enter a time in the near future.
      Note: Add five minutes to the current time in your time zone.
      For example, if the current time is 13:30 PM EST, set the start time to 13:35 PM EST.
    4. In the Time zone list, select the time zone that you are in.
    5. Click Next.
  1. In the Create import data job: Failure policy (8 of 9) window, complete the following settings:
    1. In the Fail a run after more than field, enter 10.
    2. Click Next.
  2. In the Create import data job: Review (9 of 9) window, review the configuration, then click Create.
  3. On the Data Jobs landing page, double click on the Import Account row, to see its details.
  4. On the Data Job: Import Account page, confirm there are no Runs.
  5. Wait for the time that you configured for the schedule, and then, in the upper-right corner, click Refresh.
    Note: The time is always displayed in the time zone format (America/New_York) configured in the Localization settings in the OperatorID record.
  1. On the Data Job: Import Account landing page, confirm that the system ingests the records.
    Confirm the account is imported
  2. In the Details section, in the Target profile data source field, click the Account data source.
  3. On the Data Set: Account landing page, click the Records tab to confirm that the system ingests the account data.

This Challenge is to practice what you learned in the following Module:

Available in the following mission:

Return to mission
If you are having problems with your training, please review the Pega Academy Support FAQs.

Did you find this content helpful?

Yes
No

Want to help us improve this content?
Suggest an edit

We'd prefer it if you saw us at our best.

Pega Academy has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice