Use Case - Setting up data ingestion
2 Tasks
20 mins
Scenario
The U+ Comms data warehouse team created the subscription and account data files, and their respective manifest files. IT built the technical infrastructure to upload the data daily to a repository that Pega Customer Decision Hub™ can access. A token file will initiate the data ingestion process.
Prepare the batch import data jobs to populate the subscription and account tables in Customer Decision Hub.
The IT development team has provided access to the file repository through File Browser, a third-party application. Use File Browser to browse, review, and create new files in the repository.
Tip: A variation of this scenario exists in Batch import data jobs.
Detailed Tasks
1 Other details
- As the Decisioning Architect, create two new file data sets from Profile Data Sources to access the files in the repository:
- Subscription file data set
Requirement Detail Name
Subscription file
Source type
File repository
Apply to
UComms-CDH-Data-Subscription
Repository
filerepo
File path
filerepo/IngestionData/Subscription/
Manifest file name
SubscriptionManifest.xml
Data file name
SubscriptionData.csv
CSV file details
First row contains fields (header)
Delimiter character
Comma ( , )
Quotation marks
None
Date field formats
Date time format: MM/dd/yyyy HH:mm
Date format: MM/dd/yyyy
Time format: HH:mm:ss - Account file data set
Requirement Detail Name
Account file
Source type
File repository
Apply to
UComms-CDH-Data-Account
Repository
filerepo
File path
filerepo/IngestionData/Account/
Manifest file name
AccountManifest.xml
Data file name
AccountData.csv
CSV file details
First row contains fields (header)
Delimiter character
Comma ( , )
Quotation marks
None
Date field formats
Date time format: MM/dd/yyyy HH:mm
Date format: MM/dd/yyyy
Time format: HH:mm:ssCaution: Ensure that the date and time formats are correctly set when you create the file data sets. If an error occurs during the data ingestion, you can reuse the files from the ArchivedData folder by using the File Browser.
- Subscription file data set
- Create two new batch import data jobs to ingest the data:
- Import Subscription data job
Requirement Detail Data job name
Import Subscription
Target
Subscription
Source
Subscription file
Trigger
File Detection
Failure policy
10 records per file.
- Import Account data job
Requirement Detail Data job name
Import Account
Target
Account
Source
Account file
Trigger
File Detection
Failure policy
10 records per file.
- Import Subscription data job
- For this use case, you have access to a third-party application, File Browser. With this application, you can browse, upload, and download files from the file repository. On the exercise landing page, click File Browser to log in with the following credentials and browse the files, then create new token files in the repository:
- Username: pega-filerepo
- Password: pega-filerepo
- To begin data ingestion, create a new .tok file in the IngestionData/Subscription folder. Then, create another .tok file in the IngestionData/Account folder to initiate the data job for Account data, as shown in the following figure:
Tip: To create a new file, click New file in the left navigation pane. After you create the new .tok file, click Save in the upper-right corner.
2 Acceptance criteria
- A new file data set is created to access the subscription data.
- A new file data set is created to access the account data.
- A new data job is created to ingest subscription data.
- A new data job is created to ingest account data.
- Both data jobs are triggered by a token file.
- Both subscription and account data sources are populated with data.
Available in the following mission:
If you are having problems with your training, please review the Pega Academy Support FAQs.
Want to help us improve this content?