Talend Data Mapper: Tips and Tutorials Part 1

Welcome! This is part one of a three part series all about Talend’s own data mapping tool: Talend Data Mapper. Subjects like input and output structures, using EDI documents, conditional statements and filtering will be available over the series to help develop insight into the mapping madness.

We are also sharing posts and previews on channels like Instagram, TikTok, LinkedIn and YouTube. Feel free to join the communities over there and keep up to date on Talend, SQL, Data Integration, Analytics and much more!

Why is there a Data Mapper?

The Talend Data Mapper is the low-code solution for matching and transforming hierarchic structured data.

Hierarchical data structures like JSON and XML are commonly used but can become quite complex with many loops and arrays existing both on the same level and deeply nested. Accessing these structures requires looping during parsing, which quickly turns into a technically complex product to develop and maintain.

The Talend Data Mapper is a drag and drop solution to match fields in any depth of a list or array. But beneath the surface are development functions that help with different parts of the Data Integration process. Lookups in a database, conditionals and complex java code can all be used in the tool.

The Data Mapper is a single data integration component to be used in a Job in Talend Studio, which makes it a simple to understand solution for the client and an easier to maintain job for the developer.

Data Mapping

One mistake during data mapping can lead to a replicated error, ripple through your data process and, ultimately, lead to low-quality data for analysis or costly misinterpretation.

Data Mapping is crucial for data integration, data migration, data warehouse automation, automated data extraction and other data management processes used in all layers of business. It is the bridge between systems, data models or sources, and makes data usable at the destination.

While data mapping has been common business for some time, the amount of data (and its sources) has been increasing, making the data mapping process more complex. This requires tools to that can be automated and customized for large data projects.

Talend Data Mapper Applied

The data mapper has many functions, utilities and tricks up its sleeve. Getting to know some of these may help you in deciding if the Talend Data Mapper can help you in the future.

Filtering loops

Using Talend Data Mapper makes data mapping for complex hierarchical structures as easy as dragging and dropping elements but might require more specific choices being made depending on certain conditions. That is why loop functions have a filter argument to specify expressions that run each instance of the loop. Each time the loop runs, the expression is evaluated. If it returns True, the instance is included in the output. If it returns False it is not included.

Filtering enables more control over which data gets passed on to the output and multiple filters can be applied on each loop level. Filters can be as simple as a constant or as complex as filtering on a validation rule.

Simple Filtering on Loop

In this example we create a simple filter on the main loop, to make sure the mapping only processes incoming messages that are from the specific RetailFormula ‘STORE’.

1. 1. When highlighting the root node, we see the main loop being a SimpleLoop function on the root node of the input data. This means that for each instance of the input root node, the output root node and all its child nodes run once.
   2. Below the Input Map Element argument in the SimpleLoop function is the Filter argument. From the functions panel, drag the Equal function onto the Filter argument.
   3. We now want to specify the Equal function with two arguments which it needs to evaluate. For the first value we drag the RetailFormula element from the input onto the first argument.
   4. Again, from the functions panel, drag in the constant function onto the second argument.
   5. Double-Click the constant and specify its value as a string ‘STORE’.

 

This results in a filtered output for this complete mapping since it now filters on the root node of the output structure. For each instance of the incoming data, the mapping will evaluate if the value of the element RetailFormula is equal to ‘STORE’. Only if the result is true will the mapping be executed.

Filtering on a Validation Rule

As a more complex example of filtering loops, we will set up a validation rule and apply this rule to a filter. There are three functions part of setting up a filter on the validation rule: ValidateGroup, Contains and IsValid. They will be explained during use in the example.

1. First, to prepare the Validation Rule, we assign the looping input element to a Validation Group using the ValidateGroup function. From the functions panel, drag the ValidateGroup function onto the looping input element.
2. Specify the Message and the Number describing the validation issue.
3. For the data argument we drag in the DeliveryMethod element from the input structure.
4. Next, select that same DeliveryMethod element in the input structure and open the Validate tab.
5. From the function panel we drag in the Contains function and add the DeliveryMethod to the InputValue argument.
6. Double-Click the Contains function and specify the contained string as ‘HomeDelivery’.

7. The final step is to apply the validation rule to the filter for the looping output element. Select the element and, from the functions panel, drag the IsValid function onto the filter argument.

   

For every loop instance the filter will evaluate if the mapped input value is valid according to its validation rule and return a Boolean, which then either includes the instance or excludes the instance from the output.

Aggregate on Loop

When building a mapping in the Talend Data Mapper, connecting looping elements from the input to looping elements in the output is a straightforward drag-and-drop. For each element the data mapper will define a loop expression to specify how the incoming element is handled. But when mapping a looping element from the input to a non-looping element in the output, there is no loop handling available and you need to define an aggregate function to handle the incoming loop.

The Talend Data Mapper has several functions available to aggregate on incoming loops, with the four most used being: AgConcat, AgConcateFirstPresentValue, AgCount and AgSum.

AgConcat

The AgConcat function Aggregates all loop iterations and Concatenates element values.

This function requires defining a loop expression to handle the incoming looping element. It will then loop according to the loop expression and concatenate all elements. Within the loop expression the filter commonly used to define which elements from the incoming loop are written to the output, for example filtering on indexNumber or a key for a key/value pair.