Welcome to part two of our series about the Talend Data Mapper. This article is all about different types of structures, using EDI documents and sharing data back and forth with data integration jobs in Talend Studio.
We are also sharing posts and previews on channels like Instagram, TikTok, LinkedIn and YouTube. Feel free to join the communities over there and keep up to date on Talend, SQL, Data Integration, Analytics and much more!
Talend Data Mapper Applied
The data mapper has many functions, utilities and tricks up its sleeve. Getting to know some of these may help you in deciding if the Talend Data Mapper can help you in the future. This is our continuation of helpful tips and step by step tutorials on getting the most out of the Talend Data mapping tool.
Inherited Structures
In the Talend Data Mapper we work with Data Structures. A structure is a collection of elements that describe the contents of a type of document. You can import structures from .zip archives, from example JSON and XML documents or create a structure from scratch. Another way to create a structure is by having it inherit its elements from a previously built structure.
Creating a Structure from Inheritance
In this example we create a simple filter on the main loop, to make sure the mapping only processes incoming messages that are from the specific RetailFormula ‘STORE’.
- In the Hierarchical mapper, under the Structures node, right click and select to create a new Structure.
- Tick the option to create a structure by inheriting from another structure. This does mean there has to already be a structure in place to inherit from.
- After selecting Next, navigate to the structure you want to inherit from and name your new structure. It is advised to use a standardized naming strategy for your project, to make documenting and sharing easier.
- When you click finish, the new structure will be open in your editor. On the left side of the editor, you see the representation of your structure with black arrows on the elements. This is a visual indicator to show that the elements are inherited from another structure. Inheritance means that changing the structure here, does not create a change in the parent structure. But changing the parent structure does change the child structure.
Inherited Structures makes the use of very complex mappings much easier by linking structures together. In a large integration project, the same Common Message Model (CMM) can be reused or partially inherited, while keeping the control over elements in a central structure. With the CMM, inheritance and the Talend Data Mapper it gives both flexibility and control to the developer and the user.
Working with EDI documents
Electronic Data Interchange (EDI) is the exchange of business documents in a standard electronic format between business partners. There are several EDI standards in use today, including ANSI, EDIFACT and TRADACOMS.
EDI has strict formatting and syntax rules to define exactly how each piece of information in the document can be found. Each single business document is comprised of data elements. These are individual items of information within the document. For each element there is EDI standard notation to indicate the data type followed by a data element separator.
These documents can get very large, and it would take a lot of time to manually build structures for each document. That is why Talend Data Mapper has the capability to import EDI structures, so you can focus on the mapping. Talend Data Mapper allows you to work with structures for various EDI standards, such as X12 HIPAA, HL7 and EDIFACT.
Import EDI structures from zip specifications
- Start by determining the EDI standard and version you are using. For example, the 2022 most recent version for UN/EDIFACT is D.22A. On the UNECE website you can download a .zip package containing all the specifications. [2]
- In Talend Studio, select the mapping view and in the hierarchical mapper on the left side of the interface expand the Structures node. This shows an overview of all structures in the project, which can be further organized in folders.
- Right-click the Structures node and select New > Structure.
- The first option lets you import a structure definition from a document. This can be an XML schema, but also a sample document (i.e., a JSON you might receive as incoming data carrier). This option also has the capability of importing the .zip specifications for EDI documents. Click the next button.
- In the list of possible imports find the EDIFACT specifications (zip) and click next.
- Select the local file option, click browse and locate the d22a.zip file in the downloads folder. Click on open and then again press next.
- Select the folder you want to create the structures in. At the top you can see the complete path. Here it also shows that it will create a subfolder for the d22a version. When creating structures for different versions they will be in their respective folders, making it easy to operate different versions side by side in a project. Click next, Talend Data Mapper will extract the .zip and create the structures belonging to version d22a.
-
When you know expand the structures node in the Hierarchical Mapper, in the structures folder there is now a d22a folder. If you open it up it contains a folder Messages. In there are standardized structures for all message types available in UN/EDIFACT, which you can then use as the input structure when creating a mapping.
REFERENCES
[1] https://www.edibasics.com/
[2] https://unece.org/trade/uncefact/unedifact/download
Using External Variables
In Data Integration jobs in Talend, global variables are useful for storing and sharing snippets of data or parameters to be called anywhere in the Standard Job Framework. These key/value pairs defined in the job are then also available for use within the Talend Data Mapper. Besides calling a globalVar, you can also put a global variable, which is then available for use in other components in the job.
Calling External Variables
In the Talend Data Mapper, select the field in the Output structure that needs to be filled with the value from the globalVar. From the function panel drag in the ‘GetValueFromExternalMap’ function onto the Value tab for the selected output field. The empty ‘Key’ field appears underneath the function, this can be filled with a value from the Input structure but will often be a constant with the key set in the Job.
Drag the ‘Constant’ function from the functions panel on the ‘Key’ field and enter the key from the key/value pair set in the job. When the Talend Data Mapper gets to this output field it will call the external variable from the job.
Putting a Value to an External Variable
During the mapping process, you can also put values in a new global variable straight from the Talend Data Mapper.
On the Output field you want to store in a global variable, select the Util tab. The Value tab is for mapping values to the field, the Util tab can be used here to extract values from the Data Mapper for global use in the Talend DI Job.
From the function panel, drag the ‘PutValueToExternalMap’ function onto the Util tab. This will show a Key and a Value field to define the globalVar. In this example we drag a ‘constant’ function in for the key and name it ‘OrderCurrency’. From the input structure, drag the ‘Currency’ field in for the ‘Value’.
When the Mapping runs during a Data Integration Job it will create a globalVar with the key/value pair: ‘OrderCurrency’ and the value from the ‘Currency’ input structure. This is then accessible in the DI job by calling it as a regular global variable.
Read more about the Talend Data Mapper and other Data Integration best practices in the next post! There is even more educational content available in other posts in our blog. Also follow us on Instagram, TikTok, LinkedIn, YouTube for more information and don’t hesitate to contact us!