Saturday, 26 January 2013

Understand Context variables with Examples - Part 3 (Talend Open Studio)

In the earlier posts we have provided the value of the context variables either in Talend Job itself or defined the values in Repository context variables. Today I am going to create a Talend job which will filter the records based on the values of the context variables placed in the file. We will use Talend’s tContextLoad component.

I am going to use the same input file which I have used in my previous post on repository context variables. Lets look at the input file.



I have formatted the input file as below to make it easy to read.


I will create two context variables; context_dept_id and context_salary.

Lets get started and create a new talend job and follow the below mentioned steps:

1. Create metadata for our Input delimited employee file. You can take reference of the steps to create metadata for delimited from here. Your metadata should look like following:

2.Drag the metadata created in step 1 to Job designer pane and select tFileInputDelimited component from the Popup.

3. Drag tFilterRows and tLogRow components from Palette pane. Component tFilterRows will be used to filter the records based on the values of the context variables and tLogRow will be used to display the output in output console.

4. Right click tFilterRows, select Row > main and connect it to tLogRow.

As of now, we have not defined any context variable. Lets run the job and observe our output.

Output without context variable:


As you can see that, all of the input file records have been displayed in the output as we have not defined and used context variable in the tFilterRows component.

Now lets define context variables. Open the Contexts pane.


5. Add two context variables as we did in the last post; context_dept_id and context_salary.

Also, do not provide any default values to the context variables.

6. Now we have created these context variables, its time to use them to filter the input records. Open the component properties of the tFilterRows and add two conditions as shown below. In the value section provide the name of the context variables.


Remember: We have not provided values to the context variables. As of now we have only defined these variables.

7. Try to execute the Job and observe what happens.




It will throw exception “Exception in component tFilterRow_1 : java.lang.NullPointerException”.

This is because we have not provided any values to the context variables. In the earlier posts we have provided the value of the context variables either in Talend Job itself or defined the values in Repository context variables. Today we are going to provide the values of the context variables in the file. This can help us dynamically provide values of the variables.

Context variable name and its values should be present in the key value pair in the file. As per our example I have provided the values of context variables as mentioned below. Note: Name of the context variable in the file should match with the name of the context variable defined in the Job. Below is the screenshot of our context variable file.



8. In the current Job Drag tFileInputDelimited component from Palette Pane. This component will be used to read the context variable file. Create the metadata of this component as following and provide the path of the context variable file in the File Path.


9. Now drag tContextLoad component from Palette pane and place it onto job designer. This component will set the value of the context variables defined in the job from the file.

10. Right Click tFileInputDelimited component , select Row>Main and connect it to tContextLoad component. This will create a new subjob.

11. Right click tFileInputDelimited component and Select Trigger>OnSubjobOk and connect it to tFileInputDelimited component which is reading our actual file. This will allow the job to first load the context variable values from file and then run the actual job.

12. Now try to execute the job and check the output.



You can see that only those records are present in the output where department is 12 and salary is greater than 10000. You can compare it with input file (screenshot below).

13. Now change the values in the context variables in the file as below.



You can see that only those records are present in the output where department is 10 and salary is greater than 10000. You can compare it with input file. You can find the snapshot of US input file below:


This end the 3 article tutorial on How to define and use the context variables. Links to other two (Part 1 and Part 2) articles are mentioned below:


Understand Context variables with Examples - Part 1 (Talend Open Studio)
Understand Context variables with Examples - Part 2 (Talend Open Studio)

You may also like to read..

1. Understand Context Variables Part 1 ( Define context variables inside Talend job)
2. Understand Context Variables Part 2 (Populate the values of context variables from file. tContextLoad)
3. How to Pass Context Variables to Child Jobs.
4. How to Pass context Variables/ Parameters through  command line.

5 comments: