Changes for page 12 Validation and Transformation Language (VTL)

Last modified by Artur on 2025/09/10 11:19

From 1.20 to 1.21 From 1.25 to 1.26

From version 1.21

edited by Helena
on 2025/06/16 13:26

Change comment: There is no comment for this version

To version 1.25

edited by Helena
on 2025/06/16 13:38

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)
Attachments (0 modified, 1 added, 0 removed)
- 1750070288958-132.png

Details

Page properties

Content

@@ -256,19 +256,14 @@
  At observation / data point level, calling Cj (j=1, … n) the j^^th^^ Code of the MeasureDimension:
  * The set of SDMX observations having the same values for all the Dimensions except than the MeasureDimension become one multi-measure VTL Data Point, having one Measure for each Code Cj of the SDMX MeasureDimension;
--* The values of the SDMX simple Dimensions, TimeDimension and DataAttributes not depending on the MeasureDimension (these components by definition have always the same values for all the observations of the set above) become the values of the corresponding VTL (simple)
--
--Identifiers, (time) Identifier and Attributes.
--
++* The values of the SDMX simple Dimensions, TimeDimension and DataAttributes not depending on the MeasureDimension (these components by definition have always the same values for all the observations of the set above) become the values of the corresponding VTL (simple) Identifiers, (time) Identifier and Attributes.
  * The value of the Measure of the SDMX observation belonging to the set above and having MeasureDimension=Cj becomes the value of the VTL Measure Cj
  * For the SDMX DataAttributes depending on the MeasureDimension, the value of the DataAttribute DA of the SDMX observation belonging to the set above and having MeasureDimension=Cj becomes the value of the VTL Attribute DA_Cj
  ==== 12.3.3.3 From SDMX DataAttributes to VTL Measures ====
--* In some cases, it may happen that the DataAttributes of the SDMX DataStructure need to be managed as Measures in VTL. Therefore, a variant of both the methods above consists in transforming all the SDMX DataAttributes in VTL Measures. When DataAttributes are converted to Measures, the two methods above are called Basic_A2M and Pivot_A2M (the suffix "A2M" stands for Attributes to Measures). Obviously, the resulting VTL data structure is, in general, multi-measure and does not contain
++* In some cases, it may happen that the DataAttributes of the SDMX DataStructure need to be managed as Measures in VTL. Therefore, a variant of both the methods above consists in transforming all the SDMX DataAttributes in VTL Measures. When DataAttributes are converted to Measures, the two methods above are called Basic_A2M and Pivot_A2M (the suffix "A2M" stands for Attributes to Measures). Obviously, the resulting VTL data structure is, in general, multi-measure and does not contain Attributes.
--Attributes.
--
  The Basic_A2M and Pivot_A2M behaves respectively like the Basic and Pivot methods, except that the final VTL components, which according to the Basic and Pivot methods would have had the role of Attribute, assume instead the role of Measure.
  Proper VTL features allow changing the role of specific attributes even after the SDMX to VTL mapping: they can be useful when only some of the DataAttributes need to be managed as VTL Measures.
@@ -285,11 +285,12 @@
  Mapping table:
--|**VTL**|**SDMX**
--|(Simple) Identifier|Dimension
--|(Time) Identifier|TimeDimension
--|Measure|Measure
--|Attribute|DataAttribute
++(% style="width:470.294px" %)
++|(% style="width:262px" %)**VTL**|(% style="width:205px" %)**SDMX**
++|(% style="width:262px" %)(Simple) Identifier|(% style="width:205px" %)Dimension
++|(% style="width:262px" %)(Time) Identifier|(% style="width:205px" %)TimeDimension
++|(% style="width:262px" %)Measure|(% style="width:205px" %)Measure
++|(% style="width:262px" %)Attribute|(% style="width:205px" %)DataAttribute
  If the distinction between simple identifier and time identifier is not maintained in the VTL environment, the classification between Dimension and TimeDimension exists only in SDMX, as declared in the relevant DataStructureDefinition.
@@ -317,11 +317,12 @@
  The summary mapping table of the **unpivot** mapping method is the following:
--|**VTL**|**SDMX**
--|(Simple) Identifier|Dimension
--|(Time) Identifier|TimeDimension
--|All Measure Components|MeasureDimension (having one Code for each VTL measure component) & one Measure
--|Attribute|DataAttribute depending on all SDMX Dimensions including the TimeDimension and except the MeasureDimension
++(% style="width:638.294px" %)
++|(% style="width:200px" %)**VTL**|(% style="width:435px" %)**SDMX**
++|(% style="width:200px" %)(Simple) Identifier|(% style="width:435px" %)Dimension
++|(% style="width:200px" %)(Time) Identifier|(% style="width:435px" %)TimeDimension
++|(% style="width:200px" %)All Measure Components|(% style="width:435px" %)MeasureDimension (having one Code for each VTL measure component) & one Measure
++|(% style="width:200px" %)Attribute|(% style="width:435px" %)DataAttribute depending on all SDMX Dimensions including the TimeDimension and except the MeasureDimension
  At observation / data point level:
@@ -343,12 +343,13 @@
  The mapping table is the following:
--|VTL|SDMX
--|(Simple) Identifier|Dimension
--|(Time) Identifier|TimeDimension
--|Some Measures|Measure
--|Other Measures|DataAttribute
--|Attribute|DataAttribute
++(% style="width:467.294px" %)
++|(% style="width:214px" %)VTL|(% style="width:250px" %)SDMX
++|(% style="width:214px" %)(Simple) Identifier|(% style="width:250px" %)Dimension
++|(% style="width:214px" %)(Time) Identifier|(% style="width:250px" %)TimeDimension
++|(% style="width:214px" %)Some Measures|(% style="width:250px" %)Measure
++|(% style="width:214px" %)Other Measures|(% style="width:250px" %)DataAttribute
++|(% style="width:214px" %)Attribute|(% style="width:250px" %)DataAttribute
  Even in this case, the resulting SDMX definitions must be compliant with the SDMX consistency rules. For example, the SDMX DSD must have the attributeRelationship for the DataAttributes, which does not exist in VTL.
@@ -386,11 +386,11 @@
  Therefore, the generic name of this kind of VTL datasets would be:
--'DF(1.0.0)/INDICATORvalue.COUNTRYvalue'
++> 'DF(1.0.0)/INDICATORvalue.COUNTRYvalue'
  Where DF(1.0.0) is the Dataflow and //INDICATORvalue// and //COUNTRYvalue //are placeholders for one value of the INDICATOR and COUNTRY dimensions. Instead the specific name of one of these VTL datasets would be:
--‘DF(1.0.0)/POPULATION.USA’
++> ‘DF(1.0.0)/POPULATION.USA’
  In particular, this is the VTL dataset that contains all the observations of the Dataflow DF(1.0.0) for which //INDICATOR// = POPULATION and //COUNTRY// = USA.
@@ -404,26 +404,22 @@
  SDMX Dataflow having INDICATOR=//INDICATORvalue //and COUNTRY=// COUNTRYvalue//. For example, the VTL dataset ‘DF1(1.0.0)/POPULATION.USA’ would contain all the observations of DF1(1.0.0) having INDICATOR = POPULATION and COUNTRY = USA.
--In order to obtain the data structure of these VTL Data Sets from the SDMX one, it is assumed that the SDMX DimensionComponents on which the mapping is based are dropped, i.e. not maintained in the VTL data structure; this is possible because their values are fixed for each one of the invoked VTL Data Sets{{footnote}}If these DimensionComponents would not be dropped, the various VTL Data Sets resulting from this kind of mapping would have non-matching values for the Identifiers corresponding to the mapping Dimensions (e.g. POPULATION and COUNTRY). As a consequence, taking into account that the typical binary VTL operations at dataset level (+, -, *, / and so on) are executed on the observations having matching values for the identifiers, it would not be possible to compose the resulting VTL datasets one another (e.g. it would not be possible to calculate the population ratio between USA and CANADA).{{/footnote}}. After that, the mapping method from SDMX to VTL specified for the Dataflow DF1(1.0.0) is applied (i.e.
++In order to obtain the data structure of these VTL Data Sets from the SDMX one, it is assumed that the SDMX DimensionComponents on which the mapping is based are dropped, i.e. not maintained in the VTL data structure; this is possible because their values are fixed for each one of the invoked VTL Data Sets{{footnote}}If these DimensionComponents would not be dropped, the various VTL Data Sets resulting from this kind of mapping would have non-matching values for the Identifiers corresponding to the mapping Dimensions (e.g. POPULATION and COUNTRY). As a consequence, taking into account that the typical binary VTL operations at dataset level (+, -, *, / and so on) are executed on the observations having matching values for the identifiers, it would not be possible to compose the resulting VTL datasets one another (e.g. it would not be possible to calculate the population ratio between USA and CANADA).{{/footnote}}. After that, the mapping method from SDMX to VTL specified for the Dataflow DF1(1.0.0) is applied (i.e. basic, pivot …).
--basic, pivot …).
--
  In the example above, for all the datasets of the kind
--‘DF1(1.0.0)///INDICATORvalue//.//COUNTRYvalue//’, the dimensions INDICATOR and COUNTRY would be dropped so that the data structure of all the resulting VTL Data Sets would have the identifier TIME_PERIOD only.
++> ‘DF1(1.0.0)///INDICATORvalue//.//COUNTRYvalue//’, the dimensions INDICATOR and COUNTRY would be dropped so that the data structure of all the resulting VTL Data Sets would have the identifier TIME_PERIOD only.
  It should be noted that the desired VTL Data Sets (i.e. of the kind ‘DF1(1.0.0)/// INDICATORvalue//.//COUNTRYvalue//’) can be obtained also by applying the VTL operator “**sub**” (subspace) to the Dataflow DF1(1.0.0), like in the following VTL expression:
--‘DF1(1.0.0)/POPULATION.USA’ :=
++> ‘DF1(1.0.0)/POPULATION.USA’ :=
++> DF1(1.0.0) [ sub INDICATOR=“POPULATION”, COUNTRY=“USA” ];
++>
++> ‘DF1(1.0.0)/POPULATION.CANADA’ :=
++> DF1(1.0.0) [ sub INDICATOR=“POPULATION”, COUNTRY=“CANADA” ];
++>
++> … … …
--DF1(1.0.0) [ sub INDICATOR=“POPULATION”, COUNTRY=“USA” ];
--
--‘DF1(1.0.0)/POPULATION.CANADA’ :=
--
--DF1(1.0.0) [ sub INDICATOR=“POPULATION”, COUNTRY=“CANADA” ];
--
--… … …
--
  In fact the VTL operator “sub” has exactly the same behaviour. Therefore, mapping different parts of a SDMX Dataflow to different VTL Data Sets in the direction from SDMX to VTL through the ordered concatenation notation is equivalent to a proper use of the operator “**sub**” on such a Dataflow.{{footnote}}In case the ordered concatenation notation is used, the VTL Transformation described above, e.g. ‘DF1(1.0)/POPULATION.USA’ := DF1(1.0) [ sub INDICATOR=“POPULATION”, COUNTRY=“USA”], is implicitly executed. In order to test the overall compliance of the VTL program to the VTL consistency rules, it has to be considered as part of the VTL program even if it is not explicitly coded.{{/footnote}}
  In the direction from SDMX to VTL it is allowed to omit the value of one or more DimensionComponents on which the mapping is based, but maintaining all the separating dots (therefore it may happen to find two or more consecutive dots and dots in the beginning or in the end). The absence of value means that for the corresponding Dimension all the values are kept and the Dimension is not dropped.
@@ -432,10 +432,9 @@
  This is equivalent to the application of the VTL “sub” operator only to the identifier //INDICATOR//:
--‘DF1(1.0.0)/POPULATION.’ :=
++> ‘DF1(1.0.0)/POPULATION.’ :=
++> DF1(1.0.0) [ sub INDICATOR=“POPULATION” ];
--DF1(1.0.0) [ sub INDICATOR=“POPULATION” ];
--
  Therefore the VTL Data Set ‘DF1(1.0.0)/POPULATION.’ would have the identifiers COUNTRY and TIME_PERIOD.
  Heterogeneous invocations of the same Dataflow are allowed, i.e. omitting different Dimensions in different invocations.
@@ -453,41 +453,38 @@
  The corresponding VTL Transformations, assuming that the result needs to be persistent, would be of this kind:{{footnote}}the symbol of the VTL persistent assignment is used (<-){{/footnote}}
--‘DF2(1.0.0)/INDICATORvalue.COUNTRYvalue’ <- expression
++> ‘DF2(1.0.0)/INDICATORvalue.COUNTRYvalue’ <- expression
  Some examples follow, for some specific values of INDICATOR and COUNTRY:
--‘DF2(1.0.0)/GDPPERCAPITA.USA’ <- expression11; ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ <- expression12;
--… … …
++> ‘DF2(1.0.0)/GDPPERCAPITA.USA’ <- expression11; ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ <- expression12;
++> … … …
++> ‘DF2(1.0.0)/POPGROWTH.USA’  <- expression21;
++> ‘DF2(1.0.0)/POPGROWTH.CANADA’ <- expression22;
++> … … …
--‘DF2(1.0.0)/POPGROWTH.USA’  <- expression21;
--‘DF2(1.0.0)/POPGROWTH.CANADA’ <- expression22;
--… … …
--
  As said, it is assumed that these VTL derived Data Sets have the TIME_PERIOD as the only identifier. In the mapping from VTL to SMDX, the Dimensions INDICATOR and COUNTRY are added to the VTL data structure on order to obtain the SDMX one, with the following values respectively:
  VTL dataset   INDICATOR value COUNTRY value
--‘DF2(1.0.0)/GDPPERCAPITA.USA’  GDPPERCAPITA USA
--‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ GDPPERCAPITA CANADA … … …
--‘DF2(1.0.0)/POPGROWTH.USA’  POPGROWTH USA
--‘DF2(1.0.0)/POPGROWTH.CANADA’  POPGROWTH CANADA
++> ‘DF2(1.0.0)/GDPPERCAPITA.USA’  GDPPERCAPITA USA
++> ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ GDPPERCAPITA CANADA … … …
++> ‘DF2(1.0.0)/POPGROWTH.USA’  POPGROWTH USA
++> ‘DF2(1.0.0)/POPGROWTH.CANADA’  POPGROWTH CANADA
++> … … …
--… … …
--
  It should be noted that the application of this many-to-one mapping from VTL to SDMX is equivalent to an appropriate sequence of VTL Transformations. These use the VTL operator “calc” to add the proper VTL identifiers (in the example, INDICATOR and COUNTRY) and to assign to them the proper values and the operator “union” in order to obtain the final VTL dataset (in the example DF2(1.0.0)), that can be mapped oneto-one to the homonymous SDMX Dataflow. Following the same example, these VTL Transformations would be:
--DF2bis_GDPPERCAPITA_USA := ‘DF2(1.0.0)/GDPPERCAPITA.USA’ [calc identifier INDICATOR := ”GDPPERCAPITA”, identifier COUNTRY := ”USA”];
--DF2bis_GDPPERCAPITA_CANADA := ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ [calc identifier INDICATOR:=”GDPPERCAPITA”, identifier COUNTRY:=”CANADA”]; … … …
--DF2bis_POPGROWTH_USA  := ‘DF2(1.0.0)/POPGROWTH.USA’
--[calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”USA”];
--DF2bis_POPGROWTH_CANADA’ := ‘DF2(1.0.0)/POPGROWTH.CANADA’ [calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”CANADA”]; … … …
--DF2(1.0) <- UNION   (DF2bis_GDPPERCAPITA_USA’,
--DF2bis_GDPPERCAPITA_CANADA’,
--… ,
--DF2bis_POPGROWTH_USA’,
--DF2bis_POPGROWTH_CANADA’
--…);
++> DF2bis_GDPPERCAPITA_USA := ‘DF2(1.0.0)/GDPPERCAPITA.USA’ [calc identifier INDICATOR := ”GDPPERCAPITA”, identifier COUNTRY := ”USA”];
++> DF2bis_GDPPERCAPITA_CANADA := ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ [calc identifier INDICATOR:=”GDPPERCAPITA”, identifier COUNTRY:=”CANADA”]; … … …
++> DF2bis_POPGROWTH_USA  := ‘DF2(1.0.0)/POPGROWTH.USA’  [calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”USA”];
++> DF2bis_POPGROWTH_CANADA’ := ‘DF2(1.0.0)/POPGROWTH.CANADA’ [calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”CANADA”]; … … …
++> DF2(1.0) <- UNION   (DF2bis_GDPPERCAPITA_USA’,
++> DF2bis_GDPPERCAPITA_CANADA’,
++> … ,
++> DF2bis_POPGROWTH_USA’,
++> DF2bis_POPGROWTH_CANADA’
++> …);
  In other words, starting from the datasets explicitly calculated through VTL (in the example ‘DF2(1.0)/GDPPERCAPITA.USA’ and so on), the first step consists in calculating other (non-persistent) VTL datasets (in the example DF2bis_GDPPERCAPITA_USA and so on) by adding the identifiers INDICATOR and COUNTRY with the desired values (//INDICATORvalue// and //COUNTRYvalue)//. Finally, all these non-persistent Data Sets are united and give the final result DF2(1.0){{footnote}}The result is persistent in this example but it can be also non persistent if needed.{{/footnote}}, which can be mapped one-to-one to the homonymous SDMX Dataflow having the dimension components TIME_PERIOD, INDICATOR and COUNTRY.
@@ -499,25 +499,26 @@
  With reference to the VTL “model for Variables and Value domains”, the following additional mappings have to be considered:
--|VTL|SDMX
--|**Data Set Component**|Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a Component (either a DimensionComponent or a Measure or a DataAttribute) belonging to one specific Dataflow^^43^^
--|**Represented Variable**|**Concept** with a definite Representation
--|**Value Domain**|(((
++(% style="width:706.294px" %)
++|(% style="width:257px" %)VTL|(% style="width:446px" %)SDMX
++|(% style="width:257px" %)**Data Set Component**|(% style="width:446px" %)Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a Component (either a DimensionComponent or a Measure or a DataAttribute) belonging to one specific Dataflow^^43^^
++|(% style="width:257px" %)**Represented Variable**|(% style="width:446px" %)**Concept** with a definite Representation
++|(% style="width:257px" %)**Value Domain**|(% style="width:446px" %)(((
  **Representation** (see the Structure
  Pattern in the Base Package)
  )))
--|**Enumerated Value Domain / Code List**|**Codelist**
--|**Code**|**Code** (for enumerated DimensionComponent, Measure, DataAttribute)
--|**Described Value Domain**|(((
++|(% style="width:257px" %)**Enumerated Value Domain / Code List**|(% style="width:446px" %)**Codelist**
++|(% style="width:257px" %)**Code**|(% style="width:446px" %)**Code** (for enumerated DimensionComponent, Measure, DataAttribute)
++|(% style="width:257px" %)**Described Value Domain**|(% style="width:446px" %)(((
  non-enumerated** Representation**
  (having Facets / ExtendedFacets, see the Structure Pattern in the Base Package)
  )))
--|**Value**|Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a **Code** of a Codelist (for enumerated Representations) or
--| |to a valid **value **(for non-enumerated** **Representations)
--|**Value Domain Subset / Set**|This abstraction does not exist in SDMX
--|**Enumerated Value Domain Subset / Enumerated Set**|This abstraction does not exist in SDMX
--|**Described Value Domain Subset / Described Set**|This abstraction does not exist in SDMX
--|**Set list**|This abstraction does not exist in SDMX
++|(% style="width:257px" %)**Value**|(% style="width:446px" %)Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a **Code** of a Codelist (for enumerated Representations) or
++|(% style="width:257px" %) |(% style="width:446px" %)to a valid **value **(for non-enumerated** **Representations)
++|(% style="width:257px" %)**Value Domain Subset / Set**|(% style="width:446px" %)This abstraction does not exist in SDMX
++|(% style="width:257px" %)**Enumerated Value Domain Subset / Enumerated Set**|(% style="width:446px" %)This abstraction does not exist in SDMX
++|(% style="width:257px" %)**Described Value Domain Subset / Described Set**|(% style="width:446px" %)This abstraction does not exist in SDMX
++|(% style="width:257px" %)**Set list**|(% style="width:446px" %)This abstraction does not exist in SDMX
  The main difference between VTL and SDMX relies on the fact that the VTL artefacts for defining subsets of Value Domains do not exist in SDMX, therefore the VTL features for referring to predefined subsets are not available in SDMX. These artefacts are the Value Domain Subset (or Set), either enumerated or described, the Set List (list of values belonging to enumerated subsets) and the Data Set Component (aimed at defining the set of values that the Component of a Data Set can take, possibly a subset of the codes of Value Domain).
@@ -525,8 +525,10 @@
  Therefore, it is important to be aware that some VTL operations (for example the binary operations at data set level) are consistent only if the components having the same names in the operated VTL Data Sets have also the same representation (i.e. the same Value Domain as for VTL). For example, it is possible to obtain correct results from the VTL expression
--DS_c := DS_a + DS_b (where DS_a, DS_b, DS_c are VTL Data Sets) if the matching components in DS_a and DS_b (e.g. ref_date, geo_area, sector …) refer to the same general representation. In simpler words, DS_a and DS_b must use the same values/codes (for ref_date, geo_area, sector … ), otherwise the relevant values would not match and the result of the operation would be wrong.
++> DS_c := DS_a + DS_b (where DS_a, DS_b, DS_c are VTL Data Sets)
++if the matching components in DS_a and DS_b (e.g. ref_date, geo_area, sector …) refer to the same general representation. In simpler words, DS_a and DS_b must use the same values/codes (for ref_date, geo_area, sector … ), otherwise the relevant values would not match and the result of the operation would be wrong.
++
  As mentioned, the property above is not enforced by construction in SDMX, and different representations of the same Concept can be not compatible one another (for example, it may happen that geo_area is represented by ISO-alpha-3 codes in DS_a and by ISO alpha-2 codes in DS_b). Therefore, it will be up to the definer of VTL
  Transformations to ensure that the VTL expressions are consistent with the actual representations of the correspondent SDMX Concepts.
@@ -541,8 +541,9 @@
  The VTL data types are sub-divided in scalar types (like integers, strings, etc.), which are the types of the scalar values, and compound types (like Data Sets, Components, Rulesets, etc.), which are the types of the compound structures. See below the diagram of the VTL data types, taken from the VTL User Manual:
--[[image:1750067055028-964.png]]
++[[image:1750070288958-132.png]]
++
  **Figure 22 – VTL Data Types**
  The VTL scalar types are in turn subdivided in basic scalar types, which are elementary (not defined in term of other data types) and Value Domain and Set scalar types, which are defined in terms of the basic scalar types.

1750070288958-132.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+45.9 KB

Content