Changes for page 12 Validation and Transformation Language (VTL)

Last modified by Helena on 2025/09/10 11:19

From 5.19 to 5.18 From 5.27 to 5.26

From version 5.26

edited by Helena
on 2025/05/16 09:02

Change comment: There is no comment for this version

To version 5.19

edited by Helena
on 2025/05/16 08:57

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -533,7 +533,7 @@
  DF2bis_GDPPERCAPITA_USA and so on) by adding the identifiers INDICATOR and COUNTRY with the desired values (//INDICATORvalue// and //COUNTRYvalue)//. Finally, all these non-persistent Data Sets are united and give the final result DF2(1.0){{footnote}}The result is persistent in this example but it can be also non persistent if needed.{{/footnote}}, which can be mapped one-to-one to the homonymous SDMX Dataflow having the dimension components TIME_PERIOD, INDICATOR and COUNTRY.
--Therefore, mapping different VTL datasets having the same data structure to different parts of a SDMX Dataflow, i.e. in the direction from VTL to SDMX, through the ordered concatenation notation is equivalent to a proper use of the operators “calc” and “union” on such datasets.{{footnote}}In case the ordered concatenation notation from VTL to SDMX is used, the set of Transformations described above is implicitly performed; therefore, in order to test the overall compliance of the VTL program to the VTL consistency rules, these implicit Transformations have to be considered as part of the VTL program even if they are not explicitly coded.{{/footnote}}
++Therefore, mapping different VTL datasets having the same data structure to different parts of a SDMX Dataflow, i.e. in the direction from VTL to SDMX, through the ordered concatenation notation is equivalent to a proper use of the operators “calc” and “union” on such datasets.{{footnote}}In case the ordered concatenation notation from VTL to SDMX is used, the set of Transformations described above is implicitly performed; therefore, in order to test the overall compliance of the VTL program to the VTL consistency rules, these implicit Transformations have to be considered as part of the VTL program even if they are not explicitly coded.{{/footnote}}{{footnote}}Through SDMX Constraints, it is possible to specify the values that a Component of a Dataflow can assume.{{/footnote}}
  It is worth noting that in the direction from VTL to SDMX it is mandatory to specify the value for every Dimension on which the mapping is based (in other word, in the name of the calculated VTL dataset is __not__ possible to omit the value of some of the Dimensions).
@@ -543,7 +543,7 @@
  (% style="width:1170.29px" %)
  |**VTL**|(% style="width:754px" %)**SDMX**
--|**Data Set Component**|(% style="width:754px" %)Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a Component (either a DimensionComponent or a Measure or a DataAttribute) belonging to one specific Dataflow{{footnote}}Through SDMX Constraints, it is possible to specify the values that a Component of a Dataflow can assume.{{/footnote}}
++|**Data Set Component**|(% style="width:754px" %)Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a Component (either a DimensionComponent or a Measure or a DataAttribute) belonging to one specific Dataflow
  |**Represented Variable**|(% style="width:754px" %)(((
  **Concept** with a definite
@@ -576,16 +576,16 @@
  The main difference between VTL and SDMX relies on the fact that the VTL artefacts for defining subsets of Value Domains do not exist in SDMX, therefore the VTL features for referring to predefined subsets are not available in SDMX. These artefacts are the Value Domain Subset (or Set), either enumerated or described, the Set List (list of values belonging to enumerated subsets) and the Data Set Component (aimed at defining the set of values that the Component of a Data Set can take, possibly a subset of the codes of Value Domain).
--Another difference consists in the fact that all Value Domains are considered as identifiable objects in VTL either if enumerated or not, while in SDMX the Codelist (corresponding to a VTL enumerated Value Domain) is identifiable, while the SDMX non-enumerated Representation (corresponding to a VTL non-enumerated Value Domain) is not identifiable. As a consequence, the definition of the VTL Rulesets, which in VTL can refer either to enumerated or non-enumerated value domains, in SDMX can refer only to enumerated Value Domains (i.e. to SDMX Codelists). As for the mapping between VTL variables and SDMX Concepts, it should be noted that these artefacts do not coincide perfectly. In fact, the VTL variables are represented variables, defined always on the same Value Domain (“Representation” in SDMX) independently of the data set / data structure in which they appear{{footnote}}By using represented variables, VTL can assume that data structures having the same variables as identifiers can be composed one another because the correspondent values can match.{{/footnote}}, while the SDMX Concepts can have different Representations in different DataStructures.{{footnote}}A Concept becomes a Component in a DataStructureDefinition, and Components can have different LocalRepresentations in different DataStructureDefinitions, also overriding the (possible) base representation of the Concept.{{/footnote}} This means that one SDMX Concept can correspond to many VTL Variables, one for each representation the Concept has.
++Another difference consists in the fact that all Value Domains are considered as identifiable objects in VTL either if enumerated or not, while in SDMX the Codelist (corresponding to a VTL enumerated Value Domain) is identifiable, while the SDMX non-enumerated Representation (corresponding to a VTL non-enumerated Value Domain) is not identifiable. As a consequence, the definition of the VTL Rulesets, which in VTL can refer either to enumerated or non-enumerated value domains, in SDMX can refer only to enumerated Value Domains (i.e. to SDMX Codelists). As for the mapping between VTL variables and SDMX Concepts, it should be noted that these artefacts do not coincide perfectly. In fact, the VTL variables are represented variables, defined always on the same Value Domain (“Representation” in SDMX) independently of the data set / data structure in which they appear^^[[(% class="wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink" %)^^40^^>>path:#sdfootnote40sym||name="sdfootnote40anc"]](%%)^^, while the SDMX Concepts can have different Representations in different DataStructures.^^[[(% class="wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink" %)^^41^^>>path:#sdfootnote41sym||name="sdfootnote41anc"]](%%)^^ This means that one SDMX Concept can correspond to many VTL Variables, one for each representation the Concept has.
  Therefore, it is important to be aware that some VTL operations (for example the binary operations at data set level) are consistent only if the components having the same names in the operated VTL Data Sets have also the same representation (i.e. the same Value Domain as for VTL). For example, it is possible to obtain correct results from the VTL expression
--DS_c := DS_a + DS_b (where DS_a, DS_b, DS_c are VTL Data Sets)
++DS_c := DS_a + DS_b (where DS_a, DS_b, DS_c are VTL Data Sets) if the matching components in DS_a and DS_b (e.g. ref_date, geo_area, sector …) refer to the same general representation. In simpler words, DS_a and DS_b must use the same values/codes (for ref_date, geo_area, sector … ), otherwise the relevant values would not match and the result of the operation would be wrong.
--if the matching components in DS_a and DS_b (e.g. ref_date, geo_area, sector …) refer to the same general representation. In simpler words, DS_a and DS_b must use the same values/codes (for ref_date, geo_area, sector … ), otherwise the relevant values would not match and the result of the operation would be wrong.
--
  As mentioned, the property above is not enforced by construction in SDMX, and different representations of the same Concept can be not compatible one another (for example, it may happen that geo_area is represented by ISO-alpha-3 codes in DS_a and by ISO alpha-2 codes in DS_b). Therefore, it will be up to the definer of VTL
++[[image:SDMX 3-0-0 SECTION 6 FINAL-1.0_en_59eee18f.gif||alt="Shape5" height="1" width="192"]]
++
  Transformations to ensure that the VTL expressions are consistent with the actual representations of the correspondent SDMX Concepts.
  It remains up to the SDMX-VTL definer also the assurance of the consistency between a VTL Ruleset defined on Variables and the SDMX Components on which the Ruleset is applied. In fact, a VTL Ruleset is expressed by means of the values of the Variables (i.e. SDMX Concepts), i.e. assuming definite representations for them (e.g. ISOalpha-3 for country). If the Ruleset is applied to SDMX Components that have the same name of the Concept they refer to but different representations (e.g. ISO-alpha-2 for country), the Ruleset cannot work properly.
@@ -600,8 +600,7 @@
  [[image:SDMX 3-0-0 SECTION 6 FINAL-1.0_en_e3df33ae.png||height="543" width="483"]]
--(% class="wikigeneratedid" id="HFigure222013VTLDataTypes" %)
--**Figure 22 – VTL Data Types**
++==== Figure 22 – VTL Data Types ====
  The VTL scalar types are in turn subdivided in basic scalar types, which are elementary (not defined in term of other data types) and Value Domain and Set scalar types, which are defined in terms of the basic scalar types.
@@ -608,12 +608,131 @@
  The VTL basic scalar types are listed below and follow a hierarchical structure in terms of supersets/subsets (e.g. "scalar" is the superset of all the basic scalar types):
--**Figure 23 – VTL Basic Scalar Types**
  (((
--
++//n//
++
++//a//
++
++//e//
++
++//l//
++
++//o//
++
++//o//
++
++//B//
++
++//n//
++
++//o//
++
++//i//
++
++//t//
++
++//a//
++
++//r//
++
++//u//
++
++//D//
++
++//d//
++
++//o//
++
++//i//
++
++//r//
++
++//e//
++
++//p//
++
++//_//
++
++//e//
++
++//m//
++
++//i//
++
++//T//
++
++//e//
++
++//t//
++
++//a//
++
++//D//
++
++//e//
++
++//m//
++
++//i//
++
++//T//
++
++//r//
++
++//e//
++
++//g//
++
++//e//
++
++//t//
++
++//n//
++
++//I//
++
++//r//
++
++//e//
++
++//b//
++
++//m//
++
++//u//
++
++//N//
++
++//g//
++
++//n//
++
++//i//
++
++//r//
++
++//t//
++
++//S//
++
++//r//
++
++//a//
++
++//l//
++
++//a//
++
++//c//
++
++//S//
++
++[[image:SDMX 3-0-0 SECTION 6 FINAL-1.0_en_82d45833.gif||alt="Shape6" height="231" width="184"]]
  )))
++==== Figure 23 – VTL Basic Scalar Types ====
++
  === 12.4.2 VTL basic scalar types and SDMX data types ===
  The VTL assumes that a basic scalar type has a unique internal representation and can have more external representations.
@@ -636,55 +636,70 @@
  The following table describes the default mapping for converting from the SDMX data types to the VTL basic scalar types.
--|(% style="width:501px" %)SDMX data type (BasicComponentDataType)|(% style="width:1437px" %)Default VTL basic scalar type
--|(% style="width:501px" %)(((
++|SDMX data type (BasicComponentDataType)|Default VTL basic scalar type
++|(((
  String
++
  (string allowing any character)
--)))|(% style="width:1437px" %)string
--|(% style="width:501px" %)(((
++)))|string
++|(((
  Alpha
++
  (string which only allows A-z)
--)))|(% style="width:1437px" %)string
--|(% style="width:501px" %)(((
++)))|string
++|(((
  AlphaNumeric
++
  (string which only allows A-z and 0-9)
--)))|(% style="width:1437px" %)string
--|(% style="width:501px" %)(((
++)))|string
++|(((
  Numeric
++
  (string which only allows 0-9, but is not numeric so that is can having leading zeros)
--)))|(% style="width:1437px" %)string
--|(% style="width:501px" %)(((
++)))|string
++|(((
  BigInteger
++
  (corresponds to XML Schema xs:integer datatype; infinite set of integer values)
--)))|(% style="width:1437px" %)integer
--|(% style="width:501px" %)(((
++)))|integer
++|(((
  Integer
--(corresponds to XML Schema xs:int datatype; between -2147483648 and +2147483647 (inclusive))
--)))|(% style="width:1437px" %)integer
--|(% style="width:501px" %)(((
++
++(corresponds to XML Schema xs:int datatype; between -2147483648 and +2147483647
++
++(inclusive))
++)))|integer
++|(((
  Long
++
  (corresponds to XML Schema xs:long datatype; between -9223372036854775808 and
  +9223372036854775807 (inclusive))
--)))|(% style="width:1437px" %)integer
--|(% style="width:501px" %)(((
++)))|integer
++|(((
  Short
++
  (corresponds to XML Schema xs:short datatype; between -32768 and -32767 (inclusive))
--)))|(% style="width:1437px" %)integer
--|(% style="width:501px" %)Decimal (corresponds to XML Schema xs:decimal datatype; subset of real numbers that can be represented as decimals)|(% style="width:1437px" %)number
--|(% style="width:501px" %)(((
++)))|integer
++|Decimal (corresponds to XML Schema xs:decimal datatype; subset of real numbers that can be represented as decimals)|number
++|(((
  Float
++
  (corresponds to XML Schema xs:float datatype; patterned after the IEEE single-precision 32-bit floating point type)
--)))|(% style="width:1437px" %)number
--|(% style="width:501px" %)(((
++)))|number
++|(((
  Double
++
  (corresponds to XML Schema xs:double datatype; patterned after the IEEE double-precision 64-bit floating point type)
--)))|(% style="width:1437px" %)number
--|(% style="width:501px" %)(((
++)))|number
++|(((
  Boolean
--(corresponds to the XML Schema xs:boolean datatype; support the mathematical concept of binary-valued logic: {true, false})
--)))|(% style="width:1437px" %)boolean
++(corresponds to the XML Schema xs:boolean datatype; support the mathematical concept of
++
++binary-valued logic: {true, false})
++)))|boolean
++
  | |(% colspan="2" %)(((
  URI
@@ -912,7 +912,7 @@
  |N|fixed number of digits used in the preceding textual representation of the month or the day
  | |
--The default conversion, either standard or customized, can be used to deduce automatically the representation of the components of the result of a VTL Transformation. In alternative, the representation of the resulting SDMX Dataflow can be given explicitly by providing its DataStructureDefinition. In other words, the representation specified in the DSD, if available, overrides any default conversion^^[[(% class="wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink" %)^^42^^>>path:#sdfootnote42sym||name="sdfootnote42anc"]](%%)^^.
++The default conversion, either standard or customized, can be used to deduce automatically the representation of the components of the result of a VTL Transformation. In alternative, the representation of the resulting SDMX Dataflow can be given explicitly by providing its DataStructureDefinition. In other words, the representation specified in the DSD, if available, overrides any default conversion^^[[(% class="wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink wikiinternallink" %)^^42^^>>path:#sdfootnote42sym||name="sdfootnote42anc"]](%%)^^.
  === 12.4.5 Null Values ===