Changes for page 12 Validation and Transformation Language (VTL)
Last modified by Artur on 2025/09/10 11:19
Summary
-
Page properties (1 modified, 0 added, 0 removed)
Details
- Page properties
-
- Content
-
... ... @@ -19,7 +19,6 @@ 19 19 This section does not explain the VTL language or any of the content published in the VTL guides. Rather, this is a description of how the VTL can be used in the SDMX context and applied to SDMX artefacts. 20 20 21 21 == 12.2 References to SDMX artefacts from VTL statements == 22 - 23 23 === 12.2.1 Introduction === 24 24 25 25 The VTL can manipulate SDMX artefacts (or objects) by referencing them through predefined conventional names (aliases). ... ... @@ -49,8 +49,10 @@ 49 49 50 50 The generic structure of the URN is the following: 51 51 52 -SDMXprefix.SDMX-IM-package-name.class-name=agency-id:maintainedobject-id (maintainedobject-version).*container-object-id.object-id51 +SDMXprefix.SDMX-IM-package-name.class-name=agency-id:maintainedobject-id 53 53 53 +(maintainedobject-version).*container-object-id.object-id 54 + 54 54 The **SDMXprefix** is "urn:sdmx:org", always the same for all SDMX artefacts. 55 55 56 56 The SDMX-IM-package-name** **is the concatenation of the string** **"sdmx.infomodel." with the package-name, which the artefact belongs to. For example, for referencing a Dataflow the SDMX-IM-package-name is "sdmx.infomodel.datastructure", because the class Dataflow belongs to the package "datastructure". ... ... @@ -71,19 +71,24 @@ 71 71 72 72 The maintainedobject-version is the version, according to the SDMX versioning rules, of the maintained object which the artefact belongs to (for example, possible versions might be 1.0, 2.3, 1.0.0, 2.1.0 or 3.1.2). 73 73 74 -The container-object-id does not apply to the classes that can be referenced in VTL Transformations, therefore is not present in their URN .75 +The container-object-id does not apply to the classes that can be referenced in VTL Transformations, therefore is not present in their URN 75 75 76 76 The object-id is the name of the non-maintainable artefact (when the artefact is maintainable its name is already specified as the maintainedobject-id, see above), in particular it has to be specified: 77 77 78 -* if the artefact is a Dimension, TimeDimension, Measure or DataAttribute (the object-id is the name of one of the artefacts above, which are data structure components) 79 +* if the artefact is a Dimension, TimeDimension, Measure or 80 + 81 +DataAttribute (the object-id is the name of one of the artefacts above, which are data structure components) 82 + 79 79 * if the artefact is a Concept (the object-id is the name of the Concept) 80 80 81 81 For example, by using the URN, the VTL Transformation that sums two SDMX Dataflows DF1 and DF2 and assigns the result to a third persistent Dataflow DFR, assuming that DF1, DF2 and DFR are the maintainedobject-id of the three Dataflows, that their version is 1.0.0 and their Agency is AG, would be written as{{footnote}}Since these references to SDMX objects include non-permitted characters as per the VTL ID notation, they need to be included between single quotes, according to the VTL rules for irregular names.{{/footnote}}: 82 82 83 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DFR(1.0.0)' <- 84 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF1(1.0.0)' + 85 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF2(1.0.0)' 87 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DFR(1.0.0)' <- 86 86 89 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF1(1.0.0)' + 90 + 91 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF2(1.0.0)' 92 + 87 87 === 12.2.3 Abbreviation of the URN === 88 88 89 89 The complete formulation of the URN described above is exhaustive but verbose, even for very simple statements. In order to reduce the verbosity through a simplified identifier and make the work of transformation definers easier, proper abbreviations of the URN are possible. Using this approach, the referenced artefacts remain intelligible in the VTL code by a human reader. ... ... @@ -92,13 +92,10 @@ 92 92 93 93 * The SDMXprefix can be omitted for all the SDMX objects, because it is a prefixed string (urn:sdmx:org), always the same for SDMX objects. 94 94 * The SDMX-IM-package-name** **can be omitted as well because it can be deduced from the class-name that follows it (the table of the SDMX-IM packages and classes that allows this deduction is in the SDMX 2.1 Standards - Section 5 - Registry Specifications, paragraph 6.2.3). In particular, considering the object classes of the artefacts that VTL can reference, the package is: 95 -** "datastructure" for the classes Dataflow, Dimension, TimeDimension, Measure, DataAttribute, 96 -** "conceptscheme" for the class Concept, 97 -** "codelist" for the class Codelist. 101 +** "datastructure" for the classes Dataflow, Dimension, TimeDimension, Measure, DataAttribute, o "conceptscheme" for the class Concept, o "codelist" for the class Codelist. 98 98 * The class-name can be omitted as it can be deduced from the VTL invocation. In particular, starting from the VTL class of the invoked artefact (e.g. dataset, component, identifier, measure, attribute, variable, valuedomain), which is known given the syntax of the invoking VTL operator{{footnote}}For the syntax of the VTL operators see the VTL Reference Manual{{/footnote}}, the SDMX class can be deduced from the mapping rules between VTL and SDMX (see the section "Mapping between VTL and SDMX" hereinafter){{footnote}}In case the invoked artefact is a VTL component, which can be invoked only within the invocation of a VTL data set (SDMX Dataflow), the specific SDMX class-name (e.g. Dimension, TimeDimension, Measure or DataAttribute) can be deduced from the data structure of the SDMX Dataflow, which the component belongs to.{{/footnote}}. 99 99 * If the agency-id is not specified, it is assumed by default equal to the agency-id of the TransformationScheme, UserDefinedOperatorScheme or RulesetScheme from which the artefact is invoked. For example, the agencyid can be omitted if it is the same as the invoking TransformationScheme and cannot be omitted if the artefact comes from another agency{{footnote}}If the Agency is composite (for example AgencyA.Dept1.Unit2), the agency is considered different even if only part of the composite name is different (for example AgencyA.Dept1.Unit3 is a different Agency than the previous one). Moreover the agency-id cannot be omitted in part (i.e., if a TransformationScheme owned by AgencyA.Dept1.Unit2 references an artefact coming from AgencyA.Dept1.Unit3, the specification of the agency-id becomes mandatory and must be complete, without omitting the possibly equal parts like AgencyA.Dept1){{/footnote}}. Take also into account that, according to the VTL consistency rules, the agency of the result of a Transformation must be the same as its TransformationScheme, therefore the agency-id can be omitted for all the results (left part of Transformation statements). 100 -* As for the maintainedobject-id, this is essential in some cases while in other cases it can be omitted: 101 -** if the referenced artefact is a Dataflow, which is a maintainable class, the maintainedobject-id is the dataflow-id and obviously cannot be omitted; 104 +* As for the maintainedobject-id, this is essential in some cases while in other cases it can be omitted: o if the referenced artefact is a Dataflow, which is a maintainable class, the maintainedobject-id is the dataflow-id and obviously cannot be omitted; 102 102 ** if the referenced artefact is a Dimension, TimeDimension, Measure, DataAttribute, which are not maintainable and belong to the DataStructure maintainable class, the maintainedobject-id is the dataStructure-id and can be omitted, given that these components are always invoked within the invocation of a Dataflow, whose dataStructure-id can be deduced from the SDMX structural definitions; 103 103 ** if the referenced artefact is a Concept, which is not maintainable and belong to the ConceptScheme maintainable class, the maintained object is the conceptScheme-id and cannot be omitted; 104 104 ** if the referenced artefact is a Codelist, which is a maintainable class, the maintainedobject-id is the codelist-id and obviously cannot be omitted. ... ... @@ -110,47 +110,51 @@ 110 110 111 111 For example, the full formulation that uses the complete URN shown at the end of the previous paragraph: 112 112 113 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DFR(1.0.0)' := 114 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF1(1.0.0)' + 115 -> 'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF2(1.0.0)' 116 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DFR(1.0.0)' := 116 116 118 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF1(1.0.0)' + 119 + 120 +'urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AG:DF2(1.0.0)' 121 + 117 117 by omitting all the non-essential parts would become simply: 118 118 119 - >DFR:=DF1 + DF2124 +DFR := DF1 + DF2 120 120 121 121 The references to the Codelists can be simplified similarly. For example, given the non-abbreviated reference to the Codelist AG:CL_FREQ(1.0.0), which is{{footnote}}Single quotes are needed because this reference is not a VTL regular name. 19 Single quotes are not needed in this case because CL_FREQ is a VTL regular name.{{/footnote}}: 122 122 123 - >'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=AG:CL_FREQ(1.0.0)'128 +'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=AG:CL_FREQ(1.0.0)' 124 124 125 125 if the Codelist is referenced from a RulesetScheme belonging to the agency AG, omitting all the optional parts, the abbreviated reference would become simply^^19^^: 126 126 127 - >CL_FREQ132 +CL_FREQ 128 128 129 129 As for the references to the components, it can be enough to specify the componentId, given that the dataStructure-Id can be omitted. An example of non-abbreviated reference, if the data structure is DST1 and the component is SECTOR, is the following: 130 130 131 - >'urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=AG:DST1(1.0.0).SECTOR'136 +'urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=AG:DST1(1.0.0).S 132 132 138 +ECTOR' 139 + 133 133 The corresponding fully abbreviated reference, if made from a TransformationScheme belonging to AG, would become simply: 134 134 135 - >SECTOR142 +SECTOR 136 136 137 137 For example, the Transformation for renaming the component SECTOR of the Dataflow DF1 into SEC can be written as{{footnote}}The result DFR(1.0.0) is be equal to DF1(1.0.0) save that the component SECTOR is called SEC{{/footnote}}: 138 138 139 - >'DFR(1.0.0)' := 'DF1(1.0.0)' [rename SECTOR to SEC]146 +'DFR(1.0.0)' := 'DF1(1.0.0)' [rename SECTOR to SEC] 140 140 141 141 In the references to the Concepts, which can exist for example in the definition of the VTL Rulesets, at least the conceptScheme-id and the concept-id must be specified. 142 142 143 143 An example of non-abbreviated reference, if the conceptScheme-id is CS1 and the concept-id is SECTOR, is the following: 144 144 145 - >'urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=AG:CS1(1.0.0).SECTOR'152 +'urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=AG:CS1(1.0.0).SECTOR' 146 146 147 147 The corresponding fully abbreviated reference, if made from a RulesetScheme belonging to AG, would become simply: 148 148 149 - >CS1(1.0.0).SECTOR156 +CS1(1.0.0).SECTOR 150 150 151 151 The Codes and in general all the Values can be written without any other specification, for example, the transformation to check if the values of the measures of the Dataflow DF1 are between 0 and 25000 can be written like follows: 152 152 153 - >'DFR(1.0.0)' := between ( 'DF1(1.0.0)', 0, 25000 )160 +'DFR(1.0.0)' := between ( 'DF1(1.0.0)', 0, 25000 ) 154 154 155 155 The artefact (Component, Concept, Codelist …) which the Values are referred to can be deduced from the context in which the reference is made, taking also into account the VTL syntax. In the Transformation above, for example, the values 0 and 2500 are compared to the values of the measures of DF1(1.0.0). 156 156 ... ... @@ -173,7 +173,6 @@ 173 173 In the body of the Rulesets, the Codes and in general all the Values can be written without any other specification, because the artefact, which the Values are referred (Codelist, Concept) to can be deduced from the Ruleset signature. 174 174 175 175 == 12.3 Mapping between SDMX and VTL artefacts == 176 - 177 177 === 12.3.1. When the mapping occurs === 178 178 179 179 The mapping methods between the VTL and SDMX object classes allow transforming a SDMX definition in a VTL one and vice-versa for the artefacts to be manipulated. It should be remembered that VTL programs (i.e. Transformation Schemes) are represented in SDMX through the TransformationScheme maintainable class which is composed of Transformations (nameable artefacts). Each Transformation assigns the outcome of the evaluation of a VTL expression to a result: the input operands of the expression and the result can be SDMX artefacts. Every time a SDMX object is referenced in a VTL Transformation as an input operand, there is the need to generate a VTL definition of the object, so that the VTL operations can take place. This can be made starting from the SDMX definition and applying a SDMX-VTL mapping method in the direction from SDMX to VTL. The possible mapping methods from SDMX to VTL are described in the following paragraphs and are conceived to allow the automatic deduction of the VTL definition of the object from the knowledge of the SDMX definition. ... ... @@ -198,7 +198,7 @@ 198 198 199 199 The possible mapping options are described in more detail in the following sections. 200 200 201 -=== 12.3. 3Mapping from SDMX to VTL data structures ===207 +=== 12.3.2 Mapping from SDMX to VTL data structures === 202 202 203 203 ==== 12.3.3.1 Basic Mapping ==== 204 204 ... ... @@ -206,12 +206,11 @@ 206 206 207 207 When transforming **from SDMX to VTL**, this method consists in leaving the components unchanged and maintaining their names and roles, according to the following table: 208 208 209 -(% style="width:468.294px" %) 210 -|(% style="width:196px" %)**SDMX**|(% style="width:269px" %)**VTL** 211 -|(% style="width:196px" %)Dimension|(% style="width:269px" %)(Simple) Identifier 212 -|(% style="width:196px" %)TimeDimension|(% style="width:269px" %)(Time) Identifier 213 -|(% style="width:196px" %)Measure|(% style="width:269px" %)Measure 214 -|(% style="width:196px" %)DataAttribute|(% style="width:269px" %)Attribute 215 +|**SDMX**|**VTL** 216 +|Dimension|(Simple) Identifier 217 +|TimeDimension|(Time) Identifier 218 +|Measure|Measure 219 +|DataAttribute|Attribute 215 215 216 216 The SDMX DataAttributes, in VTL they are all considered "at data point / observation level" (i.e. dependent on all the VTL Identifiers), because VTL does not have the SDMX AttributeRelationships, which defines the construct to which the DataAttribute is related (e.g. observation, dimension or set or group of dimensions, whole data set). 217 217 ... ... @@ -221,8 +221,10 @@ 221 221 222 222 An alternative mapping method from SDMX to VTL is the **Pivot **mapping, which makes sense and is different from the Basic method only for the SDMX data structures that contain a Dimension that plays the role of measure dimension (like in SDMX 2.1) and just one Measure. Through this method, these structures can be mapped to multimeasure VTL data structures. Besides that, a user may choose to use any Dimension acting as a list of Measures (e.g., a Dimension with indicators), either by considering the “Measure” role of a Dimension, or at will using any coded Dimension. Of course, in SDMX 3.0, this can only work when only one Measure is defined in the DSD. 223 223 224 -In SDMX 2.1 the MeasureDimension was a subclass of DimensionComponent like Dimension and TimeDimension. In the current SDMX version, this subclass does not exist anymore, however a Dimension can have the role of measure dimension (i.e. a Dimension that contributes to the identification of the measures). In SDMX 2.1 a DataStructure could have zero or one MeasureDimensions, in the current version of the standard, from zero to many Dimension may have the role of measure dimension. Hereinafter a Dimension that plays the role of measure dimension is referenced for simplicity as “MeasureDimension“, i.e. maintaining the capital letters and the courier font even if the MeasureDimension is not anymore a class in the SDMX Information Model of the current SDMX version. For the sake of simplicity, the description below considers just one Dimension having the role of MeasureDimension (i.e., the more simple and common case). Nevertheless, it maintains its validity also if in the DataStructure there are more dimension with the role of MeasureDimensions: in this case what is said about the MeasureDimension must be applied to the combination of all the MeasureDimensions considered as a joint variable{{footnote}}E.g., if in the data structure there exist 3 Dimensions C,D,E having the role of MeasureDimension, they should be considered as a joint MeasureDimension Z=(C,D,E); therefore when the description says “each possible value Cj of the MeasureDimension …” it means “each possible combination of values (Cj, Dk, Ew) of the joint MeasureDimension Z=(C,D,E)”.{{/footnote}}.229 +In SDMX 2.1 the MeasureDimension was a subclass of DimensionComponent like Dimension and TimeDimension. In the current SDMX version, this subclass does not exist anymore, however a Dimension can have the role of measure dimension (i.e. a Dimension that contributes to the identification of the measures). In SDMX 2.1 a DataStructure could have zero or one MeasureDimensions, in the current version of the standard, from zero to many Dimension may have the role of measure dimension. Hereinafter a Dimension that plays the role of measure dimension is referenced for simplicity as “MeasureDimension“, i.e. maintaining the capital letters and the courier font even if the MeasureDimension is not anymore a class in the SDMX Information Model of the current SDMX version. For the sake of simplicity, the description below considers just one Dimension having the role of MeasureDimension (i.e., the more simple and common case). Nevertheless, it maintains its validity also if in the DataStructure there are more dimension with the role of MeasureDimensions: in this case what is said about the MeasureDimension must be applied to the combination of all the 225 225 231 +MeasureDimensions considered as a joint variable{{footnote}}E.g., if in the data structure there exist 3 Dimensions C,D,E having the role of MeasureDimension, they should be considered as a joint MeasureDimension Z=(C,D,E); therefore when the description says “each possible value Cj of the MeasureDimension …” it means “each possible combination of values (Cj, Dk, Ew) of the joint MeasureDimension Z=(C,D,E)”.{{/footnote}}. 232 + 226 226 Among other things, the Pivot method provides also backward compatibility with the SDMX 2.1 data structures that contained a MeasureDimension. 227 227 228 228 If applied to SDMX structures that do not contain any MeasureDimension, this method behaves like the Basic mapping (see the previous paragraph). ... ... @@ -456,10 +456,13 @@ 456 456 Some examples follow, for some specific values of INDICATOR and COUNTRY: 457 457 458 458 ‘DF2(1.0.0)/GDPPERCAPITA.USA’ <- expression11; ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ <- expression12; 466 + 459 459 … … … 460 460 461 461 ‘DF2(1.0.0)/POPGROWTH.USA’ <- expression21; 470 + 462 462 ‘DF2(1.0.0)/POPGROWTH.CANADA’ <- expression22; 472 + 463 463 … … … 464 464 465 465 As said, it is assumed that these VTL derived Data Sets have the TIME_PERIOD as the only identifier. In the mapping from VTL to SMDX, the Dimensions INDICATOR and COUNTRY are added to the VTL data structure on order to obtain the SDMX one, with the following values respectively: ... ... @@ -466,9 +466,13 @@ 466 466 467 467 VTL dataset INDICATOR value COUNTRY value 468 468 479 + 469 469 ‘DF2(1.0.0)/GDPPERCAPITA.USA’ GDPPERCAPITA USA 481 + 470 470 ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ GDPPERCAPITA CANADA … … … 483 + 471 471 ‘DF2(1.0.0)/POPGROWTH.USA’ POPGROWTH USA 485 + 472 472 ‘DF2(1.0.0)/POPGROWTH.CANADA’ POPGROWTH CANADA 473 473 474 474 … … … ... ... @@ -476,15 +476,25 @@ 476 476 It should be noted that the application of this many-to-one mapping from VTL to SDMX is equivalent to an appropriate sequence of VTL Transformations. These use the VTL operator “calc” to add the proper VTL identifiers (in the example, INDICATOR and COUNTRY) and to assign to them the proper values and the operator “union” in order to obtain the final VTL dataset (in the example DF2(1.0.0)), that can be mapped oneto-one to the homonymous SDMX Dataflow. Following the same example, these VTL Transformations would be: 477 477 478 478 DF2bis_GDPPERCAPITA_USA := ‘DF2(1.0.0)/GDPPERCAPITA.USA’ [calc identifier INDICATOR := ”GDPPERCAPITA”, identifier COUNTRY := ”USA”]; 493 + 479 479 DF2bis_GDPPERCAPITA_CANADA := ‘DF2(1.0.0)/GDPPERCAPITA.CANADA’ [calc identifier INDICATOR:=”GDPPERCAPITA”, identifier COUNTRY:=”CANADA”]; … … … 495 + 480 480 DF2bis_POPGROWTH_USA := ‘DF2(1.0.0)/POPGROWTH.USA’ 497 + 481 481 [calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”USA”]; 499 + 482 482 DF2bis_POPGROWTH_CANADA’ := ‘DF2(1.0.0)/POPGROWTH.CANADA’ [calc identifier INDICATOR := ”POPGROWTH”, identifier COUNTRY := ”CANADA”]; … … … 501 + 483 483 DF2(1.0) <- UNION (DF2bis_GDPPERCAPITA_USA’, 503 + 484 484 DF2bis_GDPPERCAPITA_CANADA’, 505 + 485 485 … , 507 + 486 486 DF2bis_POPGROWTH_USA’, 509 + 487 487 DF2bis_POPGROWTH_CANADA’ 511 + 488 488 …); 489 489 490 490 In other words, starting from the datasets explicitly calculated through VTL (in the example ‘DF2(1.0)/GDPPERCAPITA.USA’ and so on), the first step consists in calculating other (non-persistent) VTL datasets (in the example DF2bis_GDPPERCAPITA_USA and so on) by adding the identifiers INDICATOR and COUNTRY with the desired values (//INDICATORvalue// and //COUNTRYvalue)//. Finally, all these non-persistent Data Sets are united and give the final result DF2(1.0){{footnote}}The result is persistent in this example but it can be also non persistent if needed.{{/footnote}}, which can be mapped one-to-one to the homonymous SDMX Dataflow having the dimension components TIME_PERIOD, INDICATOR and COUNTRY. ... ... @@ -493,7 +493,9 @@ 493 493 494 494 It is worth noting that in the direction from VTL to SDMX it is mandatory to specify the value for every Dimension on which the mapping is based (in other word, in the name of the calculated VTL dataset is not possible to omit the value of some of the Dimensions). 495 495 496 -=== 12.3.7 Mapping variables and value domains between VTL and SDMX === 520 +1. 521 +11. 522 +111. Mapping variables and value domains between VTL and SDMX 497 497 498 498 With reference to the VTL “model for Variables and Value domains”, the following additional mappings have to be considered: 499 499 ... ... @@ -502,6 +502,7 @@ 502 502 |**Represented Variable**|**Concept** with a definite Representation 503 503 |**Value Domain**|((( 504 504 **Representation** (see the Structure 531 + 505 505 Pattern in the Base Package) 506 506 ))) 507 507 |**Enumerated Value Domain / Code List**|**Codelist** ... ... @@ -508,6 +508,7 @@ 508 508 |**Code**|**Code** (for enumerated DimensionComponent, Measure, DataAttribute) 509 509 |**Described Value Domain**|((( 510 510 non-enumerated** Representation** 538 + 511 511 (having Facets / ExtendedFacets, see the Structure Pattern in the Base Package) 512 512 ))) 513 513 |**Value**|Although this abstraction exists in SDMX, it does not have an explicit definition and correspond to a **Code** of a Codelist (for enumerated Representations) or ... ... @@ -531,10 +531,10 @@ 531 531 532 532 It remains up to the SDMX-VTL definer also the assurance of the consistency between a VTL Ruleset defined on Variables and the SDMX Components on which the Ruleset is applied. In fact, a VTL Ruleset is expressed by means of the values of the Variables (i.e. SDMX Concepts), i.e. assuming definite representations for them (e.g. ISOalpha-3 for country). If the Ruleset is applied to SDMX Components that have the same name of the Concept they refer to but different representations (e.g. ISO-alpha-2 for country), the Ruleset cannot work properly. 533 533 534 -== 12.4 Mapping between SDMX and VTL Data Types == 562 +1. 563 +11. Mapping between SDMX and VTL Data Types 564 +111. VTL Data types 535 535 536 -=== 12.4.1 VTL Data types === 537 - 538 538 According to the VTL User Guide the possible operations in VTL depend on the data types of the artefacts. For example, numbers can be multiplied but text strings cannot. In the VTL Transformations, the compliance between the operators and the data types of their operands is statically checked, i.e., violations result in compile-time errors. 539 539 540 540 The VTL data types are sub-divided in scalar types (like integers, strings, etc.), which are the types of the scalar values, and compound types (like Data Sets, Components, Rulesets, etc.), which are the types of the compound structures. See below the diagram of the VTL data types, taken from the VTL User Manual: ... ... @@ -541,15 +541,17 @@ 541 541 542 542 [[image:1750067055028-964.png]] 543 543 544 - **Figure 22 – VTL Data Types**572 +==== Figure 22 – VTL Data Types ==== 545 545 546 546 The VTL scalar types are in turn subdivided in basic scalar types, which are elementary (not defined in term of other data types) and Value Domain and Set scalar types, which are defined in terms of the basic scalar types. 547 547 548 548 The VTL basic scalar types are listed below and follow a hierarchical structure in terms of supersets/subsets (e.g. "scalar" is the superset of all the basic scalar types): 549 549 550 - **Figure 23 – VTL Basic Scalar Types**578 +==== Figure 23 – VTL Basic Scalar Types ==== 551 551 552 -=== 12.4.2 VTL basic scalar types and SDMX data types === 580 +1. 581 +11. 582 +111. VTL basic scalar types and SDMX data types 553 553 554 554 The VTL assumes that a basic scalar type has a unique internal representation and can have more external representations. 555 555 ... ... @@ -567,7 +567,9 @@ 567 567 568 568 The opposite conversion, i.e. from VTL to SDMX, happens when a VTL result, i.e. a VTL Data Set output of a Transformation, must become a SDMX artefact (or part of it). The values of the VTL result must be converted into the desired (SDMX) external representations (data types) of the SDMX artefact. 569 569 570 -=== 12.4.3 Mapping SDMX data types to VTL basic scalar types === 600 +1. 601 +11. 602 +111. Mapping SDMX data types to VTL basic scalar types 571 571 572 572 The following table describes the default mapping for converting from the SDMX data types to the VTL basic scalar types. 573 573 ... ... @@ -574,6 +574,7 @@ 574 574 |SDMX data type (BasicComponentDataType)|Default VTL basic scalar type 575 575 |((( 576 576 String 609 + 577 577 (string allowing any character) 578 578 )))|string 579 579 |((( ... ... @@ -583,6 +583,7 @@ 583 583 )))|string 584 584 |((( 585 585 AlphaNumeric 619 + 586 586 (string which only allows A-z and 0-9) 587 587 )))|string 588 588 |((( ... ... @@ -592,70 +592,89 @@ 592 592 )))|string 593 593 |((( 594 594 BigInteger 629 + 595 595 (corresponds to XML Schema xs:integer datatype; infinite set of integer values) 596 596 )))|integer 597 597 |((( 598 598 Integer 634 + 599 599 (corresponds to XML Schema xs:int datatype; between -2147483648 and +2147483647 636 + 600 600 (inclusive)) 601 601 )))|integer 602 602 |((( 603 603 Long 641 + 604 604 (corresponds to XML Schema xs:long datatype; between -9223372036854775808 and 643 + 605 605 +9223372036854775807 (inclusive)) 606 606 )))|integer 607 607 |((( 608 608 Short 648 + 609 609 (corresponds to XML Schema xs:short datatype; between -32768 and -32767 (inclusive)) 610 610 )))|integer 611 611 |Decimal (corresponds to XML Schema xs:decimal datatype; subset of real numbers that can be represented as decimals)|number 612 612 |((( 613 613 Float 654 + 614 614 (corresponds to XML Schema xs:float datatype; patterned after the IEEE single-precision 32-bit floating point type) 615 615 )))|number 616 616 |((( 617 617 Double 659 + 618 618 (corresponds to XML Schema xs:double datatype; patterned after the IEEE double-precision 64-bit floating point type) 619 619 )))|number 620 620 |((( 621 621 Boolean 664 + 622 622 (corresponds to the XML Schema xs:boolean datatype; support the mathematical concept of 666 + 623 623 binary-valued logic: {true, false}) 624 624 )))|boolean 625 625 |((( 626 626 URI 671 + 627 627 (corresponds to the XML Schema xs:anyURI; absolute or relative Uniform Resource Identifier Reference) 628 628 )))|string 629 629 |((( 630 630 Count 676 + 631 631 (an integer following a sequential pattern, increasing by 1 for each occurrence) 632 632 )))|integer 633 633 |((( 634 634 InclusiveValueRange 681 + 635 635 (decimal number within a closed interval, whose bounds are specified in the SDMX representation by the facets minValue and maxValue) 636 636 )))|number 637 637 |((( 638 638 ExclusiveValueRange 686 + 639 639 (decimal number within an open interval, whose bounds are specified in the SDMX representation by the facets minValue and maxValue) 640 640 )))|number 641 641 |((( 642 642 Incremental 691 + 643 643 (decimal number the increased by a specific interval (defined by the interval facet), which is typically enforced outside of the XML validation) 644 644 )))|number 645 645 |((( 646 646 ObservationalTimePeriod 696 + 647 647 (superset of StandardTimePeriod and TimeRange) 648 648 )))|time 649 649 |((( 650 650 StandardTimePeriod 701 + 651 651 (superset of BasicTimePeriod and ReportingTimePeriod) 652 652 )))|time 653 653 |((( 654 654 BasicTimePeriod 706 + 655 655 (superset of GregorianTimePeriod and DateTime) 656 656 )))|date 657 657 |((( 658 658 GregorianTimePeriod 711 + 659 659 (superset of GregorianYear, GregorianYearMonth, and GregorianDay) 660 660 )))|date 661 661 |GregorianYear (YYYY)|date ... ... @@ -663,26 +663,32 @@ 663 663 |GregorianDay (YYYY-MM-DD)|date 664 664 |((( 665 665 ReportingTimePeriod 719 + 666 666 (superset of RepostingYear, ReportingSemester, ReportingTrimester, ReportingQuarter, ReportingMonth, ReportingWeek, ReportingDay) 667 667 )))|time_period 668 668 |((( 669 669 ReportingYear 724 + 670 670 (YYYY-A1 – 1 year period) 671 671 )))|time_period 672 672 |((( 673 673 ReportingSemester 729 + 674 674 (YYYY-Ss – 6 month period) 675 675 )))|time_period 676 676 |((( 677 677 ReportingTrimester 734 + 678 678 (YYYY-Tt – 4 month period) 679 679 )))|time_period 680 680 |((( 681 681 ReportingQuarter 739 + 682 682 (YYYY-Qq – 3 month period) 683 683 )))|time_period 684 684 |((( 685 685 ReportingMonth 744 + 686 686 (YYYY-Mmm – 1 month period) 687 687 )))|time_period 688 688 |ReportingWeek|time_period ... ... @@ -689,34 +689,42 @@ 689 689 | (YYYY-Www – 7 day period; following ISO 8601 definition of a week in a year)| 690 690 |((( 691 691 ReportingDay 751 + 692 692 (YYYY-Dddd – 1 day period) 693 693 )))|time_period 694 694 |((( 695 695 DateTime 756 + 696 696 (YYYY-MM-DDThh:mm:ss) 697 697 )))|date 698 698 |((( 699 699 TimeRange 761 + 700 700 (YYYY-MM-DD(Thh:mm:ss)?/<duration>) 701 701 )))|time 702 702 |((( 703 703 Month 766 + 704 704 (~-~-MM; speicifies a month independent of a year; e.g. February is black history month in the United States) 705 705 )))|string 706 706 |((( 707 707 MonthDay 771 + 708 708 (~-~-MM-DD; specifies a day within a month independent of a year; e.g. Christmas is December 25^^th^^; used to specify reporting year start day) 709 709 )))|string 710 710 |((( 711 711 Day 776 + 712 712 (~-~--DD; specifies a day independent of a month or year; e.g. the 15^^th^^ is payday) 713 713 )))|string 714 714 |((( 715 715 Time 781 + 716 716 (hh:mm:ss; time independent of a date; e.g. coffee break is at 10:00 AM) 717 717 )))|string 718 718 |((( 719 719 Duration 786 + 720 720 (corresponds to XML Schema xs:duration datatype) 721 721 )))|duration 722 722 |XHTML|Metadata type – not applicable ... ... @@ -724,20 +724,27 @@ 724 724 |IdentifiableReference|Metadata type – not applicable 725 725 |DataSetReference|Metadata type – not applicable 726 726 727 - **Figure 14 – Mappings from SDMX data types to VTL Basic Scalar Types**794 +додол 728 728 796 +==== Figure 14 – Mappings from SDMX data types to VTL Basic Scalar Types ==== 797 + 729 729 When VTL takes in input SDMX artefacts, it is assumed that a type conversion according to the table above always happens. In case a different VTL basic scalar type is desired, it can be achieved in the VTL program taking in input the default VTL basic scalar type above and applying to it the VTL type conversion features (see the implicit and explicit type conversion and the "cast" operator in the VTL Reference Manual). 730 730 731 -=== 12.4.4 Mapping VTL basic scalar types to SDMX data types === 800 +1. 801 +11. 802 +111. Mapping VTL basic scalar types to SDMX data types 732 732 733 733 The following table describes the default conversion from the VTL basic scalar types to the SDMX data types . 734 734 735 735 |((( 736 736 VTL basic 808 + 737 737 scalar type 738 738 )))|((( 739 739 Default SDMX data type 812 + 740 740 (BasicComponentDataType 814 + 741 741 ) 742 742 )))|Default output format 743 743 |String|String|Like XML (xs:string) ... ... @@ -747,15 +747,17 @@ 747 747 |Time|StandardTimePeriod|<date>/<date> (as defined above) 748 748 |time_period|((( 749 749 ReportingTimePeriod 824 + 750 750 (StandardReportingPeriod) 751 751 )))|((( 752 752 YYYY-Pppp 828 + 753 753 (according to SDMX ) 754 754 ))) 755 755 |Duration|Duration|Like XML (xs:duration) PnYnMnDTnHnMnS 756 756 |Boolean|Boolean|Like XML (xs:boolean) with the values "true" or "false" 757 757 758 - **Figure 14 – Mappings from SDMX data types to VTL Basic Scalar Types**834 +==== Figure 14 – Mappings from SDMX data types to VTL Basic Scalar Types ==== 759 759 760 760 In case a different default conversion is desired, it can be achieved through the CustomTypeScheme and CustomType artefacts (see also the section 761 761 ... ... @@ -813,13 +813,17 @@ 813 813 814 814 The default conversion, either standard or customized, can be used to deduce automatically the representation of the components of the result of a VTL Transformation. In alternative, the representation of the resulting SDMX Dataflow can be given explicitly by providing its DataStructureDefinition. In other words, the representation specified in the DSD, if available, overrides any default conversion{{footnote}}The representation given in the DSD should obviously be compatible with the VTL data type.{{/footnote}}. 815 815 816 -=== 12.4.3 Null Values === 892 +1. 893 +11. 894 +111. Null Values 817 817 818 818 In the conversions from SDMX to VTL it is assumed by default that a missing value in SDMX becomes a NULL in VTL. After the conversion, the NULLs can be manipulated through the proper VTL operators. 819 819 820 820 On the other side, the VTL programs can produce in output NULL values for Measures and Attributes (Null values are not allowed in the Identifiers). In the conversion from VTL to SDMX, it is assumed that a NULL in VTL becomes a missing value in SDMX. In the conversion from VTL to SDMX, the default assumption can be overridden, separately for each VTL basic scalar type, by specifying which the value that represents the NULL in SDMX is. This can be specified in the attribute "nullValue" of the CustomType artefact (see also the section Transformations and Expressions of the SDMX information model). A CustomType belongs to a CustomTypeScheme, which can be referenced by one or more TransformationScheme (i.e. VTL programs). The overriding assumption is applied for all the SDMX Dataflows calculated in the TransformationScheme. 821 821 822 -=== 12.4.5 Format of the literals used in VTL Transformations === 900 +1. 901 +11. 902 +111. Format of the literals used in VTL Transformations 823 823 824 824 The VTL programs can contain literals, i.e. specific values of certain data types written directly in the VTL definitions or expressions. The VTL does not prescribe a specific format for the literals and leave the specific VTL systems and the definers of VTL Transformations free of using their preferred formats. 825 825 ... ... @@ -833,6 +833,7 @@ 833 833 834 834 In case a literal is operand of a VTL Cast operation, the format specified in the Cast overrides all the possible otherwise specified formats. 835 835 916 + 836 836 ---- 837 837 838 838 {{putFootnotes/}}