Skip to Content
Last modified by Helena on 2025/09/10 11:19
From version 13.6
edited by Helena
on 2025/05/16 21:25
Change comment: There is no comment for this version
To version 14.5
edited by Helena
on 2025/05/19 14:25
Change comment: Update document after refactoring.

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -SDMX STANDARDS: SECTION 1
1 +SDMX 3.0 Standards. Section 1. Framework for SDMX technical standards
Parent
... ... @@ -1,0 +1,1 @@
1 +Methodology.WebHome
Content
... ... @@ -11,14 +11,12 @@
11 11  
12 12  = 1 Introduction =
13 13  
14 -The [[Statistical Data and Metadata Exchange>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] ([[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]) initiative (https:~/~/www.[[sdmx>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]].org) sets 3 standards that can facilitate the exchange of statistical data and metadata using modern 4 information technology.
14 +The [[Statistical Data and Metadata Exchange>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] ([[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]) initiative ([[https:~~/~~/www.sdmx.org>>https://https:www.sdmx.org]]) sets 3 standards that can facilitate the exchange of statistical data and metadata using modern 4 information technology.
15 15  
16 16  The [[SDMX Technical Specifications>>doc:xwiki:Glossary.SDMX Technical Specification.WebHome]] are organised into several discrete sections.
17 17  
18 -The following are published on the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] website
18 +The following are published on the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] website ([[__https:~~/~~/www.sdmx.org__>>https://https:www.sdmx.org]]).
19 19  
20 -([[__https:~~/~~/www.sdmx.org__>>https://https:www.sdmx.org]]).
21 -
22 22  **Section 1** **Framework for SDMX Technical Standards** – this document providing an introduction to the technical standards.
23 23  
24 24  **Section 2** **SDMX Information Model** -
... ... @@ -53,7 +53,7 @@
53 53  
54 54  The following sections are obsolete:
55 55  
56 -Section 3 - [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] - replaced by the [[sdmx>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-twg/[[sdmx-ml>>doc:xwiki:Glossary.SDMX-ML.WebHome]] GitHub repository
54 +Section 3 - [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] - replaced by the sdmx-twg/sdmx-ml GitHub repository
57 57  
58 58  Section 4 - [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-EDI
59 59  
... ... @@ -266,7 +266,7 @@
266 266  
267 267  Web services allow computer applications to exchange data directly over the Internet, essentially allowing modular or distributed computing in a more flexible fashion than ever before. In order to allow web services to function, however, many standards are required: for requesting and supplying data; for expressing the enveloping data which is used to package exchanged data; for describing web services to one another, to allow for easy integration into applications that use other web services as data resources.
268 268  
269 -(% style="color:#2ecc71" %)Version(%%) 3.0 has standardized on RESTful web services with a OpenAPI specification published on the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group’s GitHub repository __[[https:~~/~~/github.com/>>url:https://github.com/sdmx-twg]][[sdmx>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]][[->>url:https://github.com/sdmx-twg]][[twg>>url:https://github.com/sdmx-twg]]__[[.>>url:https://github.com/sdmx-twg]] There are five ‘resources’:
267 +(% style="color:#2ecc71" %)Version(%%) 3.0 has standardized on RESTful web services with a OpenAPI specification published on the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group’s GitHub repository [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]]. There are five ‘resources’:
270 270  
271 271  * structure – retrieval and maintenance of [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]]
272 272  * data – retrieval of data
... ... @@ -274,8 +274,10 @@
274 274  * availability – retrieval of information on the data available for a [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]
275 275  * metadata – retrieval of [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]]
276 276  
277 -The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]: https:~/~/ws-entry-point/data/[[dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
275 +The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]:
278 278  
277 +https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
278 +
279 279  = 4 The SDMX Information Model =
280 280  
281 281  [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] specifies a number of transmission formats for expressing data and [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]], the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
... ... @@ -345,7 +345,7 @@
345 345  * SDMX-ML 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
346 346  * SDMX-ML 2.1 Structure Specific Time Series data message
347 347  
348 -The following messages were deprecated in version 3.0 as a consequence of the deprecation of the SOAP web services:
348 +The following messages were deprecated in (% style="color:#2ecc71" %)version(%%) 3.0 as a consequence of the deprecation of the SOAP web services:
349 349  
350 350  * SDMX-ML Query messages
351 351  * SDMX-ML Submit Structure Request messages
... ... @@ -352,38 +352,38 @@
352 352  
353 353  = 6 Dependencies on SDMX content-oriented guidelines =
354 354  
355 -The technical standards proposed here are designed so that they can be used in conjunction with other SDMX guidelines which are more closely tied to the content and semantics of statistical data exchange. The SDMX Information Model works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, SDMX creates and maintains guidelines for cross-domain concepts, terminology, and structural definitions. There are three major parts to this effort.
355 +The technical standards proposed here are designed so that they can be used in conjunction with other [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] guidelines which are more closely tied to the content and semantics of statistical data exchange. The [[SDMX Information Model>>doc:xwiki:Glossary.SDMX Information Model.WebHome]] works equally well with any statistical (% style="color:#2ecc71" %)concept(%%), but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific (% style="color:#2ecc71" %)concepts(%%) and terminology. To achieve this goal, [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] creates and maintains guidelines for [[cross-domain concepts>>doc:xwiki:Glossary.Cross-domain concept.WebHome]], terminology, and structural definitions. There are three major parts to this effort.
356 356  
357 357  == 6.1 Cross-Domain Concepts ==
358 358  
359 -The SDMX Cross-Domain Concepts is a content guideline concerning concepts which are used across statistical domains. This list is expected to grow and to be subject to revision as SDMX is used in a growing number of domains. The use of the SDMX Cross-Domain Concepts, where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical concepts includes not only the definitions of the concepts, and their names, but also, where appropriate, their representation with standard code lists, and the role they play within data structure definitions and metadata structure definitions.
359 +The [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:xwiki:Glossary.Cross-domain concept.WebHome]] is a content guideline concerning (% style="color:#2ecc71" %)concepts(%%) which are used across [[statistical domains>>doc:xwiki:Glossary.Statistical subject-matter domain.WebHome]]. This list is expected to grow and to be subject to revision as [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] is used in a growing number of domains. The use of the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:xwiki:Glossary.Cross-domain concept.WebHome]], where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical (% style="color:#2ecc71" %)concepts(%%) includes not only the definitions of the (% style="color:#2ecc71" %)concepts(%%), and their names, but also, where appropriate, their [[representation>>doc:xwiki:Glossary.Representation.WebHome]] with standard [[code lists>>doc:xwiki:Glossary.Code list.WebHome]], and the role they play within [[data structure definitions>>doc:xwiki:Glossary.Data structure definition.WebHome]] and metadata structure definitions.
360 360  
361 -The intent of this guideline is two-fold: to provide a core set of concepts which can be used to structure statistical data and metadata, to promote interoperability between systems (“structural metadata”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized concept names and definitions for other types of metadata (“reference metadata”, as defined above.)
361 +The intent of this guideline is two-fold: to provide a core set of (% style="color:#2ecc71" %)concepts(%%) which can be used to structure statistical data and metadata, to promote interoperability between systems (“[[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]]”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized (% style="color:#2ecc71" %)concept(%%) names and definitions for other types of metadata (“[[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]]”, as defined above.)
362 362  
363 363  == 6.2 Metadata Common Vocabulary ==
364 364  
365 -The Metadata Common Vocabulary is an SDMX guideline which provides definition of terms to be used for the comparison and mapping of terminology found in data structure definitions and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the SDMX Technical Standards.
365 +The Metadata Common Vocabulary is an [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] guideline which provides definition of terms to be used for the comparison and mapping of terminology found in [[data structure definitions>>doc:xwiki:Glossary.Data structure definition.WebHome]] and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be (% style="color:#2ecc71" %)mapped(%%). This set of terms is inclusive of the terminology used within the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards.
366 366  
367 -The MCV provides definitions for terms on which the SDMX Cross-Domain Metadata Concepts work is built.
367 +The MCV provides definitions for terms on which the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Cross-Domain Metadata (% style="color:#2ecc71" %)Concepts(%%) work is built.
368 368  
369 369  == 6.3 Statistical Subject-Matter Domains ==
370 370  
371 -The Statistical Subject-Matter Domains is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard data structure definitions, concepts, etc. within their domains. Such groups already exist within the international community. SDMX would use the Statistical Subject-Matter Domains list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of SDMX-conformant technical systems within and across statistical domains. The organisation of the content of such schemes is supported in SDMX as a Category Scheme.
371 +The [[Statistical Subject-Matter Domains>>doc:xwiki:Glossary.Statistical subject-matter domain.WebHome]] is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be (% style="color:#2ecc71" %)mapped(%%), to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard [[data structure definitions>>doc:xwiki:Glossary.Data structure definition.WebHome]], (% style="color:#2ecc71" %)concepts(%%), etc. within their domains. Such groups already exist within the international community. [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] would use the [[Statistical Subject-Matter Domains>>doc:xwiki:Glossary.Statistical subject-matter domain.WebHome]] list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-conformant technical systems within and across [[statistical domains>>doc:xwiki:Glossary.Statistical subject-matter domain.WebHome]]. The organisation of the content of such schemes is supported in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] as a [[Category>>doc:xwiki:Glossary.Category.WebHome]] Scheme.
372 372  
373 -SDMX Statistical Subject-Matter Domains will be listed and maintained by the SDMX Initiative and will be subject to adjustment.
373 +[[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] [[Statistical Subject-Matter Domains>>doc:xwiki:Glossary.Statistical subject-matter domain.WebHome]] will be listed and maintained by the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Initiative and will be subject to adjustment.
374 374  
375 375  == 6.4 SDMX Concept Roles ==
376 376  
377 -These guidelines define the standard set of SDMX Concept Roles and their use. This set of standard SDMX Concepts are implemented as a cross-domain Concept Scheme that defines the set of concept roles and gives examples on concept role implementation in SDMX 2.0, 2.1 and 3.0. A concept role gives a particular context to a concept for easy and systematic interpretation by machine processing and visualization tools. For example, the concepts REPORTING_AREA and COUNTERPART_AREA are different concepts but they are both geographical characteristics, therefore they can be associated with the same concept role ID: "GEO". This allows visualization systems to interpret these concepts as geographical data in order to generate maps. The implementation of concept roles is different in versions 2.0 and 2.1/3.0 of the SDMX technical standard. Specifically for SDMX 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations.
377 +These guidelines define the standard set of [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#2ecc71" %)Concept(%%) Roles and their use. This set of standard [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#2ecc71" %)Concepts(%%) are implemented as a [[cross-domain Concept>>doc:xwiki:Glossary.Cross-domain concept.WebHome]] Scheme that defines the set of (% style="color:#2ecc71" %)concept(%%) roles and gives examples on (% style="color:#2ecc71" %)concept(%%) role implementation in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 2.0, 2.1 and 3.0. A (% style="color:#2ecc71" %)concept(%%) role gives a particular context to a (% style="color:#2ecc71" %)concept(%%) for easy and systematic interpretation by machine processing and visualization tools. For example, the (% style="color:#2ecc71" %)concepts(%%) REPORTING_AREA and COUNTERPART_AREA are different (% style="color:#2ecc71" %)concepts(%%) but they are both geographical characteristics, therefore they can be associated with the same (% style="color:#2ecc71" %)concept(%%) role ID: "GEO". This allows visualization systems to interpret these (% style="color:#2ecc71" %)concepts(%%) as geographical data in order to generate (% style="color:#2ecc71" %)maps(%%). The implementation of (% style="color:#2ecc71" %)concept(%%) roles is different in (% style="color:#2ecc71" %)versions(%%) 2.0 and 2.1/3.0 of the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] technical standard. Specifically for [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations.
378 378  
379 -Additional roles may be provided via the standard roles’ mechanism in SDMX 3.0, i.e., via Concept Schemes; the semantics of these roles have to be agreed bilateraly in data exchanges. The Concept Roles are available as an SDMX Concept Scheme on the SDMX Global Registry.
379 +Additional roles may be provided via the standard roles’ mechanism in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, i.e., via [[Concept Schemes>>doc:xwiki:Glossary.Concept scheme.WebHome]]; the semantics of these roles have to be agreed bilateraly in data exchanges. The (% style="color:#2ecc71" %)Concept(%%) Roles are available as an [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] [[Concept Scheme>>doc:xwiki:Glossary.Concept scheme.WebHome]] on the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Global Registry.
380 380  
381 381  = 7 Validation and Transformation Language =
382 382  
383 -For many years the SDMX initiative has been fostering and supporting the development of a standard calculation language, called Validation and Transformation Language (VTL). A blueprint for defining calculations was already described in the original SDMX 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate SDMX artefacts.
383 +For many years the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] initiative has been fostering and supporting the development of a standard calculation language, called [[Validation and Transformation Language>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] ([[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]]). A blueprint for defining calculations was already described in the original [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] artefacts.
384 384  
385 -These developments started in late 2012 and were put in charge of the Validation and Transformation Language Task Force (VTL TF), which included members of the SDMX Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in SDMX and applicable also to GSIM and DDI. This brought to the publication of the VTL 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of VTL 1.0 made by several organisations and triggered a large improvement of the language. A new provisional version, the VTL 1.1, was released in public consultation in 2017. The high number of comments received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the VTL 2.0 was published between April and July 2018 (see the SDMX website).
385 +These developments started in late 2012 and were put in charge of the [[Validation and Transformation Language>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] Task Force ([[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] TF), which included members of the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] and applicable also to GSIM and DDI. This brought to the publication of the [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 1.0 in 2015. Then new requirements came from a number of proofs of (% style="color:#2ecc71" %)concepts(%%) and tests of [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 1.0 made by several organisations and triggered a large improvement of the language. A new provisional (% style="color:#2ecc71" %)version(%%), the [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 1.1, was released in public consultation in 2017. The high number of [[comments>>doc:xwiki:Glossary.Comment.WebHome]] received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible (% style="color:#2ecc71" %)version(%%). Finally, the [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 2.0 was published between April and July 2018 (see the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] website).
386 386  
387 -The implementation of the VTL 2.0 in SDMX started in late 2018 and was published as an incremental revision to the SDMX 2.1 standards in July 2020. It allows users to write VTL 2.0 programs for validating and transforming SDMX data, to store these programs in a SDMX metadata registry and to exchange them through SDMX messages, also together the definition of the data structures of the involved data.
387 +The implementation of the [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 2.0 in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] started in late 2018 and was published as an incremental revision to the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 standards in July 2020. It allows users to write [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] 2.0 programs for validating and transforming [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] data, to store these programs in a [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] metadata registry and to exchange them through [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] messages, also together the definition of the data structures of the involved data.
388 388  
389 -The Transformations and Expressions package for modelling VTL programs in the SDMX information model is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
389 +The Transformations and Expressions package for modelling [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] programs in the [[SDMX information model>>doc:xwiki:Glossary.SDMX Information Model.WebHome]] is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.