Changes for page SDMX 3.1 Standards. Section 1. Framework

Last modified by Helena K. on 2026/06/08 15:16

From 35.2 to 35.3 From 35.6 to 35.7

From version 35.3

edited by Helena K.
on 2026/06/08 12:08

Change comment: There is no comment for this version

To version 35.6

edited by Helena K.
on 2026/06/08 12:09

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -340,64 +340,65 @@
  == 5.3 SDMX-CSV ==
--[[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] is the CSV transmission format specification for exchanging data and [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] only.
--[[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
++SDMX-CSV is the CSV transmission format specification for exchanging data and reference metadata only.
++SDMX-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
++
  There are two distinct message types:
--1. **//Data//**: For the exchange of data. Like [[SDMX-JSON>>doc:sdmx:Glossary 2\.1.SDMX-JSON.WebHome]], [[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] can include both [[code>>doc:sdmx:Glossary 2\.1.Code.WebHome]] IDs and labels which is helpful when using the data to create human readable charts and dashboards.
--1. **//Metadata//**: For the exchange of [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] sets.
++1. **//Data//**: For the exchange of data. Like SDMX-JSON, SDMX-CSV can include both code IDs and labels which is helpful when using the data to create human readable charts and dashboards.
++1. **//Metadata//**: For the exchange of reference metadata sets.
  == 5.4 Formats and Messages Deprecated in Version 3.0 ==
  The following formats and messages have been deprecated in version 3.0 to simplify, modernise and rationalise the standard.
--* [[SDMX-EDI>>doc:sdmx:Glossary 2\.1.SDMX-EDI.WebHome]]
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 2.1 Generic ([[Time Series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data messages (for observations, time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] and cross-sectional data)
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 2.1 Structure Specific [[Time Series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] data message
++* SDMX-EDI
++* SDMX-ML 1.0/2.0 Generic (time-series) data message
++* SDMX-ML 1.0/2.0 Compact (time-series) data message
++* SDMX-ML 1.0/2.0 Utility (time-series) data message
++* SDMX-ML 1.0/2.0 Cross-Sectional data message
++* SDMX-ML 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
++* SDMX-ML 2.1 Structure Specific Time Series data message
  The following messages were deprecated in version 3.0 as a consequence of the deprecation of the SOAP web services:
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] Query messages
--* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] Submit Structure Request messages
++* SDMX-ML Query messages
++* SDMX-ML Submit Structure Request messages
  = 6 Dependencies on SDMX content-oriented guidelines =
--The technical standards proposed here are designed so that they can be used in conjunction with other [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] guidelines which are more closely tied to the content and semantics of statistical [[data exchange>>doc:sdmx:Glossary 2\.1.Data exchange.WebHome]]. The [[SDMX Information Model>>doc:sdmx:Glossary 2\.1.SDMX Information Model.WebHome]] works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] creates and maintains guidelines for [[cross-domain concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]], terminology, and structural definitions. There are three major parts to this effort.
++The technical standards proposed here are designed so that they can be used in conjunction with other SDMX guidelines which are more closely tied to the content and semantics of statistical data exchange. The SDMX Information Model works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, SDMX creates and maintains guidelines for cross-domain concepts, terminology, and structural definitions. There are three major parts to this effort.
  == 6.1 Cross-Domain Concepts ==
--The [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]] is a content guideline concerning [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] which are used across [[statistical domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]]. This list is expected to grow and to be subject to revision as [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] is used in a growing number of domains. The use of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]], where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] includes not only the definitions of the [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], and their names, but also, where appropriate, their [[representation>>doc:sdmx:Glossary 2\.1.Representation.WebHome]] with standard [[code lists>>doc:sdmx:Glossary 2\.1.Code list.WebHome]], and the role they play within [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] and [[metadata structure definitions>>doc:sdmx:Glossary 2\.1.Metadata structure definition.WebHome]].
++The SDMX Cross-Domain Concepts is a content guideline concerning concepts which are used across statistical domains. This list is expected to grow and to be subject to revision as SDMX is used in a growing number of domains. The use of the SDMX Cross-Domain Concepts, where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical concepts includes not only the definitions of the concepts, and their names, but also, where appropriate, their representation with standard code lists, and the role they play within data structure definitions and metadata structure definitions.
--The intent of this guideline is two-fold: to provide a core set of [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] which can be used to structure statistical data and metadata, to promote interoperability between systems (“[[structural metadata>>doc:sdmx:Glossary 2\.1.Structural metadata.WebHome]]”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] names and definitions for other types of metadata (“[[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]]”, as defined above.)
++The intent of this guideline is two-fold: to provide a core set of concepts which can be used to structure statistical data and metadata, to promote interoperability between systems (“structural metadata”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized concept names and definitions for other types of metadata (“reference metadata”, as defined above.)
  == 6.2 Metadata Common Vocabulary ==
--The Metadata Common Vocabulary is an [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] guideline which provides definition of terms to be used for the comparison and mapping of terminology found in [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Technical Standards.
++The Metadata Common Vocabulary is an SDMX guideline which provides definition of terms to be used for the comparison and mapping of terminology found in data structure definitions and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the SDMX Technical Standards.
--The MCV provides definitions for terms on which the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Cross-Domain Metadata [[Concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] work is built.
++The MCV provides definitions for terms on which the SDMX Cross-Domain Metadata Concepts work is built.
  == 6.3 Statistical Subject-Matter Domains ==
--The [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]], [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], etc. within their domains. Such groups already exist within the international community. [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] would use the [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]]-conformant technical systems within and across [[statistical domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]]. The organisation of the content of such schemes is supported in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] as a [[Category Scheme>>doc:sdmx:Glossary 2\.1.Category scheme.WebHome]].
++The Statistical Subject-Matter Domains is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard data structure definitions, concepts, etc. within their domains. Such groups already exist within the international community. SDMX would use the Statistical Subject-Matter Domains list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of SDMX-conformant technical systems within and across statistical domains. The organisation of the content of such schemes is supported in SDMX as a Category Scheme.
--[[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] will be listed and maintained by the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Initiative and will be subject to [[adjustment>>doc:sdmx:Glossary 2\.1.Adjustment.WebHome]].
++SDMX Statistical Subject-Matter Domains will be listed and maintained by the SDMX Initiative and will be subject to adjustment.
  == 6.4 SDMX Concept Roles ==
--These guidelines define the standard set of [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] Roles and their use. This set of standard [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] are implemented as a [[cross-domain Concept>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]] Scheme that defines the set of [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] roles and gives examples on [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role implementation in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.0, 2.1 and 3.0. A [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role gives a particular context to a [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] for easy and systematic interpretation by machine processing and visualization tools. For example, the [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] REPORTING_AREA and COUNTERPART_AREA are different [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] but they are both geographical characteristics, therefore they can be associated with the same [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role ID: "GEO". This allows visualization systems to interpret these [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] as geographical data in order to generate [[maps>>doc:sdmx:Glossary 2\.1.Map.WebHome]]. The implementation of [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] roles is different in [[versions>>doc:sdmx:Glossary 2\.1.Version.WebHome]] 2.0 and 2.1/3.0 of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] technical standard. Specifically for [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 3.0, i.e., via [[Concept Schemes>>doc:sdmx:Glossary 2\.1.Concept scheme.WebHome]]; the semantics of these roles have to be agreed bilateraly in [[data exchanges>>doc:sdmx:Glossary 2\.1.Data exchange.WebHome]]. The [[Concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] Roles are available as an [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concept Scheme>>doc:sdmx:Glossary 2\.1.Concept scheme.WebHome]] on the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Global Registry>>doc:sdmx:Glossary 2\.1.Global registry.WebHome]].
++These guidelines define the standard set of SDMX Concept Roles and their use. This set of standard SDMX Concepts are implemented as a cross-domain Concept Scheme that defines the set of concept roles and gives examples on concept role implementation in SDMX 2.0, 2.1 and 3.0. A concept role gives a particular context to a concept for easy and systematic interpretation by machine processing and visualization tools. For example, the concepts REPORTING_AREA and COUNTERPART_AREA are different concepts but they are both geographical characteristics, therefore they can be associated with the same concept role ID: "GEO". This allows visualization systems to interpret these concepts as geographical data in order to generate maps. The implementation of concept roles is different in versions 2.0 and 2.1/3.0 of the SDMX technical standard. Specifically for SDMX 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in SDMX 3.0, i.e., via Concept Schemes; the semantics of these roles have to be agreed bilateraly in data exchanges. The Concept Roles are available as an SDMX Concept Scheme on the SDMX Global Registry.
  = 7 Validation and Transformation Language =
--For many years the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] initiative has been fostering and supporting the development of a standard calculation [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]], called [[Validation and Transformation Language>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]]). A blueprint for defining calculations was already described in the original [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]] able to manipulate [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary 2\.1.Artefact.WebHome]].
++For many years the SDMX initiative has been fostering and supporting the development of a standard calculation language, called Validation and Transformation Language (VTL). A blueprint for defining calculations was already described in the original SDMX 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate SDMX artefacts.
--These developments started in late 2012 and were put in charge of the [[Validation and Transformation Language>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] Task Force ([[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] TF), which included members of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]] to be implemented in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] and applicable also to GSIM and DDI. This brought to the publication of the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.0 made by several organisations and triggered a large improvement of the [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]]. A new provisional version, the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.1, was released in public consultation in 2017. The high number of [[comments>>doc:sdmx:Glossary 2\.1.Comment.WebHome]] received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 was published between April and July 2018 (see the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] website).
++These developments started in late 2012 and were put in charge of the Validation and Transformation Language Task Force (VTL TF), which included members of the SDMX Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in SDMX and applicable also to GSIM and DDI. This brought to the publication of the VTL 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of VTL 1.0 made by several organisations and triggered a large improvement of the language. A new provisional version, the VTL 1.1, was released in public consultation in 2017. The high number of comments received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the VTL 2.0 was published between April and July 2018 (see the SDMX website).
--The implementation of the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] started in late 2018 and was published as an incremental revision to the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.1 standards in July 2020. It allows users to write [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 programs for validating and transforming [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] data, to store these programs in a [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] metadata registry and to exchange them through [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] messages, also together the definition of the data structures of the involved data.
++The implementation of the VTL 2.0 in SDMX started in late 2018 and was published as an incremental revision to the SDMX 2.1 standards in July 2020. It allows users to write VTL 2.0 programs for validating and transforming SDMX data, to store these programs in a SDMX metadata registry and to exchange them through SDMX messages, also together the definition of the data structures of the involved data.
--The Transformations and Expressions package for modelling [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] programs in the [[SDMX information model>>doc:sdmx:Glossary 2\.1.SDMX Information Model.WebHome]] is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
++The Transformations and Expressions package for modelling VTL programs in the SDMX information model is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.