Changes for page SDMX 3.1 Standards. Section 1. Framework

Last modified by Artur on 2025/09/30 12:30

From 12.2 to 12.3 From 13.1 to 13.2

From version 12.3

edited by Helena
on 2025/06/06 17:14

Change comment: There is no comment for this version

To version 13.1

edited by Helena
on 2025/06/06 17:15

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -257,30 +257,32 @@
  Web services allow computer applications to exchange data directly over the Internet, essentially allowing modular or distributed computing in a more flexible fashion than ever before. In order to allow web services to function, however, many standards are required: for requesting and supplying data; for expressing the enveloping data which is used to package exchanged data; for describing web services to one another, to allow for easy integration into applications that use other web services as data resources.
--Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the SDMX Technical Working Group’s GitHub repository [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]]. There are five ‘resources’:
++Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group’s GitHub repository  [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]].
++There are five ‘resources’:
++
  * structure – retrieval and maintenance of structural metadata
  * data – retrieval of data
  * schema – retrieval of XML schemas to validate specific data or metadata sets
--* availability – retrieval of information on the data available for a Dataflow
++* availability – retrieval of information on the data available for a [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]
  * metadata – retrieval of reference metadata
  * registration – retrieval of data locations (URL) for specific provision agreements
--The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) Dataflow: https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
++The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]: https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
  = 4 The SDMX Information Model =
--SDMX provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because SDMX specifies a number of transmission formats for expressing data and structural metadata, the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
++[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] specifies a number of transmission formats for expressing data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
--SDMX recognizes that statistical data is structured; in SDMX this structure is termed a Data Structure Definition. “Data sets” are made up of one or more lower-level “groups”, based on their degrees of similarity. Each group is in turn comprised of one or more “series” of data. Each series or section has a “key” - values for each of a cluster of concepts, also called "dimensions" - which identifies it, and one or more “observations”, which typically combine the time of the observation, and the value of the observation (e.g., measurement). Additionally, metadata may be attached at any level of this structure as descriptive “attributes”. Code lists (enumerations) and other patterns for representation of data and metadata are also modelled.
++[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] recognizes that statistical data is structured; in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] this structure is termed a [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. “[[Data sets>>doc:sdmx:Glossary.Data set.WebHome]]” are made up of one or more lower-(% style="color:#e74c3c" %)level(%%) “groups”, based on their degrees of similarity. Each group is in turn comprised of one or more “series” of data. Each series or section has a “key” - values for each of a cluster of (% style="color:#e74c3c" %)concepts(%%), also called "[[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]]" - which identifies it, and one or more “observations”, which typically combine the time of the observation, and the value of the observation (e.g., measurement). Additionally, metadata may be attached at any (% style="color:#e74c3c" %)level(%%) of this structure as descriptive “[[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]”. [[Code lists>>doc:sdmx:Glossary.Code list.WebHome]] (enumerations) and other patterns for [[representation>>doc:sdmx:Glossary.Representation.WebHome]] of data and metadata are also modelled.
--There is some similarity between “cube” structures commonly used to process statistical data, and the Data Structure Definition idea in the SDMX Information Model. It is important to note that the data as structured according to the SDMX Information Model is optimized for exchange, potentially with partners who may have no ability to process a “cube” of data coming from complex statistical systems. SDMX time series can be understood as “slices” of the cube. Such a slice is identified by its key. A "series" key consists of the values for all dimensions specified by the key family except time. Thus, it is possible to reconstruct and describe data cubes from SDMX-structured data, and to exchange such databases using the interfaces and formats provided for that purpose in the standard. Additional objects such as hierarchical code lists, constraints and structure maps make it possible to more fully model the structure of cubes.
++There is some similarity between “cube” structures commonly used to process statistical data, and the [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] idea in the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]]. It is important to note that the data as structured according to the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is optimized for exchange, potentially with partners who may have no ability to process a “cube” of data coming from complex statistical systems. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] time series can be understood as “slices” of the cube. Such a slice is identified by its key. A "series" key consists of the values for all [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] specified by the key family except time. Thus, it is possible to reconstruct and describe data cubes from [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-structured data, and to exchange such databases using the interfaces and formats provided for that purpose in the standard. Additional objects such as [[hierarchical code lists>>doc:sdmx:Glossary.Hierarchical code list.WebHome]], (% style="color:#e74c3c" %)constraints(%%) and structure (% style="color:#e74c3c" %)maps(%%) make it possible to more fully model the structure of cubes.
--The information model also provides a view of reference metadata: a mechanism for referencing the meaningful “objects” within the SDMX view of statistical exchange processes (data providers, structures, provisioning agreements, dataflows, metadata flows, etc.) to which metadata is attached; a mechanism for describing a set of meaningful concepts, of organizing them into a presentational structure, and of indicating how their values are represented. This is based on a simple, hierarchical view of reference metadata which is common to many metadata systems and classification/categorization schemes. SDMX provides a model (and XML and JSON formats) for both describing reference metadata structures, and of reporting reference metadata according to those structures.
++The information model also provides a view of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]: a mechanism for referencing the meaningful “objects” within the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] view of statistical exchange processes ([[data providers>>doc:sdmx:Glossary.Data provider.WebHome]], structures, [[provisioning agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]], [[dataflows>>doc:sdmx:Glossary.Dataflow.WebHome]], [[metadata flows>>doc:sdmx:Glossary.Metadataflow.WebHome]], etc.) to which metadata is attached; a mechanism for describing a set of meaningful (% style="color:#e74c3c" %)concepts(%%), of organizing them into a presentational structure, and of indicating how their values are represented. This is based on a simple, hierarchical view of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] which is common to many metadata systems and classification/categorization schemes. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] provides a model (and XML and JSON formats) for both describing [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] structures, and of reporting [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] according to those structures.
--Version 2.0/2.1 introduced support for metadata related to the process aspects of statistical exchange. A step-by-step process can be modelled; information about who is providing data and reference metadata and how they are providing it can be expressed; and the technical aspects of service-level agreements (and similar types of provisioning agreements) can be represented.
++(% style="color:#e74c3c" %)Version(%%) 2.0/2.1 introduced support for metadata related to the process aspects of statistical exchange. A step-by-step process can be modelled; information about who is providing data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and how they are providing it can be expressed; and the technical aspects of service-(% style="color:#e74c3c" %)level(%%) agreements (and similar types of [[provisioning agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]]) can be represented.
--Support for the Validation and Transformation Language (VTL) in the SDMX Information Model was introduced in the July 2020 revision of 2.1 and is retained in version 3.0 with minimal changes. This allows reusable VTL ‘programs’ (a cohesive set of transformation statements designed to be executed together) and their associated constructs such as validation rulesets and user-defined operators to be managed and exchanged as SDMX structural metadata. Mappings between objects such as data sets referenced in VTL programs and the actual SDMX artefacts to which they relate is essential when it comes to actually executing programs, and this information can also be defined. Chapter 7 has more information on VTL and its integration with SDMX.
++Support for the [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]]) in the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] was introduced in the July 2020 revision of 2.1 and is retained in (% style="color:#e74c3c" %)version(%%) 3.0 with minimal changes. This allows reusable [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ‘programs’ (a cohesive set of transformation statements designed to be executed together) and their associated constructs such as validation rulesets and user-defined operators to be managed and exchanged as [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. Mappings between objects such as [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] referenced in [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] programs and the actual [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] to which they relate is essential when it comes to actually executing programs, and this information can also be defined. Chapter 7 has more information on [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] and its integration with [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]].
  A full UML conceptual design of the information model is set out in Section 2 of the Technical Specifications.
@@ -288,94 +288,94 @@
  == 5.1 SDMX-ML ==
--SDMX-ML is the XML transmission format specification for exchanging structural metadata, data and reference metadata, and interacting with SDMX registry services. It is designed as a general-purpose format for all automation and data / metadata exchange tasks, and provides the most complete coverage.
++[[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] is the XML transmission format specification for exchanging [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], and interacting with [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] services. It is designed as a general-purpose format for all automation and data / metadata exchange tasks, and provides the most complete coverage.
  There are four distinct types of message:
--1. //Structure Definition~:// For the exchange of structural metadata. A SDMX-ML structure message can carry details of any number and combination of structural metadata artefacts like DSDs, code lists and constraints.
--1. //Structure-specific Data~:// For the exchange of data. This format is specific to the Data Structure Definitions of the data sets (in other terms, it is DSD-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large data sets in XML format, provides strict validation of conformance with the DSD using a generic XML parser, and supports the transmission of partial data sets (incremental updates) as well as whole data sets.
++1. //Structure Definition~:// For the exchange of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. A [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] structure message can carry details of any number and combination of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] like DSDs, [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] and (% style="color:#e74c3c" %)constraints(%%).
++1. //Structure-specific Data~:// For the exchange of data. This format is specific to the [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] of the [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] (in other terms, it is [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] in XML format, provides strict validation of conformance with the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] using a generic XML parser, and supports the transmission of partial [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[incremental updates>>doc:sdmx:Glossary.Incremental update.WebHome]]) as well as whole [[data sets>>doc:sdmx:Glossary.Data set.WebHome]].
  Many XML tools and technologies have expectations about the functions performed by an XML schema, one of which is a very direct relationship between the XML constructs described in the XML schema and the tagged data in the XML instance. Strong data typing is also considered normal, supporting full validation of the tagged data. These message types are designed to support validation and other expected XML schema functions.
--1. //Generic Metadata~:// For the exchange of reference metadata sets. ‘Generic’ means the XML elements and XML attributes are the same regardless of the metadata set.
--1. //Registry~:// All of the possible interactions with the SDMX registry services are supported using SDMX-ML interfaces and REST API calls. Submission of structural metadata content, data registrations and subscriptions is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
++1. //Generic Metadata~:// For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets. ‘Generic’ means the XML elements and XML [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] are the same regardless of the metadata set.
++1. //Registry~:// All of the possible interactions with the [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] services are supported using [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] interfaces and REST API calls. Submission of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] content, data registrations and [[subscriptions>>doc:sdmx:Glossary.Subscription.WebHome]] is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
--== {{id name="_Toc56646"/}}5.2 SDMX-JSON ==
++== 5.2 SDMX-JSON ==
--SDMX-JSON is the JSON transmission format specification for exchanging structural metadata, data and reference metadata. It provides an alternative to SDMX-ML and is most suited to applications like web data dissemination.
++[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] is the JSON transmission format specification for exchanging [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]. It provides an alternative to [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and is most suited to applications like web data dissemination.
--SDMX-JSON messages serve the same function as those of the XML formats but have a different structure. For data, an important distinction is that they carry both component codes and labels which provides all the information needed to display the content in a single JSON response. The XML Structure-specific Data format by contrast carries only code IDs thus requiring applications obtain and hold structural metadata about the data set in order to display the content in human-readable form.
++[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages serve the same function as those of the XML formats but have a different structure. For data, an important distinction is that they carry both [[component>>doc:sdmx:Glossary.Component.WebHome]] [[codes>>doc:sdmx:Glossary.Code.WebHome]] and labels which provides all the information needed to display the content in a single JSON response. The XML Structure-specific Data format by contrast carries only [[code>>doc:sdmx:Glossary.Code.WebHome]] IDs thus requiring applications obtain and hold [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] about the [[data set>>doc:sdmx:Glossary.Data set.WebHome]] in order to display the content in human-readable form.
--SDMX-JSON does not include messages for subscription / notification or registration registry services - SDMX-ML must be used for those purposes.
++[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] does not include messages for [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] / [[notification>>doc:sdmx:Glossary.Notification.WebHome]] or registration registry services - [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] must be used for those purposes.
  There are three distinct message types:
--1. //Structure~:// For the exchange structural metadata. SDMX-JSON structure messages follow the same principles as for SDMX-ML in that a single message can transmit any number and combination of structural metadata artefacts. While the SDMX-ML and SDMX-JSON messages are structured differently, it is possible to freely convert between them.
--1. //Data: //For the exchange of data. Unlike SDMX-ML, the structure of a SDMX-JSON data message is not specific to the DSDs of the data sets so schema validation will not check for compliance of the data with the DSDs.
--1. //Metadata//: For the exchange of reference metadata sets.
++1. //Structure~:// For the exchange [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] structure messages follow the same principles as for [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] in that a single message can transmit any number and combination of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]]. While the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages are structured differently, it is possible to freely convert between them.
++1. //Data: //For the exchange of data. Unlike [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]], the structure of a [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] data message is not specific to the DSDs of the [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] so schema validation will not check for compliance of the data with the DSDs.
++1. //Metadata//: For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets.
  == 5.3 SDMX-CSV ==
--SDMX-CSV is the CSV transmission format specification for exchanging data and reference metadata only.
++[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV is the CSV transmission format specification for exchanging data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] only.
--SDMX-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
++[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
  There are two distinct message types:
--1. //Data//: For the exchange of data. Like SDMX-JSON, SDMX-CSV can include both code IDs and labels which is helpful when using the data to create human readable charts and dashboards.
--1. //Metadata//: For the exchange of reference metadata sets.
++1. //Data//: For the exchange of data. Like [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]], [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV can include both [[code>>doc:sdmx:Glossary.Code.WebHome]] IDs and labels which is helpful when using the data to create human readable charts and dashboards.
++1. //Metadata//: For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets.
  == 5.4 Formats and Messages Deprecated in Version 3.0 ==
--The following formats and messages have been deprecated in version 3.0 to simplify, modernise and rationalise the standard.
++The following formats and messages have been deprecated in (% style="color:#e74c3c" %)version(%%) 3.0 to simplify, modernise and rationalise the standard.
--* SDMX-EDI
--* SDMX-ML 1.0/2.0 Generic (time-series) data message
--* SDMX-ML 1.0/2.0 Compact (time-series) data message
--* SDMX-ML 1.0/2.0 Utility (time-series) data message
--* SDMX-ML 1.0/2.0 Cross-Sectional data message
--* SDMX-ML 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
--* SDMX-ML 2.1 Structure Specific Time Series data message
++* [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-EDI
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-series) data message
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-series) data message
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-series) data message
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 2.1 Structure Specific Time Series data message
--The following messages were deprecated in version 3.0 as a consequence of the deprecation of the SOAP web services:
++The following messages were deprecated in (% style="color:#e74c3c" %)version(%%) 3.0 as a consequence of the deprecation of the SOAP web services:
--* SDMX-ML Query messages
--* SDMX-ML Submit Structure Request messages
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Query messages
++* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Submit Structure Request messages
  = 6 Dependencies on SDMX content-oriented guidelines =
--The technical standards proposed here are designed so that they can be used in conjunction with other SDMX guidelines which are more closely tied to the content and semantics of statistical data exchange. The SDMX Information Model works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, SDMX creates and maintains guidelines for cross-domain concepts, terminology, and structural definitions. There are three major parts to this effort.
++The technical standards proposed here are designed so that they can be used in conjunction with other [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] guidelines which are more closely tied to the content and semantics of statistical data exchange. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] works equally well with any statistical (% style="color:#e74c3c" %)concept(%%), but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific (% style="color:#e74c3c" %)concepts(%%) and terminology. To achieve this goal, [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] creates and maintains guidelines for [[cross-domain concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]], terminology, and structural definitions. There are three major parts to this effort.
  == 6.1 Cross-Domain Concepts ==
--The SDMX Cross-Domain Concepts is a content guideline concerning concepts which are used across statistical domains. This list is expected to grow and to be subject to revision as SDMX is used in a growing number of domains. The use of the SDMX Cross-Domain Concepts, where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical concepts includes not only the definitions of the concepts, and their names, but also, where appropriate, their representation with standard code lists, and the role they play within data structure definitions and metadata structure definitions.
++The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]] is a content guideline concerning (% style="color:#e74c3c" %)concepts(%%) which are used across [[statistical domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]]. This list is expected to grow and to be subject to revision as [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is used in a growing number of domains. The use of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]], where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical (% style="color:#e74c3c" %)concepts(%%) includes not only the definitions of the (% style="color:#e74c3c" %)concepts(%%), and their names, but also, where appropriate, their [[representation>>doc:sdmx:Glossary.Representation.WebHome]] with standard [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], and the role they play within [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and metadata structure definitions.
--The intent of this guideline is two-fold: to provide a core set of concepts which can be used to structure statistical data and metadata, to promote interoperability between systems (“structural metadata”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized concept names and definitions for other types of metadata (“reference metadata”, as defined above.)
++The intent of this guideline is two-fold: to provide a core set of (% style="color:#e74c3c" %)concepts(%%) which can be used to structure statistical data and metadata, to promote interoperability between systems (“[[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized (% style="color:#e74c3c" %)concept(%%) names and definitions for other types of metadata (“[[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]”, as defined above.)
  == 6.2 Metadata Common Vocabulary ==
--The Metadata Common Vocabulary is an SDMX guideline which provides definition of terms to be used for the comparison and mapping of terminology found in data structure definitions and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the SDMX Technical Standards.
++The Metadata Common Vocabulary is an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] guideline which provides definition of terms to be used for the comparison and mapping of terminology found in [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be (% style="color:#e74c3c" %)mapped(%%). This set of terms is inclusive of the terminology used within the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards.
--The MCV provides definitions for terms on which the SDMX Cross-Domain Metadata
++The MCV provides definitions for terms on which the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Cross-Domain Metadata
--Concepts work is built.
++(% style="color:#e74c3c" %)Concepts(%%) work is built.
  == 6.3 Statistical Subject-Matter Domains ==
--The Statistical Subject-Matter Domains is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard data structure definitions, concepts, etc. within their domains. Such groups already exist within the international community. SDMX would use the Statistical Subject-Matter Domains list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of SDMX-conformant technical systems within and across statistical domains. The organisation of the content of such schemes is supported in SDMX as a Category Scheme.
++The [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be (% style="color:#e74c3c" %)mapped(%%), to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]], (% style="color:#e74c3c" %)concepts(%%), etc. within their domains. Such groups already exist within the international community. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] would use the [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant technical systems within and across [[statistical domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]]. The organisation of the content of such schemes is supported in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] as a [[Category>>doc:sdmx:Glossary.Category.WebHome]] Scheme.
--SDMX Statistical Subject-Matter Domains will be listed and maintained by the SDMX Initiative and will be subject to adjustment.
++[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] will be listed and maintained by the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Initiative and will be subject to adjustment.
  == 6.4 SDMX Concept Roles ==
--These guidelines define the standard set of SDMX Concept Roles and their use. This set of standard SDMX Concepts are implemented as a cross-domain Concept Scheme that defines the set of concept roles and gives examples on concept role implementation in SDMX 2.0, 2.1 and 3.0. A concept role gives a particular context to a concept for easy and systematic interpretation by machine processing and visualization tools. For example, the concepts REPORTING_AREA and COUNTERPART_AREA are different concepts but they are both geographical characteristics, therefore they can be associated with the same concept role ID: "GEO". This allows visualization systems to interpret these concepts as geographical data in order to generate maps. The implementation of concept roles is different in versions 2.0 and 2.1/3.0 of the SDMX technical standard. Specifically for SDMX 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in SDMX 3.0, i.e., via Concept Schemes; the semantics of these roles have to be agreed bilateraly in data exchanges. The Concept Roles are available as an SDMX Concept Scheme on the SDMX Global Registry.
++These guidelines define the standard set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#e74c3c" %)Concept(%%) Roles and their use. This set of standard [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#e74c3c" %)Concepts(%%) are implemented as a [[cross-domain Concept>>doc:sdmx:Glossary.Cross-domain concept.WebHome]] Scheme that defines the set of (% style="color:#e74c3c" %)concept(%%) roles and gives examples on (% style="color:#e74c3c" %)concept(%%) role implementation in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.0, 2.1 and 3.0. A (% style="color:#e74c3c" %)concept(%%) role gives a particular context to a (% style="color:#e74c3c" %)concept(%%) for easy and systematic interpretation by machine processing and visualization tools. For example, the (% style="color:#e74c3c" %)concepts(%%) REPORTING_AREA and COUNTERPART_AREA are different (% style="color:#e74c3c" %)concepts(%%) but they are both geographical characteristics, therefore they can be associated with the same (% style="color:#e74c3c" %)concept(%%) role ID: "GEO". This allows visualization systems to interpret these (% style="color:#e74c3c" %)concepts(%%) as geographical data in order to generate (% style="color:#e74c3c" %)maps(%%). The implementation of (% style="color:#e74c3c" %)concept(%%) roles is different in (% style="color:#e74c3c" %)versions(%%) 2.0 and 2.1/3.0 of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] technical standard. Specifically for [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, i.e., via [[Concept Schemes>>doc:sdmx:Glossary.Concept scheme.WebHome]]; the semantics of these roles have to be agreed bilateraly in data exchanges. The (% style="color:#e74c3c" %)Concept(%%) Roles are available as an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Concept Scheme>>doc:sdmx:Glossary.Concept scheme.WebHome]] on the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Global Registry.
  = 7 Validation and Transformation Language =
--For many years the SDMX initiative has been fostering and supporting the development of a standard calculation language, called Validation and Transformation Language (VTL). A blueprint for defining calculations was already described in the original SDMX 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate SDMX artefacts.
++For many years the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] initiative has been fostering and supporting the development of a standard calculation language, called [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]]). A blueprint for defining calculations was already described in the original [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]].
--These developments started in late 2012 and were put in charge of the Validation and Transformation Language Task Force (VTL TF), which included members of the SDMX Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in SDMX and applicable also to GSIM and DDI. This brought to the publication of the VTL 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of VTL 1.0 made by several organisations and triggered a large improvement of the language. A new provisional version, the VTL 1.1, was released in public consultation in 2017. The high number of comments received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the VTL 2.0 was published between April and July 2018 (see the SDMX website).
++These developments started in late 2012 and were put in charge of the [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] Task Force ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] TF), which included members of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] and applicable also to GSIM and DDI. This brought to the publication of the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.0 in 2015. Then new requirements came from a number of proofs of (% style="color:#e74c3c" %)concepts(%%) and tests of [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.0 made by several organisations and triggered a large improvement of the language. A new provisional (% style="color:#e74c3c" %)version(%%), the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.1, was released in public consultation in 2017. The high number of [[comments>>doc:sdmx:Glossary.Comment.WebHome]] received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible (% style="color:#e74c3c" %)version(%%). Finally, the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 was published between April and July 2018 (see the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] website).
--The implementation of the VTL 2.0 in SDMX started in late 2018 and was published as an incremental revision to the SDMX 2.1 standards in July 2020. It allows users to write VTL 2.0 programs for validating and transforming SDMX data, to store these programs in a SDMX metadata registry and to exchange them through SDMX messages, also together the definition of the data structures of the involved data.
++The implementation of the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] started in late 2018 and was published as an incremental revision to the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 standards in July 2020. It allows users to write [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 programs for validating and transforming [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data, to store these programs in a [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] metadata registry and to exchange them through [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] messages, also together the definition of the data structures of the involved data.
--The Transformations and Expressions package for modelling VTL programs in the SDMX information model is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
++The Transformations and Expressions package for modelling [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] programs in the [[SDMX information model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.