Skip to Content
Last modified by Helena K. on 2026/06/08 15:16
From version 8.1
edited by Helena K.
on 2025/06/06 17:01
Change comment: There is no comment for this version
To version 35.3
edited by Helena K.
on 2026/06/08 12:08
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,0 +1,1 @@
1 +SDMX 3.1 Standards. Section 1. Framework
Parent
... ... @@ -1,0 +1,1 @@
1 +Methodology.WebHome
Tags
... ... @@ -1,0 +1,1 @@
1 +Adjustment|Artefact|Attribute|Bilateral exchange|Categorisation|Category|Category Scheme|Code|Codelist|Comment|Component|Concept|Concept Scheme|Constraint|Coverage|Cross-domain Concept|Currency|Data Consumer|Data Provider|Data Set|Data Source|Data Structure Definition|Data exchange|Dataflow|Dimension|Frequency of observation|Global Registry|Hierarchical Codelist|Hierarchy|Incremental update|Item Scheme|Language|Maintenance agency|Map|Measure|Metadata Set|Metadata Structure Definition|Metadata repository|Metadataflow|Notification|Provision Agreement|Reference metadata|Release policy - release calendar|Reporting Taxonomy|Representation|SDMX Information Model|SDMX Registry|SDMX Technical Specification|SDMX-CSV|SDMX-EDI|SDMX-JSON|SDMX-ML|Series|Statistical Data and Metadata eXchange|Statistical classification|Statistical subject-matter domain|Statistical unit|Structural metadata|Subscription|Timeliness|Validation and Transformation Language|Version
Content
... ... @@ -10,11 +10,11 @@
10 10  
11 11  = 1 Introduction =
12 12  
13 -The Statistical Data and Metadata Exchange (SDMX) initiative (https:~/~/www.sdmx.org) sets standards that can facilitate the exchange of statistical data and metadata using modern information technology.
13 +The Statistical Data and Metadata Exchange (SDMX) initiative ([[https:~~/~~/www.sdmx.org>>https://https:www.sdmx.org||rel="noopener noreferrer" target="_blank"]]) sets standards that can facilitate the exchange of statistical data and metadata using modern information technology.
14 14  
15 15  The SDMX Technical Specifications are organised into several discrete sections.
16 16  
17 -The following are published on the SDMX website ([[__https:~~/~~/www.sdmx.org__>>https://https:www.sdmx.org]]):
17 +The following are published on the SDMX website ([[__https:~~/~~/www.sdmx.org__>>https://https:www.sdmx.org||rel="noopener noreferrer" target="_blank"]]):
18 18  
19 19  **Section 1** **Framework for SDMX Technical Standards** – this document providing an introduction to the technical standards.
20 20  
... ... @@ -24,7 +24,7 @@
24 24  
25 25  **Section 6** **SDMX Technical Notes** – detailed technical guidance for implementors of the SDMX standard.
26 26  
27 -The following are published on the GitHub repository of the SDMX Standards Technical Working Group ([[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]]): sdmx-twg/sdmx-rest – REST API
27 +The following are published on the GitHub repository of the SDMX Standards Technical Working Group ([[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg||rel="noopener noreferrer" target="_blank"]]): sdmx-twg/sdmx-rest – REST API
28 28  
29 29  Technical specifications for the SDMX RESTful web services application programming interfaces (API).
30 30  
... ... @@ -82,6 +82,7 @@
82 82  
83 83  == 2.3 Major Changes from 2.1 to 3.0 ==
84 84  
85 +
85 85  SDMX version 3.0 introduces new features, improvements and changes to the Standard in the following key areas:
86 86  
87 87  (% class="wikigeneratedid" id="HInformationModel" %)
... ... @@ -98,7 +98,7 @@
98 98  (% class="wikigeneratedid" id="HVersioningofStructuralMetadataArtefacts" %)
99 99  **Versioning of Structural Metadata Artefacts**
100 100  
101 -• Adoption of the three-number semantic versioning standard for structural metadata artefacts ([[__https:~~/~~/semver.org__>>https://https:semver.org]])
102 +Adoption of the three-number semantic versioning standard for structural metadata artefacts ([[__https:~~/~~/semver.org__>>https://https:semver.org||rel="noopener noreferrer" target="_blank"]])
102 102  
103 103  (% class="wikigeneratedid" id="HRESTWebServicesApplicationProgrammingInterface28API29" %)
104 104  **REST Web Services Application Programming Interface (API)**
... ... @@ -111,16 +111,12 @@
111 111  (% class="wikigeneratedid" id="HSOAPWebServicesAPI" %)
112 112  **SOAP Web Services API**
113 113  
114 -• The SOAP web services API has been deprecated with version 3.0 standardising on REST
115 +* The SOAP web services API has been deprecated with version 3.0 standardising on REST
115 115  
116 116  (% class="wikigeneratedid" id="HXML2CJSON2CCSVandEDITransmissionformats" %)
117 117  **XML, JSON, CSV and EDI Transmission formats**
118 118  
119 -* The SDMX-ML, SDMX-JSON and SDMX-CSV specifications have been extended and modified where needed to support the new features and changes such as reference metadata and microdata
120 -* Obsolete SDMX-ML data message variants including Generic, Compact, Utility and Cross-sectional have been deprecated standardising on Structure Specific Data as the sole XML format for data exchange
121 -* The SDMX-EDI transmission format for structures and data has been deprecated
122 -* The organisation of structures into ‘collections’ in SDMX-ML and SDMX-JSON structure messages has been flattened and simplified
123 -* The option to reference structures in SDMX-ML and SDMX-JSON messages using Agency, ID and Version has been deprecated with URN now exclusively used for all non-local referencing purpose
120 +The SDMX-ML, SDMX-JSON and SDMX-CSV specifications have been extended and modified where needed to support the new features and changes such as reference metadata and microdata Obsolete SDMX-ML data message variants including Generic, Compact, Utility and Cross-sectional have been deprecated standardising on Structure Specific Data as the sole XML format for data exchange The SDMX-EDI transmission format for structures and data has been deprecated The organisation of structures into ‘collections’ in SDMX-ML and SDMX-JSON structure messages has been flattened and simplified The option to reference structures in SDMX-ML and SDMX-JSON messages using Agency, ID and Version has been deprecated with URN now exclusively used for all non-local referencing purpose
124 124  
125 125  Several of the changes are ‘breaking’ meaning that, in specific cases, the version 3.0 specification is not backwardly compatible with earlier versions of the Standard.
126 126  
... ... @@ -144,50 +144,42 @@
144 144  * Addition of Dimension Constraint property to a Dataflow
145 145  * Addition of evolving structure property to a Data Structure Definition
146 146  * Remove version property on Categorisation
147 -* Simplification of Constraints o Removal of Advanced Release Calendar
144 +* Simplification of Constraints
145 +** Removal of Advanced Release Calendar
146 +** Removal of Role, Data Constraints only restrict data that can be reported
147 +** Restrict constraint targets to Identifiable structures (not URLs)
148 +** Addition of Availability Constraint to define actual data
148 148  
149 -o Removal of Role, Data Constraints only restrict data that can be reported// //o Restrict constraint targets to Identifiable structures (not URLs) o Addition of Availability Constraint to define actual data
150 -
151 151  (% class="wikigeneratedid" id="HDocumentation" %)
152 152  **Documentation**
153 153  
154 -• Registering Reference Metadata removed from documentation, to align with XML Registration object which is unable to reference a Metadata Provision, and REST API which is unable to query for registered reference metadata sources.
153 +* Registering Reference Metadata removed from documentation, to align with XML Registration object which is unable to reference a Metadata Provision, and REST API which is unable to query for registered reference metadata sources.
155 155  
155 +The SDMX standards specified here are designed to support the requirements of all of these automation processes and technologies.
156 +
156 156  = 3 Processes and Business Scope =
157 157  
158 158  == 3.1 Process Patterns ==
159 159  
161 +
160 160  SDMX identifies three basic process patterns regarding the exchange of statistical data and metadata. These can be described as follows:
161 161  
162 -1. //Bilateral exchange~:// All aspects of the exchange process are agreed between counterparties, including the mechanism for exchange of data and metadata, the formats, the frequency or schedule, and the mode used for communications regarding the exchange. This is perhaps the most common process pattern.
163 -1. //Gateway exchange~:// Gateway exchanges are an organized set of bilateral exchanges, in which several data and metadata collecting organizations or individuals agree to exchange the collected information with each other in a single, known format, and according to a single, known process. This pattern has the effect of reducing the burden of managing multiple bilateral exchanges (in data and metadata collection) across the sharing organizations/individuals. This is also a very common process pattern in the statistical area, where communities of institutions agree on ways to gain efficiencies within the scope of their collective responsibilities.
164 -1. //Data-sharing exchange~:// Open, freely available data formats and process patterns are known and standard. Thus, any organization or individual can use any counterparty’s data and metadata (assuming they are permitted access to it). This model requires no bilateral agreement, but only requires that data and metadata providers and consumers adhere to the standards.
164 +1. //**Bilateral exchange**~:// All aspects of the exchange process are agreed between counterparties, including the mechanism for exchange of data and metadata, the formats, the frequency or schedule, and the mode used for communications regarding the exchange. This is perhaps the most common process pattern.
165 +1. //**Gateway exchange**~:// Gateway exchanges are an organized set of bilateral exchanges, in which several data and metadata collecting organizations or individuals agree to exchange the collected information with each other in a single, known format, and according to a single, known process. This pattern has the effect of reducing the burden of managing multiple bilateral exchanges (in data and metadata collection) across the sharing organizations/individuals. This is also a very common process pattern in the statistical area, where communities of institutions agree on ways to gain efficiencies within the scope of their collective responsibilities.
166 +1. //**Data-sharing exchange**~:// Open, freely available data formats and process patterns are known and standard. Thus, any organization or individual can use any counterparty’s data and metadata (assuming they are permitted access to it). This model requires no bilateral agreement, but only requires that data and metadata providers and consumers adhere to the standards.
165 165  
166 -This document specifies the SDMX standards designed to facilitate exchanges based on any of these process patterns, and shows how SDMX offers advantages in all cases. It is possible to agree bilaterally to use a standard format (such as SDMX-ML or SDMX-JSON); it is possible for data senders in a gateway process to use a standard format for data exchange with each other, or with any data providers who agree to do so; it is possible to agree to use the full set of SDMX standards to support a common data-sharing process of exchange, whether based on an SDMX-conformant registry or some other architecture.
167 -
168 -The standards specified here specifically support a data-sharing process based on the use of central registry services. Registry services provide visibility into the data and metadata existing within the community, and support the access and use of this data and metadata by providing a set of triggers for automated processing. The data or metadata itself is not stored in a central registry – these services merely provide a useful set of metadata about the data (and additional metadata) in a known location, so that users/applications can easily locate and obtain whatever data and/or metadata is registered. The use of standards for all data, metadata, and the registry services themselves is ubiquitous, permitting a high level of automation within a data-sharing community.
169 -
170 -It should be pointed out that these different process models are not mutually exclusive – a single system capable of expressing data and metadata in SDMX-conformant formats could support all three scenarios. Different standards may be applicable to different processes (for example, many registry services interfaces are used only in a data-sharing scenario) but all have a common basis in a shared information model.
171 -
172 -In addition to looking at collection and reporting, it is also important to consider the dissemination of data. Data and metadata – no matter how they are exchanged between counterparties in the process of their development and creation – are all eventually supplied to an end user of some type. Often, this is through specific applications inside of institutions. But more and more frequently, data and metadata are also published on websites in various formats. The dissemination of data and its accompanying metadata on the web is a focus of the SDMX standards. Standards for statistical data and metadata allow improvements in the publication of data – it becomes more easily possible to process a standard format once the data is obtained, and the data and metadata are linked together, making the comprehension and further processing of the data easier.
173 -
174 -In discussions of statistical data, there are many aspects of its dissemination which impact data quality: data discovery, ease of use, and timeliness. SDMX standards provide support for all of these aspects of data dissemination. Standard data formats promote ease of use, and provide links to relevant metadata. The concept of registry services means that data and metadata can more easily be discovered. Timeliness is improved throughout the data lifecycle by increases in efficiency, promoted through the availability of metadata and ease of use.
175 -
176 -It is important to note that SDMX is primarily focused on the //exchange// and //dissemination// of statistical data and metadata. There may also be many uses for the standard model and formats specified here in the context of internal processing of data that are not concerned with the exchange between organizations and users, however. It is felt that a clear, standard formatting of data and metadata for the purposes of exchange and dissemination can also facilitate internal processing by organizations and users, but this is not the focus of the specification.
177 -
178 178  == 3.2 SDMX and Process Automation ==
179 179  
170 +
180 180  Statistical data and metadata exchanges employ many different automated processes, but some are of more general interest than others. There are some common information technologies that are nearly ubiquitous within information systems today. SDMX aims to provide standards that are most useful for these automated processes and technologies.
181 181  
182 182  Briefly, these can be described as:
183 183  
184 -1. //Batch Exchange of Data and Metadata~:// The transmission of whole or partial databases between counterparties, including incremental updating.
185 -1. //Provision of Data and Metadata on the Internet~:// Internet technology - including its use in private or semi-private TCP/IP networks - is extremely common. This technology includes XML, JSON and REST web services as primary mechanisms for automating data and metadata provision, as well as the more traditional static HTML and database-driven publishing.
186 -1. //Generic Processes~:// While many applications and processes are specific to some set of data and metadata, other types of automated services and processes are designed to handle any type of statistical data and metadata whatsoever. This is particularly true in cases where portal sites and data feeds are made available on the Internet.
187 -1. //Presentation and Transformation of Data~:// In order to make data and metadata useful to consumers, they must support automated processes that transform them into application-specific processing formats, other standard formats, and presentational formats. Although not strictly an aspect of exchange, this type of automated processing represents a set of requirements that must be supported if the information exchange between counterparties is itself to be supported.
175 +1. //**Batch Exchange of Data and Metadata**~:// The transmission of whole or partial databases between counterparties, including incremental updating.
176 +1. //**Provision of Data and Metadata on the Internet**~:// Internet technology - including its use in private or semi-private TCP/IP networks - is extremely common. This technology includes XML, JSON and REST web services as primary mechanisms for automating data and metadata provision, as well as the more traditional static HTML and database-driven publishing.
177 +1. //**Generic Processes**~:// While many applications and processes are specific to some set of data and metadata, other types of automated services and processes are designed to handle any type of statistical data and metadata whatsoever. This is particularly true in cases where portal sites and data feeds are made available on the Internet.
178 +1. //**Presentation and Transformation of Data**~:// In order to make data and metadata useful to consumers, they must support automated processes that transform them into application-specific processing formats, other standard formats, and presentational formats. Although not strictly an aspect of exchange, this type of automated processing represents a set of requirements that must be supported if the information exchange between counterparties is itself to be supported.
188 188  
189 -The SDMX standards specified here are designed to support the requirements of all of these automation processes and technologies.
190 -
191 191  == 3.3 Statistical Data and Metadata ==
192 192  
193 193  To avoid confusion about which "data" and "metadata" are the intended content of the SDMX formats specified here, a statement of scope is offered. Statistical "data" are sets of often numeric observations which typically have time associated with them. They are associated with a set of metadata values, representing specific concepts, which act as identifiers and descriptors of the data. These metadata values and concepts can be understood as the named dimensions of a multi-dimensional co-ordinate system, describing what is often called a "cube" of data.
... ... @@ -206,8 +206,20 @@
206 206  
207 207  The formal objects in the information model are presented schematically in Figure 1, and are discussed in more detail elsewhere in this document.
208 208  
209 -[[image:SDMX_3-1-0_SECTION_1_FINAL_6728d8d4.png||height="829" width="606"]]
198 +This document specifies the SDMX standards designed to facilitate exchanges based on any of these process patterns, and shows how SDMX offers advantages in all cases. It is possible to agree bilaterally to use a standard format (such as SDMX-ML or SDMX-JSON); it is possible for data senders in a gateway process to use a standard format for data exchange with each other, or with any data providers who agree to do so; it is possible to agree to use the full set of SDMX standards to support a common data-sharing process of exchange, whether based on an SDMX-conformant registry or some other architecture.
210 210  
200 +The standards specified here specifically support a data-sharing process based on the use of central registry services. Registry services provide visibility into the data and metadata existing within the community, and support the access and use of this data and metadata by providing a set of triggers for automated processing. The data or metadata itself is not stored in a central registry – these services merely provide a useful set of metadata about the data (and additional metadata) in a known location, so that users/applications can easily locate and obtain whatever data and/or metadata is registered. The use of standards for all data, metadata, and the registry services themselves is ubiquitous, permitting a high level of automation within a data-sharing community.
201 +
202 +It should be pointed out that these different process models are not mutually exclusive – a single system capable of expressing data and metadata in SDMX-conformant formats could support all three scenarios. Different standards may be applicable to different processes (for example, many registry services interfaces are used only in a data-sharing scenario) but all have a common basis in a shared information model.
203 +
204 +In addition to looking at collection and reporting, it is also important to consider the dissemination of data. Data and metadata – no matter how they are exchanged between counterparties in the process of their development and creation – are all eventually supplied to an end user of some type. Often, this is through specific applications inside of institutions. But more and more frequently, data and metadata are also published on websites in various formats. The dissemination of data and its accompanying metadata on the web is a focus of the SDMX standards. Standards for statistical data and metadata allow improvements in the publication of data – it becomes more easily possible to process a standard format once the data is obtained, and the data and metadata are linked together, making the comprehension and further processing of the data easier.
205 +
206 +In discussions of statistical data, there are many aspects of its dissemination which impact data quality: data discovery, ease of use, and timeliness. SDMX standards provide support for all of these aspects of data dissemination. Standard data formats promote ease of use, and provide links to relevant metadata. The concept of registry services means that data and metadata can more easily be discovered. Timeliness is improved throughout the data lifecycle by increases in efficiency, promoted through the availability of metadata and ease of use.
207 +
208 +It is important to note that SDMX is primarily focused on the //exchange// and //dissemination// of statistical data and metadata. There may also be many uses for the standard model and formats specified here in the context of internal processing of data that are not concerned with the exchange between organizations and users, however. It is felt that a clear, standard formatting of data and metadata for the purposes of exchange and dissemination can also facilitate internal processing by organizations and users, but this is not the focus of the specification.
209 +
210 +[[image:SDMX%203.1%20Section%201.png]]
211 +
211 211  **Figure 1: High Level Schematic of Major Artefacts in the SDMX 3.0 Information Model**
212 212  
213 213  == 3.4 The SDMX View of Statistical Exchange ==
... ... @@ -218,8 +218,10 @@
218 218  
219 219  The first version of SDMX provided for data sets - specific statistical data reported according to a specific structure, for a specific time range - and for data structure definitions - the metadata which describes the structure of statistical data sets. These are important objects in statistical exchanges, and are retained and enhanced in the second version of the standards in a backward-compatible form. A related object in statistical exchanges is the "data flow" - this supports the concept of data reporting or dissemination on an ongoing basis. "Data flows" can be understood as data sets which are not bounded by time. Data structures are owned and maintained by agencies - in a similar fashion, data flows are owned by maintenance agencies.
220 220  
221 -SDMX allows for the publication of statistical data (and the related structural metadata) but also provided for the standard, systematic representation of reference metadata. In version 2.1, reference metadata were reported independent of the statistical data. However, in 3.0 reference metadata associated directly with data such as footnotes are reported as attributes of the data set. For other reference metadata, principally that linked to structures like “concepts”, SDMX provides reference "metadata sets", "metadata structure definitions", and "metadata flows". These objects are very similar to data sets, data structure definitions, and data flows, but concern reference metadata rather than statistical observations. In the same way that data providers may publish statistical data, they may also publish reference metadata. Metadata structural definitions are maintained by agencies in a fashion similar to the way that agencies maintain data structure definitions, the structural definitions of data sets.
222 +SDMX allows for the publication of statistical data (and the related structural metadata) but also provided for the standard, systematic representation of reference metadata. In version 2.1, reference metadata were reported independent of the statistical data.
222 222  
224 +However, in 3.0 reference metadata associated directly with data such as footnotes are reported as attributes of the data set. For other reference metadata, principally that linked to structures like “concepts”, SDMX provides reference "metadata sets", "metadata structure definitions", and "metadata flows". These objects are very similar to data sets, data structure definitions, and data flows, but concern reference metadata rather than statistical observations. In the same way that data providers may publish statistical data, they may also publish reference metadata. Metadata structural definitions are maintained by agencies in a fashion similar to the way that agencies maintain data structure definitions, the structural definitions of data sets.
225 +
223 223  The structural definitions of both data and reference metadata associate specific statistical concepts with their representations, whether textual, coded, etc. These concepts are taken from a "concept scheme" which is maintained by a specific agency. Concept schemes group a set of concepts, provide their definitions and names, and allow for semantic relationships to be expressed, when some concepts are specializations of others. It is possible for a single concept scheme to be used both for data structures - key families - and for reference metadata structures.
224 224  
225 225  Inherent in any statistical exchange – and in many dissemination activities – is a concept of "service level agreement", even if this is not formalized or made explicit. SDMX incorporates this idea in objects termed "provision agreements". Data providers may provide data to many different data flows. Data flows may incorporate data coming from more than one data provider. Provision agreements are the objects which tell you which data providers are supplying what data to which data flows. Similarly, metadata provision agreements for metadata flows.
... ... @@ -244,25 +244,16 @@
244 244  * //**Provision Agreement (Metadata Provision Agreement):**// The set of information which describes the way in which data sets and metadata sets are provided by a data/metadata provider. A provision agreement can be constrained in much the same way as a data or metadata flow definition. Thus, a data provider can express the fact that it provides a particular data flow covering a specific set of countries and topics, Importantly, the actual source of registered data or metadata is attached to the provision agreement (in terms of a URL). The term “agreement” is used because this information can be understood as the basis of a “service-level agreement”. In SDMX, however, this is informational metadata to support the technical systems, as opposed to any sort of contractual information (which is outside the scope of a technical specification). In version 3.0, metadata provision agreement and data provision agreement are two separate artefacts.
245 245  * //**Data Constraint:**// Used to restrict content (such as enumerations) and are used by provision agreements, data flows, data structure definitions in order to provide a set of reporting restrictions in the context of a collection
246 246  * //**Metadata Constraint:**// Used to restrict content (such as enumerations) and are used by metadata provision agreements, metadata flows, metadata structure definitions in order to provide a set of reporting restrictions in the context of a collection
250 +* //**Available Data Constraint:**// Used to report the set of Component values that have data reported against them in the context of a Data Query. This structure allows a user to know what valid filters can be applied to a cube of data, such that the resulting cube will contain data.
251 +* //**Structure Map: **//Structure maps describes a mapping between data structure definitions or dataflows for the purpose of transforming a data set into a different structure. The mapping rules are defined using one or more component maps which each map in turn describes how one or more components from the source data structure definition map to one or more components in that of the target. Represent maps act as lookup tables and specific provision is made for mapping dates and times.
252 +* //**Representation Map:**// Representation maps describe mappings between source value(s) and target value(s) where the values are restricted to those in a code list, value list or be of a certain type such as integer or string.
253 +* //**Item Scheme Map:**// An item scheme map describes mapping rules between any item scheme with the exception of code lists and value lists which use representation maps. The version 3.0 information model provides four item scheme maps: organisation scheme map, concept scheme map, category scheme map and reporting taxonomy map. Organisation scheme map and reporting scheme map have been omitted from the information model schematic in Figure 1.
254 +* //**Reporting Taxonomy: **//A reporting taxonomy allows an organisation to link (possibly in a hierarchical way) a number of cube or data flow definitions which together form a complete “report” of data or metadata. This supports primary reporting which often comprises multiple cubes of heterogeneous data, but may also support other collection and reporting functions. It also supports the specification of publications such as a yearbook, in terms of the data or metadata contained in the publication.
255 +* //**Process:**// The process class provides a way to model statistical processes as a set of interconnected //process steps.// Although not central to the exchange and dissemination of statistical data and metadata, having a shared description of processing allows for the interoperable exchange and dissemination of reference metadata sets which describe processes-related concepts.
256 +* //**Hierarchy**//: Describes complex code hierarchies principally for data discovery purposes. The codes themselves are referenced from the code lists in which they are maintained.
257 +* //**Hierarchy Association**//: A hierarchy association links a hierarchy to something that needs it like a dimension. Furthermore, the linking can be specified in the context of another object such as a dimension in the context of a dataflow. Thus, a dimension in a data structure definition could have different hierarchies depending on the dataflow.
258 +* //**Transformation Scheme:**// A transformation scheme is a set of Validation and Transformation Language (VTL) transformations aimed at obtaining some meaningful results for the user (e.g., the validation of one or more data sets). The set of transformations is meant to be executed together (in the same run) and may contain 597 any number of transformations in order to produce any number of results. Thus, a transformation scheme can be considered as a VTL ‘program’.
247 247  
248 -• //**Available Data Constraint:**// Used to report the set of Component values that have data reported against them in the context of a Data Query. This structure allows a user to know what valid filters can be applied to a cube of data, such that the resulting cube will contain data.
249 -
250 -• //**Structure Map: **//Structure maps describes a mapping between data structure definitions or dataflows for the purpose of transforming a data set into a different structure. The mapping rules are defined using one or more component maps which each map in turn describes how one or more components from the source data structure definition map to one or more components in that of the target. Represent maps act as lookup tables and specific provision is made for mapping dates and times.
251 -
252 -• //**Representation Map:**// Representation maps describe mappings between source value(s) and target value(s) where the values are restricted to those in a code list, value list or be of a certain type such as integer or string.
253 -
254 -• //**Item Scheme Map:**// An item scheme map describes mapping rules between any item scheme with the exception of code lists and value lists which use representation maps. The version 3.0 information model provides four item scheme maps: organisation scheme map, concept scheme map, category scheme map and reporting taxonomy map. Organisation scheme map and reporting scheme map have been omitted from the information model schematic in Figure 1.
255 -
256 -• //**Reporting Taxonomy: **//A reporting taxonomy allows an organisation to link (possibly in a hierarchical way) a number of cube or data flow definitions which together form a complete “report” of data or metadata. This supports primary reporting which often comprises multiple cubes of heterogeneous data, but may also support other collection and reporting functions. It also supports the specification of publications such as a yearbook, in terms of the data or metadata contained in the publication.
257 -
258 -• //**Process:**// The process class provides a way to model statistical processes as a set of interconnected //process steps.// Although not central to the exchange and dissemination of statistical data and metadata, having a shared description of processing allows for the interoperable exchange and dissemination of reference metadata sets which describe processes-related concepts.
259 -
260 -• //**Hierarchy**//: Describes complex code hierarchies principally for data discovery purposes. The codes themselves are referenced from the code lists in which they are maintained.
261 -
262 -• //**Hierarchy Association**//: A hierarchy association links a hierarchy to something that needs it like a dimension. Furthermore, the linking can be specified in the context of another object such as a dimension in the context of a dataflow. Thus, a dimension in a data structure definition could have different hierarchies depending on the dataflow.
263 -
264 -• //**Transformation Scheme:**// A transformation scheme is a set of Validation and Transformation Language (VTL) transformations aimed at obtaining some meaningful results for the user (e.g., the validation of one or more data sets). The set of transformations is meant to be executed together (in the same run) and may contain 597 any number of transformations in order to produce any number of results. Thus, a transformation scheme can be considered as a VTL ‘program’.
265 -
266 266  == 3.5 SDMX Registry Services ==
267 267  
268 268  In order to provide visibility into the large amount of data and metadata which exists within the SDMX model of statistical exchange, it is felt that an architecture based on a set of registry services is potentially useful. A “registry” – as understood in webservices terminology – is an application which maintains and stores metadata for querying, and which can be used by any other application in the network with sufficient access privileges (though note that the mechanism of access control is outside of the scope of the SDMX standard). It can be understood as the index of a distributed database or metadata repository which is made up of all the data provider’s data sets and reference metadata sets within a statistical community, located across the Internet or similar network.
... ... @@ -282,8 +282,10 @@
282 282  
283 283  Web services allow computer applications to exchange data directly over the Internet, essentially allowing modular or distributed computing in a more flexible fashion than ever before. In order to allow web services to function, however, many standards are required: for requesting and supplying data; for expressing the enveloping data which is used to package exchanged data; for describing web services to one another, to allow for easy integration into applications that use other web services as data resources.
284 284  
285 -Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the SDMX Technical Working Group’s GitHub repository __[[https:~~/~~/github.com/sdmx>>url:https://github.com/sdmx-twg]][[->>url:https://github.com/sdmx-twg]][[twg>>url:https://github.com/sdmx-twg]]__[[.>>url:https://github.com/sdmx-twg]] There are five ‘resources’:
279 +Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the SDMX Technical Working Group’s GitHub repository  [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg||rel="noopener noreferrer" target="_blank"]].
286 286  
281 +There are five ‘resources’:
282 +
287 287  * structure – retrieval and maintenance of structural metadata
288 288  * data – retrieval of data
289 289  * schema – retrieval of XML schemas to validate specific data or metadata sets
... ... @@ -291,10 +291,11 @@
291 291  * metadata – retrieval of reference metadata
292 292  * registration – retrieval of data locations (URL) for specific provision agreements
293 293  
294 -The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) Dataflow: https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
290 +The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) Dataflow: [[https:~~/~~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A>>https://https:ws-entry-pointdatadataflowECBEXR1.0.0M.USD.EUR.SP00.A]]
295 295  
296 296  = 4 The SDMX Information Model =
297 297  
294 +
298 298  SDMX provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because SDMX specifies a number of transmission formats for expressing data and structural metadata, the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
299 299  
300 300  SDMX recognizes that statistical data is structured; in SDMX this structure is termed a Data Structure Definition. “Data sets” are made up of one or more lower-level “groups”, based on their degrees of similarity. Each group is in turn comprised of one or more “series” of data. Each series or section has a “key” - values for each of a cluster of concepts, also called "dimensions" - which identifies it, and one or more “observations”, which typically combine the time of the observation, and the value of the observation (e.g., measurement). Additionally, metadata may be attached at any level of this structure as descriptive “attributes”. Code lists (enumerations) and other patterns for representation of data and metadata are also modelled.
... ... @@ -313,20 +313,22 @@
313 313  
314 314  == 5.1 SDMX-ML ==
315 315  
313 +
316 316  SDMX-ML is the XML transmission format specification for exchanging structural metadata, data and reference metadata, and interacting with SDMX registry services. It is designed as a general-purpose format for all automation and data / metadata exchange tasks, and provides the most complete coverage.
317 317  
318 318  There are four distinct types of message:
319 319  
320 -1. //Structure Definition~:// For the exchange of structural metadata. A SDMX-ML structure message can carry details of any number and combination of structural metadata artefacts like DSDs, code lists and constraints.
321 -1. //Structure-specific Data~:// For the exchange of data. This format is specific to the Data Structure Definitions of the data sets (in other terms, it is DSD-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large data sets in XML format, provides strict validation of conformance with the DSD using a generic XML parser, and supports the transmission of partial data sets (incremental updates) as well as whole data sets.
318 +1. //**Structure Definition**~:// For the exchange of structural metadata. A SDMX-ML structure message can carry details of any number and combination of structural metadata artefacts like DSDs, code lists and constraints.
319 +1. //**Structure-specific Data**~:// For the exchange of data. This format is specific to the Data Structure Definitions of the data sets (in other terms, it is DSD-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large data sets in XML format, provides strict validation of conformance with the DSD using a generic XML parser, and supports the transmission of partial data sets (incremental updates) as well as whole data sets.
322 322  
323 323  Many XML tools and technologies have expectations about the functions performed by an XML schema, one of which is a very direct relationship between the XML constructs described in the XML schema and the tagged data in the XML instance. Strong data typing is also considered normal, supporting full validation of the tagged data. These message types are designed to support validation and other expected XML schema functions.
324 324  
325 -1. //Generic Metadata~:// For the exchange of reference metadata sets. ‘Generic’ means the XML elements and XML attributes are the same regardless of the metadata set.
326 -1. //Registry~:// All of the possible interactions with the SDMX registry services are supported using SDMX-ML interfaces and REST API calls. Submission of structural metadata content, data registrations and subscriptions is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
323 +1. //**Generic Metadata**~:// For the exchange of reference metadata sets. ‘Generic’ means the XML elements and XML attributes are the same regardless of the metadata set.
324 +1. //**Registry**~:// All of the possible interactions with the SDMX registry services are supported using SDMX-ML interfaces and REST API calls. Submission of structural metadata content, data registrations and subscriptions is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
327 327  
328 -== {{id name="_Toc56646"/}}5.2 SDMX-JSON ==
326 +== 5.2 SDMX-JSON ==
329 329  
328 +
330 330  SDMX-JSON is the JSON transmission format specification for exchanging structural metadata, data and reference metadata. It provides an alternative to SDMX-ML and is most suited to applications like web data dissemination.
331 331  
332 332  SDMX-JSON messages serve the same function as those of the XML formats but have a different structure. For data, an important distinction is that they carry both component codes and labels which provides all the information needed to display the content in a single JSON response. The XML Structure-specific Data format by contrast carries only code IDs thus requiring applications obtain and hold structural metadata about the data set in order to display the content in human-readable form.
... ... @@ -335,72 +335,70 @@
335 335  
336 336  There are three distinct message types:
337 337  
338 -1. //Structure~:// For the exchange structural metadata. SDMX-JSON structure messages follow the same principles as for SDMX-ML in that a single message can transmit any number and combination of structural metadata artefacts. While the SDMX-ML and SDMX-JSON messages are structured differently, it is possible to freely convert between them.
339 -1. //Data: //For the exchange of data. Unlike SDMX-ML, the structure of a SDMX-JSON data message is not specific to the DSDs of the data sets so schema validation will not check for compliance of the data with the DSDs.
337 +1. //**Structure**~:// For the exchange structural metadata. SDMX-JSON structure messages follow the same principles as for SDMX-ML in that a single message can transmit any number and combination of structural metadata artefacts. While the SDMX-ML and SDMX-JSON messages are structured differently, it is possible to freely convert between them.
338 +1. //**Data**: //For the exchange of data. Unlike SDMX-ML, the structure of a SDMX-JSON data message is not specific to the DSDs of the data sets so schema validation will not check for compliance of the data with the DSDs.
340 340  1. //Metadata//: For the exchange of reference metadata sets.
341 341  
342 342  == 5.3 SDMX-CSV ==
343 343  
344 -SDMX-CSV is the CSV transmission format specification for exchanging data and reference metadata only.
343 +[[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] is the CSV transmission format specification for exchanging data and [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] only.
345 345  
346 -SDMX-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
345 +[[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
347 347  
348 348  There are two distinct message types:
349 349  
350 -1. //Data//: For the exchange of data. Like SDMX-JSON, SDMX-CSV can include both code IDs and labels which is helpful when using the data to create human readable charts and dashboards.
351 -1. //Metadata//: For the exchange of reference metadata sets.
349 +1. **//Data//**: For the exchange of data. Like [[SDMX-JSON>>doc:sdmx:Glossary 2\.1.SDMX-JSON.WebHome]], [[SDMX-CSV>>doc:sdmx:Glossary 2\.1.SDMX-CSV.WebHome]] can include both [[code>>doc:sdmx:Glossary 2\.1.Code.WebHome]] IDs and labels which is helpful when using the data to create human readable charts and dashboards.
350 +1. **//Metadata//**: For the exchange of [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] sets.
352 352  
353 353  == 5.4 Formats and Messages Deprecated in Version 3.0 ==
354 354  
355 355  The following formats and messages have been deprecated in version 3.0 to simplify, modernise and rationalise the standard.
356 356  
357 -* SDMX-EDI
358 -* SDMX-ML 1.0/2.0 Generic (time-series) data message
359 -* SDMX-ML 1.0/2.0 Compact (time-series) data message
360 -* SDMX-ML 1.0/2.0 Utility (time-series) data message
361 -* SDMX-ML 1.0/2.0 Cross-Sectional data message
362 -* SDMX-ML 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
363 -* SDMX-ML 2.1 Structure Specific Time Series data message
356 +* [[SDMX-EDI>>doc:sdmx:Glossary 2\.1.SDMX-EDI.WebHome]]
357 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
358 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
359 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data message
360 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message
361 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 2.1 Generic ([[Time Series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]) data messages (for observations, time-[[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] and cross-sectional data)
362 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] 2.1 Structure Specific [[Time Series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] data message
364 364  
365 365  The following messages were deprecated in version 3.0 as a consequence of the deprecation of the SOAP web services:
366 366  
367 -* SDMX-ML Query messages
368 -* SDMX-ML Submit Structure Request messages
366 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] Query messages
367 +* [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] Submit Structure Request messages
369 369  
370 370  = 6 Dependencies on SDMX content-oriented guidelines =
371 371  
372 -The technical standards proposed here are designed so that they can be used in conjunction with other SDMX guidelines which are more closely tied to the content and semantics of statistical data exchange. The SDMX Information Model works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, SDMX creates and maintains guidelines for cross-domain concepts, terminology, and structural definitions. There are three major parts to this effort.
371 +The technical standards proposed here are designed so that they can be used in conjunction with other [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] guidelines which are more closely tied to the content and semantics of statistical [[data exchange>>doc:sdmx:Glossary 2\.1.Data exchange.WebHome]]. The [[SDMX Information Model>>doc:sdmx:Glossary 2\.1.SDMX Information Model.WebHome]] works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] creates and maintains guidelines for [[cross-domain concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]], terminology, and structural definitions. There are three major parts to this effort.
373 373  
374 374  == 6.1 Cross-Domain Concepts ==
375 375  
376 -The SDMX Cross-Domain Concepts is a content guideline concerning concepts which are used across statistical domains. This list is expected to grow and to be subject to revision as SDMX is used in a growing number of domains. The use of the SDMX Cross-Domain Concepts, where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical concepts includes not only the definitions of the concepts, and their names, but also, where appropriate, their representation with standard code lists, and the role they play within data structure definitions and metadata structure definitions.
375 +The [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]] is a content guideline concerning [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] which are used across [[statistical domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]]. This list is expected to grow and to be subject to revision as [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] is used in a growing number of domains. The use of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]], where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] includes not only the definitions of the [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], and their names, but also, where appropriate, their [[representation>>doc:sdmx:Glossary 2\.1.Representation.WebHome]] with standard [[code lists>>doc:sdmx:Glossary 2\.1.Code list.WebHome]], and the role they play within [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] and [[metadata structure definitions>>doc:sdmx:Glossary 2\.1.Metadata structure definition.WebHome]].
377 377  
378 -The intent of this guideline is two-fold: to provide a core set of concepts which can be used to structure statistical data and metadata, to promote interoperability between systems (“structural metadata”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized concept names and definitions for other types of metadata (“reference metadata”, as defined above.)
377 +The intent of this guideline is two-fold: to provide a core set of [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] which can be used to structure statistical data and metadata, to promote interoperability between systems (“[[structural metadata>>doc:sdmx:Glossary 2\.1.Structural metadata.WebHome]]”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] names and definitions for other types of metadata (“[[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]]”, as defined above.)
379 379  
380 380  == 6.2 Metadata Common Vocabulary ==
381 381  
382 -The Metadata Common Vocabulary is an SDMX guideline which provides definition of terms to be used for the comparison and mapping of terminology found in data structure definitions and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the SDMX Technical Standards.
381 +The Metadata Common Vocabulary is an [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] guideline which provides definition of terms to be used for the comparison and mapping of terminology found in [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Technical Standards.
383 383  
384 -The MCV provides definitions for terms on which the SDMX Cross-Domain Metadata
383 +The MCV provides definitions for terms on which the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Cross-Domain Metadata [[Concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] work is built.
385 385  
386 -Concepts work is built.
387 -
388 388  == 6.3 Statistical Subject-Matter Domains ==
389 389  
390 -The Statistical Subject-Matter Domains is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard data structure definitions, concepts, etc. within their domains. Such groups already exist within the international community. SDMX would use the Statistical Subject-Matter Domains list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of SDMX-conformant technical systems within and across statistical domains. The organisation of the content of such schemes is supported in SDMX as a Category Scheme.
387 +The [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]], [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], etc. within their domains. Such groups already exist within the international community. [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] would use the [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]]-conformant technical systems within and across [[statistical domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]]. The organisation of the content of such schemes is supported in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] as a [[Category Scheme>>doc:sdmx:Glossary 2\.1.Category scheme.WebHome]].
391 391  
392 -SDMX Statistical Subject-Matter Domains will be listed and maintained by the SDMX Initiative and will be subject to adjustment.
389 +[[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary 2\.1.Statistical subject-matter domain.WebHome]] will be listed and maintained by the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Initiative and will be subject to [[adjustment>>doc:sdmx:Glossary 2\.1.Adjustment.WebHome]].
393 393  
394 394  == 6.4 SDMX Concept Roles ==
395 395  
396 -These guidelines define the standard set of SDMX Concept Roles and their use. This set of standard SDMX Concepts are implemented as a cross-domain Concept Scheme that defines the set of concept roles and gives examples on concept role implementation in SDMX 2.0, 2.1 and 3.0. A concept role gives a particular context to a concept for easy and systematic interpretation by machine processing and visualization tools. For example, the concepts REPORTING_AREA and COUNTERPART_AREA are different concepts but they are both geographical characteristics, therefore they can be associated with the same concept role ID: "GEO". This allows visualization systems to interpret these concepts as geographical data in order to generate maps. The implementation of concept roles is different in versions 2.0 and 2.1/3.0 of the SDMX technical standard. Specifically for SDMX 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in SDMX 3.0, i.e., via Concept Schemes; the semantics of these roles have to be agreed bilateraly in data exchanges. The Concept Roles are available as an SDMX Concept Scheme on the SDMX Global Registry.
393 +These guidelines define the standard set of [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] Roles and their use. This set of standard [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] are implemented as a [[cross-domain Concept>>doc:sdmx:Glossary 2\.1.Cross-domain concept.WebHome]] Scheme that defines the set of [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] roles and gives examples on [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role implementation in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.0, 2.1 and 3.0. A [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role gives a particular context to a [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] for easy and systematic interpretation by machine processing and visualization tools. For example, the [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] REPORTING_AREA and COUNTERPART_AREA are different [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] but they are both geographical characteristics, therefore they can be associated with the same [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] role ID: "GEO". This allows visualization systems to interpret these [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] as geographical data in order to generate [[maps>>doc:sdmx:Glossary 2\.1.Map.WebHome]]. The implementation of [[concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] roles is different in [[versions>>doc:sdmx:Glossary 2\.1.Version.WebHome]] 2.0 and 2.1/3.0 of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] technical standard. Specifically for [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 3.0, i.e., via [[Concept Schemes>>doc:sdmx:Glossary 2\.1.Concept scheme.WebHome]]; the semantics of these roles have to be agreed bilateraly in [[data exchanges>>doc:sdmx:Glossary 2\.1.Data exchange.WebHome]]. The [[Concept>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] Roles are available as an [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Concept Scheme>>doc:sdmx:Glossary 2\.1.Concept scheme.WebHome]] on the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[Global Registry>>doc:sdmx:Glossary 2\.1.Global registry.WebHome]].
397 397  
398 398  = 7 Validation and Transformation Language =
399 399  
400 -For many years the SDMX initiative has been fostering and supporting the development of a standard calculation language, called Validation and Transformation Language (VTL). A blueprint for defining calculations was already described in the original SDMX 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate SDMX artefacts.
397 +For many years the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] initiative has been fostering and supporting the development of a standard calculation [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]], called [[Validation and Transformation Language>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]]). A blueprint for defining calculations was already described in the original [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]] able to manipulate [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary 2\.1.Artefact.WebHome]].
401 401  
402 -These developments started in late 2012 and were put in charge of the Validation and Transformation Language Task Force (VTL TF), which included members of the SDMX Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in SDMX and applicable also to GSIM and DDI. This brought to the publication of the VTL 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of VTL 1.0 made by several organisations and triggered a large improvement of the language. A new provisional version, the VTL 1.1, was released in public consultation in 2017. The high number of comments received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the VTL 2.0 was published between April and July 2018 (see the SDMX website).
399 +These developments started in late 2012 and were put in charge of the [[Validation and Transformation Language>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] Task Force ([[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] TF), which included members of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]] to be implemented in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] and applicable also to GSIM and DDI. This brought to the publication of the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.0 made by several organisations and triggered a large improvement of the [[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]]. A new provisional version, the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 1.1, was released in public consultation in 2017. The high number of [[comments>>doc:sdmx:Glossary 2\.1.Comment.WebHome]] received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 was published between April and July 2018 (see the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] website).
403 403  
404 -The implementation of the VTL 2.0 in SDMX started in late 2018 and was published as an incremental revision to the SDMX 2.1 standards in July 2020. It allows users to write VTL 2.0 programs for validating and transforming SDMX data, to store these programs in a SDMX metadata registry and to exchange them through SDMX messages, also together the definition of the data structures of the involved data.
401 +The implementation of the [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 in [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] started in late 2018 and was published as an incremental revision to the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] 2.1 standards in July 2020. It allows users to write [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] 2.0 programs for validating and transforming [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] data, to store these programs in a [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] metadata registry and to exchange them through [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] messages, also together the definition of the data structures of the involved data.
405 405  
406 -The Transformations and Expressions package for modelling VTL programs in the SDMX information model is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
403 +The Transformations and Expressions package for modelling [[VTL>>doc:sdmx:Glossary 2\.1.Validation and transformation language.WebHome]] programs in the [[SDMX information model>>doc:sdmx:Glossary 2\.1.SDMX Information Model.WebHome]] is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
SDMX_3-1-0_SECTION_1_FINAL_6728d8d4.png
Author
... ... @@ -1,1 +1,0 @@
1 -xwiki:XWiki.helena
Size
... ... @@ -1,1 +1,0 @@
1 -403.7 KB
Content
SDMX 3.1 Section 1.png
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.arturkryazhev
Size
... ... @@ -1,0 +1,1 @@
1 +101.7 KB
Content
SDMX%203.1%20Section%201.png
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.arturkryazhev
Size
... ... @@ -1,0 +1,1 @@
1 +101.7 KB
Content
SUZ.Methodology.Code.MethodologyClass[0]
SKMS.Methodology.Code.MethodologyClass[0]
© Semantic R&D Group, 2026