Skip to Content
Last modified by Artur on 2025/09/30 12:30
From version 14.2
edited by Helena
on 2025/06/06 17:25
Change comment: There is no comment for this version
To version 12.4
edited by Helena
on 2025/06/06 17:14
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -98,7 +98,7 @@
98 98  (% class="wikigeneratedid" id="HVersioningofStructuralMetadataArtefacts" %)
99 99  **Versioning of Structural Metadata Artefacts**
100 100  
101 -Adoption of the three-number semantic versioning standard for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] ([[__https:~~/~~/semver.org__>>https://https:semver.org]])
101 +• Adoption of the three-number semantic versioning standard for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] ([[__https:~~/~~/semver.org__>>https://https:semver.org]])
102 102  
103 103  (% class="wikigeneratedid" id="HRESTWebServicesApplicationProgrammingInterface28API29" %)
104 104  **REST Web Services Application Programming Interface (API)**
... ... @@ -108,107 +108,91 @@
108 108  * Improvements to [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] queries
109 109  * Support for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] maintenance using HTTP PUT, POST and DELETE verbs
110 110  
111 -(% class="wikigeneratedid" id="HSOAPWebServicesAPI" %)
112 -**SOAP Web Services API**
113 113  
114 -* The SOAP web services API has been deprecated with (% style="color:#e74c3c" %)version(%%) 3.0 standardising on REST
115 115  
116 -(% class="wikigeneratedid" id="HXML2CJSON2CCSVandEDITransmissionformats" %)
117 -**XML, JSON, CSV and EDI Transmission formats**
113 +The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards specified here are designed to support the requirements of all of these automation processes and technologies.
118 118  
119 -The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]], [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] and [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV specifications have been extended and modified where needed to support the new features and changes such as [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and microdata Obsolete [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] data message variants including Generic, Compact, Utility and Cross-sectional have been deprecated standardising on Structure Specific Data as the sole XML format for data exchange The [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] transmission format for structures and data has been deprecated The organisation of structures into ‘collections’ in [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] structure messages has been flattened and simplified The option to reference structures in [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages using Agency, ID and (% style="color:#e74c3c" %)Version(%%) has been deprecated with URN now exclusively used for all non-local referencing purpose
115 +== 3.3 Statistical Data and Metadata ==
120 120  
121 -Several of the changes are ‘breaking’ meaning that, in specific cases, the (% style="color:#e74c3c" %)version(%%) 3.0 specification is not backwardly compatible with earlier (% style="color:#e74c3c" %)versions(%%) of the Standard.
117 +To avoid confusion about which "data" and "metadata" are the intended content of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] formats specified here, a statement of scope is offered. Statistical "data" are sets of often numeric observations which typically have time associated with them. They are associated with a set of metadata values, representing specific (% style="color:#e74c3c" %)concepts(%%), which act as identifiers and descriptors of the data. These metadata values and (% style="color:#e74c3c" %)concepts(%%) can be understood as the named [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] of a multi-dimensional co-ordinate system, describing what is often called a "cube" of data.
122 122  
123 -The principle breaking changes are:
119 +[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifies a standard technique for modelling, expressing, and understanding the structure of this multi-dimensional "cube", allowing automated processing of data from a variety of sources. This approach is widely applicable across types of data and attempts to provide the simplest and most easily comprehensible technique that will support the exchange of this broad set of data and related metadata.
124 124  
125 -* REST API – The REST API is not backwardly compatible due to modifications to the URLs and query parameters resulting in breaking changes in four of the five main resources.
126 -* SOAP API – Deprecation of the SOAP API means that existing systems designed to use SOAP will not work with (% style="color:#e74c3c" %)version(%%) 3.0 registries.
127 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] – [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 and earlier structure, data and metadata XML messages are not valid in (% style="color:#e74c3c" %)version(%%) 3.0. Specifically: legacy data messages including Generic, Compact and Utility are no longer supported. The remaining Structure Specific data message has been changed to support new features such as reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] as part of the [[dataset>>doc:sdmx:Glossary.Data set.WebHome]], Structure messages have a number of breaking changes, principally modification to the information model, removal of the agency-(% style="color:#e74c3c" %)version(%%)-id option for referencing [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] and changes to the way the structures are organised into ‘collections’ within the message.
128 -* [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] – [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 structure, data and metadata JSON messages are not valid in (% style="color:#e74c3c" %)version(%%) 3.0. The data message has been changed to support the improved REST data queries, in particular the ability to retrieve in one operation data from multiple [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] with potentially different [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]]. Breaking changes similar to those for the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] transmission format have been made to the structure message.
129 -* [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV - The CSV data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] messages are not backwardly compatible with those under (% style="color:#e74c3c" %)version(%%) 2.1 due to changes to the structure of the messages needed to support new features such as the improved REST API data queries.
130 -* [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] – Deprecation of the EDI transmission format means that existing systems designed to send or receive structures or data in EDI will not work with (% style="color:#e74c3c" %)version(%%) 3.0 registries.
131 -* Information Model – Several structures have been changed in the (% style="color:#e74c3c" %)version(%%) 3.0 model and three removed. For these reasons the (% style="color:#e74c3c" %)version(%%) 3.0 model is not directly compatible with (% style="color:#e74c3c" %)version(%%) 2.1 or earlier, although conversion of specific [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] is possible under some circumstances. Loss of information during the conversion process however means that in cases like structure mapping, the conversion is not reversible i.e. it is not possible to recreate the 2.1 structure once it has been converted to the 3.0 model.
121 +The term "metadata" is very broad indeed. A distinction can be made between “structural” metadata – those (% style="color:#e74c3c" %)concepts(%%) used in the description and identification of statistical data and metadata – and “reference” metadata – the larger set of (% style="color:#e74c3c" %)concepts(%%) that describe and qualify statistical [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and processing more generally, and which are often associated not with specific observations or series of data, but with entire collections of data or even the institutions which provide that data.
132 132  
133 -The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0 Major Changes document provides more information including an analysis of the breaking changes.
123 +The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for the structuring not only of data, but also of “reference” metadata. While these [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] structures exist independent of the data and its [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], they are often linked. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for the attachment of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] to any part of the data or [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], as well as for the reporting and exchange of the [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and its structural descriptions. This function of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards supports many aspects of data quality initiatives, allowing as it does for the exchange of metadata in its broadest sense, of which quality-related metadata is a major part.
134 134  
135 -== 2.4 Major Changes from 3.0 to 3.1 ==
125 +Metadata are associated not only with data, but also with the process of providing and managing the flow of data. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for a set of metadata concerned with “data provisioning” – metadata which are useful to those who need to understand the content and form of a [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]]’s output. Each [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]] can describe in standard fashion the content of and dependencies within the data and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] which they produce, and supply information about the scheduling and mechanism by which their data and metadata are provided. This allows for automation of some validation and control functions, as well as supporting management of data reporting.
136 136  
137 -(% class="wikigeneratedid" id="HInformationModel-1" %)
138 -**Information Model**
127 +[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] also recognizes the importance of classification schemes in organizing and managing the exchange and dissemination of data and metadata. It is possible to express information about classification schemes and domain [[categories>>doc:sdmx:Glossary.Category.WebHome]] in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]], along with their relationships to data and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]], as well as to categorize other objects in the model.
139 139  
140 -* Addition of [[Dimension>>doc:sdmx:Glossary.Dimension.WebHome]] (% style="color:#e74c3c" %)Constraint(%%) property to a Dataflow
141 -* Addition of evolving structure property to a Data Structure Definition
142 -* Remove (% style="color:#e74c3c" %)version(%%) property on Categorisation
143 -* Simplification of (% style="color:#e74c3c" %)Constraints
144 -** Removal of Advanced Release Calendar
145 -** Removal of Role, Data (% style="color:#e74c3c" %)Constraints(%%) only restrict data that can be reported
146 -** Restrict (% style="color:#e74c3c" %)constraint(%%) targets to Identifiable structures (not URLs)
147 -** Addition of Availability (% style="color:#e74c3c" %)Constraint(%%) to define actual data
129 +The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards offer a common model, a choice of syntax and, for XML, a choice of data formats which support the exchange of any type of statistical data meeting the definition above; several optimized formats are specified based on the specific requirements of each implementation, as described below in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] section.
148 148  
149 -(% class="wikigeneratedid" id="HDocumentation" %)
150 -**Documentation**
131 +The formal objects in the information model are presented schematically in Figure 1, and are discussed in more detail elsewhere in this document.
151 151  
152 -* Registering [[Reference Metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] removed from documentation, to align with XML Registration object which is unable to reference a Metadata Provision, and REST API which is unable to query for registered [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sources.
133 +This document specifies the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards designed to facilitate exchanges based on any of these process patterns, and shows how [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] offers advantages in all cases. It is possible to agree bilaterally to use a standard format (such as [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] or [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]]); it is possible for data senders in a gateway process to use a standard format for data exchange with each other, or with any [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]] who agree to do so; it is possible to agree to use the full set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards to support a common data-sharing process of exchange, whether based on an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant registry or some other architecture.
153 153  
154 -The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards specified here are designed to support the requirements of all of these automation processes and technologies.
135 +The standards specified here specifically support a data-sharing process based on the use of central registry services. Registry services provide visibility into the data and metadata existing within the community, and support the access and use of this data and metadata by providing a set of triggers for automated processing. The data or metadata itself is not stored in a central registry – these services merely provide a useful set of metadata about the data (and additional metadata) in a known location, so that users/applications can easily locate and obtain whatever data and/or metadata is registered. The use of standards for all data, metadata, and the registry services themselves is ubiquitous, permitting a high (% style="color:#e74c3c" %)level(%%) of automation within a data-sharing community.
155 155  
156 -= 3 Processes and Business Scope =
137 +It should be pointed out that these different process models are not mutually exclusive – a single system capable of expressing data and metadata in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant formats could support all three scenarios. Different standards may be applicable to different processes (for example, many registry services interfaces are used only in a data-sharing scenario) but all have a common basis in a shared information model.
157 157  
158 -== 3.1 Process Patterns ==
139 +In addition to looking at collection and reporting, it is also important to consider the dissemination of data. Data and metadata – no matter how they are exchanged between counterparties in the process of their development and creation – are all eventually supplied to an end user of some type. Often, this is through specific applications inside of institutions. But more and more frequently, data and metadata are also published on websites in various formats. The dissemination of data and its accompanying metadata on the web is a focus of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards. Standards for statistical data and metadata allow improvements in the publication of data – it becomes more easily possible to process a standard format once the data is obtained, and the data and metadata are linked together, making the comprehension and further processing of the data easier.
159 159  
160 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifies three basic process patterns regarding the exchange of statistical data and metadata. These can be described as follows:
141 +In discussions of statistical data, there are many aspects of its dissemination which impact data quality: data discovery, ease of use, and [[timeliness>>doc:sdmx:Glossary.Timeliness.WebHome]]. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards provide support for all of these aspects of data dissemination. Standard data formats promote ease of use, and provide links to relevant metadata. The (% style="color:#e74c3c" %)concept(%%) of registry services means that data and metadata can more easily be discovered. [[Timeliness>>doc:sdmx:Glossary.Timeliness.WebHome]] is improved throughout the data lifecycle by increases in efficiency, promoted through the availability of metadata and ease of use.
161 161  
162 -1. //**Bilateral exchange**~:// All aspects of the exchange process are agreed between counterparties, including the mechanism for exchange of data and metadata, the formats, the frequency or schedule, and the mode used for communications regarding the exchange. This is perhaps the most common process pattern.
163 -1. //**Gateway exchange**~:// Gateway exchanges are an organized set of bilateral exchanges, in which several data and metadata collecting organizations or individuals agree to exchange the collected information with each other in a single, known format, and according to a single, known process. This pattern has the effect of reducing the burden of managing multiple bilateral exchanges (in data and metadata collection) across the sharing organizations/individuals. This is also a very common process pattern in the statistical area, where communities of institutions agree on ways to gain efficiencies within the scope of their collective responsibilities.
164 -1. //**Data-sharing exchange**~:// Open, freely available data formats and process patterns are known and standard. Thus, any organization or individual can use any counterparty’s data and metadata (assuming they are permitted access to it). This model requires no bilateral agreement, but only requires that data and metadata providers and consumers adhere to the standards.
143 +It is important to note that [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is primarily focused on the //exchange// and //dissemination// of statistical data and metadata. There may also be many uses for the standard model and formats specified here in the context of internal processing of data that are not concerned with the exchange between organizations and users, however. It is felt that a clear, standard formatting of data and metadata for the purposes of exchange and dissemination can also facilitate internal processing by organizations and users, but this is not the focus of the specification.
165 165  
145 +== 3.2 SDMX and Process Automation ==
166 166  
147 +Statistical data and metadata exchanges employ many different automated processes, but some are of more general interest than others. There are some common information technologies that are nearly ubiquitous within information systems today. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] aims to provide standards that are most useful for these automated processes and technologies.
167 167  
168 -[[image:SDMX 3.1 Section 1.png]]
149 +Briefly, these can be described as:
169 169  
170 -**Figure 1: High Level Schematic of Major Artefacts in the SDMX 3.0 Information Model**
151 +//Batch Exchange of Data and Metadata~:// The transmission of whole or partial databases between counterparties, including incremental updating. //Provision of Data and Metadata on the Internet~:// Internet technology - including its use in private or semi-private TCP/IP networks - is extremely common. This technology includes XML, JSON and REST web services as primary mechanisms for automating data and metadata provision, as well as the more traditional static HTML and database-driven publishing. //Generic Processes~:// While many applications and processes are specific to some set of data and metadata, other types of automated services and processes are designed to handle any type of statistical data and metadata whatsoever. This is particularly true in cases where portal sites and data feeds are made available on the Internet. //Presentation and Transformation of Data~:// In order to make data and metadata useful to consumers, they must support automated processes that transform them into application-specific processing formats, other standard formats, and presentational formats. Although not strictly an aspect of exchange, this type of automated processing represents a set of requirements that must be supported if the information exchange between counterparties is itself to be supported.
171 171  
153 +o Removal of Role, Data (% style="color:#e74c3c" %)Constraints(%%) only restrict data that can be reported// //o Restrict (% style="color:#e74c3c" %)constraint(%%) targets to Identifiable structures (not URLs) o Addition of Availability (% style="color:#e74c3c" %)Constraint(%%) to define actual data
172 172  
173 -== 3.2 SDMX and Process Automation ==
155 +(% class="wikigeneratedid" id="HDocumentation" %)
156 +**Documentation**
174 174  
175 -Statistical data and metadata exchanges employ many different automated processes, but some are of more general interest than others. There are some common information technologies that are nearly ubiquitous within information systems today. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] aims to provide standards that are most useful for these automated processes and technologies.
158 +• Registering [[Reference Metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] removed from documentation, to align with XML Registration object which is unable to reference a Metadata Provision, and REST API which is unable to query for registered [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sources.
176 176  
177 -Briefly, these can be described as:
160 += 3 Processes and Business Scope =
178 178  
179 -1. //**Batch Exchange of Data and Metadata**~:// The transmission of whole or partial databases between counterparties, including incremental updating.
180 -1. //**Provision of Data and Metadata on the Internet**~:// Internet technology - including its use in private or semi-private TCP/IP networks - is extremely common. This technology includes XML, JSON and REST web services as primary mechanisms for automating data and metadata provision, as well as the more traditional static HTML and database-driven publishing. //Generic Processes~:// While many applications and processes are specific to some set of data and metadata, other types of automated services and processes are designed to handle any type of statistical data and metadata whatsoever. This is particularly true in cases where portal sites and data feeds are made available on the Internet.
181 -1. //**Presentation and Transformation of Data**~:// In order to make data and metadata useful to consumers, they must support automated processes that transform them into application-specific processing formats, other standard formats, and presentational formats. Although not strictly an aspect of exchange, this type of automated processing represents a set of requirements that must be supported if the information exchange between counterparties is itself to be supported.
162 +== 3.1 Process Patterns ==
182 182  
183 -== 3.3 Statistical Data and Metadata ==
164 +[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifies three basic process patterns regarding the exchange of statistical data and metadata. These can be described as follows:
184 184  
185 -To avoid confusion about which "data" and "metadata" are the intended content of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] formats specified here, a statement of scope is offered. Statistical "data" are sets of often numeric observations which typically have time associated with them. They are associated with a set of metadata values, representing specific (% style="color:#e74c3c" %)concepts(%%), which act as identifiers and descriptors of the data. These metadata values and (% style="color:#e74c3c" %)concepts(%%) can be understood as the named [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] of a multi-dimensional co-ordinate system, describing what is often called a "cube" of data.
166 +//Bilateral exchange~:// All aspects of the exchange process are agreed between counterparties, including the mechanism for exchange of data and metadata, the formats, the frequency or schedule, and the mode used for communications regarding the exchange. This is perhaps the most common process pattern. //Gateway exchange~:// Gateway exchanges are an organized set of bilateral exchanges, in which several data and metadata collecting organizations or individuals agree to exchange the collected information with each other in a single, known format, and according to a single, known process. This pattern has the effect of reducing the burden of managing multiple bilateral exchanges (in data and metadata collection) across the sharing organizations/individuals. This is also a very common process pattern in the statistical area, where communities of institutions agree on ways to gain efficiencies within the scope of their collective responsibilities. //Data-sharing exchange~:// Open, freely available data formats and process patterns are known and standard. Thus, any organization or individual can use any counterparty’s data and metadata (assuming they are permitted access to it). This model requires no bilateral agreement, but only requires that data and metadata providers and consumers adhere to the standards.
186 186  
187 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifies a standard technique for modelling, expressing, and understanding the structure of this multi-dimensional "cube", allowing automated processing of data from a variety of sources. This approach is widely applicable across types of data and attempts to provide the simplest and most easily comprehensible technique that will support the exchange of this broad set of data and related metadata.
168 +The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0 Major Changes document provides more information including an analysis of the breaking changes.
188 188  
189 -The term "metadata" is very broad indeed. A distinction can be made between “structural” metadata – those (% style="color:#e74c3c" %)concepts(%%) used in the description and identification of statistical data and metadata – and “reference” metadata – the larger set of (% style="color:#e74c3c" %)concepts(%%) that describe and qualify statistical [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and processing more generally, and which are often associated not with specific observations or series of data, but with entire collections of data or even the institutions which provide that data.
170 +== 2.4 Major Changes from 3.0 to 3.1 ==
190 190  
191 -The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for the structuring not only of data, but also of “reference” metadata. While these [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] structures exist independent of the data and its [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], they are often linked. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for the attachment of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] to any part of the data or [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], as well as for the reporting and exchange of the [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and its structural descriptions. This function of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards supports many aspects of data quality initiatives, allowing as it does for the exchange of metadata in its broadest sense, of which quality-related metadata is a major part.
172 +(% class="wikigeneratedid" id="HInformationModel-1" %)
173 +**Information Model**
192 192  
193 -Metadata are associated not only with data, but also with the process of providing and managing the flow of data. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides for a set of metadata concerned with “data provisioning” – metadata which are useful to those who need to understand the content and form of a [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]]’s output. Each [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]] can describe in standard fashion the content of and dependencies within the data and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] which they produce, and supply information about the scheduling and mechanism by which their data and metadata are provided. This allows for automation of some validation and control functions, as well as supporting management of data reporting.
175 +Addition of [[Dimension>>doc:sdmx:Glossary.Dimension.WebHome]] (% style="color:#e74c3c" %)Constraint(%%) property to a Dataflow Addition of evolving structure property to a Data Structure Definition Remove (% style="color:#e74c3c" %)version(%%) property on Categorisation Simplification of (% style="color:#e74c3c" %)Constraints(%%) o Removal of Advanced Release Calendar
194 194  
195 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] also recognizes the importance of classification schemes in organizing and managing the exchange and dissemination of data and metadata. It is possible to express information about classification schemes and domain [[categories>>doc:sdmx:Glossary.Category.WebHome]] in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]], along with their relationships to data and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]], as well as to categorize other objects in the model.
177 +Several of the changes are ‘breaking’ meaning that, in specific cases, the (% style="color:#e74c3c" %)version(%%) 3.0 specification is not backwardly compatible with earlier (% style="color:#e74c3c" %)versions(%%) of the Standard.
196 196  
197 -The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards offer a common model, a choice of syntax and, for XML, a choice of data formats which support the exchange of any type of statistical data meeting the definition above; several optimized formats are specified based on the specific requirements of each implementation, as described below in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] section.
179 +The principle breaking changes are:
198 198  
199 -The formal objects in the information model are presented schematically in Figure 1, and are discussed in more detail elsewhere in this document.
181 +REST API – The REST API is not backwardly compatible due to modifications to the URLs and query parameters resulting in breaking changes in four of the five main resources. SOAP API – Deprecation of the SOAP API means that existing systems designed to use SOAP will not work with (% style="color:#e74c3c" %)version(%%) 3.0 registries. [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] – [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 and earlier structure, data and metadata XML messages are not valid in (% style="color:#e74c3c" %)version(%%) 3.0. Specifically: legacy data messages including Generic, Compact and Utility are no longer supported. The remaining Structure Specific data message has been changed to support new features such as reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] as part of the [[dataset>>doc:sdmx:Glossary.Data set.WebHome]], Structure messages have a number of breaking changes, principally modification to the information model, removal of the agency-(% style="color:#e74c3c" %)version(%%)-id option for referencing [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] and changes to the way the structures are organised into ‘collections’ within the message. [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] – [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 structure, data and metadata JSON messages are not valid in (% style="color:#e74c3c" %)version(%%) 3.0. The data message has been changed to support the improved REST data queries, in particular the ability to retrieve in one operation data from multiple [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] with potentially different [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]]. Breaking changes similar to those for the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] transmission format have been made to the structure message. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV - The CSV data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] messages are not backwardly compatible with those under (% style="color:#e74c3c" %)version(%%) 2.1 due to changes to the structure of the messages needed to support new features such as the improved REST API data queries. [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] – Deprecation of the EDI transmission format means that existing systems designed to send or receive structures or data in EDI will not work with (% style="color:#e74c3c" %)version(%%) 3.0 registries. Information Model – Several structures have been changed in the (% style="color:#e74c3c" %)version(%%) 3.0 model and three removed. For these reasons the (% style="color:#e74c3c" %)version(%%) 3.0 model is not directly compatible with (% style="color:#e74c3c" %)version(%%) 2.1 or earlier, although conversion of specific [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] is possible under some circumstances. Loss of information during the conversion process however means that in cases like structure mapping, the conversion is not reversible i.e. it is not possible to recreate the 2.1 structure once it has been converted to the 3.0 model.
200 200  
201 -This document specifies the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards designed to facilitate exchanges based on any of these process patterns, and shows how [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] offers advantages in all cases. It is possible to agree bilaterally to use a standard format (such as [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] or [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]]); it is possible for data senders in a gateway process to use a standard format for data exchange with each other, or with any [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]] who agree to do so; it is possible to agree to use the full set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards to support a common data-sharing process of exchange, whether based on an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant registry or some other architecture.
183 +(% class="wikigeneratedid" id="HSOAPWebServicesAPI" %)
184 +**SOAP Web Services API**
202 202  
203 -The standards specified here specifically support a data-sharing process based on the use of central registry services. Registry services provide visibility into the data and metadata existing within the community, and support the access and use of this data and metadata by providing a set of triggers for automated processing. The data or metadata itself is not stored in a central registry – these services merely provide a useful set of metadata about the data (and additional metadata) in a known location, so that users/applications can easily locate and obtain whatever data and/or metadata is registered. The use of standards for all data, metadata, and the registry services themselves is ubiquitous, permitting a high (% style="color:#e74c3c" %)level(%%) of automation within a data-sharing community.
186 +• The SOAP web services API has been deprecated with (% style="color:#e74c3c" %)version(%%) 3.0 standardising on REST
204 204  
205 -It should be pointed out that these different process models are not mutually exclusive – a single system capable of expressing data and metadata in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant formats could support all three scenarios. Different standards may be applicable to different processes (for example, many registry services interfaces are used only in a data-sharing scenario) but all have a common basis in a shared information model.
188 +(% class="wikigeneratedid" id="HXML2CJSON2CCSVandEDITransmissionformats" %)
189 +**XML, JSON, CSV and EDI Transmission formats**
206 206  
207 -In addition to looking at collection and reporting, it is also important to consider the dissemination of data. Data and metadata – no matter how they are exchanged between counterparties in the process of their development and creation – are all eventually supplied to an end user of some type. Often, this is through specific applications inside of institutions. But more and more frequently, data and metadata are also published on websites in various formats. The dissemination of data and its accompanying metadata on the web is a focus of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards. Standards for statistical data and metadata allow improvements in the publication of data – it becomes more easily possible to process a standard format once the data is obtained, and the data and metadata are linked together, making the comprehension and further processing of the data easier.
191 +The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]], [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] and [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV specifications have been extended and modified where needed to support the new features and changes such as [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and microdata Obsolete [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] data message variants including Generic, Compact, Utility and Cross-sectional have been deprecated standardising on Structure Specific Data as the sole XML format for data exchange The [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] transmission format for structures and data has been deprecated The organisation of structures into ‘collections’ in [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] structure messages has been flattened and simplified The option to reference structures in [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages using Agency, ID and (% style="color:#e74c3c" %)Version(%%) has been deprecated with URN now exclusively used for all non-local referencing purpose
208 208  
209 -In discussions of statistical data, there are many aspects of its dissemination which impact data quality: data discovery, ease of use, and [[timeliness>>doc:sdmx:Glossary.Timeliness.WebHome]]. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards provide support for all of these aspects of data dissemination. Standard data formats promote ease of use, and provide links to relevant metadata. The (% style="color:#e74c3c" %)concept(%%) of registry services means that data and metadata can more easily be discovered. [[Timeliness>>doc:sdmx:Glossary.Timeliness.WebHome]] is improved throughout the data lifecycle by increases in efficiency, promoted through the availability of metadata and ease of use.
193 +[[image:SDMX 3.1 Section 1.png]]
210 210  
211 -It is important to note that [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is primarily focused on the //exchange// and //dissemination// of statistical data and metadata. There may also be many uses for the standard model and formats specified here in the context of internal processing of data that are not concerned with the exchange between organizations and users, however. It is felt that a clear, standard formatting of data and metadata for the purposes of exchange and dissemination can also facilitate internal processing by organizations and users, but this is not the focus of the specification.
195 +**Figure 1: High Level Schematic of Major Artefacts in the SDMX 3.0 Information Model**
212 212  
213 213  == 3.4 The SDMX View of Statistical Exchange ==
214 214  
... ... @@ -273,32 +273,30 @@
273 273  
274 274  Web services allow computer applications to exchange data directly over the Internet, essentially allowing modular or distributed computing in a more flexible fashion than ever before. In order to allow web services to function, however, many standards are required: for requesting and supplying data; for expressing the enveloping data which is used to package exchanged data; for describing web services to one another, to allow for easy integration into applications that use other web services as data resources.
275 275  
276 -Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group’s GitHub repository  [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]].
260 +Version 3.1 has standardized on RESTful web services with a OpenAPI specification published on the SDMX Technical Working Group’s GitHub repository [[__https:~~/~~/github.com/sdmx-twg__>>https://https:github.comsdmx-twg]]. There are five ‘resources’:
277 277  
278 -There are five ‘resources’:
279 -
280 280  * structure – retrieval and maintenance of structural metadata
281 281  * data – retrieval of data
282 282  * schema – retrieval of XML schemas to validate specific data or metadata sets
283 -* availability – retrieval of information on the data available for a [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]
265 +* availability – retrieval of information on the data available for a Dataflow
284 284  * metadata – retrieval of reference metadata
285 285  * registration – retrieval of data locations (URL) for specific provision agreements
286 286  
287 -The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]: https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
269 +The following conceptual example uses the ‘data’ resource to query a data repository for a series identified by the key ‘M.USD.EUR.SP00.A’ in the EXR (ECB exchange rates) Dataflow: https:~/~/ws-entry-point/data/dataflow/ECB/EXR/1.0.0/M.USD.EUR.SP00.A
288 288  
289 289  = 4 The SDMX Information Model =
290 290  
291 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] specifies a number of transmission formats for expressing data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
273 +SDMX provides a way of modelling statistical data, and defines the set of metadata constructs used for this purpose. Because SDMX specifies a number of transmission formats for expressing data and structural metadata, the model is used as a mechanism for guaranteeing that transformation between the different formats is lossless. In this sense, all of the formats are syntax-bound expressions of the common information model.
292 292  
293 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] recognizes that statistical data is structured; in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] this structure is termed a [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. “[[Data sets>>doc:sdmx:Glossary.Data set.WebHome]]” are made up of one or more lower-(% style="color:#e74c3c" %)level(%%) “groups”, based on their degrees of similarity. Each group is in turn comprised of one or more “series” of data. Each series or section has a “key” - values for each of a cluster of (% style="color:#e74c3c" %)concepts(%%), also called "[[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]]" - which identifies it, and one or more “observations”, which typically combine the time of the observation, and the value of the observation (e.g., measurement). Additionally, metadata may be attached at any (% style="color:#e74c3c" %)level(%%) of this structure as descriptive “[[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]”. [[Code lists>>doc:sdmx:Glossary.Code list.WebHome]] (enumerations) and other patterns for [[representation>>doc:sdmx:Glossary.Representation.WebHome]] of data and metadata are also modelled.
275 +SDMX recognizes that statistical data is structured; in SDMX this structure is termed a Data Structure Definition. “Data sets” are made up of one or more lower-level “groups”, based on their degrees of similarity. Each group is in turn comprised of one or more “series” of data. Each series or section has a “key” - values for each of a cluster of concepts, also called "dimensions" - which identifies it, and one or more “observations”, which typically combine the time of the observation, and the value of the observation (e.g., measurement). Additionally, metadata may be attached at any level of this structure as descriptive “attributes”. Code lists (enumerations) and other patterns for representation of data and metadata are also modelled.
294 294  
295 -There is some similarity between “cube” structures commonly used to process statistical data, and the [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] idea in the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]]. It is important to note that the data as structured according to the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is optimized for exchange, potentially with partners who may have no ability to process a “cube” of data coming from complex statistical systems. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] time series can be understood as “slices” of the cube. Such a slice is identified by its key. A "series" key consists of the values for all [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] specified by the key family except time. Thus, it is possible to reconstruct and describe data cubes from [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-structured data, and to exchange such databases using the interfaces and formats provided for that purpose in the standard. Additional objects such as [[hierarchical code lists>>doc:sdmx:Glossary.Hierarchical code list.WebHome]], (% style="color:#e74c3c" %)constraints(%%) and structure (% style="color:#e74c3c" %)maps(%%) make it possible to more fully model the structure of cubes.
277 +There is some similarity between “cube” structures commonly used to process statistical data, and the Data Structure Definition idea in the SDMX Information Model. It is important to note that the data as structured according to the SDMX Information Model is optimized for exchange, potentially with partners who may have no ability to process a “cube” of data coming from complex statistical systems. SDMX time series can be understood as “slices” of the cube. Such a slice is identified by its key. A "series" key consists of the values for all dimensions specified by the key family except time. Thus, it is possible to reconstruct and describe data cubes from SDMX-structured data, and to exchange such databases using the interfaces and formats provided for that purpose in the standard. Additional objects such as hierarchical code lists, constraints and structure maps make it possible to more fully model the structure of cubes.
296 296  
297 -The information model also provides a view of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]: a mechanism for referencing the meaningful “objects” within the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] view of statistical exchange processes ([[data providers>>doc:sdmx:Glossary.Data provider.WebHome]], structures, [[provisioning agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]], [[dataflows>>doc:sdmx:Glossary.Dataflow.WebHome]], [[metadata flows>>doc:sdmx:Glossary.Metadataflow.WebHome]], etc.) to which metadata is attached; a mechanism for describing a set of meaningful (% style="color:#e74c3c" %)concepts(%%), of organizing them into a presentational structure, and of indicating how their values are represented. This is based on a simple, hierarchical view of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] which is common to many metadata systems and classification/categorization schemes. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] provides a model (and XML and JSON formats) for both describing [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] structures, and of reporting [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] according to those structures.
279 +The information model also provides a view of reference metadata: a mechanism for referencing the meaningful “objects” within the SDMX view of statistical exchange processes (data providers, structures, provisioning agreements, dataflows, metadata flows, etc.) to which metadata is attached; a mechanism for describing a set of meaningful concepts, of organizing them into a presentational structure, and of indicating how their values are represented. This is based on a simple, hierarchical view of reference metadata which is common to many metadata systems and classification/categorization schemes. SDMX provides a model (and XML and JSON formats) for both describing reference metadata structures, and of reporting reference metadata according to those structures.
298 298  
299 -(% style="color:#e74c3c" %)Version(%%) 2.0/2.1 introduced support for metadata related to the process aspects of statistical exchange. A step-by-step process can be modelled; information about who is providing data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] and how they are providing it can be expressed; and the technical aspects of service-(% style="color:#e74c3c" %)level(%%) agreements (and similar types of [[provisioning agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]]) can be represented.
281 +Version 2.0/2.1 introduced support for metadata related to the process aspects of statistical exchange. A step-by-step process can be modelled; information about who is providing data and reference metadata and how they are providing it can be expressed; and the technical aspects of service-level agreements (and similar types of provisioning agreements) can be represented.
300 300  
301 -Support for the [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]]) in the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] was introduced in the July 2020 revision of 2.1 and is retained in (% style="color:#e74c3c" %)version(%%) 3.0 with minimal changes. This allows reusable [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ‘programs’ (a cohesive set of transformation statements designed to be executed together) and their associated constructs such as validation rulesets and user-defined operators to be managed and exchanged as [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. Mappings between objects such as [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] referenced in [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] programs and the actual [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] to which they relate is essential when it comes to actually executing programs, and this information can also be defined. Chapter 7 has more information on [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] and its integration with [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]].
283 +Support for the Validation and Transformation Language (VTL) in the SDMX Information Model was introduced in the July 2020 revision of 2.1 and is retained in version 3.0 with minimal changes. This allows reusable VTL ‘programs’ (a cohesive set of transformation statements designed to be executed together) and their associated constructs such as validation rulesets and user-defined operators to be managed and exchanged as SDMX structural metadata. Mappings between objects such as data sets referenced in VTL programs and the actual SDMX artefacts to which they relate is essential when it comes to actually executing programs, and this information can also be defined. Chapter 7 has more information on VTL and its integration with SDMX.
302 302  
303 303  A full UML conceptual design of the information model is set out in Section 2 of the Technical Specifications.
304 304  
... ... @@ -306,94 +306,94 @@
306 306  
307 307  == 5.1 SDMX-ML ==
308 308  
309 -[[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] is the XML transmission format specification for exchanging [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], and interacting with [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] services. It is designed as a general-purpose format for all automation and data / metadata exchange tasks, and provides the most complete coverage.
291 +SDMX-ML is the XML transmission format specification for exchanging structural metadata, data and reference metadata, and interacting with SDMX registry services. It is designed as a general-purpose format for all automation and data / metadata exchange tasks, and provides the most complete coverage.
310 310  
311 311  There are four distinct types of message:
312 312  
313 -1. //Structure Definition~:// For the exchange of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. A [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] structure message can carry details of any number and combination of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] like DSDs, [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] and (% style="color:#e74c3c" %)constraints(%%).
314 -1. //Structure-specific Data~:// For the exchange of data. This format is specific to the [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] of the [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] (in other terms, it is [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] in XML format, provides strict validation of conformance with the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] using a generic XML parser, and supports the transmission of partial [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[incremental updates>>doc:sdmx:Glossary.Incremental update.WebHome]]) as well as whole [[data sets>>doc:sdmx:Glossary.Data set.WebHome]].
295 +1. //Structure Definition~:// For the exchange of structural metadata. A SDMX-ML structure message can carry details of any number and combination of structural metadata artefacts like DSDs, code lists and constraints.
296 +1. //Structure-specific Data~:// For the exchange of data. This format is specific to the Data Structure Definitions of the data sets (in other terms, it is DSD-specific) and is created by following mappings between the metadata constructs defined in the Structure Definition message and the technical specification of the format. It supports the exchange of large data sets in XML format, provides strict validation of conformance with the DSD using a generic XML parser, and supports the transmission of partial data sets (incremental updates) as well as whole data sets.
315 315  
316 316  Many XML tools and technologies have expectations about the functions performed by an XML schema, one of which is a very direct relationship between the XML constructs described in the XML schema and the tagged data in the XML instance. Strong data typing is also considered normal, supporting full validation of the tagged data. These message types are designed to support validation and other expected XML schema functions.
317 317  
318 -1. //Generic Metadata~:// For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets. ‘Generic’ means the XML elements and XML [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] are the same regardless of the metadata set.
319 -1. //Registry~:// All of the possible interactions with the [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] services are supported using [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] interfaces and REST API calls. Submission of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] content, data registrations and [[subscriptions>>doc:sdmx:Glossary.Subscription.WebHome]] is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
300 +1. //Generic Metadata~:// For the exchange of reference metadata sets. ‘Generic’ means the XML elements and XML attributes are the same regardless of the metadata set.
301 +1. //Registry~:// All of the possible interactions with the SDMX registry services are supported using SDMX-ML interfaces and REST API calls. Submission of structural metadata content, data registrations and subscriptions is performed by a synchronous exchange of documents – a “request” message answered by a “response” message.
320 320  
321 -== 5.2 SDMX-JSON ==
303 +== {{id name="_Toc56646"/}}5.2 SDMX-JSON ==
322 322  
323 -[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] is the JSON transmission format specification for exchanging [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]. It provides an alternative to [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and is most suited to applications like web data dissemination.
305 +SDMX-JSON is the JSON transmission format specification for exchanging structural metadata, data and reference metadata. It provides an alternative to SDMX-ML and is most suited to applications like web data dissemination.
324 324  
325 -[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages serve the same function as those of the XML formats but have a different structure. For data, an important distinction is that they carry both [[component>>doc:sdmx:Glossary.Component.WebHome]] [[codes>>doc:sdmx:Glossary.Code.WebHome]] and labels which provides all the information needed to display the content in a single JSON response. The XML Structure-specific Data format by contrast carries only [[code>>doc:sdmx:Glossary.Code.WebHome]] IDs thus requiring applications obtain and hold [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] about the [[data set>>doc:sdmx:Glossary.Data set.WebHome]] in order to display the content in human-readable form.
307 +SDMX-JSON messages serve the same function as those of the XML formats but have a different structure. For data, an important distinction is that they carry both component codes and labels which provides all the information needed to display the content in a single JSON response. The XML Structure-specific Data format by contrast carries only code IDs thus requiring applications obtain and hold structural metadata about the data set in order to display the content in human-readable form.
326 326  
327 -[[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] does not include messages for [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] / [[notification>>doc:sdmx:Glossary.Notification.WebHome]] or registration registry services - [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] must be used for those purposes.
309 +SDMX-JSON does not include messages for subscription / notification or registration registry services - SDMX-ML must be used for those purposes.
328 328  
329 329  There are three distinct message types:
330 330  
331 -1. //Structure~:// For the exchange [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] structure messages follow the same principles as for [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] in that a single message can transmit any number and combination of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]]. While the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] messages are structured differently, it is possible to freely convert between them.
332 -1. //Data: //For the exchange of data. Unlike [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]], the structure of a [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] data message is not specific to the DSDs of the [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] so schema validation will not check for compliance of the data with the DSDs.
333 -1. //Metadata//: For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets.
313 +1. //Structure~:// For the exchange structural metadata. SDMX-JSON structure messages follow the same principles as for SDMX-ML in that a single message can transmit any number and combination of structural metadata artefacts. While the SDMX-ML and SDMX-JSON messages are structured differently, it is possible to freely convert between them.
314 +1. //Data: //For the exchange of data. Unlike SDMX-ML, the structure of a SDMX-JSON data message is not specific to the DSDs of the data sets so schema validation will not check for compliance of the data with the DSDs.
315 +1. //Metadata//: For the exchange of reference metadata sets.
334 334  
335 335  == 5.3 SDMX-CSV ==
336 336  
337 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV is the CSV transmission format specification for exchanging data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] only.
319 +SDMX-CSV is the CSV transmission format specification for exchanging data and reference metadata only.
338 338  
339 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
321 +SDMX-CSV provides a simple columnar format for data and metadata that can be readily created and interpreted by standard software tools such as Microsoft Excel. Nevertheless, data and metadata can still be converted between the CSV and the JSON / XML formats without loss.
340 340  
341 341  There are two distinct message types:
342 342  
343 -1. //Data//: For the exchange of data. Like [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]], [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV can include both [[code>>doc:sdmx:Glossary.Code.WebHome]] IDs and labels which is helpful when using the data to create human readable charts and dashboards.
344 -1. //Metadata//: For the exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets.
325 +1. //Data//: For the exchange of data. Like SDMX-JSON, SDMX-CSV can include both code IDs and labels which is helpful when using the data to create human readable charts and dashboards.
326 +1. //Metadata//: For the exchange of reference metadata sets.
345 345  
346 346  == 5.4 Formats and Messages Deprecated in Version 3.0 ==
347 347  
348 -The following formats and messages have been deprecated in (% style="color:#e74c3c" %)version(%%) 3.0 to simplify, modernise and rationalise the standard.
330 +The following formats and messages have been deprecated in version 3.0 to simplify, modernise and rationalise the standard.
349 349  
350 -* [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-EDI
351 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-series) data message
352 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-series) data message
353 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-series) data message
354 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message
355 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
356 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] 2.1 Structure Specific Time Series data message
332 +* SDMX-EDI
333 +* SDMX-ML 1.0/2.0 Generic (time-series) data message
334 +* SDMX-ML 1.0/2.0 Compact (time-series) data message
335 +* SDMX-ML 1.0/2.0 Utility (time-series) data message
336 +* SDMX-ML 1.0/2.0 Cross-Sectional data message
337 +* SDMX-ML 2.1 Generic (Time Series) data messages (for observations, time-series and cross-sectional data)
338 +* SDMX-ML 2.1 Structure Specific Time Series data message
357 357  
358 -The following messages were deprecated in (% style="color:#e74c3c" %)version(%%) 3.0 as a consequence of the deprecation of the SOAP web services:
340 +The following messages were deprecated in version 3.0 as a consequence of the deprecation of the SOAP web services:
359 359  
360 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Query messages
361 -* [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Submit Structure Request messages
342 +* SDMX-ML Query messages
343 +* SDMX-ML Submit Structure Request messages
362 362  
363 363  = 6 Dependencies on SDMX content-oriented guidelines =
364 364  
365 -The technical standards proposed here are designed so that they can be used in conjunction with other [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] guidelines which are more closely tied to the content and semantics of statistical data exchange. The [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] works equally well with any statistical (% style="color:#e74c3c" %)concept(%%), but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific (% style="color:#e74c3c" %)concepts(%%) and terminology. To achieve this goal, [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] creates and maintains guidelines for [[cross-domain concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]], terminology, and structural definitions. There are three major parts to this effort.
347 +The technical standards proposed here are designed so that they can be used in conjunction with other SDMX guidelines which are more closely tied to the content and semantics of statistical data exchange. The SDMX Information Model works equally well with any statistical concept, but to encourage interoperability, it is also necessary to standardize and harmonize the use of specific concepts and terminology. To achieve this goal, SDMX creates and maintains guidelines for cross-domain concepts, terminology, and structural definitions. There are three major parts to this effort.
366 366  
367 367  == 6.1 Cross-Domain Concepts ==
368 368  
369 -The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]] is a content guideline concerning (% style="color:#e74c3c" %)concepts(%%) which are used across [[statistical domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]]. This list is expected to grow and to be subject to revision as [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is used in a growing number of domains. The use of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Cross-Domain Concepts>>doc:sdmx:Glossary.Cross-domain concept.WebHome]], where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical (% style="color:#e74c3c" %)concepts(%%) includes not only the definitions of the (% style="color:#e74c3c" %)concepts(%%), and their names, but also, where appropriate, their [[representation>>doc:sdmx:Glossary.Representation.WebHome]] with standard [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], and the role they play within [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and metadata structure definitions.
351 +The SDMX Cross-Domain Concepts is a content guideline concerning concepts which are used across statistical domains. This list is expected to grow and to be subject to revision as SDMX is used in a growing number of domains. The use of the SDMX Cross-Domain Concepts, where appropriate, provides a framework to further promote interoperability among organisations using the technical standards presented here. The harmonization of statistical concepts includes not only the definitions of the concepts, and their names, but also, where appropriate, their representation with standard code lists, and the role they play within data structure definitions and metadata structure definitions.
370 370  
371 -The intent of this guideline is two-fold: to provide a core set of (% style="color:#e74c3c" %)concepts(%%) which can be used to structure statistical data and metadata, to promote interoperability between systems (“[[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized (% style="color:#e74c3c" %)concept(%%) names and definitions for other types of metadata (“[[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]”, as defined above.)
353 +The intent of this guideline is two-fold: to provide a core set of concepts which can be used to structure statistical data and metadata, to promote interoperability between systems (“structural metadata”, as described above); and to promote the exchange of metadata more widely, with a set of harmonized concept names and definitions for other types of metadata (“reference metadata”, as defined above.)
372 372  
373 373  == 6.2 Metadata Common Vocabulary ==
374 374  
375 -The Metadata Common Vocabulary is an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] guideline which provides definition of terms to be used for the comparison and mapping of terminology found in [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be (% style="color:#e74c3c" %)mapped(%%). This set of terms is inclusive of the terminology used within the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards.
357 +The Metadata Common Vocabulary is an SDMX guideline which provides definition of terms to be used for the comparison and mapping of terminology found in data structure definitions and in other aspects of statistical metadata management. Essentially, it provides ISOcompliant definitions for a wide range of statistical terms, which may be used directly, or against which other terminology systems may be mapped. This set of terms is inclusive of the terminology used within the SDMX Technical Standards.
376 376  
377 -The MCV provides definitions for terms on which the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Cross-Domain Metadata
359 +The MCV provides definitions for terms on which the SDMX Cross-Domain Metadata
378 378  
379 -(% style="color:#e74c3c" %)Concepts(%%) work is built.
361 +Concepts work is built.
380 380  
381 381  == 6.3 Statistical Subject-Matter Domains ==
382 382  
383 -The [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be (% style="color:#e74c3c" %)mapped(%%), to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]], (% style="color:#e74c3c" %)concepts(%%), etc. within their domains. Such groups already exist within the international community. [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] would use the [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant technical systems within and across [[statistical domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]]. The organisation of the content of such schemes is supported in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] as a [[Category>>doc:sdmx:Glossary.Category.WebHome]] Scheme.
365 +The Statistical Subject-Matter Domains is a listing of the breadth of statistical information for the purposes of organizing widespread statistical exchange and categorization. It acts as a standard scheme against which the categorization schemes of various counterparties can be mapped, to facilitate interoperable data and metadata exchange. It serves another useful purpose, however, which is to allow an organization of corresponding “domain groups”, each of which could define standard data structure definitions, concepts, etc. within their domains. Such groups already exist within the international community. SDMX would use the Statistical Subject-Matter Domains list to facilitate the efforts of these groups to develop the kinds of content standards which could support the interoperation of SDMX-conformant technical systems within and across statistical domains. The organisation of the content of such schemes is supported in SDMX as a Category Scheme.
384 384  
385 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Statistical Subject-Matter Domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] will be listed and maintained by the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Initiative and will be subject to adjustment.
367 +SDMX Statistical Subject-Matter Domains will be listed and maintained by the SDMX Initiative and will be subject to adjustment.
386 386  
387 387  == 6.4 SDMX Concept Roles ==
388 388  
389 -These guidelines define the standard set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#e74c3c" %)Concept(%%) Roles and their use. This set of standard [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#e74c3c" %)Concepts(%%) are implemented as a [[cross-domain Concept>>doc:sdmx:Glossary.Cross-domain concept.WebHome]] Scheme that defines the set of (% style="color:#e74c3c" %)concept(%%) roles and gives examples on (% style="color:#e74c3c" %)concept(%%) role implementation in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.0, 2.1 and 3.0. A (% style="color:#e74c3c" %)concept(%%) role gives a particular context to a (% style="color:#e74c3c" %)concept(%%) for easy and systematic interpretation by machine processing and visualization tools. For example, the (% style="color:#e74c3c" %)concepts(%%) REPORTING_AREA and COUNTERPART_AREA are different (% style="color:#e74c3c" %)concepts(%%) but they are both geographical characteristics, therefore they can be associated with the same (% style="color:#e74c3c" %)concept(%%) role ID: "GEO". This allows visualization systems to interpret these (% style="color:#e74c3c" %)concepts(%%) as geographical data in order to generate (% style="color:#e74c3c" %)maps(%%). The implementation of (% style="color:#e74c3c" %)concept(%%) roles is different in (% style="color:#e74c3c" %)versions(%%) 2.0 and 2.1/3.0 of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] technical standard. Specifically for [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, i.e., via [[Concept Schemes>>doc:sdmx:Glossary.Concept scheme.WebHome]]; the semantics of these roles have to be agreed bilateraly in data exchanges. The (% style="color:#e74c3c" %)Concept(%%) Roles are available as an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[Concept Scheme>>doc:sdmx:Glossary.Concept scheme.WebHome]] on the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Global Registry.
371 +These guidelines define the standard set of SDMX Concept Roles and their use. This set of standard SDMX Concepts are implemented as a cross-domain Concept Scheme that defines the set of concept roles and gives examples on concept role implementation in SDMX 2.0, 2.1 and 3.0. A concept role gives a particular context to a concept for easy and systematic interpretation by machine processing and visualization tools. For example, the concepts REPORTING_AREA and COUNTERPART_AREA are different concepts but they are both geographical characteristics, therefore they can be associated with the same concept role ID: "GEO". This allows visualization systems to interpret these concepts as geographical data in order to generate maps. The implementation of concept roles is different in versions 2.0 and 2.1/3.0 of the SDMX technical standard. Specifically for SDMX 3.0, this set of roles is considered a normative list that must be interpreted in the same way by all organisations. Additional roles may be provided via the standard roles’ mechanism in SDMX 3.0, i.e., via Concept Schemes; the semantics of these roles have to be agreed bilateraly in data exchanges. The Concept Roles are available as an SDMX Concept Scheme on the SDMX Global Registry.
390 390  
391 391  = 7 Validation and Transformation Language =
392 392  
393 -For many years the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] initiative has been fostering and supporting the development of a standard calculation language, called [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]]). A blueprint for defining calculations was already described in the original [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]].
375 +For many years the SDMX initiative has been fostering and supporting the development of a standard calculation language, called Validation and Transformation Language (VTL). A blueprint for defining calculations was already described in the original SDMX 2.1 specifications (package 13 of the Information Model - “Transformations and Expressions”). It was just a basic framework that required further developments to became operational in order to achieve a calculation language able to manipulate SDMX artefacts.
394 394  
395 -These developments started in late 2012 and were put in charge of the [[Validation and Transformation Language>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] Task Force ([[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] TF), which included members of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] and applicable also to GSIM and DDI. This brought to the publication of the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.0 in 2015. Then new requirements came from a number of proofs of (% style="color:#e74c3c" %)concepts(%%) and tests of [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.0 made by several organisations and triggered a large improvement of the language. A new provisional (% style="color:#e74c3c" %)version(%%), the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 1.1, was released in public consultation in 2017. The high number of [[comments>>doc:sdmx:Glossary.Comment.WebHome]] received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible (% style="color:#e74c3c" %)version(%%). Finally, the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 was published between April and July 2018 (see the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] website).
377 +These developments started in late 2012 and were put in charge of the Validation and Transformation Language Task Force (VTL TF), which included members of the SDMX Technical Working Group (TWG) and Statistical Working Group (SWG), besides experts coming from the DDI and GSIM communities. The intent was to define a standard language to be implemented in SDMX and applicable also to GSIM and DDI. This brought to the publication of the VTL 1.0 in 2015. Then new requirements came from a number of proofs of concepts and tests of VTL 1.0 made by several organisations and triggered a large improvement of the language. A new provisional version, the VTL 1.1, was released in public consultation in 2017. The high number of comments received triggered another phase of intensive work, with the main goal of achieving a more robust and forward compatible version. Finally, the VTL 2.0 was published between April and July 2018 (see the SDMX website).
396 396  
397 -The implementation of the [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] started in late 2018 and was published as an incremental revision to the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 standards in July 2020. It allows users to write [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] 2.0 programs for validating and transforming [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data, to store these programs in a [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] metadata registry and to exchange them through [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] messages, also together the definition of the data structures of the involved data.
379 +The implementation of the VTL 2.0 in SDMX started in late 2018 and was published as an incremental revision to the SDMX 2.1 standards in July 2020. It allows users to write VTL 2.0 programs for validating and transforming SDMX data, to store these programs in a SDMX metadata registry and to exchange them through SDMX messages, also together the definition of the data structures of the involved data.
398 398  
399 -The Transformations and Expressions package for modelling [[VTL>>doc:sdmx:Glossary.Validation and transformation language.WebHome]] programs in the [[SDMX information model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.
381 +The Transformations and Expressions package for modelling VTL programs in the SDMX information model is explained in Section 2 of the Technical Specifications with further detailed usage and implementation guidance given in Section 6.