Skip to Content
Last modified by Helena on 2025/09/10 11:19
From version 1.1
edited by Helena
on 2025/05/14 13:26
Change comment: Changed document syntax from [XWiki 2.1] to [xwiki/2.1].
To version 21.5
edited by Helena
on 2025/05/19 14:07
Change comment: Update document after refactoring.

Summary

Details

Page properties
Title
... ... @@ -1,0 +1,1 @@
1 +SDMX 3.0 Standards. Summary of major changes and new functionality
Parent
... ... @@ -1,0 +1,1 @@
1 +Methodology.WebHome
Tags
... ... @@ -1,0 +1,1 @@
1 +Artefact|Code|Code list|Hierarchy|Reference metadata|SDMX-EDI|SDMX-JSON|SDMX-ML|Statistical data and metadata exchange|Structural metadata
Content
... ... @@ -1,0 +1,692 @@
1 +{{box title="**Contents**"}}
2 +{{toc/}}
3 +{{/box}}
4 +
5 +**Revision History**
6 +
7 +(% style="width:738.039px" %)
8 +|(% style="width:187px" %)Revision|(% style="width:157px" %)Date|(% style="width:390px" %)Contents
9 +|(% style="width:187px" %)DRAFT 1.0|(% style="width:157px" %)May 2021|(% style="width:390px" %)Draft release updated for SDMX 3.0 for public consultation
10 +|(% style="width:187px" %)1.0|(% style="width:157px" %)October 2021|(% style="width:390px" %)Public Release for SDMX 3.0
11 +
12 += 1 Overview =
13 +
14 +[[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#2ecc71" %)version(%%) 3.0 introduces new features, improvements and changes to the Standard in the following key areas:
15 +
16 +**Information Model**
17 +
18 +* Simplification and improvement of the [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] model
19 +* Support for microdata
20 +* Support for geospatial data
21 +* Support for [[code list>>doc:xwiki:Glossary.Code list.WebHome]] extension and discriminated union of [[code>>doc:xwiki:Glossary.Code.WebHome]] lists
22 +* Improvements to structure mapping
23 +* Improvements to [[code>>doc:xwiki:Glossary.Code.WebHome]] [[hierarchies>>doc:xwiki:Glossary.Hierarchy.WebHome]] for data discovery
24 +* Improvements to constraints
25 +
26 +**Versioning of Structural Metadata Artefacts**
27 +
28 +* Adoption of the three-number semantic versioning standard for [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] ([[https:~~/~~/semver.org/>>https://semver.org/||rel="noopener noreferrer" target="_blank"]])
29 +
30 +**REST Web Services Application Programming Interface (API)**
31 +
32 +* Change to a single ‘structure’ resource for structure queries simplifying the REST API specification by reducing the number of resources to five
33 +* Improvements to data queries
34 +* Improvements to [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] queries
35 +* Support for [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]] maintenance using HTTP PUT, POST and DELETE verbs
36 +
37 +**SOAP Web Services API**
38 +
39 +* The SOAP web services API has been deprecated with (% style="color:#2ecc71" %)version(%%) 3.0 standardising on REST**;**
40 +
41 +**XML, JSON, CSV and EDI Transmission formats**
42 +
43 +* The [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]], [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] and [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-CSV specifications have been extended and modified where needed to support the new features and changes such as [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] and microdata
44 +* Obsolete [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] data message variants including Generic, Compact, Utility and Cross-sectional have been deprecated standardising on Structure Specific Data as the sole XML format for data exchange
45 +* The [[SDMX-EDI>>doc:xwiki:Glossary.SDMX-EDI.WebHome]] transmission format for structures and data has been deprecated
46 +* The organisation of structures into ‘collections’ in [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] structure messages has been flattened and simplified
47 +* The option to reference structures in [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] messages using Agency, ID and (% style="color:#2ecc71" %)Version(%%) has been deprecated with URN now exclusively used for all non-local referencing purpose
48 +
49 +**Breaking Changes**
50 +
51 +Many of the changes made are ‘breaking’ meaning that, while conversion between (% style="color:#2ecc71" %)versions(%%) may be possible in certain circumstances, the 3.0 specification is not directly backwardly compatible with earlier (% style="color:#2ecc71" %)versions(%%) of the Standard.
52 +
53 +A summary of the main breaking changes is given in chapter 2.
54 +
55 +**Content of the Document**
56 +
57 +The remainder of the document provides a summary of the main changes. More detailed information can be found the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 3.0 Technical Specifications, in particular:
58 +
59 +* Section 2 – Information Model
60 +* Section 5 – Registry Specification
61 +* Section 6 – Technical Notes
62 +* [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-TWG GitHub for the REST API and the XML, JSON and CSV formats
63 +
64 += 2 Summary of Breaking Changes in 3.0 =
65 +
66 +(% style="color:#2ecc71" %)Version(%%) 3.0 introduces breaking changes into the web services API, transmission formats and information model. A summary is given in the table below.
67 +
68 +== 2.1 Web Services API ==
69 +
70 +(% style="width:948.039px" %)
71 +|**REST API**|(% style="width:818px" %)(((
72 +The REST API is not backwardly compatible due to modifications to the URLs and query parameters resulting in breaking changes in four of the five main resources:
73 +
74 +* Structure queries
75 +* Data queries
76 +* Metadata queries
77 +* Availability queries
78 +
79 +Schema queries are backwardly compatible.
80 +
81 +//Guidance for implementors//
82 +
83 +REST API implementors may provide partial backward compatibility by using web server URL rewriting rules to translate (% style="color:#2ecc71" %)version(%%) 2.1 structure queries to the 3.0 equivalent.
84 +
85 +Implementors are also recommended to (% style="color:#2ecc71" %)version(%%) their API services providing users with an explicit choice of which (% style="color:#2ecc71" %)version(%%) to use.
86 +)))
87 +|**SOAP API**|(% style="width:818px" %)The SOAP API has been deprecated.
88 +
89 +== 2.2 Transmission Formats ==
90 +
91 +(((
92 +(% style="width:952.039px" %)
93 +|(% style="width:130px" %)**SDMX-ML**|(% style="width:820px" %)(((
94 +The following legacy XML data messages have been deprecated:
95 +
96 +[[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-series) data message
97 +[[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-series) data message
98 +[[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-series) data message
99 +[[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 2.1 Generic data messages (for observations, time-series and cross-sectional data)
100 +
101 +Structure Specific is the only data message option in (% style="color:#2ecc71" %)version(%%) 3.0 but is not backwardly compatible with (% style="color:#2ecc71" %)version(%%) 2.1 due to several changes including deprecation of the option to reference structures like the [[DSD>>doc:xwiki:Glossary.Data structure definition.WebHome]], [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]] and [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]] using their Agency, ID and (% style="color:#2ecc71" %)Version(%%). The time series variant of the message has also been deprecated.
102 +
103 +The [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] structure message is not backwardly compatible primarily due to:
104 +
105 +* Changes to the information model
106 +* Changes to the way the structures are organised into ‘collections’ within the message
107 +* Deprecation of the Agency, ID, (% style="color:#2ecc71" %)Version(%%) option for referencing of structures in messages
108 +)))
109 +|(% style="width:130px" %)**SDMX-JSON**|(% style="width:820px" %)(((
110 +The JSON data message is not backwardly compatible with (% style="color:#2ecc71" %)version(%%) 2.1 primarily due to changes needed to support the improved REST API data queries, in particular the ability to retrieve in one operation data from multiple [[datasets>>doc:xwiki:Glossary.Data set.WebHome]] with potentially different Data Structure Definitions.
111 +
112 +The JSON structure message is not backwardly compatible primarily due to:
113 +
114 +* Changes to the information model
115 +* Changes to the way the structures are organised into ‘collections’ within the message
116 +* Deprecation of the Agency, ID, (% style="color:#2ecc71" %)Version(%%) option for referencing of structures in messages
117 +)))
118 +|(% style="width:130px" %)**SDMX-EDI**|(% style="width:820px" %)The EDI format for both structures and data has been deprecated.
119 +|(% style="width:130px" %)**SDMX-CSV**|(% style="width:820px" %)The CSV data and [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] messages are not backwardly compatible with those under (% style="color:#2ecc71" %)version(%%) 2.1 due to changes to the structure of the messages needed to support new features such as the improved REST API data queries.
120 +)))
121 +
122 +== 2.3 Information Model ==
123 +
124 +(((
125 +(% style="width:955.039px" %)
126 +|(% style="width:132px" %)**Data Structure Definition**|(% style="width:819px" %)(((
127 +The (% style="color:#2ecc71" %)version(%%) 3.0 [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]] ([[DSD>>doc:xwiki:Glossary.Data structure definition.WebHome]]) model is not directly backwardly compatible with 2.1 primarily due to the deprecation of the special MeasureDimension.
128 +
129 +//Conversion guidance for implementors//
130 +
131 +(% style="color:#2ecc71" %)Version(%%) 2.1 DSDs can be converted to the 3.0 model by creating a [[measure>>doc:xwiki:Glossary.Measure.WebHome]] with the “[[MEASURE>>doc:xwiki:Glossary.Measure.WebHome]]” (% style="color:#2ecc71" %)concept(%%) role applied as described in paragraph 3.5.
132 +
133 +(% style="color:#2ecc71" %)Version(%%) 3.0 DSDs cannot be reliably converted to the 2.1 model due to the introduction of new features such as multiple [[measures>>doc:xwiki:Glossary.Measure.WebHome]] and value arrays for [[measures>>doc:xwiki:Glossary.Measure.WebHome]] and [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]].
134 +)))
135 +|(% style="width:132px" %)**Structure mapping model**|(% style="width:819px" %)(((
136 +The structure mapping model has changed significantly in (% style="color:#2ecc71" %)version(%%) 3.0 with deprecation of the [[Structure Set>>doc:xwiki:Glossary.Structure set.WebHome]] [[maintainable artefact>>doc:xwiki:Glossary.Maintainable artefact.WebHome]] and introduction of five new ones: [[Representation>>doc:xwiki:Glossary.Representation.WebHome]] (% style="color:#2ecc71" %)Map(%%) and four variants of [[item scheme>>doc:xwiki:Glossary.Item scheme.WebHome]] (% style="color:#2ecc71" %)map(%%).
137 +
138 +//Conversion guidance for implementors//
139 +
140 +(% style="color:#2ecc71" %)Version(%%) 2.1 [[structure sets>>doc:xwiki:Glossary.Structure set.WebHome]] can be practically converted to the (% style="color:#2ecc71" %)version(%%) 3.0 structure mapping model.
141 +
142 +Conversion from the (% style="color:#2ecc71" %)version(%%) 3.0 structure mapping model to 2.1 is generally possible. However, when attempting to convert mapping rules from 2.1 to 3.0 and back to 2.1, the resulting [[Structure Set>>doc:xwiki:Glossary.Structure set.WebHome]] will not be precisely the same as the original. In converting to (% style="color:#2ecc71" %)version(%%) 3.0, the system must generate IDs for each of the new [[maintainable artefacts>>doc:xwiki:Glossary.Maintainable artefact.WebHome]], but details of the original [[Structure Set>>doc:xwiki:Glossary.Structure set.WebHome]] [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] are lost.
143 +)))
144 +|(% style="width:132px" %)**Reference metadata model**|(% style="width:819px" %)(((
145 +The [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] model has changed in (% style="color:#2ecc71" %)version(%%) 3.0 with modifications to the role of the [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]], [[Metadata Structure Definition>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] and [[Metadataflow>>doc:xwiki:Glossary.Metadataflow.WebHome]] [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]]. Metadata [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]] and Metadata Provider Scheme have been added. Metadatasets are now identifiable.
146 +
147 +(% style="color:#2ecc71" %)Version(%%) 2.1 [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] models are not valid in (% style="color:#2ecc71" %)version(%%) 3.0.
148 +
149 +//Conversion guidance for implementors//
150 +
151 +A (% style="color:#2ecc71" %)version(%%) 2.1 [[Metadata Structure Definition>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] can be converted to the (% style="color:#2ecc71" %)version(%%) 3.0 model under some circumstances, but target information is either lost or has to be translated into a [[metadataflow>>doc:xwiki:Glossary.Metadataflow.WebHome]]. Further, conversion of a [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]] for collecting [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] against a [[dataset>>doc:xwiki:Glossary.Data set.WebHome]] would need to make changes to the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]]’s [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]]. As the [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]] may not actually be specified, judgement would need to be taken, perhaps determining the most likely candidate by examining which already have metadata reported against their [[datasets>>doc:xwiki:Glossary.Data set.WebHome]]. A 2.1 metadata report could be converted to a (% style="color:#2ecc71" %)version(%%) 3.0 Metadataset if it is attached to a structure, but requires a Metadata [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]] which would need to be created if not already in existence.
152 +
153 +Conversion from the (% style="color:#2ecc71" %)version(%%) 3.0 model to (% style="color:#2ecc71" %)version(%%) 2.1 cannot be performed reliably. The process would need target information to be derived from analysis of the [[Metadataflows>>doc:xwiki:Glossary.Metadataflow.WebHome]] and Metadata [[Provision Agreements>>doc:xwiki:Glossary.Provision agreement.WebHome]]. Depending on the complexity it may not be possible to express that information in a (% style="color:#2ecc71" %)version(%%) 2.1 [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]].
154 +)))
155 +|(% style="width:132px" %)**Constraint model**|(% style="width:819px" %)(((
156 +The (% style="color:#2ecc71" %)version(%%) 2.1 Content (% style="color:#2ecc71" %)Constraint(%%) [[artefact>>doc:xwiki:Glossary.Artefact.WebHome]] has been deprecated in (% style="color:#2ecc71" %)version(%%) 3.0 and replaced by the Data (% style="color:#2ecc71" %)Constraint(%%) for data, and the Metadata (% style="color:#2ecc71" %)Constraint(%%) for reference metadata.
157 +
158 +//Conversion guidance for implementors//
159 +
160 +2.1 Content (% style="color:#2ecc71" %)Constraints(%%) can be converted without loss to the equivalent (% style="color:#2ecc71" %)version(%%) 3.0 Data (% style="color:#2ecc71" %)Constraint(%%) model.
161 +
162 +Conversion from 3.0 to 2.1 presents challenges where wildcards have been used, in those cases requiring expansion of the wildcard into explicit values.
163 +)))
164 +|(% style="width:132px" %)**Hierarchical codelist structures**|(% style="width:819px" %)(((
165 +The (% style="color:#2ecc71" %)version(%%) 2.1 Hierarchical Codelist [[artefact>>doc:xwiki:Glossary.Artefact.WebHome]] has been deprecated in (% style="color:#2ecc71" %)version(%%) 3.0 and replaced by two new [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]], [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] and [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] Association.
166 +
167 +//Conversion guidance for implementors//
168 +
169 +(% style="color:#2ecc71" %)Version(%%) 2.1 Hierarchical Codelists can be successfully converted to the (% style="color:#2ecc71" %)version(%%) 3.0 [[hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] model. Information on which [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] to link the [[hierarchies>>doc:xwiki:Glossary.Hierarchy.WebHome]] to on what context would need to be added as a separate procedure.
170 +
171 +Conversion from the (% style="color:#2ecc71" %)version(%%) 3.0 model to (% style="color:#2ecc71" %)version(%%) 2.1 is possible, but with loss of the linking information
172 +)))
173 +)))
174 +
175 += 3 Information Model =
176 +
177 +== 3.1 Version 3.0 Information Model ==
178 +
179 +[[image:SDMX 3-0-0 Major Changes FINAL-1.0_en_6fc573fe.png||height="404" width="718"]]
180 +
181 +//Figure 1 Version 3.0 simplified Information Model UML class diagram with ‘heat map’ illustrating the areas with most change//
182 +
183 +The schematic above is a simplified UML class diagram of the [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 3.0 information model illustrating the major areas of change as a ‘heat (% style="color:#2ecc71" %)map(%%)’. Darker colours indicate where new structures have been added in (% style="color:#2ecc71" %)version(%%) 3.0 or where structures have been significantly changed.
184 +
185 +A number of ancillary structures including organisation schemes, process and [[reporting taxonomy>>doc:xwiki:Glossary.Reporting taxonomy.WebHome]] are unchanged and have not been shown. Similarly, Organisation Scheme (% style="color:#2ecc71" %)Map(%%) and [[Reporting Taxonomy>>doc:xwiki:Glossary.Reporting taxonomy.WebHome]] (% style="color:#2ecc71" %)Map(%%) have been omitted for simplicity. A schematic of the 2.1 model is given in Appendix A for comparison purposes.
186 +
187 +== 3.2 Key Changes from Version 2.1 ==
188 +
189 +New Maintainable Artefacts
190 +
191 +* Structure (% style="color:#2ecc71" %)Map
192 +* [[Representation>>doc:xwiki:Glossary.Representation.WebHome]] (% style="color:#2ecc71" %)Map
193 +* Organisation Scheme (% style="color:#2ecc71" %)Map
194 +* (% style="color:#2ecc71" %)Concept(%%) Scheme (% style="color:#2ecc71" %)Map
195 +* Category Scheme (% style="color:#2ecc71" %)Map
196 +* Reporting Taxonomy (% style="color:#2ecc71" %)Map
197 +* Value List
198 +* [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]]
199 +* [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] Association
200 +* Metadata (% style="color:#2ecc71" %)Constraint
201 +* Data (% style="color:#2ecc71" %)Constraint
202 +* Metadata Provision Agreement
203 +* Metadata Provider Scheme
204 +* Metadataset
205 +
206 +New Identifiable Artefacts
207 +
208 +* GeoFeatureSetCode
209 +* GeoGridCode
210 +* Metadata Provider
211 +
212 +Removed Maintainable Artefacts
213 +
214 +* Structure Set – replaced by Structure (% style="color:#2ecc71" %)Map(%%) and the four [[item scheme>>doc:xwiki:Glossary.Item scheme.WebHome]] maps
215 +* Hierarchical Codelist – replaced by [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] and [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] Association
216 +* (% style="color:#2ecc71" %)Constraint(%%) – replaced by Data (% style="color:#2ecc71" %)Constraint(%%) and Metadata (% style="color:#2ecc71" %)Constraint
217 +
218 +Changed Maintainable Artefacts
219 +
220 +* Data Structure Definition – support for microdatasets and [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] linked to data
221 +* Metadataflow – simplifies exchange of [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]], in particular those linked to structures
222 +* Metadata Structure Definition – simplified model for [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]]
223 +* Codelist – support for codelist extension and geospatial specialised codelists (GeographicCodelist, GeoGridCodelist)
224 +* [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] Mapping Scheme – [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] (% style="color:#2ecc71" %)Concept(%%) Mapping Scheme removed to align the [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] / [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] interface with the 3.0 model
225 +
226 +New [[Component>>doc:xwiki:Glossary.Component.WebHome]] [[Representation>>doc:xwiki:Glossary.Representation.WebHome]] Types
227 +
228 +* GeospatialInformation – a string type where the value is an expression defining a set of geographical features using a purpose-designed syntax
229 +
230 +== 3.3 Areas Unchanged from Version 2.1 ==
231 +
232 +The following areas of the information model are unchanged from (% style="color:#2ecc71" %)version(%%) 2.1:
233 +
234 +* Categories
235 +* Concepts
236 +* Data providers
237 +* Agencies
238 +* Data consumers
239 +* [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] transformation and expressions – with the exception of [[VTL>>doc:xwiki:Glossary.Validation and transformation language.WebHome]] mapping scheme as already noted
240 +* Reporting taxonomy
241 +* Process
242 +
243 +== 3.4 Reference Metadata ==
244 +
245 +[[Reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] has been substantially re-designed for (% style="color:#2ecc71" %)version(%%) 3.0 to simplify the model and better support practical use cases.
246 +
247 +(% class="wikigeneratedid" id="HSimplifyMetadataStructureDefinition" %)
248 +**Simplify Metadata Structure Definition**
249 +The [[Metadata Structure Definition>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] ([[MSD>>doc:xwiki:Glossary.Metadata structure definition.WebHome]]) has been simplified to remove target information, and the support of multiple report structures. The [[MSD>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] now only contains Metadata [[Attributes>>doc:xwiki:Glossary.Attribute.WebHome]] which are used to define the structure of a report.
250 +
251 +[[image:SDMX 3-0-0 Major Changes FINAL-1.0_en_ad5f5c97.png||height="346" width="494"]]
252 +
253 +//Figure 2 version 2.1 Metadata Structure Definition (MSD)//
254 +
255 +[[image:SDMX 3-0-0 Major Changes FINAL-1.0_en_f2695ed5.png||height="172" width="374"]]
256 +
257 +//Figure 3 the simplified version 3.0 MSD//
258 +
259 +(% class="wikigeneratedid" id="HChangetoreferencemetadatareportedagainstdata" %)
260 +**Change to reference metadata reported against data**
261 +
262 +(% class="wikigeneratedid" id="HChangetoreferencemetadatareportedagainstdata" %)
263 +[[Reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] associated with [[datasets>>doc:xwiki:Glossary.Data set.WebHome]], data series or observations are now reported with the data. The [[dataset>>doc:xwiki:Glossary.Data set.WebHome]]’s [[DSD>>doc:xwiki:Glossary.Data structure definition.WebHome]] must reference an [[MSD>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] to define the structure of its [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]]. In practice [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] for data are transmitted as part of the data message. The metadata [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] are treated in a similar way to the data [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] appearing in the message at the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]], data series or individual observation (% style="color:#2ecc71" %)level(%%) as appropriate. In contrast to simple data [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]], metadata [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] defined by an [[MSD>>doc:xwiki:Glossary.Metadata structure definition.WebHome]] can be organised into a hierarchical structure as illustrated in Figure 3 above. For this reason, metadata [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] appear in data messages structured in the same way as metadata messages.
264 +
265 +The [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] example below is an excerpt from a structure specific data message illustrating reporting of [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] with a hierarchical structure at the observation (% style="color:#2ecc71" %)level(%%).
266 +
267 +For completeness, the excerpt also shows:
268 +
269 +* OBS_STATUS – a simple observation-(% style="color:#2ecc71" %)level(%%) data [[attribute>>doc:xwiki:Glossary.Attribute.WebHome]]
270 +* TITLE – a multi-lingual data [[attribute>>doc:xwiki:Glossary.Attribute.WebHome]]
271 +* SOURCE_AGENCY – a multi-value data [[attribute>>doc:xwiki:Glossary.Attribute.WebHome]]
272 +
273 +> <Obs xsi:type="dsd:ObsType" OBS_VALUE="112" OBS_STAUS=”A” TIME_PERIOD="2010-09">
274 +> <!—- complex multi-value and multi-lingual data attributes ~-~->
275 +> <Comp id="TITLE" xsi:type="ns1:TITLE_ATTRIBUTE">
276 +> <Value>
277 +> <common:Text xml:lang="en">Some English Text</common:Text>
278 +> <common:Text xml:lang="fr">Quelques textes en anglais</common:Text>
279 +> </Value>
280 +> </Comp>
281 +> <Comp id="SOURCE_AGENCY" xsi:type="ns1:SOURCE_AGENCY_ATTRIBUTE">
282 +> <Value>4F0</Value>
283 +> <Value>4D0</Value>
284 +> <Value>CZ2</Value>
285 +> </Comp>
286 +> <!—- metadata attributes are reported like in metadata messages ~-~->
287 +> <Metadata>
288 +> <Attribute id="COLLECTION">
289 +> <Attribute id="METHOD">
290 +> <Text lang="en">AAA</Text>
291 +> </Attribute>
292 +> </Attribute>
293 +> <Attribute id="CONTACT">
294 +> <Value>CONTACT 1</Value>
295 +> <Attribute id="NAME">
296 +> <Value>Contact 1 Name 1</Value>
297 +> </Attribute>
298 +> <Attribute id="NAME">
299 +> <Value>Contact 1 Name 2</Value>
300 +> </Attribute>
301 +> </Attribute>
302 +> <Attribute id="CONTACT">
303 +> <Value>CONTACT 2</Value>
304 +> <Attribute id="NAME">
305 +> <Value>Contact 2 Name 1</Value>
306 +> </Attribute>
307 +> <Attribute id="NAME">
308 +> <Value>Contact 2 Name 2</Value>
309 +> </Attribute>
310 +> </Attribute>
311 +> </Metadata>
312 +> </Obs>
313 +
314 +(% class="wikigeneratedid" id="HNew-MetadataProvisionAgreement" %)
315 +**New - Metadata Provision Agreement**
316 +
317 +(% class="wikigeneratedid" id="HNew-MetadataProvisionAgreement" %)
318 +In (% style="color:#2ecc71" %)version(%%) 2.1 a [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]] could be used to report information against a [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]] or [[Metadataflow>>doc:xwiki:Glossary.Metadataflow.WebHome]]. From (% style="color:#2ecc71" %)version(%%) 3.0 this is managed by two separate structures, the Data [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]] and the Metadata [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]].
319 +
320 +(% class="wikigeneratedid" id="HMovetargettoMetadataflowandMetadataProvisionAgreement" %)
321 +**Move target to Metadataflow and Metadata Provision Agreement**
322 +For [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] that is reported against structures, the allowable targets information which is used to specify what structures the [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] can be reported against, has moved to the [[Metadataflow>>doc:xwiki:Glossary.Metadataflow.WebHome]] and can be further refined in the Metadata [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]].
323 +
324 +(% class="wikigeneratedid" id="HAddmaintainablepropertiestoreferencemetadata" %)
325 +**Add maintainable properties to reference metadata**
326 +A Metadataset now has mandatory identification information, (owner id, id, (% style="color:#2ecc71" %)version(%%)) enabling metadata providers to uniquely identify their reports for create, update or delete maintenance operations.
327 +
328 +== 3.5 Microdata Exchange ==
329 +
330 +Several changes have been made the [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]] to support microdata use cases in addition to aggregated time series.
331 +
332 +(% class="wikigeneratedid" id="HMultiplemeasures" %)
333 +**Multiple measures**Multiple [[measures>>doc:xwiki:Glossary.Measure.WebHome]] are a common characteristic of microdatasets. To support this use case, the MeasureDimension has been deprecated and replaced with the option to define zero or more [[measures>>doc:xwiki:Glossary.Measure.WebHome]]. [[Measures>>doc:xwiki:Glossary.Measure.WebHome]] now act like any other [[component>>doc:xwiki:Glossary.Component.WebHome]] in that they use (% style="color:#2ecc71" %)concepts(%%), can have their own local coded or uncoded [[representation>>doc:xwiki:Glossary.Representation.WebHome]] defined within the [[Data Structure Definition>>doc:xwiki:Glossary.Data structure definition.WebHome]], and can be either mandatory or conditional. Creating a [[measure>>doc:xwiki:Glossary.Measure.WebHome]] with the “[[MEASURE>>doc:xwiki:Glossary.Measure.WebHome]]” (% style="color:#2ecc71" %)concept(%%) role applied emulates the (% style="color:#2ecc71" %)version(%%) 2.1 MeasureDimension behaviour as illustrated in the [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] example below:
334 +
335 +> <str:MeasureList id=”MeasureDescriptor”>
336 +> <str:Measure id=”OBS_VALUE” minOccurs=”1” maxOccurs=”1” usage=”mandatory” > <str:ConceptIdentity>
337 +> <Ref id=”OBS_VALUE” maintainableParentID=”CONCEPTS” agencyID=”SDMX” maintainableParentVersion=”1.0.0” />
338 +> </str:ConceptIdentity>
339 +> <str:LocalRepresentation>
340 +> <str:TextFormat textType=”String” isMultiLingual=”true” />
341 +> </str:LocalRepresentation>
342 +> <str:ConceptRole>
343 +> <Ref id=”MEASURE” maintainableParentID=”SDMX_CONCEPT_ROLES” agencyID=”SDMX” maintainableParentVersion=”1.0.0” />
344 +> </str:ConceptRole>
345 +> </str:Measure>
346 +> ...
347 +> <str:Measure>
348 +> </str:MeasureList>
349 +
350 +(% class="wikigeneratedid" id="HMulti-valuemeasuresandattributes" %)
351 +**Multi-value measures and attributes**
352 +Both [[measures>>doc:xwiki:Glossary.Measure.WebHome]] and [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] have been extended with the option to take ‘arrays’ of 193 multiple coded or uncoded values. This supports use cases like multiple observation 194 status flags. New //minOccurs// and //maxOccurs// properties define the valid number of 195 values. The //usage// property separately defines whether the [[measure>>doc:xwiki:Glossary.Measure.WebHome]] or [[attribute>>doc:xwiki:Glossary.Attribute.WebHome]] is //mandatory// or optional. In the [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] [[measure>>doc:xwiki:Glossary.Measure.WebHome]] example above, the properties //minOccurs=”1” maxOccurs=”1” usage=”mandatory”// specify that OBS_VALUE must be 198 reported, and can only consist of a single value.
353 +
354 +(% class="wikigeneratedid" id="HAttributesrelationshiptomeasures" %)
355 +**Attributes relationship to measures**
356 +In addition to attaching [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] to a specific (% style="color:#2ecc71" %)level(%%) within the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]], their relationship 202 to [[measures>>doc:xwiki:Glossary.Measure.WebHome]] can also be defined.
357 +
358 +(% class="wikigeneratedid" id="HValuelists" %)
359 +**Value lists**
360 +Value lists help in modelling microdata by providing an enumeration similar to [[code lists>>doc:xwiki:Glossary.Code list.WebHome]] 206 but allowing any string values without being restricted to the rules of [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] identifiers. That allows ValueItems (the equivalent to [[Code>>doc:xwiki:Glossary.Code.WebHome]]) to contain symbols like ‘¥’ and ‘€’, but also means they are not identifiable.
361 +
362 +== 3.6 Geospatial Data Exchange ==
363 +
364 +The (% style="color:#2ecc71" %)version(%%) 3.0 model has been extended to provide explicit support for geospatial data.
365 +
366 +(% class="wikigeneratedid" id="HGeospatialInformationtype" %)
367 +**GeospatialInformation type**
368 +A new GeospatialInformation string type has been added which can be used as the [[representation>>doc:xwiki:Glossary.Representation.WebHome]] for any [[dimension>>doc:xwiki:Glossary.Dimension.WebHome]], [[attribute>>doc:xwiki:Glossary.Attribute.WebHome]] or measure [[component>>doc:xwiki:Glossary.Component.WebHome]]. The value which is a string expression conforming to the syntax defined in Section 6 of the technical specifications precisely defines a ‘Geo Feature Set’ – a collection of geographical features like points, lines or polygons. Its use is recommended in conjunction with the “GEO_FEATURE_SET” (% style="color:#2ecc71" %)concept(%%) role.
369 +
370 +(% class="wikigeneratedid" id="HGeospatialcodelists" %)
371 +**Geospatial code lists**
372 +
373 +(% class="wikigeneratedid" id="HGeospatialcodelists" %)
374 +Two new specialised types of [[code list>>doc:xwiki:Glossary.Code list.WebHome]] have been added where the definition of each [[code>>doc:xwiki:Glossary.Code.WebHome]] includes additional geospatial information in addition to the standard ID, name and description:
375 +
376 +* GeographicCodelist – each item includes an element to represent a specific Geo Feature Set which is described using the same expression syntax as for GeospatialInformation type.
377 +* GeoGridCodelist – A [[code list>>doc:xwiki:Glossary.Code list.WebHome]] defining a geographical grid composed of cells representing regular squared portions of the Earth. Each item references a cell within the grid.
378 +
379 +== 3.7 Structure Mapping ==
380 +
381 +The [[Structure Set>>doc:xwiki:Glossary.Structure set.WebHome]] in (% style="color:#2ecc71" %)version(%%) 2.1 is a container for many mapping structures including Data Structure (% style="color:#2ecc71" %)Map(%%), Codelist (% style="color:#2ecc71" %)Map(%%) and (% style="color:#2ecc71" %)Concept(%%) (% style="color:#2ecc71" %)Map(%%). For (% style="color:#2ecc71" %)version(%%) 3.0 the [[Structure Set>>doc:xwiki:Glossary.Structure set.WebHome]] [[artefact>>doc:xwiki:Glossary.Artefact.WebHome]] has been deprecated and replaced with a number of new maintainables giving better flexibility and reusability, specifically: Structure (% style="color:#2ecc71" %)Map(%%), [[Concept Scheme>>doc:xwiki:Glossary.Concept scheme.WebHome]] (% style="color:#2ecc71" %)Map(%%), [[Representation>>doc:xwiki:Glossary.Representation.WebHome]] (% style="color:#2ecc71" %)Map(%%), [[Reporting Taxonomy>>doc:xwiki:Glossary.Reporting taxonomy.WebHome]] (% style="color:#2ecc71" %)Map(%%), [[Category Scheme>>doc:xwiki:Glossary.Category scheme.WebHome]] (% style="color:#2ecc71" %)Map(%%) and Organisation Scheme (% style="color:#2ecc71" %)Map(%%).
382 +
383 +The (% style="color:#2ecc71" %)version(%%) 2.1 Codelist (% style="color:#2ecc71" %)Map(%%) been replaced with [[Representation>>doc:xwiki:Glossary.Representation.WebHome]] (% style="color:#2ecc71" %)Map(%%) which allows mappings to be defined between any combination of [[Code Lists>>doc:xwiki:Glossary.Code list.WebHome]], Value Lists and noncoded [[representations>>doc:xwiki:Glossary.Representation.WebHome]] such as text strings and numbers.
384 +
385 +(% class="wikigeneratedid" id="HMany-to-manysourceandtargetcomponents" %)
386 +**Many-to-many source and target components**
387 +Structure mapping rules may be defined with both multiple source [[components>>doc:xwiki:Glossary.Component.WebHome]] and multiple target [[components>>doc:xwiki:Glossary.Component.WebHome]] in contrast to (% style="color:#2ecc71" %)version(%%) 2.1 where only one source and target was allowed. That supports many-to-many (n-n) mapping use cases where the output of a mapping rule may be dependent on the combination of a number of input [[components>>doc:xwiki:Glossary.Component.WebHome]]. For instance:
388 +
389 +Set the output [[component>>doc:xwiki:Glossary.Component.WebHome]] INDICATOR=”DE_A” if the input [[components>>doc:xwiki:Glossary.Component.WebHome]] are FREQ=”A” and REF_AREA=”DE”.
390 +
391 +Similarly, an n-n rule may also set the values of any number of output [[components>>doc:xwiki:Glossary.Component.WebHome]]:
392 +
393 +Set the output [[components>>doc:xwiki:Glossary.Component.WebHome]] FREQ=”A”, REF_AREA=”DE” if the input [[component>>doc:xwiki:Glossary.Component.WebHome]] INDICATOR=”DE_A”.
394 +
395 +**Fixed source and target**
396 +The Structure (% style="color:#2ecc71" %)Map(%%) may now define input or output [[components>>doc:xwiki:Glossary.Component.WebHome]] which have a fixed value.
397 +
398 +(% class="wikigeneratedid" id="HTimerepresentationsmapping" %)
399 +**Time representations mapping**
400 +Non [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] time [[representations>>doc:xwiki:Glossary.Representation.WebHome]] may now be described in a Structure (% style="color:#2ecc71" %)Map(%%), allowing them to be (% style="color:#2ecc71" %)mapped(%%) into [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] [[time formats>>doc:xwiki:Glossary.Time format.WebHome]].
401 +
402 +(% class="wikigeneratedid" id="HRegularexpressionandsubstringmappings" %)
403 +**Regular expression and substring mappings**
404 +All item (% style="color:#2ecc71" %)maps(%%) allow the use of regular expressions and substrings to match source values, specifically: [[Concept Scheme>>doc:xwiki:Glossary.Concept scheme.WebHome]] (% style="color:#2ecc71" %)Map(%%), [[Reporting Taxonomy>>doc:xwiki:Glossary.Reporting taxonomy.WebHome]] (% style="color:#2ecc71" %)Map(%%), [[Category Scheme>>doc:xwiki:Glossary.Category scheme.WebHome]] (% style="color:#2ecc71" %)Map(%%) and Organisation Scheme (% style="color:#2ecc71" %)Map(%%).
405 +
406 +(% class="wikigeneratedid" id="HItemmapsvalidityperiod" %)
407 +**Item maps validity period**
408 +Item (% style="color:#2ecc71" %)maps(%%) may further define the period for which the mapping is valid, meaning the mapping rule will only be applied if the row of information being (% style="color:#2ecc71" %)mapped(%%) is within the period.
409 +
410 +== 3.8 Constraints ==
411 +
412 +(% style="color:#2ecc71" %)Constraints(%%) in (% style="color:#2ecc71" %)version(%%) 3.0 are modelled using two separate [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] which replace the (% style="color:#2ecc71" %)version(%%) 2.1 content (% style="color:#2ecc71" %)constraint(%%):
413 +
414 +* data (% style="color:#2ecc71" %)constraint(%%) for data; and
415 +* metadata (% style="color:#2ecc71" %)constraint(%%) for [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]].
416 +
417 +Metadata (% style="color:#2ecc71" %)constraint(%%) differs from its data counterpart in having a simplified cube region model better suited to [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] reporting use cases and not carrying details of the constrained targets – that information instead being defined directly within the [[metadataflow>>doc:xwiki:Glossary.Metadataflow.WebHome]] and Metadata [[Provision Agreement>>doc:xwiki:Glossary.Provision agreement.WebHome]]. Thus, metadata related (% style="color:#2ecc71" %)constraints(%%) only specify (% style="color:#2ecc71" %)constraints(%%) to the values of metadata attributes.
418 +
419 +The ‘%’ wildcard character can now be used when defining cube region (% style="color:#2ecc71" %)constraints(%%) to match multiple [[codes>>doc:xwiki:Glossary.Code.WebHome]] with a single expression, for instance for [[economic activity>>doc:xwiki:Glossary.Economic activity.WebHome]], ISIC4_% matches all [[codes>>doc:xwiki:Glossary.Code.WebHome]] beginning with ‘ISIC4_’ avoiding the need to maintain an explicit list.
420 +
421 +The validity period definition has been moved from the (% style="color:#2ecc71" %)constraint(%%) to the individual constraining terms, specifically CubeRegion, DataKeySet and MetadataTargetRegion providing more granular control.
422 +
423 +Attachment (% style="color:#2ecc71" %)constraints(%%) have been deprecated due to a lack of use cases.
424 +
425 +== 3.9 Code List Extension ==
426 +
427 +In addition to the two new specialised geospatial forms, the option has been added to define a [[code list>>doc:xwiki:Glossary.Code list.WebHome]] as an extension of, or by inheriting [[codes>>doc:xwiki:Glossary.Code.WebHome]] from, other lists. An optional prefix can be added to inherited [[codes>>doc:xwiki:Glossary.Code.WebHome]] to disambiguate duplicates.
428 +
429 +This feature allows new [[code lists>>doc:xwiki:Glossary.Code list.WebHome]] to be easily derived from existing lists without the need to make and manually maintain copies. When querying for extended [[code list>>doc:xwiki:Glossary.Code list.WebHome]] structures using the REST API, the option has been added to retrieve either the definition or the materialised list. Traditional literal lists of [[codes>>doc:xwiki:Glossary.Code.WebHome]] continue to be supported.
430 +
431 +== 3.10 Discriminated Union of Code Lists ==
432 +
433 +Combining [[code list>>doc:xwiki:Glossary.Code list.WebHome]] extension with wildcarded (% style="color:#2ecc71" %)constraints(%%) solves the discriminated union of [[code lists>>doc:xwiki:Glossary.Code list.WebHome]] problem where a classification or breakdown has multiple “variants” which are all valid but mutually exclusive. A common example is [[economic activity>>doc:xwiki:Glossary.Economic activity.WebHome]] where several alternative classification schemes are in use including ISIC revisions 1 to 4 and NACE as used in the European Community.
434 +
435 +== 3.11 Code Hierarchies ==
436 +
437 +[[Code>>doc:xwiki:Glossary.Code.WebHome]] [[hierarchies>>doc:xwiki:Glossary.Hierarchy.WebHome]] allow the definition of complex [[hierarchies>>doc:xwiki:Glossary.Hierarchy.WebHome]] of [[codes>>doc:xwiki:Glossary.Code.WebHome]] from potentially multiple lists for data discovery purposes. Hierarchical Codelist has been deprecated and replaced by two new [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]]: [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] – the actual [[hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] of [[codes>>doc:xwiki:Glossary.Code.WebHome]], and [[Hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] Association links [[hierarchies>>doc:xwiki:Glossary.Hierarchy.WebHome]] directly to any other identifiable object, a capability missing from the (% style="color:#2ecc71" %)version(%%) 2.1 model. Further, the linkage can be within a particular context, for instance linking a [[hierarchy>>doc:xwiki:Glossary.Hierarchy.WebHome]] to a [[dimension>>doc:xwiki:Glossary.Dimension.WebHome]] within the context of a specific [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]] ([[dimension>>doc:xwiki:Glossary.Dimension.WebHome]] REF_AREA in the context of the ECB:EXR [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]).
438 +
439 += 4 Versioning of Structural Metadata Artefacts =
440 +
441 +(% style="color:#2ecc71" %)Version(%%) 3.0 adopts semantic versioning principles for versioning of metadata [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] following the rules set out at __[[https:~~/~~/semver.org>>https://semver.org]] __However, this is not mandatory, and organisations may continue to use the pre-existing two-digit versioning strategy, or not to (% style="color:#2ecc71" %)version(%%) [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] by omitting the (% style="color:#2ecc71" %)//version//(%%) property. The (% style="color:#2ecc71" %)version(%%) number no longer defaults to 1.0 if not explicitly set.
442 +
443 +Semantic (% style="color:#2ecc71" %)version(%%) numbers are three digits:
444 +
445 +MAJOR.MINOR.PATCH
446 +
447 +Where
448 +
449 +* The first digit (major) indicates that changes (either new features or bug fixes) are not backward compatible.
450 +* The second digit (minor) indicates that features have been added in a backward compatible manner.
451 +* The third digit (patch) indicates that bugs have been fixed in a backward compatible manner.
452 +
453 +Examples:
454 +
455 +SDMX:CL_AREA(1.0.0)
456 +SDMX:CL_AREA(2.3.2)
457 +
458 +== Dependency management ==
459 +
460 +Additional constructs are possible for dependency management when referencing structures. For instance:
461 +
462 +2.3+.1  Means the currently latest available (% style="color:#2ecc71" %)version(%%) >= “2.3.1” and < “3.0.0” (all backwards compatible (% style="color:#2ecc71" %)versions(%%) >= “2.3.1”).
463 +2+.3.1  Means the currently latest available (% style="color:#2ecc71" %)version(%%) >= “2.3.1” (even if not backwards compatible).
464 +
465 +== Draft structures ==
466 +
467 +A key principle is that semantically versioned structures are immutable and must not be changed without a corresponding change to the (% style="color:#2ecc71" %)version(%%) number, except where explicitly marked as draft using extensions to the (% style="color:#2ecc71" %)version(%%) number.
468 +
469 +MAJOR.MINOR.PATCH-EXTENSION
470 +
471 +1.10.0-draft  Means that (% style="color:#2ecc71" %)version(%%) 1.10.0 is still being modified and may change – equivalent to setting isFinal=false in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 2.1.
472 +1.10.0-unstable  Alternative to -draft.
473 +1.10.0-notfinal  Alternative to -draft.
474 +
475 +The [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 isFinal property is deprecated in 3.0.
476 +
477 += 5 REST Web Services API =
478 +
479 +== 5.1 Simplified list of resources ==
480 +
481 +The (% style="color:#2ecc71" %)version(%%) 3.0 REST API has just five main resources:
482 +
483 +* structure
484 +* data
485 +* schema
486 +* availability
487 +* metadata
488 +
489 +All structure and item queries have been organised under the structure resource in contrast to the (% style="color:#2ecc71" %)version(%%) 2.1 API which specified a separate resource for each structure.
490 +
491 +This and changes in the URLs and query parameters on the data, availability and metadata resources means that, with the exception of schema queries, the (% style="color:#2ecc71" %)version(%%) 3.0 API is not backwardly compatible.
492 +
493 +== 5.2 Improved data queries ==
494 +
495 +Data queries have been changed to provide more granular selections from contexts wider than just a [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]].
496 +
497 +(% class="wikigeneratedid" id="HExtendthecontextofdataretrieval" %)
498 +**Extend the context of data retrieval**
499 +(% style="color:#2ecc71" %)Version(%%) 2.1 data queries always retrieved data from a single specific [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]. In (% style="color:#2ecc71" %)version(%%) 3.0, the query context may be specified as:
500 +
501 +* [[Dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]];
502 +* Data Structure Definition – i.e., all [[Dataflows>>doc:xwiki:Glossary.Dataflow.WebHome]] that use it; or
503 +* Provision Agreement – i.e., all [[Dataflows>>doc:xwiki:Glossary.Dataflow.WebHome]] associated with it.
504 +
505 +Data queries may also search across [[datasets>>doc:xwiki:Glossary.Data set.WebHome]], for instance “retrieve all data about a country”.
506 +
507 +(% class="wikigeneratedid" id="HComponent-basedfilters" %)
508 +**Component-based filters**
509 +Expressions filtering on individual [[components>>doc:xwiki:Glossary.Component.WebHome]] can now be included as part of the data query URL.
510 +
511 +/data/dataflow/ESTAT/ICP?c[REF_AREA]=CH&c[CONF_STATUS]=F
512 +
513 +(% class="wikigeneratedid" id="HSupportforoperators" %)
514 +**Support for operators**
515 +Filter expressions can also include operators.
516 +
517 +/data/dataflow/ESTAT/ICP?c[REF_AREA]=DE&c[ICP_ITEM]=sw:01&c[TIME_PERIOD]=ge:2015
518 +
519 +Operators include:
520 +
521 +(% style="width:531.039px" %)
522 +|(% style="width:151px" %)eq|(% style="width:378px" %)Equals
523 +|(% style="width:151px" %)ne|(% style="width:378px" %)Not equal to
524 +|(% style="width:151px" %)le|(% style="width:378px" %)Less than
525 +|(% style="width:151px" %)ge|(% style="width:378px" %)Greater than or equal to
526 +|(% style="width:151px" %)sw|(% style="width:378px" %)Starts with
527 +
528 +(% class="wikigeneratedid" id="HSupportformultiplekeys" %)
529 +**Support for multiple keys**
530 +Queries can now specify multiple [[series keys>>doc:xwiki:Glossary.Series key.WebHome]].
531 +
532 +(% class="wikigeneratedid" %)
533 +/data/dataflow/ESTAT/ICP/1.0.0/M…A.ANR,M…A.INX,M…B.CTG
534 +
535 +== 5.3 Improved reference metadata queries ==
536 +
537 +[[Reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] queries have been improved with a number of new options to retrieve metadata reports.
538 +
539 +**Get metadata reports by ID**
540 +
541 +/metadata/metadataset/ESTAT/QUALITY_REPORT/1.0.0
542 +
543 +**Get metadata reports by Dataflow**
544 +
545 +/metadata/metadataflow/ECB/METHODOLOGY/*/FR2
546 +
547 +**Get metadata reports about a Data Structure Definition**
548 +
549 +/metadata/structure/datastructure/BIS/BIS_CBS/1.0
550 +
551 +== 5.4 Structural metadata maintenance ==
552 +
553 +Support has been added for maintenance of [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]].
554 +
555 +HTTP verbs PUT, POST and DELETE may be used to submit [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] or [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] structure messages to an [[SDMX registry>>doc:xwiki:Glossary.SDMX Registry.WebHome]] for the purposes of adding, updating or deleting [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]].
556 +
557 += 6 XML, JSON, CSV and EDI Transmission formats =
558 +
559 +== 6.1 SDMX-ML ==
560 +
561 +The [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] XML messages have been modified and updated for (% style="color:#2ecc71" %)version(%%) 3.0. While they broadly follow the same principles, there have been significant changes which break backward compatibility.
562 +
563 +(% class="wikigeneratedid" id="HStructuremessage" %)
564 +**Structure message**
565 +The [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] structure message is used for transmission of [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]]. It closely reflects the [[SDMX information model>>doc:xwiki:Glossary.SDMX Information Model.WebHome]] and has therefore been significantly updated for (% style="color:#2ecc71" %)version(%%) 3.0 with the addition of new structures, modifications where structures have changed, and removal of deprecated structures like Structure Set.
566 +
567 +Additionally, the way the individual [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] are organised into ‘collections’ within the message has been significantly revised with a simpler flat structure adopted as set out in the following table:
568 +
569 +(% style="width:1102.04px" %)
570 +|(% style="width:321px" %)**Artefact type**|(% style="width:351px" %)**Version 2.1 Collection**|(% style="width:426px" %)**Version 3.0 Collection**
571 +|(% style="width:321px" %)AgencyScheme|(% style="width:351px" %)OrganisationSchemes|(% style="width:426px" %)AgencySchemes
572 +|(% style="width:321px" %)DataConsumerScheme|(% style="width:351px" %)OrganisationSchemes|(% style="width:426px" %)DataConsumerSchemes
573 +|(% style="width:321px" %)DataProviderScheme|(% style="width:351px" %)OrganisationSchemes|(% style="width:426px" %)DataProviderSchemes
574 +|(% style="width:321px" %)MetadataProviderScheme|(% style="width:351px" %)OrganisationSchemes|(% style="width:426px" %)MetadataProviderSchemes
575 +|(% style="width:321px" %)OrganisationUnitScheme|(% style="width:351px" %)OrganisationSchemes|(% style="width:426px" %)OrganisationUnitSchemes
576 +|(% style="width:321px" %)GeographicCodelist|(% style="width:351px" %)Codelists|(% style="width:426px" %)GeographicCodelists
577 +|(% style="width:321px" %)GeoGridCodelist|(% style="width:351px" %)Codelists|(% style="width:426px" %)GeoGridCodelists
578 +|(% style="width:321px" %)ConceptScheme|(% style="width:351px" %)Concepts|(% style="width:426px" %)ConceptSchemes
579 +|(% style="width:321px" %)ValueList|(% style="width:351px" %)Codelists|(% style="width:426px" %)ValueLists
580 +|(% style="width:321px" %)StructureMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)StructureMaps
581 +|(% style="width:321px" %)RepresentationMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)RepresentationMaps
582 +|(% style="width:321px" %)ConceptSchemeMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)ConceptSchemeMaps
583 +|(% style="width:321px" %)CategorySchemeMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)CategorySchemeMaps
584 +|(% style="width:321px" %)OrganisationSchemeMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)OrganisationSchemeMaps
585 +|(% style="width:321px" %)ReportingTaxonomyMap|(% style="width:351px" %)StructureMappings|(% style="width:426px" %)ReportingTaxonomyMaps
586 +|(% style="width:321px" %)DataConstraint|(% style="width:351px" %)Constraints|(% style="width:426px" %)DataConstraints
587 +|(% style="width:321px" %)MetadataConstraint|(% style="width:351px" %)Constraints|(% style="width:426px" %)MetadataConstraints
588 +|(% style="width:321px" %)MetadataProvisionAgreement|(% style="width:351px" %)ProvisionAgreement|(% style="width:426px" %)MetadataProvisionAgreements
589 +|(% style="width:321px" %)CustomTypeScheme|(% style="width:351px" %)CustomTypes|(% style="width:426px" %)CustomTypeSchemes
590 +|(% style="width:321px" %)VtlMappingScheme|(% style="width:351px" %)VtlMappings|(% style="width:426px" %)VtlMappingSchemes
591 +|(% style="width:321px" %)NamePersonalisationScheme|(% style="width:351px" %)NamePersonalisations|(% style="width:426px" %)NamePersonalisationSchemes
592 +|(% style="width:321px" %)RulesetScheme|(% style="width:351px" %)Rulesets|(% style="width:426px" %)RulesetSchemes
593 +|(% style="width:321px" %)TransformationScheme|(% style="width:351px" %)Transformations|(% style="width:426px" %)TransformationSchemes
594 +|(% style="width:321px" %)UserDefinedOperatorScheme|(% style="width:351px" %)UserDefinedOperators|(% style="width:426px" %)UserDefinedOperatorSchemes
595 +
596 +No changes have been made to the way the following [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] are organised in the structure message:
597 +
598 +(% style="width:1106.04px" %)
599 +|(% style="width:326px" %)**Artefact type**|(% style="width:776px" %)**Collection**
600 +|(% style="width:326px" %)Dataflow|(% style="width:776px" %)Dataflows
601 +|(% style="width:326px" %)Metadataflow|(% style="width:776px" %)Metadataflows
602 +|(% style="width:326px" %)CategoryScheme|(% style="width:776px" %)CategorySchemes
603 +|(% style="width:326px" %)Categorisation|(% style="width:776px" %)Categorisations
604 +|(% style="width:326px" %)Codelist|(% style="width:776px" %)Codelists
605 +|(% style="width:326px" %)Hierarchy|(% style="width:776px" %)Hierarchies
606 +|(% style="width:326px" %)HierarchyAssociation|(% style="width:776px" %)HierarchyAssociations
607 +|(% style="width:326px" %)MetadataStructure|(% style="width:776px" %)MetadataStructures
608 +|(% style="width:326px" %)DataStructure|(% style="width:776px" %)DataStructures
609 +|(% style="width:326px" %)ReportingTaxonomy|(% style="width:776px" %)ReportingTaxonomies
610 +|(% style="width:326px" %)Process|(% style="width:776px" %)Processes
611 +|(% style="width:326px" %)ProvisionAgreement|(% style="width:776px" %)ProvisionAgreements
612 +
613 +From (% style="color:#2ecc71" %)version(%%) 3.0, collections can appear in any order within a structure message.
614 +
615 +(% class="wikigeneratedid" id="HDatamessages" %)
616 +**Data messages**
617 +
618 +(% class="wikigeneratedid" id="HDatamessages" %)
619 +All legacy [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] data messages have been deprecated with the exception of Structure Specific Data which becomes the sole standard format for transmission of [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] data in XML in (% style="color:#2ecc71" %)version(%%) 3.0.
620 +
621 +Specifically, the following data messages are not supported in (% style="color:#2ecc71" %)version(%%) 3.0:
622 +
623 +* [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Generic (time-series) data message
624 +* [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Compact (time-series) data message
625 +* [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Utility (time-series) data message
626 +* [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 1.0/2.0 Cross-Sectional data message
627 +* [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] 2.1 Generic data messages (for observations, time-series and crosssectional data)
628 +
629 +The Structure Specific Data message has been extended to support the transmission of microdata sets, in particular those with multiple [[measures>>doc:xwiki:Glossary.Measure.WebHome]] and array values for [[measures>>doc:xwiki:Glossary.Measure.WebHome]] and [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]].
630 +
631 +As detailed in paragraph 3.4, the message now additionally allows data’s [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] to be reported as an integral part of the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]]. Like data [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]], these metadata [[attributes>>doc:xwiki:Glossary.Attribute.WebHome]] are included in the data message at the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]], series or observation (% style="color:#2ecc71" %)level(%%) as appropriate.
632 +
633 +The time series variant of the Structure Specific Data message is no longer used.
634 +
635 +(% class="wikigeneratedid" id="HReferencemetadatamessage" %)
636 +**Reference metadata message**
637 +The Generic Metadata message remains the standard format for transmission of [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] sets in XML but has been modified to support the revised (% style="color:#2ecc71" %)version(%%) 3.0 [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] model.
638 +
639 +(% class="wikigeneratedid" id="HRegistrystructuralmetadata2018query2019messages" %)
640 +**Registry structural metadata ‘query’ messages**
641 +As a consequence of the deprecation of the SOAP API and standardisation on REST, the [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]] ‘query’ messages have all been removed. In (% style="color:#2ecc71" %)version(%%) 3.0, querying an [[SDMX Registry>>doc:xwiki:Glossary.SDMX Registry.WebHome]] for [[structural metadata>>doc:xwiki:Glossary.Structural metadata.WebHome]] is performed solely using REST GET.
642 +
643 +(% class="wikigeneratedid" id="HStructurereferencing" %)
644 +**Structure referencing**
645 +The option to reference structures using Agency, ID and (% style="color:#2ecc71" %)Version(%%) has been removed. From [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] (% style="color:#2ecc71" %)version(%%) 3.0 URN is used for all referencing purposes with the exception of local references such as where groups reference [[dimensions>>doc:xwiki:Glossary.Dimension.WebHome]] within a [[DSD>>doc:xwiki:Glossary.Data structure definition.WebHome]].
646 +
647 +== 6.2 SDMX-JSON ==
648 +
649 +Like [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]], the [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] messages have been significantly modified and updated for (% style="color:#2ecc71" %)version(%%) 3.0. They are not backwardly compatible with (% style="color:#2ecc71" %)version(%%) 2.1.
650 +
651 +(% class="wikigeneratedid" id="HStructuremessage-1" %)
652 +**Structure message**
653 +The [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] structure message closely replicates the [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] equivalent. Like that of [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] it has been updated to align it with the (% style="color:#2ecc71" %)version(%%) 3.0 information model with addition, deletion and modification of [[artefacts>>doc:xwiki:Glossary.Artefact.WebHome]] as required. The organisation of the structure collections has also been revised as detailed in paragraph 6.1.
654 +
655 +(% class="wikigeneratedid" id="HDatamessage" %)
656 +**Data message**
657 +The [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] data message has similarly be updated. Additional changes have been made to allow a single message to carry data from multiple [[datasets>>doc:xwiki:Glossary.Data set.WebHome]] with potentially different [[Data Structure Definitions>>doc:xwiki:Glossary.Data structure definition.WebHome]] to support REST data queries of the form “retrieve all data about a country”. For this reason, the (% style="color:#2ecc71" %)version(%%) 3.0 [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] is not backwardly compatible with (% style="color:#2ecc71" %)version(%%) 2.1 data messages. Support has been added for the transmission of microdata and reporting of [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] on data as an integral part of the dataset.
658 +
659 +(% class="wikigeneratedid" id="HReferencemetadatamessage-1" %)
660 +**Reference metadata message**
661 +The [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] metadata message has also been updated to support the (% style="color:#2ecc71" %)version(%%) 3.0 [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] and Metadataset specifications.
662 +
663 +(% class="wikigeneratedid" id="HStructurereferencing-1" %)
664 +**Structure referencing**
665 +As for [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]], the option to reference structures using Agency, ID and (% style="color:#2ecc71" %)Version(%%) has been removed with URN used for all non-local referencing purposes.
666 +
667 +== 6.3 SDMX-CSV ==
668 +
669 +CSV in [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]] is used transmission of data and [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] only.
670 +
671 +(% class="wikigeneratedid" id="HDatamessage-1" %)
672 +**Data message**
673 +The [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-CSV data message has been modified to align with the (% style="color:#2ecc71" %)version(%%) 3.0 information model, support the enhanced REST API and ensure that data can be freely converted to and from the XML and JSON formats without loss. These changes include:
674 +
675 +* An additional column identifying the type if the [[artefact>>doc:xwiki:Glossary.Artefact.WebHome]] defining the structure of the data: “[[dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]]”, “datastructure” or “dataprovision”;
676 +* A column for the structure [[artefact>>doc:xwiki:Glossary.Artefact.WebHome]]’s identification of the form
677 +
678 +ESTAT:NA_MAIN(1.6.0) which replaces the [[dataflow>>doc:xwiki:Glossary.Dataflow.WebHome]] identifier in (% style="color:#2ecc71" %)version(%%) 2.1; and
679 +
680 +* A column for the [[dataset>>doc:xwiki:Glossary.Data set.WebHome]] action: information, append, replace or delete, which is consistent with both the the [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] data messages.
681 +
682 +(% class="wikigeneratedid" id="HReferencemetadatamessage-2" %)
683 +**Reference metadata message**
684 +The [[SDMX>>doc:xwiki:Glossary.Statistical data and metadata exchange.WebHome]]-CSV metadata message is new for (% style="color:#2ecc71" %)version(%%) 3.0 and, like the [[SDMX-ML>>doc:xwiki:Glossary.SDMX-ML.WebHome]] and [[SDMX-JSON>>doc:xwiki:Glossary.SDMX-JSON.WebHome]] equivalents, is used for the transmission [[reference metadata>>doc:xwiki:Glossary.Reference metadata.WebHome]] sets.
685 +
686 +== 6.4 EDI deprecation ==
687 +
688 +The EDI format for transmission of both structures and data has been deprecated. (% style="color:#2ecc71" %)Version(%%) 3.0 is therefore not backwardly compatible with legacy EDI messages.
689 +
690 += Appendix A – Version 2.1 Information Model =
691 +
692 +[[image:SDMX 3-0-0 Major Changes FINAL-1.0_en_5f21cdf9.png||height="319" width="718"]]
SDMX 3-0-0 Major Changes FINAL-1.0_en_5f21cdf9.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.helena
Size
... ... @@ -1,0 +1,1 @@
1 +103.2 KB
Content
SDMX 3-0-0 Major Changes FINAL-1.0_en_6fc573fe.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.helena
Size
... ... @@ -1,0 +1,1 @@
1 +148.8 KB
Content
SDMX 3-0-0 Major Changes FINAL-1.0_en_ad5f5c97.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.helena
Size
... ... @@ -1,0 +1,1 @@
1 +22.0 KB
Content
SDMX 3-0-0 Major Changes FINAL-1.0_en_f2695ed5.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.helena
Size
... ... @@ -1,0 +1,1 @@
1 +5.7 KB
Content
SUZ.Methodology.Code.MethodologyClass[0]