Skip to Content

Changes for page Schema and Documentation

Last modified by Elena on 2025/09/10 11:19
From version 3.1
edited by Helena
on 2025/07/02 16:25
Change comment: There is no comment for this version
To version 8.1
edited by Helena
on 2025/07/03 15:13
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -Part VII. Samples
1 +Schema and Documentation
Tags
... ... @@ -1,0 +1,1 @@
1 +Attribute|Code list|Component|Data set|Data structure definition|Dimension|Incremental update|Maintenance agency|Metadata set|Metadata structure definition|Observation value|Reference metadata|SDMX Information Model|SDMX-EDI|SDMX-ML|Statistical data and metadata exchange|Structural metadata|Time period
Content
... ... @@ -1,0 +1,112 @@
1 +{{box title="**Contents**"}}
2 +{{toc/}}
3 +{{/box}}
4 +
5 += 1 BACKGROUND =
6 +
7 +== 1.1 History and Version 3.0 Developments ==
8 +
9 +The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards (% style="color:#e74c3c" %)Version(%%) 1.0 established an information model which described aggregated statistical [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] needed to exchange them in a standard fashion. This drew on the earlier example of the GESMES/TS standard. Based on the [[SDMX information model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]], several formats were developed: XML formats for exchange of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], and queries for these ([[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]), and EDIFACT formats for the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]]). These standards supported a number of exchange patterns, characterized as "bilateral", "gateway", and "data-sharing" models, as described in the Framework document in the (% style="color:#e74c3c" %)Version(%%) 1.0 standards package.
10 +
11 +(% style="color:#e74c3c" %)Versions(%%) 2.0 and 2.1 built on this foundation to provide a higher degree of support for all of these models, with an emphasis on data sharing in the form of a set of standard registry services interfaces. Support for new types of metadata exchange and reporting were added, with a focus on "[[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]" concerned with quality, methodology, and other issues. Further, the ability to provide metadata about the relationships between [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and structures was expanded, providing more support for data cubes. Overall, the scope of the (% style="color:#e74c3c" %)Version(%%) 2.1 [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards was much broader, and included a larger set of message types in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] formats.
12 +
13 +For (% style="color:#e74c3c" %)Version(%%) 3.0, the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] specification has been rationalised and tightened while maintaining the fundamentals and principles established in 2.1. The documents for describing [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] remain the same except where there have been changes to the information model either to enhance existing structures, add new ones, or remove those deprecated such as the Structure Set.
14 +
15 +The deprecation of the SOAP web services specification in (% style="color:#e74c3c" %)Version(%%) 3.0 has resulted in the removal of all [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Query documents. Standardisation on the REST API and HTTP GET for registry structure queries means separate 'query request' documents are no longer required.
16 +
17 +The opportunity has also been taken to rationalise the plethora of alternative data documents accumulated from (% style="color:#e74c3c" %)Versions(%%) 1.0, 2.0 and 2.1. All, including Generic, Utility, Compact and Cross-Sectional have been deprecated except for Structure Specific Data which becomes the sole standard XML format for data updates, revisions and bilateral exchanges in (% style="color:#e74c3c" %)Version(%%) 3.0.
18 +
19 +== 1.2 The XML Design ==
20 +
21 +All of these document types will share a common "envelope" at the message (% style="color:#e74c3c" %)level(%%) ("SDMXMessage.xsd"), as well as a set of common low-(% style="color:#e74c3c" %)level(%%) [[components>>doc:sdmx:Glossary.Component.WebHome]] ("SDMXCommon[subset].xsd") so that header information and basic structure of a message will always be the same.
22 +
23 +* Schema and sub-modules for describing all types of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] - for [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]]), for [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] ([[metadata structure definitions>>doc:sdmx:Glossary.Metadata structure definition.WebHome]]), for related groups of metadata and data structures, and for all types of structural objects involved in registry-based exchanges ("SDMXStructure[*subset*].xsd")
24 +* Structure specific data schema for for updates, revisions/bilateral exchange ("SDMXDataStructureSpecific.xsd")
25 +* Generic schema and sub-modules for registry interfaces ("SDMXRegistry [*subset*].xsd")
26 +* Generic schema for [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sets ("SDMXMetadataGeneric.xsd")
27 +
28 +== 1.3 Fostering the Use of a Standard SDMX-ML ==
29 +
30 +In addition to these different formats, standard mappings and corresponding transformation tools have been developed for the creation of data structure-specific schemas from structure descriptions, to transform XML data instances from one XML data description format to another, and from these formats into the corresponding [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages. This (% style="color:#e74c3c" %)level(%%) of free tools support will foster the early use of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] and permit the data to be easily used across all processes, which is otherwise a difficult requirement to meet. Ultimately, it is the fact that all formats share a common information model that enables this approach to meet the wide set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] requirements.
31 +
32 += 2 NORMATIVE REFERENCES =
33 +
34 +W3C XML Schema Definition Language, (% style="color:#e74c3c" %)version(%%) 1.0 (URL: http://www.w3c.org/XML/Schema#dev), World Wide Web Consortium
35 +
36 +W3C Extensible Markup Language, (% style="color:#e74c3c" %)version(%%) 1.0, Third Edition (URL: http://www.w3c.org/TR/2004/REC-xml-20040204/), World Wide Web Consortium
37 +
38 += 3 CONFORMANCE =
39 +
40 +
41 +Part IV (SDMX_2_1_SECTION_03A_PART_IV_DATA) of this document is normative, providing rules for the creation of conformant [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] XML instances and W3C XML Schemas.
42 +
43 += 4 DESIGN OVERVIEW =
44 +
45 +== 4.1 Scope and Requirements ==
46 +
47 +To understand the relationships between the several document types, it is important to have some familiarity with the requirements they are designed to fulfil.
48 +
49 +* Large amounts of data must be captured in a reasonably compact format, because of the potential size of databases being exchanged.
50 +* It must be possible to send [[incremental updates>>doc:sdmx:Glossary.Incremental update.WebHome]], rather than entire, complete databases. The validation of such exchanges demands not that an entire [[data set>>doc:sdmx:Glossary.Data set.WebHome]] be exchanged, but only that enough information be sent to ensure accurate updating and revision processes.
51 +* Structural information as well as data will need to be transmitted.
52 +* There must be a reliable transformation to and from the GESMES/TS EDIFACT syntax.
53 +* It should be possible to present natural-language information in multiple, equivalent languages.
54 +* To support web services and similar technological approaches, there is a requirement to send queries to information sources as well as data and structures.
55 +* Users (and registry services) may not know about a specific data structure, and will need to be able to handle data across data structures, and even (for, say, a comparison service) to put data structured according to multiple data structures in a single XML instance.
56 +* The XML should conform to the Information Model as closely as possible, and promote simple mapping from the XML instances to model based objects
57 +* The XML should behave as "normally" as possible within standard XML tools such as web development environments, parsers, guided editing tools, etc.
58 +* Data should be able to be structured in one or two (% style="color:#e74c3c" %)levels(%%), with the two (% style="color:#e74c3c" %)level(%%) format allowing for a single [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] which serves to disambitguate the observations. In addtion, it should still be possible to restrict this more generalized structure to time series only formats. However, doing so should not add any additional burden in terms of how data is processed.
59 +* Data structure and metadata structure specific formats should be able to be generically processed, without having to understand the entire structure; which is to say that the structure-specific schemas should serve to validate the information, but not be required to process it.
60 +* XML formats should promote re-use of common semantics, (% style="color:#e74c3c" %)concepts(%%), and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] to the greatest possible extent, while still recognizing the agency which maintains a specific resource (a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]], a data structure, a [[data set>>doc:sdmx:Glossary.Data set.WebHome]], etc.)
61 +* XML formats must support interactions of applications with standard registry services, based on standard interfaces. These must function both as web services, and as services operating over http and similar protocols.
62 +* XML formats must support the reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] which is not structural in nature, but which constitutes a primary information flow of metadata attached to other parts of the statistical collection, reporting, processing, exchange, and dissemination. Quality initiatives, methodological metadata, administrative metadata, and similar types of metadata reporting must be supported, and must be user-configurable.
63 +* XML formats for describing the relationships between groups of [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], by mapping (% style="color:#e74c3c" %)concepts(%%) and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] between these structures, and by allowing for common querying of data and metadata described with not only a single structural definition, but with a related set of structural definitions, based on these mappings.
64 +* Allow for time-related (% style="color:#e74c3c" %)concepts(%%) which are not related to the time of the observation to be used in data structures.
65 +* Allow for simple, un-coded incremental identifiers in [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]], to be used to dis-ambiguate data series/observations which do not have a simple 1-to-1 relationship with the [[time period>>doc:sdmx:Glossary.Time period.WebHome]] of the observation.
66 +* Allow for un-coded identifiers and descriptors to be associated with [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] which establish an external entity or identifier to disambiguate between otherwise identical series/observations (ie, when a [[data set>>doc:sdmx:Glossary.Data set.WebHome]] describes a group of organisations, or a set of accounts, which might otherwise have identical key values).
67 +* Allow for non-numeric [[observation values>>doc:sdmx:Glossary.Observation value.WebHome]] (usually but not always coded)
68 +* Allow "cube"-based systems (such as OLAP) to interoperate with less sophisticated systems, without necessarily losing the richness of metadata found in the more sophisticated systems.
69 +
70 +This is a very broad set of requirements, and in examining these it becomes evident that some of the requirements are very much at cross-purposes. It is almost impossible to design a single XML document type for any single function (exchange of data, exchange of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], querying, etc.) which will satisfy all of these requirements. At the same time, it was very much felt that whatever design was adopted should have a clear relationship with the information model.
71 +
72 +== 4.2 Design Approach ==
73 +
74 +The basic design for the XML Schema started with the information model. Just as the model does, the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] objects are built up from the abstract classes on which they are based. This approach means that more advanced XML tools that understand such inheritance can use thes constructs to more easily deserialize the XML instances into object based on the models.
75 +
76 +Another fundamental approach was to make constructs as consistent as possible so that tools built to process the XML could make use of reuse. This approach extends not only to the basic information model properties such as identificaitons and (% style="color:#e74c3c" %)version(%%), but also to areas such as referencing and query messages.
77 +
78 +Through the years of the standard, it was identified that users typically implemented on the previously named CompactData message and/or the GenericData message. It was determined that the UtilityData message was rarely used,and the CrossSectionalData message was used, but often in inconsistent manners due to the ambiguity in its definition. Therefore, the apporoach for this verison was to harmonize these data formats as much as possible. The result is two basic data formats; a generic format and a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] specific format. As much as possible these formats have been structured so that they are very similar in structure. In addition to the harmonisation of the generic and structure specific data formats, the requirements of the CompactData and CrossSectionalData messages have been combined into one format. This format is now flexible in that it allows for any single [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] to exist at the observation. In order to accomodate existing applications that did only use the CompactData message due to its time series orientation, time series only variations of the structure specific and generic data messages have been created. Care was taken to ensure that these time series only variations could be processed just as the general format counterparts are so as to introduce no additional requirements.
79 +
80 +These same principles were applied to the [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] messages as well. In addition, the base structure specific schemas from which data and metadata structure specific variations are created, have been structured so that they enforce a specific structure and can be easily processed in a generic mannner.
81 +
82 +Another new approach that has been introduced in this (% style="color:#e74c3c" %)version(%%) is to remove some of the generalities from the messages that existed to allow for web services to be more specific as to the contract of their functions. As opposed to have a single query message that can accomodate any [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] query and data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] queries serving as the input into a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query function­­-a specific message is created that only allows for what a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query should reasonably be expected to handle.
83 +
84 +== 4.3 SDMX-ML Packaging: Namespace Modules ==
85 +
86 +In the proposed XML Schema design, there is a packaging scheme based on the idea that XML namespaces can be used as "modules", so that any given user or application need only be familiar with a subset of the entire library in order to use it. This approach fit very well with the design described above, and is often used in major XML standards for other domains.
87 +
88 +The other major benefit of namespaces - especially in light of the requirement that [[maintenance agencies>>doc:sdmx:Glossary.Maintenance agency.WebHome]] be tracked across the potential reuse of the structures and data they maintained - is that it allows [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] to own certain namespace modules, and allows other [[maintenance agencies>>doc:sdmx:Glossary.Maintenance agency.WebHome]] to own namespaces specific to the key-families or [[metadata structure definitions>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] they also maintain.
89 +
90 +The result is a set of namespace packages which agree with the design approach described above. Some large schemas are divided into sub-modules, all within the same namespace, for ease of use. Each module or sub-module is a single instance of the W3C XML Schema Language's schema element, associated with the apropriate XML namespace. Where these modules and sub-modules have dependencies on one another, they use the XML Schema importing mechanism to draw on constructs described in another module or sub-module. Further, the namespces themselves reflect a sort of relationsip. For example, all data constructs are in a namespace based on http://www.sdmx.org/resources/sdmxml/schemas/v3_1/data. These relationships in the namespace are meant to reflect the relationship of the constructs.
91 +
92 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the common message constructs, including the common header information ("SDMXMessage.xsd") - uses all other [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] namespace modules
93 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the message footer constructs, including the common header information ("SDMXMessageFooter.xsd") - used only by the message namespace.
94 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the descriptions of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] such as key families, (% style="color:#e74c3c" %)concepts(%%), and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] ("SDMXStructure [*sub-module*].xsd")
95 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing constructs shared in common across all of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] message types ("SDMXCommon[*sub-module*].xsd") - needed for all other [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] namespace modules (also included for convenience is the XML namespace ["xml.xsd"] provided by the W3C for including the xml:lang [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] in schemas).
96 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module for structured (data structure-specific) data ("SDMXDataStructureSpecific.xsd")
97 +* A set of namespaced modules created and maintained by those who create data structure-specific schemas - not maintained by [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]
98 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module providing a generic format for reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], regardless of [[metadata structure definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] ("SDMXMetadataGeneric.xsd").
99 +* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module providing standard interfaces for interactions with a set of registry services ("SDMXRegistry[*sub-module*].xsd").
100 +
101 +The following sections describe in detail the proposed XML formats, which should be examined alongside the documentation provided. These proposed schemas are divided into the generic schemas, for which a complete set of schema definitions can be provided, and data structure-specific schemas, for which a core structure is provided (with schema code), plus a guide to how a specific data structure or [[metadata structure definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] can be (% style="color:#e74c3c" %)mapped(%%) onto the core structure.
102 +
103 += 5 RELATED DOCUMENTS =
104 +
105 +To all for more managable doucments, the remainder of this document has been divided into seven different parts, each a separate document. These parts are as follows:
106 +
107 +1. SDMX_3_1_SECTION_03A_PART_I_MESSAGE: This contains the description of the message namespaces schemas
108 +1. SDMX_3_1_SECTION_03A_PART_II_COMMON: This details the common namespace schemas
109 +1. SDMX_3_1_SECTION_03A_PART_III_STRUCTURE: This details the structure namespace (structural metadata) schemas
110 +1. SDMX_3_1_SECTION_03A_PART_IV_DATA: This details the data and metadata namespaces, as well as detailing the rules for creating strucutre specific schemas
111 +1. SDMX_3_1_SECTION_03A_PART_V_REGISTRY: This details the registry namespace schemas
112 +1. SDMX_3_1_SECTION_03A_PART_VI_SAMPLES: This provides a brief explanation of the currently available sample files.
SUZ.Methodology.Code.MethodologyClass[0]
index
... ... @@ -1,1 +1,1 @@
1 -7
1 +1