Changes for page Schema and Documentation

Last modified by Elena on 2025/09/10 11:19

From 8.1 to 9.1 From 10.2 to 11.1

From version 9.1

edited by Artur
on 2025/07/14 10:19

Change comment: Imported from XAR

To version 10.2

edited by Elena
on 2025/07/25 16:58

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--xwiki:XWiki.arturkryazhev
++xwiki:XWiki.elenasemanticproorg

Content

@@ -6,19 +6,19 @@
  == 1.1 History and Version 3.0 Developments ==
--The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards (% style="color:#e74c3c" %)Version(%%) 1.0 established an information model which described aggregated statistical [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] needed to exchange them in a standard fashion. This drew on the earlier example of the GESMES/TS standard. Based on the [[SDMX information model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]], several formats were developed: XML formats for exchange of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], and queries for these ([[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]), and EDIFACT formats for the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]]). These standards supported a number of exchange patterns, characterized as "bilateral", "gateway", and "data-sharing" models, as described in the Framework document in the (% style="color:#e74c3c" %)Version(%%) 1.0 standards package.
++The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards Version 1.0 established an information model which described aggregated statistical [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] needed to exchange them in a standard fashion. This drew on the earlier example of the GESMES/TS standard. Based on the [[SDMX information model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]], several formats were developed: XML formats for exchange of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], and queries for these ([[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]), and EDIFACT formats for the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]]). These standards supported a number of exchange patterns, characterized as "bilateral", "gateway", and "data-sharing" models, as described in the Framework document in the Version 1.0 standards package.
--(% style="color:#e74c3c" %)Versions(%%) 2.0 and 2.1 built on this foundation to provide a higher degree of support for all of these models, with an emphasis on data sharing in the form of a set of standard registry services interfaces. Support for new types of metadata exchange and reporting were added, with a focus on "[[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]" concerned with quality, methodology, and other issues. Further, the ability to provide metadata about the relationships between [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and structures was expanded, providing more support for data cubes. Overall, the scope of the (% style="color:#e74c3c" %)Version(%%) 2.1 [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards was much broader, and included a larger set of message types in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] formats.
++Versions 2.0 and 2.1 built on this foundation to provide a higher degree of support for all of these models, with an emphasis on data sharing in the form of a set of standard registry services interfaces. Support for new types of metadata exchange and reporting were added, with a focus on "[[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]" concerned with quality, methodology, and other issues. Further, the ability to provide metadata about the relationships between [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and structures was expanded, providing more support for data cubes. Overall, the scope of the Version 2.1 [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Technical Standards was much broader, and included a larger set of message types in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] formats.
--For (% style="color:#e74c3c" %)Version(%%) 3.0, the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] specification has been rationalised and tightened while maintaining the fundamentals and principles established in 2.1. The documents for describing [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] remain the same except where there have been changes to the information model either to enhance existing structures, add new ones, or remove those deprecated such as the Structure Set.
++For Version 3.0, the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] specification has been rationalised and tightened while maintaining the fundamentals and principles established in 2.1. The documents for describing [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] remain the same except where there have been changes to the information model either to enhance existing structures, add new ones, or remove those deprecated such as the Structure Set.
--The deprecation of the SOAP web services specification in (% style="color:#e74c3c" %)Version(%%) 3.0 has resulted in the removal of all [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Query documents. Standardisation on the REST API and HTTP GET for registry structure queries means separate 'query request' documents are no longer required.
++The deprecation of the SOAP web services specification in Version 3.0 has resulted in the removal of all [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Query documents. Standardisation on the REST API and HTTP GET for registry structure queries means separate 'query request' documents are no longer required.
--The opportunity has also been taken to rationalise the plethora of alternative data documents accumulated from (% style="color:#e74c3c" %)Versions(%%) 1.0, 2.0 and 2.1. All, including Generic, Utility, Compact and Cross-Sectional have been deprecated except for Structure Specific Data which becomes the sole standard XML format for data updates, revisions and bilateral exchanges in (% style="color:#e74c3c" %)Version(%%) 3.0.
++The opportunity has also been taken to rationalise the plethora of alternative data documents accumulated from Versions 1.0, 2.0 and 2.1. All, including Generic, Utility, Compact and Cross-Sectional have been deprecated except for Structure Specific Data which becomes the sole standard XML format for data updates, revisions and bilateral exchanges in Version 3.0.
  == 1.2 The XML Design ==
--All of these document types will share a common "envelope" at the message (% style="color:#e74c3c" %)level(%%) ("SDMXMessage.xsd"), as well as a set of common low-(% style="color:#e74c3c" %)level(%%) [[components>>doc:sdmx:Glossary.Component.WebHome]] ("SDMXCommon[subset].xsd") so that header information and basic structure of a message will always be the same.
++All of these document types will share a common "envelope" at the message level ("SDMXMessage.xsd"), as well as a set of common low-level [[components>>doc:sdmx:Glossary.Component.WebHome]] ("SDMXCommon[subset].xsd") so that header information and basic structure of a message will always be the same.
  * Schema and sub-modules for describing all types of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] - for [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]]), for [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] ([[metadata structure definitions>>doc:sdmx:Glossary.Metadata structure definition.WebHome]]), for related groups of metadata and data structures, and for all types of structural objects involved in registry-based exchanges ("SDMXStructure[*subset*].xsd")
  * Structure specific data schema for for updates, revisions/bilateral exchange ("SDMXDataStructureSpecific.xsd")
@@ -27,13 +27,13 @@
  == 1.3 Fostering the Use of a Standard SDMX-ML ==
--In addition to these different formats, standard mappings and corresponding transformation tools have been developed for the creation of data structure-specific schemas from structure descriptions, to transform XML data instances from one XML data description format to another, and from these formats into the corresponding [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages. This (% style="color:#e74c3c" %)level(%%) of free tools support will foster the early use of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] and permit the data to be easily used across all processes, which is otherwise a difficult requirement to meet. Ultimately, it is the fact that all formats share a common information model that enables this approach to meet the wide set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] requirements.
++In addition to these different formats, standard mappings and corresponding transformation tools have been developed for the creation of data structure-specific schemas from structure descriptions, to transform XML data instances from one XML data description format to another, and from these formats into the corresponding [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages. This level of free tools support will foster the early use of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] and permit the data to be easily used across all processes, which is otherwise a difficult requirement to meet. Ultimately, it is the fact that all formats share a common information model that enables this approach to meet the wide set of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] requirements.
  = 2 NORMATIVE REFERENCES =
--W3C XML Schema Definition Language, (% style="color:#e74c3c" %)version(%%) 1.0 (URL: http://www.w3c.org/XML/Schema#dev), World Wide Web Consortium
++W3C XML Schema Definition Language, version 1.0 (URL: http://www.w3c.org/XML/Schema#dev), World Wide Web Consortium
--W3C Extensible Markup Language, (% style="color:#e74c3c" %)version(%%) 1.0, Third Edition (URL: http://www.w3c.org/TR/2004/REC-xml-20040204/), World Wide Web Consortium
++W3C Extensible Markup Language, version 1.0, Third Edition (URL: http://www.w3c.org/TR/2004/REC-xml-20040204/), World Wide Web Consortium
  = 3 CONFORMANCE =
@@ -55,13 +55,13 @@
  * Users (and registry services) may not know about a specific data structure, and will need to be able to handle data across data structures, and even (for, say, a comparison service) to put data structured according to multiple data structures in a single XML instance.
  * The XML should conform to the Information Model as closely as possible, and promote simple mapping from the XML instances to model based objects
  * The XML should behave as "normally" as possible within standard XML tools such as web development environments, parsers, guided editing tools, etc.
--* Data should be able to be structured in one or two (% style="color:#e74c3c" %)levels(%%), with the two (% style="color:#e74c3c" %)level(%%) format allowing for a single [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] which serves to disambitguate the observations. In addtion, it should still be possible to restrict this more generalized structure to time series only formats. However, doing so should not add any additional burden in terms of how data is processed.
++* Data should be able to be structured in one or two levels, with the two level format allowing for a single [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] which serves to disambitguate the observations. In addtion, it should still be possible to restrict this more generalized structure to time series only formats. However, doing so should not add any additional burden in terms of how data is processed.
  * Data structure and metadata structure specific formats should be able to be generically processed, without having to understand the entire structure; which is to say that the structure-specific schemas should serve to validate the information, but not be required to process it.
--* XML formats should promote re-use of common semantics, (% style="color:#e74c3c" %)concepts(%%), and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] to the greatest possible extent, while still recognizing the agency which maintains a specific resource (a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]], a data structure, a [[data set>>doc:sdmx:Glossary.Data set.WebHome]], etc.)
++* XML formats should promote re-use of common semantics, [[concepts>>doc:sdmx:Glossary.Concept.WebHome]], and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] to the greatest possible extent, while still recognizing the agency which maintains a specific resource (a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]], a data structure, a [[data set>>doc:sdmx:Glossary.Data set.WebHome]], etc.)
  * XML formats must support interactions of applications with standard registry services, based on standard interfaces. These must function both as web services, and as services operating over http and similar protocols.
  * XML formats must support the reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] which is not structural in nature, but which constitutes a primary information flow of metadata attached to other parts of the statistical collection, reporting, processing, exchange, and dissemination. Quality initiatives, methodological metadata, administrative metadata, and similar types of metadata reporting must be supported, and must be user-configurable.
--* XML formats for describing the relationships between groups of [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], by mapping (% style="color:#e74c3c" %)concepts(%%) and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] between these structures, and by allowing for common querying of data and metadata described with not only a single structural definition, but with a related set of structural definitions, based on these mappings.
--* Allow for time-related (% style="color:#e74c3c" %)concepts(%%) which are not related to the time of the observation to be used in data structures.
++* XML formats for describing the relationships between groups of [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] and [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], by mapping [[concepts>>doc:sdmx:Glossary.Concept.WebHome]] and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] between these structures, and by allowing for common querying of data and metadata described with not only a single structural definition, but with a related set of structural definitions, based on these mappings.
++* Allow for time-related [[concepts>>doc:sdmx:Glossary.Concept.WebHome]] which are not related to the time of the observation to be used in data structures.
  * Allow for simple, un-coded incremental identifiers in [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]], to be used to dis-ambiguate data series/observations which do not have a simple 1-to-1 relationship with the [[time period>>doc:sdmx:Glossary.Time period.WebHome]] of the observation.
  * Allow for un-coded identifiers and descriptors to be associated with [[data structure definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] which establish an external entity or identifier to disambiguate between otherwise identical series/observations (ie, when a [[data set>>doc:sdmx:Glossary.Data set.WebHome]] describes a group of organisations, or a set of accounts, which might otherwise have identical key values).
  * Allow for non-numeric [[observation values>>doc:sdmx:Glossary.Observation value.WebHome]] (usually but not always coded)
@@ -73,13 +73,13 @@
  The basic design for the XML Schema started with the information model. Just as the model does, the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] objects are built up from the abstract classes on which they are based. This approach means that more advanced XML tools that understand such inheritance can use thes constructs to more easily deserialize the XML instances into object based on the models.
--Another fundamental approach was to make constructs as consistent as possible so that tools built to process the XML could make use of reuse. This approach extends not only to the basic information model properties such as identificaitons and (% style="color:#e74c3c" %)version(%%), but also to areas such as referencing and query messages.
++Another fundamental approach was to make constructs as consistent as possible so that tools built to process the XML could make use of reuse. This approach extends not only to the basic information model properties such as identificaitons and [[version>>doc:sdmx:Glossary.Version.WebHome]], but also to areas such as referencing and query messages.
  Through the years of the standard, it was identified that users typically implemented on the previously named CompactData message and/or the GenericData message. It was determined that the UtilityData message was rarely used,and the CrossSectionalData message was used, but often in inconsistent manners due to the ambiguity in its definition. Therefore, the apporoach for this verison was to harmonize these data formats as much as possible. The result is two basic data formats; a generic format and a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] specific format. As much as possible these formats have been structured so that they are very similar in structure. In addition to the harmonisation of the generic and structure specific data formats, the requirements of the CompactData and CrossSectionalData messages have been combined into one format. This format is now flexible in that it allows for any single [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] to exist at the observation. In order to accomodate existing applications that did only use the CompactData message due to its time series orientation, time series only variations of the structure specific and generic data messages have been created. Care was taken to ensure that these time series only variations could be processed just as the general format counterparts are so as to introduce no additional requirements.
  These same principles were applied to the [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] messages as well. In addition, the base structure specific schemas from which data and metadata structure specific variations are created, have been structured so that they enforce a specific structure and can be easily processed in a generic mannner.
--Another new approach that has been introduced in this (% style="color:#e74c3c" %)version(%%) is to remove some of the generalities from the messages that existed to allow for web services to be more specific as to the contract of their functions. As opposed to have a single query message that can accomodate any [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] query and data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] queries serving as the input into a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query function-a specific message is created that only allows for what a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query should reasonably be expected to handle.
++Another new approach that has been introduced in this version is to remove some of the generalities from the messages that existed to allow for web services to be more specific as to the contract of their functions. As opposed to have a single query message that can accomodate any [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] query and data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] queries serving as the input into a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query function-a specific message is created that only allows for what a [[codelist>>doc:sdmx:Glossary.Code list.WebHome]] query should reasonably be expected to handle.
  == 4.3 SDMX-ML Packaging: Namespace Modules ==
@@ -91,7 +91,7 @@
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the common message constructs, including the common header information ("SDMXMessage.xsd") - uses all other [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] namespace modules
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the message footer constructs, including the common header information ("SDMXMessageFooter.xsd") - used only by the message namespace.
--* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the descriptions of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] such as key families, (% style="color:#e74c3c" %)concepts(%%), and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] ("SDMXStructure [*sub-module*].xsd")
++* An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing the descriptions of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] such as key families, [[concepts>>doc:sdmx:Glossary.Concept.WebHome]], and [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] ("SDMXStructure [*sub-module*].xsd")
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module containing constructs shared in common across all of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] message types ("SDMXCommon[*sub-module*].xsd") - needed for all other [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] namespace modules (also included for convenience is the XML namespace ["xml.xsd"] provided by the W3C for including the xml:lang [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] in schemas).
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module for structured (data structure-specific) data ("SDMXDataStructureSpecific.xsd")
  * A set of namespaced modules created and maintained by those who create data structure-specific schemas - not maintained by [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]
@@ -98,7 +98,7 @@
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module providing a generic format for reporting of [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], regardless of [[metadata structure definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] ("SDMXMetadataGeneric.xsd").
  * An [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Namespace Module providing standard interfaces for interactions with a set of registry services ("SDMXRegistry[*sub-module*].xsd").
--The following sections describe in detail the proposed XML formats, which should be examined alongside the documentation provided. These proposed schemas are divided into the generic schemas, for which a complete set of schema definitions can be provided, and data structure-specific schemas, for which a core structure is provided (with schema code), plus a guide to how a specific data structure or [[metadata structure definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] can be (% style="color:#e74c3c" %)mapped(%%) onto the core structure.
++The following sections describe in detail the proposed XML formats, which should be examined alongside the documentation provided. These proposed schemas are divided into the generic schemas, for which a complete set of schema definitions can be provided, and data structure-specific schemas, for which a core structure is provided (with schema code), plus a guide to how a specific data structure or [[metadata structure definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]] can be mapped onto the core structure.
  = 5 RELATED DOCUMENTS =