Skip to Content

Changes for page Schema and Documentation

Last modified by Helena K. on 2026/06/09 12:21
From version 15.3
edited by Helena K.
on 2026/06/09 12:19
Change comment: There is no comment for this version
To version 15.2
edited by Helena K.
on 2026/06/09 12:11
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -46,58 +46,58 @@
46 46  To understand the relationships between the several document types, it is important to have some familiarity with the requirements they are designed to fulfil.
47 47  
48 48  * Large amounts of data must be captured in a reasonably compact format, because of the potential size of databases being exchanged.
49 -* It must be possible to send [[incremental updates>>doc:sdmx:Glossary 2\.1.Incremental update.WebHome]], rather than entire, complete databases. The validation of such exchanges demands not that an entire [[data set>>doc:sdmx:Glossary 2\.1.Data set.WebHome]] be exchanged, but only that enough information be sent to ensure accurate updating and revision processes.
49 +* It must be possible to send incremental updates, rather than entire, complete databases. The validation of such exchanges demands not that an entire data set be exchanged, but only that enough information be sent to ensure accurate updating and revision processes.
50 50  * Structural information as well as data will need to be transmitted.
51 51  * There must be a reliable transformation to and from the GESMES/TS EDIFACT syntax.
52 -* It should be possible to present natural-[[language>>doc:sdmx:Glossary 2\.1.Language.WebHome]] information in multiple, equivalent [[languages>>doc:sdmx:Glossary 2\.1.Language.WebHome]].
52 +* It should be possible to present natural-language information in multiple, equivalent languages.
53 53  * To support web services and similar technological approaches, there is a requirement to send queries to information sources as well as data and structures.
54 54  * Users (and registry services) may not know about a specific data structure, and will need to be able to handle data across data structures, and even (for, say, a comparison service) to put data structured according to multiple data structures in a single XML instance.
55 55  * The XML should conform to the Information Model as closely as possible, and promote simple mapping from the XML instances to model based objects
56 56  * The XML should behave as "normally" as possible within standard XML tools such as web development environments, parsers, guided editing tools, etc.
57 -* Data should be able to be structured in one or two levels, with the two level format allowing for a single [[dimension>>doc:sdmx:Glossary 2\.1.Dimension.WebHome]] which serves to disambitguate the observations. In addtion, it should still be possible to restrict this more generalized structure to [[time series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] only formats. However, doing so should not add any additional burden in terms of how data is processed.
57 +* Data should be able to be structured in one or two levels, with the two level format allowing for a single dimension which serves to disambitguate the observations. In addtion, it should still be possible to restrict this more generalized structure to time series only formats. However, doing so should not add any additional burden in terms of how data is processed.
58 58  * Data structure and metadata structure specific formats should be able to be generically processed, without having to understand the entire structure; which is to say that the structure-specific schemas should serve to validate the information, but not be required to process it.
59 -* XML formats should promote re-use of common semantics, [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], and [[codelists>>doc:sdmx:Glossary 2\.1.Code list.WebHome]] to the greatest possible extent, while still recognizing the agency which maintains a specific resource (a [[codelist>>doc:sdmx:Glossary 2\.1.Code list.WebHome]], a data structure, a [[data set>>doc:sdmx:Glossary 2\.1.Data set.WebHome]], etc.)
59 +* XML formats should promote re-use of common semantics, concepts, and codelists to the greatest possible extent, while still recognizing the agency which maintains a specific resource (a codelist, a data structure, a data set, etc.)
60 60  * XML formats must support interactions of applications with standard registry services, based on standard interfaces. These must function both as web services, and as services operating over http and similar protocols.
61 -* XML formats must support the reporting of [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] which is not structural in nature, but which constitutes a primary information flow of metadata attached to other parts of the statistical collection, reporting, processing, exchange, and dissemination. Quality initiatives, methodological metadata, administrative metadata, and similar types of metadata reporting must be supported, and must be user-configurable.
62 -* XML formats for describing the relationships between groups of [[metadata sets>>doc:sdmx:Glossary 2\.1.Metadata set.WebHome]] and [[data sets>>doc:sdmx:Glossary 2\.1.Data set.WebHome]], by mapping [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] and [[codelists>>doc:sdmx:Glossary 2\.1.Code list.WebHome]] between these structures, and by allowing for common querying of data and metadata described with not only a single structural definition, but with a related set of structural definitions, based on these mappings.
63 -* Allow for time-related [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]] which are not related to the time of the observation to be used in data structures.
64 -* Allow for simple, un-coded incremental identifiers in [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]], to be used to dis-ambiguate data [[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]/observations which do not have a simple 1-to-1 relationship with the [[time period>>doc:sdmx:Glossary 2\.1.Time period.WebHome]] of the observation.
65 -* Allow for un-coded identifiers and descriptors to be associated with [[data structure definitions>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] which establish an external entity or identifier to disambiguate between otherwise identical [[series>>doc:sdmx:Glossary 2\.1.Series.WebHome]]/observations (ie, when a [[data set>>doc:sdmx:Glossary 2\.1.Data set.WebHome]] describes a group of organisations, or a set of accounts, which might otherwise have identical key values).
66 -* Allow for non-numeric [[observation values>>doc:sdmx:Glossary 2\.1.Observation value.WebHome]] (usually but not always coded)
61 +* XML formats must support the reporting of reference metadata which is not structural in nature, but which constitutes a primary information flow of metadata attached to other parts of the statistical collection, reporting, processing, exchange, and dissemination. Quality initiatives, methodological metadata, administrative metadata, and similar types of metadata reporting must be supported, and must be user-configurable.
62 +* XML formats for describing the relationships between groups of metadata sets and data sets, by mapping concepts and codelists between these structures, and by allowing for common querying of data and metadata described with not only a single structural definition, but with a related set of structural definitions, based on these mappings.
63 +* Allow for time-related concepts which are not related to the time of the observation to be used in data structures.
64 +* Allow for simple, un-coded incremental identifiers in data structure definitions, to be used to dis-ambiguate data series/observations which do not have a simple 1-to-1 relationship with the time period of the observation.
65 +* Allow for un-coded identifiers and descriptors to be associated with data structure definitions which establish an external entity or identifier to disambiguate between otherwise identical series/observations (ie, when a data set describes a group of organisations, or a set of accounts, which might otherwise have identical key values).
66 +* Allow for non-numeric observation values (usually but not always coded)
67 67  * Allow "cube"-based systems (such as OLAP) to interoperate with less sophisticated systems, without necessarily losing the richness of metadata found in the more sophisticated systems.
68 68  
69 -This is a very broad set of requirements, and in examining these it becomes evident that some of the requirements are very much at cross-purposes. It is almost impossible to design a single XML document type for any single function (exchange of data, exchange of [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]], querying, etc.) which will satisfy all of these requirements. At the same time, it was very much felt that whatever design was adopted should have a clear relationship with the information model.
69 +This is a very broad set of requirements, and in examining these it becomes evident that some of the requirements are very much at cross-purposes. It is almost impossible to design a single XML document type for any single function (exchange of data, exchange of reference metadata, querying, etc.) which will satisfy all of these requirements. At the same time, it was very much felt that whatever design was adopted should have a clear relationship with the information model.
70 70  
71 71  == 4.2 Design Approach ==
72 72  
73 -The basic design for the XML Schema started with the information model. Just as the model does, the [[structural metadata>>doc:sdmx:Glossary 2\.1.Structural metadata.WebHome]] objects are built up from the abstract classes on which they are based. This approach means that more advanced XML tools that understand such inheritance can use thes constructs to more easily deserialize the XML instances into object based on the models.
73 +The basic design for the XML Schema started with the information model. Just as the model does, the structural metadata objects are built up from the abstract classes on which they are based. This approach means that more advanced XML tools that understand such inheritance can use thes constructs to more easily deserialize the XML instances into object based on the models.
74 74  
75 -Another fundamental approach was to make constructs as consistent as possible so that tools built to process the XML could make use of reuse. This approach extends not only to the basic information model properties such as identificaitons and [[version>>doc:sdmx:Glossary 2\.1.Version.WebHome]], but also to areas such as referencing and query messages.
75 +Another fundamental approach was to make constructs as consistent as possible so that tools built to process the XML could make use of reuse. This approach extends not only to the basic information model properties such as identificaitons and version, but also to areas such as referencing and query messages.
76 76  
77 -Through the years of the standard, it was identified that users typically implemented on the previously named CompactData message and/or the GenericData message. It was determined that the UtilityData message was rarely used,and the CrossSectionalData message was used, but often in inconsistent manners due to the ambiguity in its definition. Therefore, the apporoach for this verison was to harmonize these data formats as much as possible. The result is two basic data formats; a generic format and a [[data structure definition>>doc:sdmx:Glossary 2\.1.Data structure definition.WebHome]] specific format. As much as possible these formats have been structured so that they are very similar in structure. In addition to the harmonisation of the generic and structure specific data formats, the requirements of the CompactData and CrossSectionalData messages have been combined into one format. This format is now flexible in that it allows for any single [[dimension>>doc:sdmx:Glossary 2\.1.Dimension.WebHome]] to exist at the observation. In order to accomodate existing applications that did only use the CompactData message due to its [[time series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] orientation, [[time series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] only variations of the structure specific and generic data messages have been created. Care was taken to ensure that these [[time series>>doc:sdmx:Glossary 2\.1.Series.WebHome]] only variations could be processed just as the general format counterparts are so as to introduce no additional requirements.
77 +Through the years of the standard, it was identified that users typically implemented on the previously named CompactData message and/or the GenericData message. It was determined that the UtilityData message was rarely used,and the CrossSectionalData message was used, but often in inconsistent manners due to the ambiguity in its definition. Therefore, the apporoach for this verison was to harmonize these data formats as much as possible. The result is two basic data formats; a generic format and a data structure definition specific format. As much as possible these formats have been structured so that they are very similar in structure. In addition to the harmonisation of the generic and structure specific data formats, the requirements of the CompactData and CrossSectionalData messages have been combined into one format. This format is now flexible in that it allows for any single dimension to exist at the observation. In order to accomodate existing applications that did only use the CompactData message due to its time series orientation, time series only variations of the structure specific and generic data messages have been created. Care was taken to ensure that these time series only variations could be processed just as the general format counterparts are so as to introduce no additional requirements.
78 78  
79 -These same principles were applied to the [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] messages as well. In addition, the base structure specific schemas from which data and metadata structure specific variations are created, have been structured so that they enforce a specific structure and can be easily processed in a generic mannner.
79 +These same principles were applied to the reference metadata messages as well. In addition, the base structure specific schemas from which data and metadata structure specific variations are created, have been structured so that they enforce a specific structure and can be easily processed in a generic mannner.
80 80  
81 -Another new approach that has been introduced in this version is to remove some of the generalities from the messages that existed to allow for web services to be more specific as to the contract of their functions. As opposed to have a single query message that can accomodate any [[structural metadata>>doc:sdmx:Glossary 2\.1.Structural metadata.WebHome]] query and data and [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]] queries serving as the input into a [[codelist>>doc:sdmx:Glossary 2\.1.Code list.WebHome]] query function­­-a specific message is created that only allows for what a [[codelist>>doc:sdmx:Glossary 2\.1.Code list.WebHome]] query should reasonably be expected to handle.
81 +Another new approach that has been introduced in this version is to remove some of the generalities from the messages that existed to allow for web services to be more specific as to the contract of their functions. As opposed to have a single query message that can accomodate any structural metadata query and data and reference metadata queries serving as the input into a codelist query function­­-a specific message is created that only allows for what a codelist query should reasonably be expected to handle.
82 82  
83 83  == 4.3 SDMX-ML Packaging: Namespace Modules ==
84 84  
85 85  In the proposed XML Schema design, there is a packaging scheme based on the idea that XML namespaces can be used as "modules", so that any given user or application need only be familiar with a subset of the entire library in order to use it. This approach fit very well with the design described above, and is often used in major XML standards for other domains.
86 86  
87 -The other major benefit of namespaces - especially in light of the requirement that [[maintenance agencies>>doc:sdmx:Glossary 2\.1.Maintenance agency.WebHome]] be tracked across the potential reuse of the structures and data they maintained - is that it allows [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] to own certain namespace modules, and allows other [[maintenance agencies>>doc:sdmx:Glossary 2\.1.Maintenance agency.WebHome]] to own namespaces specific to the key-families or [[metadata structure definitions>>doc:sdmx:Glossary 2\.1.Metadata structure definition.WebHome]] they also maintain.
87 +The other major benefit of namespaces - especially in light of the requirement that maintenance agencies be tracked across the potential reuse of the structures and data they maintained - is that it allows SDMX to own certain namespace modules, and allows other maintenance agencies to own namespaces specific to the key-families or metadata structure definitions they also maintain.
88 88  
89 -The result is a set of namespace packages which agree with the design approach described above. Some large schemas are divided into sub-modules, all within the same namespace, for ease of use. Each module or sub-module is a single instance of the W3C XML Schema [[Language>>doc:sdmx:Glossary 2\.1.Language.WebHome]]'s schema element, associated with the apropriate XML namespace. Where these modules and sub-modules have dependencies on one another, they use the XML Schema importing mechanism to draw on constructs described in another module or sub-module. Further, the namespces themselves reflect a sort of relationsip. For example, all data constructs are in a namespace based on http://www.sdmx.org/resources/sdmxml/schemas/v3_1/data. These relationships in the namespace are meant to reflect the relationship of the constructs.
89 +The result is a set of namespace packages which agree with the design approach described above. Some large schemas are divided into sub-modules, all within the same namespace, for ease of use. Each module or sub-module is a single instance of the W3C XML Schema Language's schema element, associated with the apropriate XML namespace. Where these modules and sub-modules have dependencies on one another, they use the XML Schema importing mechanism to draw on constructs described in another module or sub-module. Further, the namespces themselves reflect a sort of relationsip. For example, all data constructs are in a namespace based on http://www.sdmx.org/resources/sdmxml/schemas/v3_1/data. These relationships in the namespace are meant to reflect the relationship of the constructs.
90 90  
91 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module containing the common message constructs, including the common header information ("SDMXMessage.xsd") - uses all other [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] namespace modules
92 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module containing the message footer constructs, including the common header information ("SDMXMessageFooter.xsd") - used only by the message namespace.
93 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module containing the descriptions of [[structural metadata>>doc:sdmx:Glossary 2\.1.Structural metadata.WebHome]] such as key families, [[concepts>>doc:sdmx:Glossary 2\.1.Concept.WebHome]], and [[codelists>>doc:sdmx:Glossary 2\.1.Code list.WebHome]] ("SDMXStructure [*sub-module*].xsd")
94 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module containing constructs shared in common across all of the [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] message types ("SDMXCommon[*sub-module*].xsd") - needed for all other [[SDMX-ML>>doc:sdmx:Glossary 2\.1.SDMX-ML.WebHome]] namespace modules (also included for convenience is the XML namespace ["xml.xsd"] provided by the W3C for including the xml:lang [[attribute>>doc:sdmx:Glossary 2\.1.Attribute.WebHome]] in schemas).
95 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module for structured (data structure-specific) data ("SDMXDataStructureSpecific.xsd")
96 -* A set of namespaced modules created and maintained by those who create data structure-specific schemas - not maintained by [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]]
97 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module providing a generic format for reporting of [[reference metadata>>doc:sdmx:Glossary 2\.1.Reference metadata.WebHome]], regardless of [[metadata structure definition>>doc:sdmx:Glossary 2\.1.Metadata structure definition.WebHome]] ("SDMXMetadataGeneric.xsd").
98 -* An [[SDMX>>doc:sdmx:Glossary 2\.1.Statistical data and metadata exchange.WebHome]] Namespace Module providing standard interfaces for interactions with a set of registry services ("SDMXRegistry[*sub-module*].xsd").
91 +* An SDMX Namespace Module containing the common message constructs, including the common header information ("SDMXMessage.xsd") - uses all other SDMX-ML namespace modules
92 +* An SDMX Namespace Module containing the message footer constructs, including the common header information ("SDMXMessageFooter.xsd") - used only by the message namespace.
93 +* An SDMX Namespace Module containing the descriptions of structural metadata such as key families, concepts, and codelists ("SDMXStructure [*sub-module*].xsd")
94 +* An SDMX Namespace Module containing constructs shared in common across all of the SDMX message types ("SDMXCommon[*sub-module*].xsd") - needed for all other SDMX-ML namespace modules (also included for convenience is the XML namespace ["xml.xsd"] provided by the W3C for including the xml:lang attribute in schemas).
95 +* An SDMX Namespace Module for structured (data structure-specific) data ("SDMXDataStructureSpecific.xsd")
96 +* A set of namespaced modules created and maintained by those who create data structure-specific schemas - not maintained by SDMX
97 +* An SDMX Namespace Module providing a generic format for reporting of reference metadata, regardless of metadata structure definition ("SDMXMetadataGeneric.xsd").
98 +* An SDMX Namespace Module providing standard interfaces for interactions with a set of registry services ("SDMXRegistry[*sub-module*].xsd").
99 99  
100 -The following sections describe in detail the proposed XML formats, which should be examined alongside the documentation provided. These proposed schemas are divided into the generic schemas, for which a complete set of schema definitions can be provided, and data structure-specific schemas, for which a core structure is provided (with schema code), plus a guide to how a specific data structure or [[metadata structure definition>>doc:sdmx:Glossary 2\.1.Metadata structure definition.WebHome]] can be mapped onto the core structure.
100 +The following sections describe in detail the proposed XML formats, which should be examined alongside the documentation provided. These proposed schemas are divided into the generic schemas, for which a complete set of schema definitions can be provided, and data structure-specific schemas, for which a core structure is provided (with schema code), plus a guide to how a specific data structure or metadata structure definition can be mapped onto the core structure.
101 101  
102 102  = 5 RELATED DOCUMENTS =
103 103  
© Semantic R&D Group, 2026