Changes for page Guidelines for SDMX Data Structure Definitions

Last modified by Artur K. on 2026/05/29 14:28

From 1.7 to 1.8 From 1.16 to 1.17

From version 1.8

edited by Helena K.
on 2026/01/15 12:38

Change comment: There is no comment for this version

To version 1.16

edited by Helena K.
on 2026/01/15 12:48

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -247,26 +247,23 @@
  **Table 4. General comparison of data structuring approaches**
--|(% style="width:360px" %)**Many pure concepts**|(% style="width:1255px" %)**Few composite concepts**
--|(% style="width:360px" %)cleaner data structure|(% style="width:1255px" %)(((
++|(% style="width:416px" %)**Many pure concepts**|(% style="width:1199px" %)**Few composite concepts**
++|(% style="width:416px" %)cleaner data structure|(% style="width:1199px" %)(((
  Mixed dimensions may be composed inconsistently making the decomposition into purer concepts and code lists difficult (requiring complex mapping etc.). Information that corresponds to the same concept may be included in different dimensions, e.g. reference year is contained in the indicator dimension in the first example but in the unit in the second example below. The optimal common data structure would consist of Economic Indicator, Unit, and Base period.
  [[image:1768469652632-803.png||height="106" width="352"]]
--
--
  )))
--|(% style="width:360px" %)shorter and simpler code lists|(% style="width:1255px" %)code lists longer and more complex, may require hierarchy to be “readable”
--|(% style="width:360px" %)more flexible in terms of defining constraints, but constraints more complex|(% style="width:1255px" %)simpler constraints, but some constraints may be difficult to be represented because of mixed dimensions. Consider for instance a constraint “Base period = 1995” in the above example, where some observations include the base period in the Economic Indicator dimension, others in the Unit dimension. Instead of specifying a constraint on a pure Base Period dimension, the constraints may have to be specified at observation (or time series) level
--|(% style="width:360px" %)more flexible in terms of mapping to other data structures (used by other systems), further processing and analysis (e.g. tabulation, dissemination format), and future needs|(% style="width:1255px" %)“mixed” dimensions make data structure less flexible in these respects
--|(% style="width:360px" %)longer (i.e. more complex) observation keys|(% style="width:1255px" %)shorter keys
--|(% style="width:360px" %)special values of code lists such as “not applicable”, “total” may be rather heavily used|(% style="width:1255px" %)less usage of these special values
--|(% style="width:360px" %)creates sparse data if many observations use “not applicable”|(% style="width:1255px" %)way to avoid sparseness
--|(% style="width:360px" %)many constraints may be necessary due to sparseness|(% style="width:1255px" %)typically fewer constraints required because data are less sparse
--|(% style="width:360px" %) |(% style="width:1255px" %)
--|(% style="width:360px" %) |(% style="width:1255px" %)
++|(% style="width:416px" %)shorter and simpler code lists|(% style="width:1199px" %)code lists longer and more complex, may require hierarchy to be “readable”
++|(% style="width:416px" %)more flexible in terms of defining constraints, but constraints more complex|(% style="width:1199px" %)simpler constraints, but some constraints may be difficult to be represented because of mixed dimensions. Consider for instance a constraint “Base period = 1995” in the above example, where some observations include the base period in the Economic Indicator dimension, others in the Unit dimension. Instead of specifying a constraint on a pure Base Period dimension, the constraints may have to be specified at observation (or time series) level
++|(% style="width:416px" %)more flexible in terms of mapping to other data structures (used by other systems), further processing and analysis (e.g. tabulation, dissemination format), and future needs|(% style="width:1199px" %)“mixed” dimensions make data structure less flexible in these respects
++|(% style="width:416px" %)longer (i.e. more complex) observation keys|(% style="width:1199px" %)shorter keys
++|(% style="width:416px" %)special values of code lists such as “not applicable”, “total” may be rather heavily used|(% style="width:1199px" %)less usage of these special values
++|(% style="width:416px" %)creates sparse data if many observations use “not applicable”|(% style="width:1199px" %)way to avoid sparseness
++|(% style="width:416px" %)many constraints may be necessary due to sparseness|(% style="width:1199px" %)typically fewer constraints required because data are less sparse
++|(% style="width:416px" %)many dimensions are tantamount to many attachment levels for attributes (i.e. DSD more flexible in terms of attribute attachment)|(% style="width:1199px" %)less dimensions = less possible attribute attachment levels
++|(% style="width:416px" %)more difficult to handle by an end user|(% style="width:1199px" %)presumably more easily comprehensible and manageable by an end user
++|(% style="width:416px" %)more flexible in terms of defining queries; can be mapped to any “mixed” representation|(% style="width:1199px" %)less flexible in terms of search and retrieval
--
--
  The latter two aspects mentioned in the table could be summarized as the “many pure dimensions” approach being more difficult to handle for a “basic” user, but providing fewer options for an “advanced” user. When it comes to dissemination to end users, a purer data structure is the appropriate format for consumption by applications and advanced users. For less advanced user groups it makes sense to hide the (for them: unnecessary) complexity by means of concatenating dimensions, for instance to create a time series view.
  Comparing single-purpose and single-domain exchange scenarios with multi-domain and/or multi-purpose scenarios, pure concepts are typically easier to achieve in the former, whereas composite concepts/dimensions may make life easier in the latter, especially because certain cross-classification concepts may only apply to some domains and/or purposes covered. “Purpose” mean“mixed” dimensions make data structure less
@@ -279,18 +279,13 @@
  |**Level of data exchange**|**Pure vs. composite concepts approach**
  |**within an organization**|(((
  Depends on diversity of systems involved in data exchange.
--
  The approach that requires the least mapping (and similar processing) steps between the two communicating data structures is preferable in terms of a “quick win” solution.
--
  In general, a more granular model is preferable due to its flexibility that helps support potential future needs (with respect to processing, analysis, exchange, dissemination, etc.).
--
  However, an internal exchange should not be made more complex than necessary. If the structures of the communicating systems are comparable, it may not make sense to create an artificial intermediary structure that is more pure, but also more complex than both underlying structures.
--
  Still, as a longer-term strategy it seems reasonable to define a set of internal “standard” code lists that all systems can map to. This allows bilateral communication via the shared concepts and code lists meaning that every data structure only has to be mapped once – to the internal standard – to be able to communicate with all other participating (i.e. mapped) systems.
  )))
  |**between organizations at national level**|(((
  The pros and cons at this level of exchange are comparable to those at the “within organization” level. If the data structures of the communicating systems are comparable, there is no need to introduce complexity by a conceptually optimal, pure data structure. However, if the data structures deviate to a greater extent (and they often do), they should both be decomposed to find a “common denominator”, a more granular “exchange vocabulary” which they can be mapped to.
--
  If related international or national standards exist, they should be used, even though national labels and/or additional levels of detail may be required in the code lists.
  )))
  |**between international organization and national organizations of member countries**|International organizations should collect data at a level of granularity and purity that is most suitable for the intended (and potential future) analyses. The tradeoff with the higher complexity of constraints required to check structural validity of collected data needs to be taken into account as well. Also it is recommended to consider the burden that a more complex data structure may put on national data providers. However, once a DSD is defined, its lifetime is expected to be a number of years. The main effort of the data provider is to specify the mapping from the production data structure to the DSD. Once this is done the data exchange can be automated and the complexity of the DSD does not matter that much.
@@ -304,15 +304,14 @@
  **Table 6. Data structuring approaches by role in data exchange**
--|**Role in data exchange**|**Pure vs. composite concepts approach**
--|**Data provider**|(((
++|(% style="width:215px" %)**Role in data exchange**|(% style="width:1400px" %)**Pure vs. composite concepts approach**
++|(% style="width:215px" %)**Data provider**|(% style="width:1400px" %)(((
  If the composition of the concepts in the data provider's production system largely differs from the one in the DSD, mapping it to a few composite concepts may be more complex than mapping it to many pure concepts. (Mapping to just one mixed concept is straightforward, though.) This is due to the need to decompose and recombine concepts in case of a “mixed concepts” DSD. If the data provider’s internal data structure is very granular or very similar to the DSD, it does not make a huge difference if the concepts in that DSD are pure or not.
--
  For a “final” data provider disseminating data to the public, the flexibility offered by a pure data structure in terms of defining different output formats may be beneficial.
  )))
--|**Data collector**|Defining constraints for data validation is more complex for a highdimensional, pure DSD. But such a DSD provides more flexibility in terms of consumption and reuse, i.e. mapping to the data collector’s internal data model mapping easier.
--|**DSD maintenance**|Pure concepts usually have shorter, less complex code lists and are thus easier to maintain. In contrast, the maintenance of constraints, hierarchical code lists, and derived, composite concepts (e.g. for dissemination) requires more effort.
--|**End user (“the public”)**|Consumption and reuse are more flexible in a pure data structure, but it is more difficult to identify observation keys that actually have data because of the created sparseness. (Constraints may help in this respect.) Frequent occurrences of “non applicable” values may also make data usage cumbersome.
++|(% style="width:215px" %)**Data collector**|(% style="width:1400px" %)Defining constraints for data validation is more complex for a highdimensional, pure DSD. But such a DSD provides more flexibility in terms of consumption and reuse, i.e. mapping to the data collector’s internal data model mapping easier.
++|(% style="width:215px" %)**DSD maintenance**|(% style="width:1400px" %)Pure concepts usually have shorter, less complex code lists and are thus easier to maintain. In contrast, the maintenance of constraints, hierarchical code lists, and derived, composite concepts (e.g. for dissemination) requires more effort.
++|(% style="width:215px" %)**End user (“the public”)**|(% style="width:1400px" %)Consumption and reuse are more flexible in a pure data structure, but it is more difficult to identify observation keys that actually have data because of the created sparseness. (Constraints may help in this respect.) Frequent occurrences of “non applicable” values may also make data usage cumbersome.
  == 4.2 Number and relations of DSDs ==
@@ -334,36 +334,22 @@
  **Table 7. Data structuring approaches by level of data exchange**
--|**Level of data exchange**|**Data structuring approa one DSD**|(% colspan="2" %)(((
--**ch**
--
--**master + satellite DSDs**
--)))|**multiple, indep. DSDs**
++|(% colspan="1" rowspan="2" %)**Level of data exchange**|(% colspan="4" rowspan="1" %)**Data structuring approach**
++|**one DSD**|(% colspan="2" %)**master + satellite DSDs**|**multiple, indep. DSDs**
  |**within organization**|(((
--best for single-domain, single-purpose can be created on the
--
--fly from structured databases
++best for single-domain, single-purpose can be created on the fly from structured databases
  )))|(% colspan="2" %)use if harmonization is important in covered domains or purposes or if such a set of DSDs is already available at international level|easier to do than master + satellite approach each domain/purpose can maintain DSDs independently can be created on the fly from structured databases
  |**between national organizations**|(% colspan="4" %)the same applies as to the “within organization” scenario
--|**Level of data exchange**|(% colspan="3" %)(((
--**Data structuring approach**
--
--**one DSD                        master + satellite DSDs**
--)))|**multiple, indep. DSDs**
  |**between int. organization and national organizations**|(% colspan="2" %)best for single domain, single purpose scenarios that are usually rather restricted with very clear specification of what needs to be exchanged|preferable over multiDSD approach in case of multi-domain and/or multi-purpose scenarios with highly correlated data flows for maintenance reasons|(((
--for multi-domain and/or multipurpose scenarios; only recommended if overlap of domains/purposes is minor (e.g. just w.r.t. cross-domain concepts)
--
--equivalent to multiple “one DSD” solutions, one for each domain / purpose
++for multi-domain and/or multipurpose scenarios; only recommended if overlap of domains/purposes is minor (e.g. just w.r.t. cross-domain concepts) equivalent to multiple “one DSD” solutions, one for each domain / purpose
  )))
  |**between international organizations**|(% colspan="3" %)comparable to “national to international” scenario|
  |**dissemination to public**|(% colspan="2" %)for single-domain, single-purpose cases in more complex cases this may be the preferable approach for data discovery tools (one data structure to find and access all data)|(% colspan="2" %)(((
  in multi-purpose or –domain scenarios:
--if it is relevant for the public to see the relationship between the data structures: use master + satellites approach
--
--otherwise the multi-DSD option is preferable, although with the highest possible degree of re-use of code lists and concepts
--
--in both cases: important to include only concepts, code lists, and codes actually available / used by the data
++* if it is relevant for the public to see the relationship between the data structures: use master + satellites approach
++* otherwise the multi-DSD option is preferable, although with the highest possible degree of re-use of code lists and concepts
++* in both cases: important to include only concepts, code lists, and codes actually available / used by the data
  )))
  In general, finding the “perfect” data structure is less important for bilateral data exchange. Independent, custom-tailored DSDs may do the job quite well, as harmonization and standardization are typically not of high importance. If the data exchange is just a part of a more comprehensive scenario (e.g. multi-purpose, multi-domain, gateway, or data-sharing scenarios), a master DSD with satellite DSDs is preferable.
@@ -372,20 +372,17 @@
  **Table 8. Data structuring approaches by role in data exchange**
--|**Role in data exchange**|**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
--|**Data provider**|It is easier to set up a data submission process against a single DSD (= less initial costs) than against multiple DSDs.
--|**Data collector**|(((
++|(% style="width:216px" %)**Role in data exchange**|(% style="width:1399px" %)**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
++|(% style="width:216px" %)**Data provider**|(% style="width:1399px" %)It is easier to set up a data submission process against a single DSD (= less initial costs) than against multiple DSDs.
++|(% style="width:216px" %)**Data collector**|(% style="width:1399px" %)(((
  Data validation is easier with DSDs that only cover what needs to be collected. This is achieved via constraints in the master + satellites approach or via tailor-made independent DSDs. If a single DSD is used in a multi-domain or –purpose scenario, necessary constraints can be specified in the data flow definition or data provision agreement.
--
  Further processing of collected data is more flexible and easier if relations are transparent and code lists are shared as in the one DSD or master + satellite DSDs approaches. The “shared context” created through the master DSD increases harmonization and standardization and this way facilitates combined usage of data.
  )))
--|**Role in data exchange**|**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
--|**DSD maintenance**|(((
++|(% style="width:216px" %)**DSD maintenance**|(% style="width:1399px" %)(((
  The complexity and initial costs for developing and maintaining master + satellite DSDs are higher than for independent DSDs as this involves managing constraints and managing impacts of changes in shared code lists to all DSDs.
--
  In the multiple independent DSDs approach, development and maintenance efforts may be distributed. This can be seen as an advantage, but on the other hand requires coordination in case the DSDs are only partially independent (i.e. share some code lists).
  )))
--|**End user (“the public”)**|For data discovery and retrieval the user needs to know what data is actually available (instead of what might be collected/disseminated with a certain data structure). This means that the potential sparseness should be hidden from the user. A reduced DSD derived from the data structure used in the background is more useful in most cases. Whether this is done via one DSD and constraints, master + satellite DSDs, or independent DSDs does not matter that much for the user.
++|(% style="width:216px" %)**End user (“the public”)**|(% style="width:1399px" %)For data discovery and retrieval the user needs to know what data is actually available (instead of what might be collected/disseminated with a certain data structure). This means that the potential sparseness should be hidden from the user. A reduced DSD derived from the data structure used in the background is more useful in most cases. Whether this is done via one DSD and constraints, master + satellite DSDs, or independent DSDs does not matter that much for the user.
  = 5 MINIMUM STRUCTURAL AND SEMANTIC REQUIREMENTS =
@@ -415,19 +415,19 @@
  **Table 9. Minimum requirements for DSDs~*~***
--|**Question**|**Concept**|**COG**|**Code list**|**Time series      Cross-section**
--|Where?|reference area|X|revision|mand. attribute or dimension
--|What?|“indicator”|-|domain|one or multiple dimensions
--|How?|unit of measure|X|development|mand. attribute or dimension
--|How?|unit multiplier|X|available|mandatory attribute
--|How?|decimals|X|available|mandatory attribute
--|How?|//adjustment//|X|development|mand. att.        not relevant
--|When?|time period|X|format|dimension       mand. att.
--|When?|time format|X|available|mandatory attribute
--|When?|time period – collection|X|development|mand. att.         cond. att.
--|When?|data update – last update|X|time stamp|mandatory attribute
--|How often?|//frequency//|X|available|mand. att. or    not relevant
--|(% colspan="2" %)How much? observation value|-|numeric|dimension measure
++|(% style="width:205px" %)**Question**|(% style="width:272px" %)**Concept**|(% style="width:178px" %)**COG**|(% style="width:270px" %)**Code list**|(% style="width:690px" %)**Time series      Cross-section**
++|(% style="width:205px" %)Where?|(% style="width:272px" %)reference area|(% style="width:178px" %)X|(% style="width:270px" %)revision|(% style="width:690px" %)mand. attribute or dimension
++|(% style="width:205px" %)What?|(% style="width:272px" %)“indicator”|(% style="width:178px" %)-|(% style="width:270px" %)domain|(% style="width:690px" %)one or multiple dimensions
++|(% style="width:205px" %)How?|(% style="width:272px" %)unit of measure|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. attribute or dimension
++|(% style="width:205px" %)How?|(% style="width:272px" %)unit multiplier|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
++|(% style="width:205px" %)How?|(% style="width:272px" %)decimals|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
++|(% style="width:205px" %)How?|(% style="width:272px" %)//adjustment//|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. att.        not relevant
++|(% style="width:205px" %)When?|(% style="width:272px" %)time period|(% style="width:178px" %)X|(% style="width:270px" %)format|(% style="width:690px" %)dimension       mand. att.
++|(% style="width:205px" %)When?|(% style="width:272px" %)time format|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
++|(% style="width:205px" %)When?|(% style="width:272px" %)time period – collection|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. att.         cond. att.
++|(% style="width:205px" %)When?|(% style="width:272px" %)data update – last update|(% style="width:178px" %)X|(% style="width:270px" %)time stamp|(% style="width:690px" %)mandatory attribute
++|(% style="width:205px" %)How often?|(% style="width:272px" %)//frequency//|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mand. att. or    not relevant
++|(% colspan="2" style="width:477px" %)How much? observation value|(% style="width:178px" %)-|(% style="width:270px" %)numeric|(% style="width:690px" %)dimension measure
  ~*~*Concepts in //italics// are only relevant for time series DSDs. An “X” in the COG column means the concept is defined in the COG. Code list “development” means that the SWG will develop a code list to be recommended in the COG; “revision” means that the code list is recommended by the COG and under revision by the SWG; “format” means that a format is defined by another concept; “text”, “time stamp”, and “numeric” provide data types used for uncoded concepts.
@@ -435,25 +435,19 @@
  **Table 10. Suggested additional concepts for certain scenarios~*~***
--|**Question**|**Concept**|**COG**|**Code list**|**TS                  CS**|**Scenario**
++|**Question**|**Concept**|**COG**|**Code list**|**TS**|**CS**|**Scenario**
  |Who?|compiling agency|X|development|(((
--conditional          conditional
--
--    (sibling)          (obs. level)
--)))|data provider different from data compiler
++conditional (sibling)
++)))|conditional (obs. level)|data provider different from data compiler
  |Who?|(((
--confidentiality
--
--status – observation
--)))|X|available|mandatory (obs. level)|except dissemination
--|How?|observation status|X|available|conditional (obs. level)|except orig. collection
++confidentiality status – observation
++)))|X|available|(% colspan="2" rowspan="1" %)mandatory (obs. level)|except dissemination
++|How?|observation status|X|available|(% colspan="2" rowspan="1" %)conditional (obs. level)|except orig. collection
  |How much?|(((
--//observation pre-//
++//observation pre-break value//
++)))|-|numeric|cond. (obs.)|not relevant|except orig. collection
++|What and how?|//time series title//|X|text|cond. (TS)|not relevant|dissemination
--//break value//
--)))|-|numeric|cond. (obs.)      not relevant|except orig. collection
--|What and how?|//time series title//|X|text|cond. (TS)        not relevant|dissemination
--
  ~** The legend of Table 9 applies to Table 10 as well. The suggested attachment level of attributes (if any) is provided in parentheses in the TS (time series) or CS (cross-section) columns. In case an attribute does not vary at that level in a certain use case, it should be attached at the highest possible level.
  == 5.2 Attribute attachment levels and definition of groups ==
@@ -475,10 +475,8 @@
  * //ID//: a unique identifier of the message
  * //Test//: a Boolean attribute that indicates whether the message is for test purposes or not
  * //Prepared//: the date the message was prepared
--* //Sender//: the identification of the organization that is transmitting the message
++* //Sender//: the identification of the organization that is transmitting the message (recommended: code from the agency code list in the SDMX COG)
--(recommended: code from the agency code list in the SDMX COG)
--
  From a business perspective, the inclusion of the //Name// element is highly recommended, as it can help to understand the purpose of the exchange message. Other header elements such as //Receiver// are optional.
  = 6 STEP-BY-STEP GUIDE =