Wiki source code of 3 Guide for SDMX Format Standards
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | {{box title="**Contents**"}} | ||
| 2 | {{toc/}} | ||
| 3 | {{/box}} | ||
| 4 | |||
| 5 | == 3.1 Introduction == | ||
| 6 | |||
| 7 | This guide exists to provide information to implementers of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] format standards – [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]], [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] and [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV – that are concerned with data, i.e., [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and [[Data Sets>>doc:sdmx:Glossary.Data set.WebHome]]. This section is intended to provide information that will help users of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] understand and implement the standards. It is not normative, and it does not provide any rules for the use of the standards, such as those found in //[[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]: Schema and Documentation//. | ||
| 8 | |||
| 9 | == 3.2 SDMX Information Model for Format Implementers == | ||
| 10 | |||
| 11 | === 3.2.1 Introduction === | ||
| 12 | |||
| 13 | The purpose of this sub-section is to provide an introduction to the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] relating to [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and [[Data Sets>>doc:sdmx:Glossary.Data set.WebHome]] for those whose primary interest is in the use of the XML, JSON or CSV formats. For those wishing to have a deeper understanding of the Information Model, the full [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] document, and other sections in this guide provide a more in-depth view, along with UML diagrams and supporting explanation. For those who are unfamiliar with DSDs, an appendix to the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides a tutorial which may serve as a useful introduction. | ||
| 14 | |||
| 15 | The [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is used to describe the basic data and metadata structures used in all of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data formats. The Information Model concerns itself with statistical data and its [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], and that is what is described here. Both [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and data have some additional metadata in common, related to their management and administration. These aspects of the data model are not addressed in this section and covered elsewhere in this guide or in the full [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] document. | ||
| 16 | |||
| 17 | Note that in the descriptions below, text in courier and italics are the names used in the information model (e.g., //[[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]//). | ||
| 18 | |||
| 19 | == 3.3 SDMX Formats: Expressive Capabilities and Function == | ||
| 20 | |||
| 21 | [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] offers several equivalent formats for describing data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], optimized for use in different applications. Although all of these formats are derived directly from the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]], and are thus equivalent, the syntaxes used to express the model place some restrictions on their use. Also, different optimizations provide different capabilities. This section describes these differences and provides some rules for applications which may need to support more than one [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] format or syntax. This section is constrained to the [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] and the [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]. | ||
| 22 | |||
| 23 | === 3.3.1 Format Optimizations and Differences === | ||
| 24 | |||
| 25 | The following section provides a brief overview of the differences between the various [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] formats. | ||
| 26 | |||
| 27 | (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)Version(%%) 2.0 was characterised by 4 data messages, each with a distinct format: Generic, Compact, Cross-Sectional and Utility. Because of the design, data in some formats could not always be related to another format. In (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)version(%%) 2.1, this issue has been addressed by merging some formats and eliminating others. As a result, in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 there were just two types of data formats: //GenericData// and //StructureSpecificData// (i.e., specific to one [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]). As of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, based also on the reallife usage of 2.1 XML formats but also the new formats introduced (JSON and CSV), only one XML format remains, i.e., //StructureSpecificData//. Furthermore, the time specific sub-formats have also been deprecated due to the lack of usage. | ||
| 28 | |||
| 29 | [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] and [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV feature also only one flavour, each. It should be noted, though, that both XML and JSON messages allow for series oriented as well as flat representations. | ||
| 30 | |||
| 31 | //**Structure Definition**// | ||
| 32 | |||
| 33 | * The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Structure Message is currently the main way of modelling a [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]. The [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)version(%%) follows the same principles, while the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV does not support structures, yet. | ||
| 34 | * The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Structure Message allows for the structures on which a [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] depends – that is, [[codelists>>doc:sdmx:Glossary.Code list.WebHome]] and (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%) – to be either included in the message or to be referenced by the message containing the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. XML syntax is designed to leverage URIs and other Internetbased referencing mechanisms, and these are used in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] message. This option is also available in [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]]. The latter, though, further supports conveying data with some [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] within a single message. | ||
| 35 | * All structures can be inserted, replaced or deleted, unless structural dependencies are not respected. | ||
| 36 | |||
| 37 | //**Validation**// | ||
| 38 | |||
| 39 | * The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] structure specific messages will allow validation of XML syntax and data typing to be performed with a generic XML parser and enforce agreement between the structural definition and the data to a moderate degree with the same tool. | ||
| 40 | * Similarly, the [[SDMX-JSON>>doc:sdmx:Glossary.SDMX-JSON.WebHome]] message can be validated using JSON Schema and hence may also be generically parsed and validated. | ||
| 41 | * The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV format cannot be validated by generic tools. | ||
| 42 | |||
| 43 | //**Update and Delete Messages**// | ||
| 44 | |||
| 45 | * All data messages allow for both append/replace/delete messages. | ||
| 46 | * These messages allow also transmitting only data or only documentation (i.e., [[Attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values without Observation values). | ||
| 47 | |||
| 48 | //**Character Encodings**// | ||
| 49 | |||
| 50 | All formats use the UTF-8 encoding. The [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV may use a different encoding if this is reported properly in the mime type of a web service response. | ||
| 51 | |||
| 52 | //**Data Typing**// | ||
| 53 | |||
| 54 | The XML syntax and JSON syntax have similar data-typing mechanisms. Hence, there is no need for conventions in order to allow transition from one format to another, like those required for EDIFACT in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1. On the other hand, JSON schema has a simpler set of data types (as explained in section 2, paragraph “3.6.3.3 [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] Constructs”) but complements its data types with a fixed set of formats or regular expressions. In addition, the JSON schema has also types that are not natively supported in XML schema and need to be implemented as complex types in the latter. | ||
| 55 | |||
| 56 | The section below provides examples of those cases that are not natively supported by either the XML or JSON data types. More details on the data mapping between XML and JSON schemas are also explained in section “4.1.1 Data Types”. | ||
| 57 | |||
| 58 | == 3.4 SDMX Best Practices == | ||
| 59 | |||
| 60 | === 3.4.1 Reporting and Dissemination Guidelines === | ||
| 61 | |||
| 62 | ==== 3.4.1.1 Central Institutions and Their Role in Statistical Data Exchanges ==== | ||
| 63 | |||
| 64 | Central institutions are the organisations to which other partner institutions "report" statistics. These statistics are used by central institutions either to compile aggregates and/or they are put together and made available in a uniform manner (e.g., on-line or on a CD-ROM or through file transfers). Therefore, central institutions receive data from other institutions and, usually, they also "disseminate" data to individual and/or institutions for end-use. Within a country, a NSI or a national central bank (NCB) plays, of course, a central institution role as it collects data from other entities and it disseminates statistical information to end users. In [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] the role of central institution is very important: every statistical message is based on underlying structural definitions (statistical (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%), [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], DSDs) which have been devised by a particular agency, usually a central institution. Such an institution plays the role of the reference "structural definitions [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] for the corresponding messages which are exchanged. Of course, two institutions could exchange data using/referring to structural information devised by a third institution. | ||
| 65 | |||
| 66 | Central institutions can play a double role: | ||
| 67 | |||
| 68 | * collecting and further disseminating statistics; | ||
| 69 | * devising structural definitions for use in data exchanges. | ||
| 70 | |||
| 71 | ==== 3.4.1.2 Defining Data Structure Definitions (DSDs) ==== | ||
| 72 | |||
| 73 | The following guidelines are suggested for building a [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]. However, it is expected that these guidelines will be considered by central institutions when devising new DSDs. | ||
| 74 | |||
| 75 | __[[Dimensions>>doc:sdmx:Glossary.Dimension.WebHome]], [[Attributes>>doc:sdmx:Glossary.Attribute.WebHome]] and [[Codelists>>doc:sdmx:Glossary.Code list.WebHome]]__ | ||
| 76 | |||
| 77 | * **Avoid [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] that are not appropriate for all the series in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]**. If some [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] are not applicable (this is evident from the need to have a [[code>>doc:sdmx:Glossary.Code.WebHome]] in a [[code list>>doc:sdmx:Glossary.Code list.WebHome]] which is marked as "not applicable", "not relevant" or "total") for some series then consider moving these series to a new [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] in which these [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] are dropped from the key structure. This is a judgement call as it is sometimes difficult to achieve this without increasing considerably the number of DSDs. | ||
| 78 | * **Devise DSDs with a small number of [[Dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] for public viewing of data**. A [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] with the number [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] in excess 6 or 7 is often difficult for nonspecialist users to understand. In these cases, it is better to have a larger number of DSDs with smaller "cubes" of data, or to eliminate [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] and aggregate the data at a higher (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)level(%%). Dissemination of data on the web is a growing use case for the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards: the differentiation of observations by dimensionality, which are necessary for statisticians and economists, are often obscure to public consumers who may not always understand the semantic of the differentiation. | ||
| 79 | * **Avoid composite [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]]**. Each [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] should correspond to a single characteristic of the data, not to a combination of characteristics. | ||
| 80 | * Consider the inclusion of the following [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]. Once the key structure of a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] has been decided, then the set of (preferably mandatory) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] of this [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] has to be defined. In general, some statistical (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%) are deemed necessary across all [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] to qualify the contained information. Examples of these are: | ||
| 81 | ** A descriptive [[title>>doc:sdmx:Glossary.Title.WebHome]] for the series (this is most useful for dissemination of data for viewing e.g., on the web). | ||
| 82 | ** Collection (e.g., end of period, averaged or summed over period). o Unit (e.g., [[currency>>doc:sdmx:Glossary.Currency.WebHome]] of denomination). | ||
| 83 | ** Unit multiplier (e.g., expressed in millions). o Availability (which institutions can a series become available to). o [[Decimals>>doc:sdmx:Glossary.Decimals.WebHome]] (i.e., number of [[decimal>>doc:sdmx:Glossary.Decimals.WebHome]] digits used in numerical observations). | ||
| 84 | ** [[Observation Status>>doc:sdmx:Glossary.Observation status.WebHome]] (e.g., estimate, provisional, normal). | ||
| 85 | |||
| 86 | Moreover, additional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] may be considered as mandatory when a specific [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] is defined. | ||
| 87 | |||
| 88 | * **Avoid creating a new [[code list>>doc:sdmx:Glossary.Code list.WebHome]] where one already exists**. It is highly recommended that structural definitions and [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] be consistent with internationally agreed standard methodologies, wherever they exist, e.g., System of National Accounts 1993; Balance of Payments Manual, Fifth Edition; Monetary and Financial Statistics Manual; Government Finance Statistics Manual, etc. When setting-up a new data exchange, the following order of priority is suggested when considering the use of [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]: | ||
| 89 | ** international standard [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]; o international [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] supplemented by other international and/or regional institutions; | ||
| 90 | ** standardised lists used already by international institutions; o new [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] agreed between two international or regional institutions; o new [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] which extend existing [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], by adding only missing [[codes>>doc:sdmx:Glossary.Code.WebHome]]; | ||
| 91 | ** new specific [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]. | ||
| 92 | |||
| 93 | The same [[code list>>doc:sdmx:Glossary.Code list.WebHome]] can be used for several statistical (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%), within a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] or across DSDs. Note that [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] has recognised that these classifications are often quite large and the usage of [[codes>>doc:sdmx:Glossary.Code.WebHome]] in any one [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] is only a small extract of the full [[code list>>doc:sdmx:Glossary.Code list.WebHome]]. In this (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)version(%%) of the standard, it is possible to exchange and disseminate a **partial [[code list>>doc:sdmx:Glossary.Code list.WebHome]]** which is extracted from the full [[code list>>doc:sdmx:Glossary.Code list.WebHome]] and which supports the [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] values valid for a particular [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]. | ||
| 94 | |||
| 95 | [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] Structure | ||
| 96 | |||
| 97 | * The following items have to be specified by a structural definitions [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] when defining a new [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]: | ||
| 98 | * [[Data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] ([[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]) identification: | ||
| 99 | ** [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] identifier | ||
| 100 | ** [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] name | ||
| 101 | * A list of metadata (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%) assigned as [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] of the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 102 | ** (statistical) (% style="color:#e74c3c" %)[[concept>>doc:sdmx:Glossary.Concept.WebHome]](%%) identifier | ||
| 103 | ** [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier (id, (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)version(%%), [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]]) if the [[representation>>doc:sdmx:Glossary.Representation.WebHome]] is coded | ||
| 104 | * A list of (statistical) (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%) assigned as [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] for the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 105 | ** (statistical) (% style="color:#e74c3c" %)[[concept>>doc:sdmx:Glossary.Concept.WebHome]](%%) identifier | ||
| 106 | ** [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier if the (% style="color:#e74c3c" %)[[concept>>doc:sdmx:Glossary.Concept.WebHome]](%%) is coded | ||
| 107 | ** usage: mandatory, optional | ||
| 108 | ** relationship to [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] and measures | ||
| 109 | ** maximum text length for the uncoded (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]] | ||
| 110 | ** maximum [[code>>doc:sdmx:Glossary.Code.WebHome]] length for the coded (% style="color:#e74c3c" %)[[concepts>>doc:sdmx:Glossary.Concept.WebHome]] | ||
| 111 | * A list of the [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] used in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 112 | ** [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier | ||
| 113 | ** [[code list>>doc:sdmx:Glossary.Code list.WebHome]] name | ||
| 114 | ** [[code>>doc:sdmx:Glossary.Code.WebHome]] values and descriptions | ||
| 115 | * Definition of [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]. Two (or more) partners performing data exchanges in a certain context need to agree on: | ||
| 116 | ** the list of [[dataset>>doc:sdmx:Glossary.Data set.WebHome]] identifiers they will be using; | ||
| 117 | ** for each [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]]: | ||
| 118 | *** its content (e.g., by (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)Constraints(%%)) and description | ||
| 119 | *** the relevant [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] that defines the structure of the data reported or disseminated according the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] | ||
| 120 | |||
| 121 | ==== 3.4.1.3 Exchanging Attributes ==== | ||
| 122 | |||
| 123 | ===== 3.4.1.3.1 Attributes on series and group levels ===== | ||
| 124 | |||
| 125 | * Static properties. | ||
| 126 | * Upon creation of a series the sender has to provide to the receiver values for all mandatory [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]. In case they are available, values for conditional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] should also be provided. Whereas initially this information may be provided by means other than [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]/JSON/CSV messages (e.g., paper, telephone) it is expected that partner institutions will be in a position to provide this information in the available formats over time. | ||
| 127 | * A centre may agree with its data exchange partners special procedures for authorising the setting of [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]' initial values. | ||
| 128 | * Communication of changes to the centre. | ||
| 129 | * Following the creation of a series, the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values do not have to be reported again by senders, as long as they do not change. | ||
| 130 | * Whenever changes in [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values for a series (or group) occur, the reporting institutions should report either all [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values again (this is the recommended option) or only the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values which have changed. This applies both to the mandatory and the conditional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]. For example, if a previously reported value for a conditional [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] is no longer valid, this has to be reported to the centre. | ||
| 131 | * A centre may agree with its data exchange partners special procedures for authorising modifications in the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values. | ||
| 132 | * Communication of observation (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)level(%%) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]]", "observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]]", "observation pre-break" is recommended. | ||
| 133 | * Whenever an observation is exchanged, the corresponding [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]] is recommended to also be exchanged attached to the observation, regardless of whether it has changed or not since the previous data exchange. | ||
| 134 | * If the [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]] changes and the observation remains unchanged, both [[components>>doc:sdmx:Glossary.Component.WebHome]] would have to be reported (unless the observation is deleted). | ||
| 135 | |||
| 136 | For [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] having also the observation (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)level(%%) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] "observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]] and "observation pre-break" defined, this rule applies to these [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] as well: if an institution receives from another institution an observation with an [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]] [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] only attached, this means that the associated observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]] and pre-break observation [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] either never existed or from now they do not have a value for this observation. | ||
| 137 | |||
| 138 | === 3.4.2 Best Practices for Batch Data Exchange === | ||
| 139 | |||
| 140 | ==== 3.4.2.1 Introduction ==== | ||
| 141 | |||
| 142 | Batch data exchange is the exchange and maintenance of entire databases between counterparties. It is an activity that often employs [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-CSV format, and might also use the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] [[dataset>>doc:sdmx:Glossary.Data set.WebHome]]. The following points apply equally to both formats. | ||
| 143 | |||
| 144 | ==== 3.4.2.2 Positioning of the Dimension "Frequency" ==== | ||
| 145 | |||
| 146 | In [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, the "frequency" [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] is not special in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. Many central institutions devising structural definitions have decided to assign to this [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] the first position in the key structure. Nevertheless, a standard role (i.e., that of ‘Frequency’) may facilitate the easy identification of this [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]]. This is necessary to frequency's crucial role in several database systems and in attaching [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] at the "sibling" group (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)level(%%). | ||
| 147 | |||
| 148 | ==== 3.4.2.3 Identification of Data Structure Definitions (DSDs) ==== | ||
| 149 | |||
| 150 | In order to facilitate the easy and immediate recognition of the structural definition [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] that defined a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]], some central institutions devising structural definitions use the first characters of the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] identifiers to identify their institution: e.g., BIS_EER, EUROSTAT_BOP_01, ECB_BOP1, etc. Nevertheless, using the AgencyId may disambiguate any [[Artefact>>doc:sdmx:Glossary.Artefact.WebHome]] in a more efficient and machine readable way. | ||
| 151 | |||
| 152 | ==== 3.4.2.4 Identification of the Dataflows ==== | ||
| 153 | |||
| 154 | In order to facilitate the easy and immediate recognition of the institution administrating a [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]], some central institutions prefer to use the first characters of the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] identifiers to identify their institution: e.g. BIS_EER, ECB_BOP1, ECB_BOP1, etc. Nevertheless, using the AgencyId may disambiguate any [[Artefact>>doc:sdmx:Glossary.Artefact.WebHome]] in a more efficient and machine readable way. | ||
| 155 | |||
| 156 | The statistical information in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is broken down into two fundamental parts – [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] (comprising the DataStructureDefinition, and associated (% style="color:#e74c3c" %)[[Concepts>>doc:sdmx:Glossary.Concept.WebHome]](%%) and [[Codelists>>doc:sdmx:Glossary.Code list.WebHome]]) – see Framework for Standards – and observational data (the [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]). This is an important distinction, with specific terminology associated with each part. Data, which is typically a set of numeric observations at specific points in time, is organised into [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] ([[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]). These [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] are structured according to a specific DataStructureDefinition and are described in the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] (via (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)Constraints(%%)). The DataStructureDefinition describes the metadata that allows an understanding of what is expressed in the [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]], whilst the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] provides the identifier and other important information (such as the periodicity of reporting) that is common to all its [[Components>>doc:sdmx:Glossary.Component.WebHome]]. | ||
| 157 | |||
| 158 | Note that the role of the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] and [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]] is very specific in the model, and the terminology used may not be the same as used in all organisations, and specifically the term [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]] is used differently in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] than in GESMES/TS. Essentially the GESMES/TS term [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] is, in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]], the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] whilst the term [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is used to describe the "container" for an instance of the data. | ||
| 159 | |||
| 160 | ==== 3.4.2.5 Special Issues ==== | ||
| 161 | |||
| 162 | ===== 3.4.2.5.1 "Frequency" related issues ===== | ||
| 163 | |||
| 164 | * //**Special frequencies.**// The issue of data collected at special (regular or irregular) intervals at a lower than daily frequency (e.g., 24 or 36 or 48 observations per year, on irregular days during the year) is not extensively discussed here. However, for data exchange purposes: | ||
| 165 | ** such data can be (% style="color: rgb(231, 76, 60); color: rgb(231, 76, 60)" %)mapped(%%) into a series with daily frequency; this daily series will only hold observations for those days on which the measured event takes place; | ||
| 166 | ** if the collection intervals are regular, additional values to the existing frequency [[code list>>doc:sdmx:Glossary.Code list.WebHome]](s) could be added in the future. | ||
| 167 | * //**Tick data.**// The issue of data collected at irregular intervals at a higher than daily frequency (e.g., tick-by-tick data) is not discussed here either. |