Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | = 3 Guide for SDMX Format Standards = | ||
| 2 | |||
| 3 | == 3.1 Introduction == | ||
| 4 | |||
| 5 | This guide exists to provide information to implementers of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] format standards – [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] and [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] – that are concerned with data, i.e. [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and [[Data Sets>>doc:sdmx:Glossary.Data set.WebHome]]. This section is intended to provide information which will help users of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] understand and implement the standards. It is not normative, and it does not provide any rules for the use of the standards, such as those found in //[[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]: Schema and Documentation// and //[[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]]: Syntax and Documentation//. | ||
| 6 | |||
| 7 | == 3.2 SDMX Information Model for Format Implementers == | ||
| 8 | |||
| 9 | === 3.2.1 Introduction === | ||
| 10 | |||
| 11 | The purpose of this sub-section is to provide an introduction to the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] relating to [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] and [[Data Sets>>doc:sdmx:Glossary.Data set.WebHome]] for those whose primary interest is in the use of the XML or EDI formats. For those wishing to have a deeper understanding of the Information Model, the full [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] document, and other sections in this guide provide a more in-depth view, along with UML diagrams and supporting explanation. For those who are unfamiliar with DSDs, an appendix to the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] provides a tutorial which may serve as a useful introduction. | ||
| 12 | |||
| 13 | The [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] is used to describe the basic data and metadata structures used in all of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data formats. The Information Model concerns itself with statistical data and its [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], and that is what is described here. Both [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and data have some additional metadata in common, related to their management and administration. These aspects of the data model are not addressed in this section and covered elsewhere in this guide or in the full [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]] document. | ||
| 14 | |||
| 15 | The [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] and [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] parts of the information model are consistent with the GESMES/TS (% style="color:#e74c3c" %)version(%%) 3.0 Data Model (called [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] in the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standard), with these exceptions: | ||
| 16 | |||
| 17 | * the “[[sibling group>>doc:sdmx:Glossary.Sibling group.WebHome]]” construct has been generalized to permit any [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] or [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] to be wildcarded, and not just frequency, as in GESMES/TS. It has been renamed a “group” to distinguish it from the “[[sibling group>>doc:sdmx:Glossary.Sibling group.WebHome]]” where only frequency is wildcarded. The set of allowable partial “group” keys must be declared in the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]], and [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] may be attached to any of these [[group keys>>doc:sdmx:Glossary.Group key.WebHome]]; | ||
| 18 | * furthermore, whilst the “group” has been retained for compatibility with (% style="color:#e74c3c" %)version(%%) 2.0 and with [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]], it has, at (% style="color:#e74c3c" %)version(%%) 2.1, been replaced by the “[[Attribute Relationship>>doc:sdmx:Glossary.Attribute relationship.WebHome]]” definition which is explained later | ||
| 19 | * the section on data [[representation>>doc:sdmx:Glossary.Representation.WebHome]] is now a convention, to support interoperability with EDIFACT-syntax implementations ( see section 3.3.2); | ||
| 20 | |||
| 21 | [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]-specific data formats are derived from the model, and some supporting features for declaring multiple [[measures>>doc:sdmx:Glossary.Measure.WebHome]] have been added to the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] descriptions Clearly, this is not a coincidence. The GESMES/TS Data Model provides the foundation for the EDIFACT messages in [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]], and also is the starting point for the development of [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]]. | ||
| 22 | |||
| 23 | Note that in the descriptions below, text in courier and italicised are the names used in the information model (e.g. //[[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]//). | ||
| 24 | |||
| 25 | == 3.3 SDMX-ML and SDMX-EDI: Comparison of Expressive Capabilities and Function == | ||
| 26 | |||
| 27 | [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] offers several equivalent formats for describing data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], optimized for use in different applications. Although all of these formats are derived directly from the SDM-IM, and are thus equivalent, the syntaxes used to express the model place some restrictions on their use. Also, different optimizations provide different capabilities. This section describes these differences, and provides some rules for applications which may need to support more than one [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] format or syntax. This section is constrained to the Data Structure Definitionand the Date Set. | ||
| 28 | |||
| 29 | === 3.3.1 Format Optimizations and Differences === | ||
| 30 | |||
| 31 | The following section provides a brief overview of the differences between the various [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] formats. | ||
| 32 | |||
| 33 | (% style="color:#e74c3c" %)Version(%%) 2.0 was characterised by 4 data messages, each with a distinct format: Generic, Compact, Cross-Sectional and Utility. Because of the design, data in some formats could not always be related to another format. In (% style="color:#e74c3c" %)version(%%) 2.1, this issue has been addressed by merging some formats and eliminating others. As a result, in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 there are just two types of data formats: //GenericData// and //StructureSpecificData// (i.e. specific to one [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]). | ||
| 34 | |||
| 35 | Both of these formats are now flexible enough to allow for data to be oriented in series with any [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] used to disambiguate the observations (as opposed to only time or a cross sectional [[measure>>doc:sdmx:Glossary.Measure.WebHome]] in (% style="color:#e74c3c" %)version(%%) 2.0). The formats have also been expanded to allow for ungrouped observations. | ||
| 36 | |||
| 37 | To allow for applications which only understand time series data, variations of these formats have been introduced in the form of two data messages; //GenericTimeSeriesData// and //StructureSpecificTimeSeriesData//. It is important to note that these variations are built on the same root structure and can be processed in the same manner as the base format so that they do NOT introduce additional processing requirements. | ||
| 38 | |||
| 39 | **//Structure Definition//** | ||
| 40 | |||
| 41 | The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Structure Message supports the use of [[annotations>>doc:sdmx:Glossary.Annotation.WebHome]] to the structure, which is not supported by the [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] syntax. | ||
| 42 | |||
| 43 | The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Structure Message allows for the structures on which a [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] depends – that is, codelists and (% style="color:#e74c3c" %)concepts(%%) – to be either included in the message or to be referenced by the message containing the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. XML syntax is designed to leverage URIs and other Internet-based referencing mechanisms, and these are used in the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] message. This option is not available to those using the [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] structure message. | ||
| 44 | |||
| 45 | **//Validation//** | ||
| 46 | |||
| 47 | [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] – as is typical of EDIFACT syntax messages – leaves validation to dedicated applications (“validation” being the checking of syntax, data typing, and adherence of the data message to the structure as described in the structural definition.) | ||
| 48 | |||
| 49 | The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Generic Data Message also leaves validation above the XML syntax (% style="color:#e74c3c" %)level(%%) to the application. | ||
| 50 | |||
| 51 | The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]-specific messages will allow validation of XML syntax and datatyping to be performed with a generic XML parser, and enforce agreement between the structural definition and the data to a moderate degree with the same tool. | ||
| 52 | |||
| 53 | //Update and Delete Messages and Documentation Messages// | ||
| 54 | |||
| 55 | All [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data messages allow for both delete messages and messages consisting of only data or only documentation. | ||
| 56 | |||
| 57 | **//Character Encodings//** | ||
| 58 | |||
| 59 | All [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages use the UTF-8 encoding, while [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] uses the ISO 8879-1 character encoding. There is a greater capacity with UTF-8 to express some character sets (see the “APPENDIX: (% style="color:#e74c3c" %)MAP(%%) OF ISO 8859-1 (UNOC) CHARACTER SET (LATIN 1 OR “WESTERN”) in the document “SYNTAX AND DOCUMENTATION (% style="color:#e74c3c" %)VERSION(%%) 2.0”.) Many transformation tools are available which allow XML instances with UTF-8 encodings to be expressed as ISO 8879-1-encoded characters, and to transform UTF-8 into ISO 8879-1. Such tools should be used when transforming [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages into [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] messages and vice-versa. | ||
| 60 | |||
| 61 | **//Data Typing//** | ||
| 62 | |||
| 63 | The XML syntax and EDIFACT syntax have different data-typing mechanisms. The section below provides a set of conventions to be observed when support for messages in both syntaxes is required. For more information on the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] [[representations>>doc:sdmx:Glossary.Representation.WebHome]] of data, see below. | ||
| 64 | |||
| 65 | === 3.3.2 Data Types === | ||
| 66 | |||
| 67 | The XML syntax has a very different mechanism for data-typing than the EDIFACT syntax, and this difference may create some difficulties for applications which support both EDIFACT-based and XML-based [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data formats. This section provides a set of conventions for the expression in data in all formats, to allow for clean interoperability between them. | ||
| 68 | |||
| 69 | It should be noted that this section does not address character encodings – it is assumed that conversion software will include the use of transformations which will (% style="color:#e74c3c" %)map(%%) between the ISO 8879-1 encoding of the [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] format and the UTF-8 encoding of the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] formats. | ||
| 70 | |||
| 71 | Note that the following conventions may be followed for ease of interoperation between EDIFACT and XML [[representations>>doc:sdmx:Glossary.Representation.WebHome]] of the data and metadata. For implementations in which no transformation between EDIFACT and XML syntaxes is foreseen, the restrictions below need not apply. | ||
| 72 | |||
| 73 | 1. **Identifiers** are: | ||
| 74 | 1*. Maximum 18 characters; | ||
| 75 | 1*. Any of A..Z (upper case alphabetic), 0..9 (numeric), _ (underscore); | ||
| 76 | 1*. The first character is alphabetic. | ||
| 77 | 1. **Names** are: | ||
| 78 | 1*. Maximum 70 characters. | ||
| 79 | 1*. From ISO 8859-1 character set (including accented characters) | ||
| 80 | 1. **Descriptions **are: | ||
| 81 | 1*. Maximum 350 characters; | ||
| 82 | 1*. From ISO 8859-1 character set. | ||
| 83 | 1. **Code values** are: | ||
| 84 | 1*. Maximum 18 characters; | ||
| 85 | 1*. Any of A..Z (upper case alphabetic), 0..9 (numeric), _ (underscore), / (solidus, slash), = (equal sign), - (hyphen); | ||
| 86 | |||
| 87 | However, [[code>>doc:sdmx:Glossary.Code.WebHome]] values providing values to a [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] must use only the following characters: | ||
| 88 | |||
| 89 | A..Z (upper case alphabetic), 0..9 (numeric), _ (underscore) | ||
| 90 | |||
| 91 | **5. Observation values** are: | ||
| 92 | |||
| 93 | * [[Decimal>>doc:sdmx:Glossary.Decimals.WebHome]] numerics (signed only if they are negative); | ||
| 94 | * The maximum number of significant figures is: | ||
| 95 | * 15 for a positive number | ||
| 96 | * 14 for a positive [[decimal>>doc:sdmx:Glossary.Decimals.WebHome]] or a negative integer | ||
| 97 | * 13 for a negative [[decimal>>doc:sdmx:Glossary.Decimals.WebHome]] | ||
| 98 | * Scientific notation may be used. | ||
| 99 | |||
| 100 | **6. Uncoded statistical concept** text values are: | ||
| 101 | |||
| 102 | * Maximum 1050 characters; | ||
| 103 | * From ISO 8859-1 character set. | ||
| 104 | |||
| 105 | **7. Time series keys**: | ||
| 106 | |||
| 107 | In principle, the maximum permissible length of time [[series keys>>doc:sdmx:Glossary.Series key.WebHome]] used in a data exchange does not need to be restricted. However, for working purposes, an effort is made to limit the maximum length to 35 characters; in this length, also (for SDMXEDI) one (separator) position is included between all successive [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] values; this means that the maximum length allowed for a pure [[series key>>doc:sdmx:Glossary.Series key.WebHome]] (concatenation of [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] values) can be less than 35 characters. The separator character is a colon (“:”) by conventional usage. | ||
| 108 | |||
| 109 | == 3.4 SDMX-ML and SDMX-EDI Best Practices == | ||
| 110 | |||
| 111 | === 3.4.1 Reporting and Dissemination Guidelines === | ||
| 112 | |||
| 113 | ==== 3.4.1.1 Central Institutions and Their Role in Statistical Data Exchanges ==== | ||
| 114 | |||
| 115 | Central institutions are the organisations to which other partner institutions "report" statistics. These statistics are used by central institutions either to compile aggregates and/or they are put together and made available in a uniform manner (e.g. on-line or on a CD-ROM or through file transfers). Therefore, central institutions receive data from other institutions and, usually, they also "disseminate" data to individual and/or institutions for end-use. Within a country, a NSI or a national central bank (NCB) plays, of course, a central institution role as it collects data from other entities and it disseminates statistical information to end users. In [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] the role of central institution is very important: every statistical message is based on underlying structural definitions (statistical (% style="color:#e74c3c" %)concepts(%%), [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], DSDs) which have been devised by a particular agency, usually a central institution. Such an institution plays the role of the reference "structural definitions [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] for the corresponding messages which are exchanged. Of course, two institutions could exchange data using/referring to structural information devised by a third institution. | ||
| 116 | |||
| 117 | Central institutions can play a double role: | ||
| 118 | |||
| 119 | * collecting and further disseminating statistics; | ||
| 120 | * devising structural definitions for use in data exchanges. | ||
| 121 | |||
| 122 | ==== 3.4.1.2 Defining Data Structure Definitions (DSDs) ==== | ||
| 123 | |||
| 124 | The following guidelines are suggested for building a [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]. However, it is expected that these guidelines will be considered by central institutions when devising new DSDs. | ||
| 125 | |||
| 126 | (% class="wikigeneratedid" id="HDimensions2CAttributesandCodeLists" %) | ||
| 127 | __Dimensions, Attributes and Code Lists__ | ||
| 128 | |||
| 129 | **//Avoid [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] that are not appropriate for all the series in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]].//** If some [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] are not applicable (this is evident from the need to have a [[code>>doc:sdmx:Glossary.Code.WebHome]] in a [[code list>>doc:sdmx:Glossary.Code list.WebHome]] which is marked as “not applicable”, “not relevant” or “total”) for some series then consider moving these series to a new [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] in which these [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] are dropped from the key structure. This is a judgement call as it is sometimes difficult to achieve this without increasing considerably the number of DSDs. | ||
| 130 | |||
| 131 | **//Devise DSDs with a small number of [[Dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] for public viewing of data.//** A [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] with the number [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] in excess 6 or 7 is often difficult for non specialist users to understand. In these cases it is better to have a larger number of DSDs with smaller “cubes” of data, or to eliminate [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] and aggregate the data at a higher (% style="color:#e74c3c" %)level(%%). Dissemination of data on the web is a growing use case for the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standards: the differentiation of observations by dimensionality which are necessary for statisticians and economists are often obscure to public consumers who may not always understand the semantic of the differentiation. | ||
| 132 | |||
| 133 | **//Avoid composite [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]].//** Each [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] should correspond to a single characteristic of the data, not to a combination of characteristics. | ||
| 134 | |||
| 135 | **//Consider the inclusion of the following [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]//**. Once the key structure of a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] has been decided, then the set of (preferably mandatory) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] of this [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] has to be defined. In general, some statistical (% style="color:#e74c3c" %)concepts(%%) are deemed necessary across all [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] to qualify the contained information. Examples of these are: | ||
| 136 | |||
| 137 | * A descriptive [[title>>doc:sdmx:Glossary.Title.WebHome]] for the series (this is most useful for dissemination of data for viewing e.g. on the web) | ||
| 138 | * Collection (e.g. end of period, averaged or summed over period) | ||
| 139 | * Unit (e.g. [[currency>>doc:sdmx:Glossary.Currency.WebHome]] of denomination) | ||
| 140 | * Unit multiplier (e.g. expressed in millions) | ||
| 141 | * Availability (which institutions can a series become available to) | ||
| 142 | * Decimals (i.e. number of [[decimal>>doc:sdmx:Glossary.Decimals.WebHome]] digits used in numerical observations) | ||
| 143 | * [[Observation Status>>doc:sdmx:Glossary.Observation status.WebHome]] (e.g. estimate, provisional, normal) | ||
| 144 | |||
| 145 | Moreover, additional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] may be considered as mandatory when a specific [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] is defined. | ||
| 146 | |||
| 147 | **//Avoid creating a new [[code list>>doc:sdmx:Glossary.Code list.WebHome]] where one already exists.//** It is highly recommended that structural definitions and [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] be consistent with internationally agreed standard methodologies, wherever they exist, e.g., System of National Accounts 1993; Balance of Payments Manual, Fifth Edition; Monetary and Financial Statistics Manual; Government Finance Statistics Manual, etc. When setting-up a new data exchange, the following order of priority is suggested when considering the use of [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]: | ||
| 148 | |||
| 149 | * international standard [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]; | ||
| 150 | * international [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] supplemented by other international and/or regional institutions; | ||
| 151 | * standardised lists used already by international institutions; | ||
| 152 | * new [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] agreed between two international or regional institutions; | ||
| 153 | * new specific [[code lists>>doc:sdmx:Glossary.Code list.WebHome]]. | ||
| 154 | |||
| 155 | The same [[code list>>doc:sdmx:Glossary.Code list.WebHome]] can be used for several statistical (% style="color:#e74c3c" %)concepts(%%), within a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] or across DSDs. Note that [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] has recognised that these classifications are often quite large and the usage of [[codes>>doc:sdmx:Glossary.Code.WebHome]] in any one [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] is only a small extract of the full [[code list>>doc:sdmx:Glossary.Code list.WebHome]]. In this (% style="color:#e74c3c" %)version(%%) of the standard it is possible to exchange and disseminate a **partial [[code list>>doc:sdmx:Glossary.Code list.WebHome]]** which is extracted from the full [[code list>>doc:sdmx:Glossary.Code list.WebHome]] and which supports the [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] values valid for a particular [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]. | ||
| 156 | |||
| 157 | __Data Structure Definition Structure__ | ||
| 158 | |||
| 159 | The following items have to be specified by a structural definitions [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] when defining a new [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]: | ||
| 160 | |||
| 161 | [[Data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] ([[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]) identification: | ||
| 162 | |||
| 163 | * [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] identifier | ||
| 164 | * [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] name | ||
| 165 | |||
| 166 | A list of metadata (% style="color:#e74c3c" %)concepts(%%) assigned as [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] of the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 167 | |||
| 168 | * (statistical) (% style="color:#e74c3c" %)concept(%%) identifier | ||
| 169 | * ordinal number of the [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] in the key structure ([[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] only) | ||
| 170 | * [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier (Id, (% style="color:#e74c3c" %)version(%%), [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]]) if the [[representation>>doc:sdmx:Glossary.Representation.WebHome]] is coded | ||
| 171 | |||
| 172 | A list of (statistical) (% style="color:#e74c3c" %)concepts(%%) assigned as [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] for the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 173 | |||
| 174 | * (statistical) (% style="color:#e74c3c" %)concept(%%) identifier | ||
| 175 | * [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier if the (% style="color:#e74c3c" %)concept(%%) is coded | ||
| 176 | * assignment status: mandatory or conditional | ||
| 177 | * attachment (% style="color:#e74c3c" %)level | ||
| 178 | * maximum text length for the uncoded (% style="color:#e74c3c" %)concepts | ||
| 179 | * maximum [[code>>doc:sdmx:Glossary.Code.WebHome]] length for the coded (% style="color:#e74c3c" %)concepts | ||
| 180 | |||
| 181 | A list of the [[code lists>>doc:sdmx:Glossary.Code list.WebHome]] used in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. For each: | ||
| 182 | |||
| 183 | * [[code list>>doc:sdmx:Glossary.Code list.WebHome]] identifier | ||
| 184 | * [[code list>>doc:sdmx:Glossary.Code list.WebHome]] name | ||
| 185 | * [[code>>doc:sdmx:Glossary.Code.WebHome]] values and descriptions | ||
| 186 | |||
| 187 | Definition of [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definitions. Two (or more) partners performing data exchanges in a certain context need to agree on: | ||
| 188 | |||
| 189 | * the list of [[data set>>doc:sdmx:Glossary.Data set.WebHome]] identifiers they will be using; | ||
| 190 | * for each [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]]: | ||
| 191 | * its content and description | ||
| 192 | * the relevant [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] that defines the structure of the data reported or disseminated according the the [[dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] definition | ||
| 193 | |||
| 194 | ==== 3.4.1.3 Exchanging Attributes ==== | ||
| 195 | |||
| 196 | ===== //3.4.1.3.1 Attributes on series, sibling and data set level // ===== | ||
| 197 | |||
| 198 | //Static properties//. | ||
| 199 | |||
| 200 | * Upon creation of a series the sender has to provide to the receiver values for all mandatory [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]. In case they are available, values for conditional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] should also be provided. Whereas initially this information may be provided by means other than [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] or [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] messages (e.g. paper, telephone) it is expected that partner institutions will be in a position to provide this information in [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] or [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] format over time. | ||
| 201 | * A centre may agree with its data exchange partners special procedures for authorising the setting of [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]' initial values. | ||
| 202 | * [[Attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values at a [[data set>>doc:sdmx:Glossary.Data set.WebHome]] (% style="color:#e74c3c" %)level(%%) are set and maintained exclusively by the centre administrating the exchanged [[data set>>doc:sdmx:Glossary.Data set.WebHome]]. | ||
| 203 | |||
| 204 | //Communication of changes// to the centre. | ||
| 205 | |||
| 206 | * Following the creation of a series, the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values do not have to be reported again by senders, as long as they do not change. | ||
| 207 | * Whenever changes in [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values for a series (or [[sibling group>>doc:sdmx:Glossary.Sibling group.WebHome]]) occur, the reporting institutions should report either all [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values again (this is the recommended option) or only the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values which have changed. This applies both to the mandatory and the conditional [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]]. For example, if a previously reported value for a conditional [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] is no longer valid, this has to be reported to the centre. | ||
| 208 | * A centre may agree with its data exchange partners special procedures for authorising modifications in the [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] values. | ||
| 209 | |||
| 210 | Communication of observation (% style="color:#e74c3c" %)level(%%) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] “[[observation status>>doc:sdmx:Glossary.Observation status.WebHome]]”, "observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]]", "observation pre-break". | ||
| 211 | |||
| 212 | * In [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]], the observation (% style="color:#e74c3c" %)level(%%) [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] “[[observation status>>doc:sdmx:Glossary.Observation status.WebHome]]” is part of the fixed syntax of the ARR segment used for observation reporting. Whenever an observation is exchanged, the corresponding [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]] must also be exchanged attached to the observation, regardless of whether it has changed or not since the previous data exchange. This rule also applies to the use of the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] formats, although the syntax does not necessarily require this. | ||
| 213 | * If the “[[observation status>>doc:sdmx:Glossary.Observation status.WebHome]]” changes and the observation remains unchanged, both [[components>>doc:sdmx:Glossary.Component.WebHome]] would have to be reported. | ||
| 214 | * For [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]] having also the observation (% style="color:#e74c3c" %)level(%%) [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] “observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]]” and "observation pre-break" defined, this rule applies to these [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] as well: if an institution receives from another institution an observation with an [[observation status>>doc:sdmx:Glossary.Observation status.WebHome]] [[attribute>>doc:sdmx:Glossary.Attribute.WebHome]] only attached, this means that the associated observation [[confidentiality>>doc:sdmx:Glossary.Confidentiality.WebHome]] and prebreak observation [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] either never existed or from now they do not have a value for this observation. | ||
| 215 | |||
| 216 | === 3.4.2 Best Practices for Batch Data Exchange === | ||
| 217 | |||
| 218 | ==== 3.4.2.1 Introduction ==== | ||
| 219 | |||
| 220 | Batch data exchange is the exchange and maintenance of entire databases between counterparties. It is an activity that often employs [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] formats, and might also use the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]]-specific [[data set>>doc:sdmx:Glossary.Data set.WebHome]]. The following points apply equally to both formats. | ||
| 221 | |||
| 222 | ==== 3.4.2.2 Positioning of the Dimension "Frequency" ==== | ||
| 223 | |||
| 224 | The position of the “frequency” [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] is unambiguously identified in the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]]. Moreover, most central institutions devising structural definitions have decided to assign to this [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] the first position in the key structure. This facilitates the easy identification of this [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]], something that it is necessary to frequency's crucial role in several database systems and in attaching [[attributes>>doc:sdmx:Glossary.Attribute.WebHome]] at the “sibling” group (% style="color:#e74c3c" %)level(%%). | ||
| 225 | |||
| 226 | ==== 3.4.2.3 Identification of Data Structure Definitions (DSDs) ==== | ||
| 227 | |||
| 228 | In order to facilitate the easy and immediate recognition of the structural definition [[maintenance agency>>doc:sdmx:Glossary.Maintenance agency.WebHome]] that defined a [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]], most central institutions devising structural definitions use the first characters of the [[data structure definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] identifiers to identify their institution: e.g. BIS_EER, EUROSTAT_BOP_01, ECB_BOP1, etc. | ||
| 229 | |||
| 230 | ==== 3.4.2.4 Identification of the Data Flows ==== | ||
| 231 | |||
| 232 | In order to facilitate the easy and immediate recognition of the institution administrating a [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definitions, many central institutions prefer to use the first characters of the [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definition identifiers to identify their institution: e.g. BIS_EER, ECB_BOP1, ECB_BOP1, etc. Note that in GESMES/TS the [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] plays the role of the [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definition (see //[[DataSet>>doc:sdmx:Glossary.Data set.WebHome]] //in the [[SDMX-IM>>doc:sdmx:Glossary.SDMX Information Model.WebHome]]//)//. | ||
| 233 | |||
| 234 | The statistical information in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is broken down into two fundamental parts - [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] (comprising the [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]], and associated (% style="color:#e74c3c" %)Concepts(%%) and [[Code Lists>>doc:sdmx:Glossary.Code list.WebHome]]) - see Framework for Standards -, and observational data (the [[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]). This is an important distinction, with specific terminology associated with each part. Data - which is typically a set of numeric observations at specific points in time - is organized into [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] (//[[DataSet>>doc:sdmx:Glossary.Data set.WebHome]]//) These [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] are structured according to a specific [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] (//DataStructureDefinition//) and are described in the [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definition (//DataflowDefinition)// The [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] describes the metadata that allows an understanding of what is expressed in the [[data set>>doc:sdmx:Glossary.Data set.WebHome]], whilst the [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] definition provides the identifier and other important information (such as the periodicity of reporting) that is common to all of its [[component>>doc:sdmx:Glossary.Component.WebHome]] [[data sets>>doc:sdmx:Glossary.Data set.WebHome]]. | ||
| 235 | |||
| 236 | Note that the role of the [[Data Flow>>doc:sdmx:Glossary.Dataflow.WebHome]] (called //DataflowDefintion// in the model) and [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] is very specific in the model, and the terminology used may not be the same as used in all organisations, and specifically the term [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] is used differently in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] than in GESMES/TS. Essentially the GESMES/TS term [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] is, in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]], the [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] Definition" whist the term [[Data Set>>doc:sdmx:Glossary.Data set.WebHome]] in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] is used to describe the "container" for an instance of the data. | ||
| 237 | |||
| 238 | ==== 3.4.2.5 Special Issues ==== | ||
| 239 | |||
| 240 | ===== 3.4.2.5.1 "Frequency" related issues ===== | ||
| 241 | |||
| 242 | **//Special frequencies.//** The issue of data collected at special (regular or irregular) intervals at a lower than daily frequency (e.g. 24 or 36 or 48 observations per year, on irregular days during the year) is not extensively discussed here. However, for data exchange purposes: | ||
| 243 | |||
| 244 | * such data can be (% style="color:#e74c3c" %)mapped(%%) into a series with daily frequency; this daily series will only hold observations for those days on which the measured event takes place; | ||
| 245 | * if the collection intervals are regular, additional values to the existing frequency [[code list>>doc:sdmx:Glossary.Code list.WebHome]](s) could be added in the future. | ||
| 246 | |||
| 247 | **//Tick data.//** The issue of data collected at irregular intervals at a higher than daily frequency (e.g. tick-by-tick data) is not discussed here either. However, for data exchange purposes, such series can already be exchanged in the [[SDMX-EDI>>doc:sdmx:Glossary.SDMX-EDI.WebHome]] format by using the option to send observations with the associated time stamp. |