Skip to Content
Last modified by Artur K. on 2026/05/29 14:28
From version 1.1
edited by Helena K.
on 2026/01/16 16:22
Change comment: There is no comment for this version
To version 3.1
edited by Artur K.
on 2026/05/29 14:28
Change comment: Copied from sdmxsrlocalization:Methodology.Possible Ways of Implementing the Observation Status Concept.WebHome

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.helena
1 +xwiki:XWiki.arturkryazhev
Content
... ... @@ -1,15 +1,16 @@
1 -
1 +{{box title="**Contents**"}}
2 +{{toc/}}
3 +{{/box}}
2 2  
3 -**Document History**
5 += Document History =
4 4  
5 -|**Version**|**Date**|**Comment**
6 -|1.0|1/10/2014|Initial version
7 -|2.0|1/5/2019|Guideline title change (more accurate); add new OBS_STATUS codes to hierarchy; changed 3.1 approach to include OBS_STATUS code (this compatibility break is the major version change); text clarification; include a document history
7 +(% style="width:972.446px" %)
8 +|**Version**|**Date**|(% style="width:773px" %)**Comment**
9 +|1.0|1/10/2014|(% style="width:773px" %)Initial version
10 +|2.0|1/5/2019|(% style="width:773px" %)Guideline title change (more accurate); add new OBS_STATUS codes to hierarchy; changed 3.1 approach to include OBS_STATUS code (this compatibility break is the major version change); text clarification; include a document history
8 8  
12 += 1) Introduction =
9 9  
10 -
11 -**1) Introduction**
12 -
13 13  First of all, it is important to note that the "Observation status" code list (CL_OBS_STATUS{{footnote}}https://sdmx.org/?page_id=3215{{/footnote}}) has an heterogeneous character as it mixes concepts which are not always mutually exclusive (e.g. a missing value can generate a break in time series, an estimated value can be of low reliability). Thus, to cope with the issue of allocating more than one flag to one statistical value, this code list should ideally be broken down into various sub-code lists corresponding to the various concepts covered. It was not done so because it was felt that it would unnecessarily increase the number of (very short) code lists for low benefits in terms of technical and conceptual orthodoxy.
14 14  
15 15  However, in view of the central importance of this code list, it is essential to provide implementers with all possible ways of implementing this code list so that they can decide, based on their specific implementation needs, which option best suits their requirements. These various options are presented in the sections below, and their pros and contras explicated.
... ... @@ -18,36 +18,37 @@
18 18  
19 19  The SDMX standard allows for the use of zero or more observation level attributes, using any identifiers. However, SDMX-EDI imposes the mandatory use of the observation level attribute called OBS_STATUS. In the past, SDMX-EDI has limited itself, for practical reasons, to the use of the observation level attributes OBS_STATUS, CONF_STATUS, PRE_BREAK_VALUE and COMMENT_OBS, but SDMX-EDI can handle any number of observation level attributes, as long as OBS_STATUS is included.
20 20  
21 -**2) One flag only per value**
22 += 2) One flag only per value =
22 22  
23 23  In case implementers want to use only one single flag per value, they should use the hierarchy below to determine the code to be used. This approach (choice of only one event, namely the most important one) offers a good compromise between simplicity for the user, completeness of provided information and presentational easiness of management on the user interface side. The main drawback of this approach is the loss of information resulting from the use of only one flag when several flags may apply to a given value.
24 24  
25 25  Example: From now on, value x is compiled on the basis of a methodology diverging from the previous one (e.g. following an alignment with international standards), which generates a break in time series. However, the value in this period is suppressed, e.g. for confidentiality reasons. In this case, two flags, namely B (//Time series break//) and Q (missing value; suppressed), should be used. If only one flag is to be indicated, then use should be made of the hierarchy below to determine which flag to use. In this case, this would be B since B has precedence over Q in the hierarchy.
26 26  
27 -|(% rowspan="2" %)**Observation status hierarchy**|(% colspan="2" %)**Relevant in conjunction with...**
28 -|**numeric values**|**missing values**
29 -|**B** / time series break (highest importance)|Yes|Yes
30 -|**O** / missing value| |Yes
31 -|**M** / missing value; data cannot exist| |Yes
32 -|**L** / missing value; data exist but were not collected| |Yes
33 -|**H** / missing value; holiday or weekend| |Yes
34 -|**Q** / missing value; suppressed| |Yes
35 -|**J** / derogation|Yes|Yes
36 -|**S** / strike and other special events|Yes|Yes
37 -|**D** / definition differs|Yes|
38 -|**K** / data included in another category| |Yes
39 -|**W** / Includes data from another category|Yes|
40 -|**I** / imputed value|Yes|
41 -|**F** / forecast value|Yes|
42 -|**E** / estimated value|Yes|
43 -|**P** / provisional value|Yes|
44 -|**N** / not significant|Yes|
45 -|**U** / low reliability|Yes|
46 -|**V** / unvalidated value|Yes|
47 -|**G** / experimental value|Yes|
48 -|**A** / normal value|Yes|
28 +(% style="width:1157.45px" %)
29 +|(% rowspan="2" style="width:512px" %)**Observation status hierarchy**|(% colspan="2" style="width:642px" %)**Relevant in conjunction with...**
30 +|(% style="width:281px" %)**numeric values**|(% style="width:361px" %)**missing values**
31 +|(% style="width:512px" %)**B** / time series break (highest importance)|(% style="width:281px" %)Yes|(% style="width:361px" %)Yes
32 +|(% style="width:512px" %)**O** / missing value|(% style="width:281px" %) |(% style="width:361px" %)Yes
33 +|(% style="width:512px" %)**M** / missing value; data cannot exist|(% style="width:281px" %) |(% style="width:361px" %)Yes
34 +|(% style="width:512px" %)**L** / missing value; data exist but were not collected|(% style="width:281px" %) |(% style="width:361px" %)Yes
35 +|(% style="width:512px" %)**H** / missing value; holiday or weekend|(% style="width:281px" %) |(% style="width:361px" %)Yes
36 +|(% style="width:512px" %)**Q** / missing value; suppressed|(% style="width:281px" %) |(% style="width:361px" %)Yes
37 +|(% style="width:512px" %)**J** / derogation|(% style="width:281px" %)Yes|(% style="width:361px" %)Yes
38 +|(% style="width:512px" %)**S** / strike and other special events|(% style="width:281px" %)Yes|(% style="width:361px" %)Yes
39 +|(% style="width:512px" %)**D** / definition differs|(% style="width:281px" %)Yes|(% style="width:361px" %)
40 +|(% style="width:512px" %)**K** / data included in another category|(% style="width:281px" %) |(% style="width:361px" %)Yes
41 +|(% style="width:512px" %)**W** / Includes data from another category|(% style="width:281px" %)Yes|(% style="width:361px" %)
42 +|(% style="width:512px" %)**I** / imputed value|(% style="width:281px" %)Yes|(% style="width:361px" %)
43 +|(% style="width:512px" %)**F** / forecast value|(% style="width:281px" %)Yes|(% style="width:361px" %)
44 +|(% style="width:512px" %)**E** / estimated value|(% style="width:281px" %)Yes|(% style="width:361px" %)
45 +|(% style="width:512px" %)**P** / provisional value|(% style="width:281px" %)Yes|(% style="width:361px" %)
46 +|(% style="width:512px" %)**N** / not significant|(% style="width:281px" %)Yes|(% style="width:361px" %)
47 +|(% style="width:512px" %)**U** / low reliability|(% style="width:281px" %)Yes|(% style="width:361px" %)
48 +|(% style="width:512px" %)**V** / unvalidated value|(% style="width:281px" %)Yes|(% style="width:361px" %)
49 +|(% style="width:512px" %)**G** / experimental value|(% style="width:281px" %)Yes|(% style="width:361px" %)
50 +|(% style="width:512px" %)**A** / normal value|(% style="width:281px" %)Yes|(% style="width:361px" %)
49 49  
50 -**3) Multiple flagging**
52 += 3) Multiple flagging =
51 51  
52 52  There might be cases however where implementers will want to attach multiple flags to one statistical value. To cope with this situation, three solutions have been analysed, based on:
53 53  
... ... @@ -57,7 +57,7 @@
57 57  
58 58  Technically the three approaches are possible. However, considering the severe limitations that the third approach would implicate, only one of the first two approaches will be recommended for use (as said earlier also with a view to improving harmonisation across implementations and backwards compatibility).
59 59  
60 -**//3.1) Duplication approach (recommended solution)//**
62 +== 3.1) Duplication approach (recommended solution) ==
61 61  
62 62  In this case, the OBS_STATUS concept is duplicated as many times as needed. These duplicated concepts should be named "OBS_STATUS", "OBS_STATUS_1", "OBS_STATUS_2", "OBS_STATUS_3", etc. All these concepts have to be inserted in the DSD and linked to the CL_OBS_STATUS code list. Only one value is allowed per code list. In order to have backwards compatibility with systems only processing a single observation status flag and also to keep DSDs with multiple flags compatible with DSDs using a single flag, it is strongly recommended to sort the flags according to the observation status hierarchy table above in a data message. For example, if the multiple flags should be G (experimental), V (unvalidated) and D (definition differs), then the order according to the hierarchy would be: OBS_STATUS = D, OBS_STATUS_1 = V, OBS_STATUS_2 = G. A system only parsing a single flag could still rely on the OBS_STATUS concept identifier to catch the flag with the highest priority.
63 63  
... ... @@ -67,7 +67,7 @@
67 67  
68 68  This approach is the recommended general solution for implementations where multiple flagging is required.
69 69  
70 -**//3.2) Decomposition approach (recommended solution if exchange does not support SDMX-EDI)//**
72 +== 3.2) Decomposition approach (recommended solution if exchange does not support SDMX-EDI) ==
71 71  
72 72  Here, CL_OBS_STATUS code list is broken down into its basic components, distinguished on the basis of the different concepts used and their mutually exclusive character. The list of "building blocks" composing the CL_OBS_STATUS code list as it stands at present could be represented as separate concepts as follows:
73 73  
... ... @@ -89,20 +89,16 @@
89 89  
90 90  Although not recommended as the preferred solution, this approach can be implemented in cases where the general solution cannot be applied, or is not the appropriate solution, in a particular context.
91 91  
92 -|(((
94 +{{box}}
93 93  **Comments on the choice of the recommended solution**
94 94  
95 -//Both "Decomposition" and "Duplication" options provide acceptable workarounds to the problem of multiple flagging, and appear to be quite similar in practice. The t//rade-off in this context was between orthodoxy and ease of implementation.
97 +Both "Decomposition" and "Duplication" options provide acceptable workarounds to the problem of multiple flagging, and appear to be quite similar in practice. The trade-off in this context was between orthodoxy and ease of implementation.
98 +Conceptually the "Decomposition" approach is the strongest of the two as it not only allows separating concepts, but also helps arranging codes into more homogeneous code lists. It also requires that implementers define pure concepts and name them accordingly.
99 +This document recommends the "Duplication" approach mainly on the practical grounds of ease of implementation because its use of “OBS_STATUS” is compatible with the cross-domain concept scheme, whereas the Decomposition approach is not. The recommended approach could be reconsidered in the future, would the technical standard better accommodate the decomposition approach.
100 +{{/box}}
96 96  
97 -//Conceptually the "Decomposition" approach is the strongest of the two as it not only allows separating concepts, but also helps arranging codes into more homogeneous code lists. It also requires that implementers define pure concepts and name them accordingly.//
102 +== 3.3) Extended single code list approach (strongly discouraged) ==
98 98  
99 -//This document recommends the "Duplication" approach mainly on the practical grounds of ease of implementation because its use of “OBS_STATUS” is compatible with the cross-domain concept scheme, whereas the Decomposition approach is not. The recommended approach could be reconsidered in the future, would the technical standard better accommodate the //decomposition //approach.//
100 -)))
101 -
102 -
103 -
104 -**//3.3) Extended single code list approach (strongly discouraged)//**
105 -
106 106  The extended version of CL_OBS_STATUS (see below) provides the full list of logically possible combinations of codes in a specific SDMX implementation.
107 107  
108 108  An advantage of this solution would be that only meaningful combinations of flags are included in the list. Users would not be able to choose combinations which would not make sense (such as "missing" and "estimated").
... ... @@ -134,7 +134,7 @@
134 134  
135 135  Considering the severe limitations implied by the third approach, only the first two approaches are recommended for use.
136 136  
137 -**4) Conclusion**
135 += 4) Conclusion =
138 138  
139 139  From the analysis of the various approaches presented above, it appears clearly that the extended single code list approach cannot be recommended for use.
140 140  
... ... @@ -150,7 +150,6 @@
150 150  * **Recommended solution if exchange does not support SDMX-EDI**
151 151  * **Strongly discouraged**
152 152  
151 +----
153 153  
154 -
155 -
156 156  {{putFootnotes/}}
© Semantic R&D Group, 2026