Skip to Content

Wiki source code of 16 Annex 6: Worked Use Case

Version 4.2 by Helena on 2025/05/23 22:09
Show last authors
1 {{box title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 == 16.1 Scope ==
6
7 The scope of this Chapter is to show the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] interactions with the Registry, database, and [[metadata repository>>doc:sdmx:Glossary.Metadata repository.WebHome]] for the use case scenario outlined in Chapter 3. The scenario is repeated below.
8
9 The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] examples used are (% style="color:#e74c3c" %)version(%%) 2.1.
10
11 == 16.2 Scenario ==
12
13 This scenario is the dissemination of data and related [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] using [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] web services. It shows how the data, [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] are used to build a web dissemination system.
14
15 [[image:image-20250318221512-1.jpeg||data-xwiki-image-style-alignment="center"]]
16
17 (% style="text-align: center;" %)
18 **{{id name="image_47"/}}Figure 47: Process flow of an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Web Data Dissemination System**
19
20 (% style="width:753.446px" %)
21 |**Process**|(% style="width:670px" %)**Description**
22 |1|(% style="width:670px" %)Retrieve the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] from a [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] source (e.g. an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]), and create database tables.
23 |2|(% style="width:670px" %)Read an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[data set>>doc:sdmx:Glossary.Data set.WebHome]] file and load the data into the database
24 |3|(% style="width:670px" %)Data discovery system continually synchronises its metadata with the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] source. A user makes a data selection from choices built from the information held in an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] ([[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] such as [[category scheme>>doc:sdmx:Glossary.Category scheme.WebHome]], [[dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]], [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]], [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]], [[provision agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]] and data registration)
25 |3|(% style="width:670px" %)These choices are logical choices, built from the [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] selections.
26 |5|(% style="width:670px" %)The logical choice is formatted as an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data query. This is passed to the Data Base which responds with an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[data set>>doc:sdmx:Glossary.Data set.WebHome]].
27 |6|(% style="width:670px" %)[[Reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] relevant to the data returned is retrieved from a [[metadata repository>>doc:sdmx:Glossary.Metadata repository.WebHome]].
28 |7|(% style="width:670px" %)The data and metadata are passed to a visualization tool to display the data in tables, charts, graphs, (% style="color:#e74c3c" %)maps(%%) etc. Often a download is offered in various formats. The download options often include also the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] or MSD.
29
30 == 16.3 Structural Metadata ==
31
32 === 16.3.1 Schematic ===
33
34 The following [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and provisioning metadata is used in the scenario.
35
36 [[image:image-20250318221512-2.jpeg||data-xwiki-image-style-alignment="center"]]
37
38 (% style="text-align: center;" %)
39 **{{id name="image_48"/}}Figure 48: Structural and Provisioning Metadata Used in the Scenario**
40
41 === 16.3.2 Data Structure Definition ===
42
43 For the reasons of clarity a sub set of the contents of the [[Code Lists>>doc:sdmx:Glossary.Code list.WebHome]] and [[Concept Schemes>>doc:sdmx:Glossary.Concept scheme.WebHome]] are used in this example, and only a few of the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] [[Attributes>>doc:sdmx:Glossary.Attribute.WebHome]] are used.
44
45 (% class="wikigeneratedid" id="HDimensions" %)
46 **Dimensions**
47
48 (% style="text-align: center;" %)
49 [[image:image-20250318221512-3.jpeg]]** **
50
51 (% class="wikigeneratedid" id="HAttributes" %)
52 **Attributes**
53
54 (% style="text-align: center;" %)
55 [[image:image-20250318221512-4.jpeg]]** **
56
57 (% class="wikigeneratedid" id="HGroup" %)
58 **Group**
59
60 (% style="text-align: center;" %)
61 [[image:image-20250318221512-5.jpeg]]** **
62
63 (% class="wikigeneratedid" id="HPrimaryMeasure" %)
64 **Primary Measure**
65
66 (% style="text-align: center;" %)
67 [[image:image-20250318221512-6.jpeg]]** **
68
69 == 16.4 Scenario Steps ==
70
71 === 16.4.1 Create Data Base ===
72
73 [[image:image-20250318221512-7.jpeg||data-xwiki-image-style-alignment="center"]]
74
75 (% style="text-align: center;" %)
76 **{{id name="image_49"/}}Figure 49: User Case - Create Database Tables from DSD**
77
78 The following [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] is extracted from the Registry using the REST query http:~/~/[ ws-entry-point]/DataStructure_ECB/ECB_EXR1
79
80 (% class="wikigeneratedid" id="HDimensions" %)
81 **Dimensions**
82
83 [[image:image-20250318221512-8.jpeg||data-xwiki-image-style-alignment="center"]]
84
85 Note that for brevity only the XML of the FREQ and TIME_PERIOD are shown in full.
86
87 (% class="wikigeneratedid" id="HNotes" %)
88 **Notes**
89
90 1. A [[Dimension>>doc:sdmx:Glossary.Dimension.WebHome]] can state its position in the key, but the actual sequence is defined by the sequence in the i.e. the inferred sequence in the  takes precedence over the value of the position attribute.
91 1. This [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] is converted from (% style="color:#e74c3c" %)version(%%) 2.0. The (% style="color:#e74c3c" %)version(%%) 2.0 (% style="color:#e74c3c" %)Concepts(%%) were standalone in the form  <(% style="color:#e74c3c" %)Concept(%%) agencyID=”ECB” (% style="color:#e74c3c" %)version(%%)=”1.0”/>
92 1. The standalone (% style="color:#e74c3c" %)concept(%%) is not supported in 2.1 and any standalone (% style="color:#e74c3c" %)concept(%%) is placed in a [[Concept Scheme>>doc:sdmx:Glossary.Concept scheme.WebHome]] with the Id “STANDALONE_CONCEPT_SCHEME” of the (% style="color:#e74c3c" %)version(%%) corresponding to the (% style="color:#e74c3c" %)version(%%) of the original standalone (% style="color:#e74c3c" %)concept(%%).
93
94 (% class="wikigeneratedid" id="HGroup-1" %)
95 **Group**
96
97 (% style="text-align: center;" %)
98 [[image:image-20250318221512-9.jpeg]]** **
99
100 **Notes:**
101
102 1. A Group in version 2.1 plays the same role as it does in version 2.0 except that it does not group Series in the data set. The Group in version 2.1 is used solely as a mechanism to attach Attributes in the data set.
103 1. The Group in version 2.1 is retained for compatibility with version 2.0 and for avoiding repetition in the new Attribute Relationship construct which is introduced in version 2.1.
104
105 (% class="wikigeneratedid" id="HAttributes" %)
106 **Attributes**
107
108 [[image:image-20250318221512-10.jpeg||data-xwiki-image-style-alignment="center"]]
109
110 [[image:image-20250318221512-11.jpeg||data-xwiki-image-style-alignment="center"]]
111
112 Note that for brevity only the XML of the DECIMALS (relationship to a group) COLLECTION (relationship to a set of Dimensions) and OBS_CONF (relationship to the Primary Measure) are shown in full.
113
114 (% class="wikigeneratedid" id="HNotes-1" %)
115 **Notes**
116
117 1. The Attribute has a relationship to either a data set of one or more Dimensions as described in Chapter 4.
118 1. DECIMALS has a relationship to a Group with the Id of Group.
119 1. COLLECTION has a relationship to a set of Dimensions – in this case this is, in reality, the series key as this has been converted from version 2.0. It is possible for an attribute to have a relationship with just one, a few, or all of the Dimensions.
120 1. OBS_CONF has a relationship with the Primary Measure.
121
122 (% class="wikigeneratedid" id="HMeasure" %)
123 **Measure**
124
125 [[image:image-20250318221512-12.jpeg||data-xwiki-image-style-alignment="center"]]
126
127 A typical database schema that can be set up using this DSD is shown below.
128
129 (% class="wikigeneratedid" id="HSeriesKeyTable" %)
130 **Series Key Table**
131
132 [[image:image-20250318221902-34.png||data-xwiki-image-style-alignment="center"]]
133
134 (% style="text-align: center;" %)
135 **{{id name="image_50"/}}Figure 50: Relational Table Structure created from a DSD Dimensions and Related Attributes**
136
137 (% class="wikigeneratedid" id="HGroup201CGroup201DTable" %)
138 **Group “Group” Table**
139
140 (% style="text-align: center;" %)
141 [[image:image-20250318221512-13.jpeg]]** **
142
143 (% style="text-align: center;" %)
144 **{{id name="image_51"/}}Figure 51: Relational Table Structure created from a DSD Group and Related Attributes**
145
146 (% class="wikigeneratedid" id="HObservation" %)
147 **Observation**
148
149 [[image:image-20250318221512-14.jpeg||data-xwiki-image-style-alignment="center"]]
150
151 (% style="text-align: center;" %)
152 **{{id name="image_52"/}}Figure 52: Relational Table Structure created from a DSD Primary Measure and Related Attributes**
153
154 === 16.4.2 Load Database ===
155
156 [[image:image-20250318221512-15.jpeg||data-xwiki-image-style-alignment="center"]]
157
158 (% style="text-align: center;" %)
159 **{{id name="image_53"/}}Figure 53: Loading an SDMX Dataset into a Database **
160
161 The following is an extract from the SDMX dataset.
162
163 [[image:image-20250319123255-1.png||data-xwiki-image-style-alignment="center"]]
164
165 [[image:image-20250319123339-2.png||data-xwiki-image-style-alignment="center"]]
166
167 Note that this is a sub set of the actual data message.
168
169 (% class="wikigeneratedid" id="HNotes-2" %)
170 **Notes**
171
172 1. The reference to the DSD is given in the message header and given a local structureId
173 1. The local structureId (DSD) is referenced from the DataSet
174 1. A Group does not contain Series in version 2.1: it is used solely to declare attributes
175 1. The first <Obs> in the first <Series> has an OBS_VALUE of “NaN”. This is an XML expression that declared a value of “not a number”, thus allowing a “missing value” to be declared.
176 1. A dataset can contain observations from different frequencies.
177
178 The diagrams below show the database content based on the schema created.
179
180 (% class="wikigeneratedid" id="HSeriesKeyTable-1" %)
181 **Series Key Table**
182
183 [[image:image-20250318221512-16.jpeg||data-xwiki-image-style-alignment="center"]]
184
185 (% class="wikigeneratedid" id="HGroupTable" %)
186 **Group Table**
187
188 [[image:image-20250318221512-17.jpeg||data-xwiki-image-style-alignment="center"]]
189
190 (% class="wikigeneratedid" id="HObservationTable" %)
191 **Observation Table**
192
193 (% style="text-align: center;" %)
194 [[image:image-20250318221512-18.jpeg]]** **
195
196 The database is now available for query.
197
198 === 16.4.3 Register Data Source ===
199
200 The following registration is submitted to the Registry.
201
202 [[image:image-20250318221512-19.jpeg||data-xwiki-image-style-alignment="center"]]
203
204 The SDMX web service of the database can now be searched for in the Registry.
205
206 === 16.4.4 Retrieve and Visualise Category Scheme and Dataflows ===
207
208 [[image:image-20250318221512-20.jpeg||data-xwiki-image-style-alignment="center"]]
209
210 (% style="text-align: center;" %)
211 **{{id name="image_54"/}}Figure 54: Building a Search Screen from Structural Metadata**
212
213 The following REST query will return the Category Scheme and Categorisations that reference any of the Categories in the scheme.
214
215 [[http:~~/~~/ ~[ws-entry-point~]/categoryscheme/ECB/SDW_ECONOMIC_CONCEPTS/1.0?references=parents>>http:// [ws-entry-point]/categoryscheme/ECB/SDW_ECONOMIC_CONCEPTS/1.0?references=parents||rel="noopener noreferrer" target="_blank"]]
216
217 [[image:image-20250318221512-21.jpeg||data-xwiki-image-style-alignment="center"]]
218
219 [[image:image-20250318221512-22.jpeg||data-xwiki-image-style-alignment="center"]]
220
221 The application will now retrieve the Dataflows in the list of Categorisations – in this case there is only one – effective exchange rates.
222
223 The following REST query will return the dataflow.
224
225 [[http:~~/~~/ ~[ws-entry-point~]/dataflow/ECB/2034482/1.0>>http:// [ws-entry-point]/dataflow/ECB/2034482/1.0||rel="noopener noreferrer" target="_blank"]]
226
227 [[image:image-20250318221512-23.jpeg||data-xwiki-image-style-alignment="center"]]
228
229 The actual choreography of the query application is dependent upon the implementation but the following is assumed for this example.
230
231 1. Query application queries the Registry for the Provision Agreement.
232 1. Query application queries the Registry for data Registrations.
233 1. If there is a data source registered then the dataflow is displayed and can be selected.
234
235 The following REST query will return any provision agreements referenced from the dataflow
236
237 [[http:~~/~~/~[ws-entry-point~]/dataflow/ECB/2034482/1.0?references=provisionagreement>>http://[ws-entry-point]/dataflow/ECB/2034482/1.0?references=provisionagreement||rel="noopener noreferrer" target="_blank"]]
238
239 [[image:image-20250318221512-24.jpeg||data-xwiki-image-style-alignment="center"]]
240
241 (% class="wikigeneratedid" id="HNotes-3" %)
242 **Notes**
243
244 ~1. The <URN> contains the same identification information as the individual attributes in the <Ref> tag in the context of the <Dataflow> and <Structure> tags (the <Dataflow> identifies that the <Structure> is a DataStructure).
245
246 The following REST query will return the Data Provider Scheme that contains the Data Provider connected to the Provision Agreement. Note that this is not necessary in order to query for the data unless the Data Provider contains information required by the query application (such as the name of the Data Provider).
247
248 [[http:~~/~~/~[ws-entry-point~]/dataprovider/ECB>>http://[ws-entry-point]/dataprovider/ECB||rel="noopener noreferrer" target="_blank"]]
249
250 (% class="wikigeneratedid" id="HNotes-4" %)
251 **Notes**
252
253 ~1. As an Agency can have only one Data Provider Scheme and this has a mandatory Id of DATA_PROVIDER_SCHEME and must be version 1.0 there is no need to specify any other parameters to the REST query.
254
255 This returns the following as part of the Data Provider Scheme.
256
257 [[image:image-20250318221512-25.jpeg||data-xwiki-image-style-alignment="center"]]
258
259 The following Registry query will return the REST data source referenced in a Registration for the Provision Agreement
260
261 [[image:image-20250318221512-26.jpeg||data-xwiki-image-style-alignment="center"]]
262
263 The response from this query is:
264
265 [[image:image-20250318221512-27.jpeg||data-xwiki-image-style-alignment="center"]]
266
267 The query application now has the details of the web service to call when making a query, and it also knows that the query expected uses the REST interface.
268
269 [[http:~~/~~/~[ws-entry-point~] /~[RESTParameters~]>>http://[ws-entry-point] /[RESTParameters||rel="noopener noreferrer" target="_blank"]]]
270
271 Where [ws-entry-point] is the URL of the SDMX web service and [RESTParameters] are the query parameters.
272
273 The application can now visualize the information it has retrieved from the Registry in a way that is meaningful to the user. An example visualization is shown below.
274
275 [[image:image-20250318221512-28.jpeg||data-xwiki-image-style-alignment="center"]]
276
277 (% style="text-align: center;" %)
278 **{{id name="image_55"/}}Figure 55: Example Query Screen Built from SDMX Category Scheme and Dataflows**
279
280 The user will select the Category and this then shows the Dataflows that are available and which have data available for query (i.e. there is a Registration). There can be many such Dataflows but in this example there is only one.
281
282 === 16.4.5 Build Dimension Selection and Logical Query ===
283
284 [[image:image-20250318221512-29.jpeg||data-xwiki-image-style-alignment="center"]]
285
286 Clicking on the Dataflow will cause the query application to query the Registry for the DSD associated to the Dataflow. The Dimensions and the relevant codes associated with the Dimensions are then shown for detailed data selections.
287
288 [[image:image-20250318221512-30.jpeg||data-xwiki-image-style-alignment="center"]]
289
290 (% style="text-align: center;" %)
291 **{{id name="image_56"/}}Figure 56: Selection Screen build from SDMX DSD**
292
293 The user can click on each of the Dimensions and the relevant code selections are displayed. Note that in the example above the application is assumed to have queries for all the keys so that it can process them in order to grey-out the Dimension selections for which there is no data. These are greyed-out based on the current selections in the other Dimensions. This shows the importance of content constraints in a web dissemination system, as they can be used to guide the user to make only the selections that will return data. The same result can be achieved if these constraints are available for query (e.g. in a Registry), but this is a less effective way of processing if the contenrts of the database are dynamic.
294
295 === 16.4.6 Query Database ===
296
297 [[image:image-20250318221512-31.jpeg||data-xwiki-image-style-alignment="center"]]
298
299 The logical query of the user is converted to an SDMX query and sent to the Data Web Service of the data publisher. The Data Web Service may also query the Metadata Repository in order to return both data and related metadata to the user application. The Data Web Service may also require structural metadata (DSD and MSD and related Code Lists and Concepts) from the structural metadata repository, in this example this an SDMX Registry.
300
301 (% class="wikigeneratedid" id="HDataQueryLogicalSelections" %)
302 **Data Query Logical Selections**
303
304 (% style="width:551.446px" %)
305 |(% style="width:261px" %)**Dimension**|(% style="width:286px" %)**Values**
306 |(% style="width:261px" %)FREQ|(% style="width:286px" %)M
307 |(% style="width:261px" %)CURRENCY|(% style="width:286px" %)Z26,Z28
308 |(% style="width:261px" %)CURRENCY_DENOM|(% style="width:286px" %)EUR,CZK,DKK
309 |(% style="width:261px" %)EXR-TYPE|(% style="width:286px" %)DFCO,EN00
310 |(% style="width:261px" %)EXR_SUFFIX|(% style="width:286px" %)A
311 |(% style="width:261px" %)Time|(% style="width:286px" %)Jan 2008-Dec 2010
312
313 (% class="wikigeneratedid" id="HRESTQuery" %)
314 **REST Query**
315
316 [[http:~~/~~/ws-entry-point/data/2034482/M.Z26+Z28.EUR+CZK+DKK.DFC0+EN00+A /ECB?startPeriod=2008-01&endPeriod=2010-12&detail=dataonly>>http://ws-entry-point/data/2034482/M.Z26+Z28.EUR+CZK+DKK.DFC0+EN00+A /ECB?startPeriod=2008-01&endPeriod=2010-12&detail=dataonly||rel="noopener noreferrer" target="_blank"]]
317
318 === 16.4.7 Visualise the Resultant Data Set ===
319
320 [[image:image-20250318221512-32.jpeg||data-xwiki-image-style-alignment="center" height="310" width="346"]]
321
322 The returned data and metadata can be presented to the user in many ways – tables, graphs, charts, maps etc. SDMX does not support visualization directly but it is possible to define roles for a Dimension and Attribute in the DSD. For instance, it is possible to assign a role of “Geography” or “Title” or “Measure Unit” etc. which will aid a data visualization service to present this information in a meaningful way to the user.
323
324 [[image:image-20250318221512-33.jpeg||data-xwiki-image-style-alignment="center"]]
325
326 (% style="text-align: center;" %)
327 **{{id name="image_57"/}}Figure 57: Example Graph Built from Data Returned from a Query**