Changes for page 16 Annex 6: Worked Use Case

Last modified by Helena on 2025/09/10 11:19

From 1.1 to 1.2 From 4.2 to 4.3

From version 1.2

edited by Artur
on 2025/05/20 14:28

Change comment: Update document after refactoring.

To version 4.2

edited by Helena
on 2025/05/23 22:09

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--xwiki:XWiki.arturkryazhev
++xwiki:XWiki.helena

Content

@@ -4,37 +4,35 @@
  == 16.1 Scope ==
--The scope of this Chapter is to show the SDMX-ML interactions with the Registry, database, and metadata repository for the use case scenario outlined in Chapter 3. The scenario is repeated below.
++The scope of this Chapter is to show the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] interactions with the Registry, database, and [[metadata repository>>doc:sdmx:Glossary.Metadata repository.WebHome]] for the use case scenario outlined in Chapter 3. The scenario is repeated below.
--The SDMX-ML examples used are version 2.1.
++The [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] examples used are (% style="color:#e74c3c" %)version(%%) 2.1.
  == 16.2 Scenario ==
--This scenario is the dissemination of data and related reference metadata using SDMX web services. It shows how the data, reference metadata, and structural metadata are used to build a web dissemination system.
++This scenario is the dissemination of data and related [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] using [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] web services. It shows how the data, [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]], and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] are used to build a web dissemination system.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-1.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
--**{{id name="image_47"/}}Figure 47: Process flow of an SDMX Web Data Dissemination System**
++**{{id name="image_47"/}}Figure 47: Process flow of an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] Web Data Dissemination System**
  (% style="width:753.446px" %)
  |**Process**|(% style="width:670px" %)**Description**
--|1|(% style="width:670px" %)Retrieve the DSD from a structural metadata source (e.g. an SDMX Registry), and create database tables.
--|2|(% style="width:670px" %)Read an SDMX data set file and load the data into the database
--|3|(% style="width:670px" %)Data discovery system continually synchronises its metadata with the structural metadata source. A user makes a data selection from choices built from the information held in an SDMX Registry (structural metadata such as category scheme, dataflow, DSD, data provider, provision agreements and data registration)
--|3|(% style="width:670px" %)These choices are logical choices, built from the dimension selections.
--|5|(% style="width:670px" %)The logical choice is formatted as an SDMX data query. This is passed to the Data Base which responds with an SDMX data set.
--|6|(% style="width:670px" %)Reference metadata relevant to the data returned is retrieved from a metadata repository.
--|7|(% style="width:670px" %)The data and metadata are passed to a visualization tool to display the data in tables, charts, graphs, maps etc. Often a download is offered in various formats. The download options often include also the DSD or MSD.
++|1|(% style="width:670px" %)Retrieve the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] from a [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] source (e.g. an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]), and create database tables.
++|2|(% style="width:670px" %)Read an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[data set>>doc:sdmx:Glossary.Data set.WebHome]] file and load the data into the database
++|3|(% style="width:670px" %)Data discovery system continually synchronises its metadata with the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] source. A user makes a data selection from choices built from the information held in an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] ([[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] such as [[category scheme>>doc:sdmx:Glossary.Category scheme.WebHome]], [[dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]], [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]], [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]], [[provision agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]] and data registration)
++|3|(% style="width:670px" %)These choices are logical choices, built from the [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]] selections.
++|5|(% style="width:670px" %)The logical choice is formatted as an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data query. This is passed to the Data Base which responds with an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[data set>>doc:sdmx:Glossary.Data set.WebHome]].
++|6|(% style="width:670px" %)[[Reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] relevant to the data returned is retrieved from a [[metadata repository>>doc:sdmx:Glossary.Metadata repository.WebHome]].
++|7|(% style="width:670px" %)The data and metadata are passed to a visualization tool to display the data in tables, charts, graphs, (% style="color:#e74c3c" %)maps(%%) etc. Often a download is offered in various formats. The download options often include also the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] or MSD.
  == 16.3 Structural Metadata ==
  === 16.3.1 Schematic ===
--The following structural metadata and provisioning metadata is used in the scenario.
++The following [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] and provisioning metadata is used in the scenario.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-2.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -42,24 +42,28 @@
  === 16.3.2 Data Structure Definition ===
--For the reasons of clarity a sub set of the contents of the Code Lists and Concept Schemes are used in this example, and only a few of the DSD Attributes are used.
++For the reasons of clarity a sub set of the contents of the [[Code Lists>>doc:sdmx:Glossary.Code list.WebHome]] and [[Concept Schemes>>doc:sdmx:Glossary.Concept scheme.WebHome]] are used in this example, and only a few of the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] [[Attributes>>doc:sdmx:Glossary.Attribute.WebHome]] are used.
--===== **Dimensions** =====
++(% class="wikigeneratedid" id="HDimensions" %)
++**Dimensions**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-3.jpeg]]** **
--===== **Attributes** =====
++(% class="wikigeneratedid" id="HAttributes" %)
++**Attributes**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-4.jpeg]]** **
--===== **Group** =====
++(% class="wikigeneratedid" id="HGroup" %)
++**Group**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-5.jpeg]]** **
--===== **Primary Measure** =====
++(% class="wikigeneratedid" id="HPrimaryMeasure" %)
++**Primary Measure**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-6.jpeg]]** **
@@ -68,28 +68,29 @@
  === 16.4.1 Create Data Base ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-7.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
  **{{id name="image_49"/}}Figure 49: User Case - Create Database Tables from DSD**
--The following SDMX-ML is extracted from the Registry using the REST query http:~/~/[ ws-entry-point]/DataStructure_ECB/ECB_EXR1
++The following [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] is extracted from the Registry using the REST query http:~/~/[ ws-entry-point]/DataStructure_ECB/ECB_EXR1
--===== Dimensions =====
++(% class="wikigeneratedid" id="HDimensions" %)
++**Dimensions**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-8.jpeg||data-xwiki-image-style-alignment="center"]]
  Note that for brevity only the XML of the FREQ and TIME_PERIOD are shown in full.
--===== **Notes** =====
++(% class="wikigeneratedid" id="HNotes" %)
++**Notes**
--1. A Dimension can state its position in the key, but the actual sequence is defined by the sequence in the <DimensionList> i.e. the inferred sequence in the <DimensionList> takes precedence over the value of the position attribute.
--1. This DSD is converted from version 2.0. The version 2.0 Concepts were standalone in the form  <Concept agencyID=”ECB” version=”1.0”/>
--1. The standalone concept is not supported in 2.1 and any standalone concept is placed in a Concept Scheme with the Id “STANDALONE_CONCEPT_SCHEME” of the version corresponding to the version of the original standalone concept.
++1. A [[Dimension>>doc:sdmx:Glossary.Dimension.WebHome]] can state its position in the key, but the actual sequence is defined by the sequence in the i.e. the inferred sequence in the  takes precedence over the value of the position attribute.
++1. This [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] is converted from (% style="color:#e74c3c" %)version(%%) 2.0. The (% style="color:#e74c3c" %)version(%%) 2.0 (% style="color:#e74c3c" %)Concepts(%%) were standalone in the form  <(% style="color:#e74c3c" %)Concept(%%) agencyID=”ECB” (% style="color:#e74c3c" %)version(%%)=”1.0”/>
++1. The standalone (% style="color:#e74c3c" %)concept(%%) is not supported in 2.1 and any standalone (% style="color:#e74c3c" %)concept(%%) is placed in a [[Concept Scheme>>doc:sdmx:Glossary.Concept scheme.WebHome]] with the Id “STANDALONE_CONCEPT_SCHEME” of the (% style="color:#e74c3c" %)version(%%) corresponding to the (% style="color:#e74c3c" %)version(%%) of the original standalone (% style="color:#e74c3c" %)concept(%%).
--===== **Group** =====
++(% class="wikigeneratedid" id="HGroup-1" %)
++**Group**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-9.jpeg]]** **
@@ -99,17 +99,17 @@
 . A Group in version 2.1 plays the same role as it does in version 2.0 except that it does not group Series in the data set. The Group in version 2.1 is used solely as a mechanism to attach Attributes in the data set.
 . The Group in version 2.1 is retained for compatibility with version 2.0 and for avoiding repetition in the new Attribute Relationship construct which is introduced in version 2.1.
--===== Attributes =====
++(% class="wikigeneratedid" id="HAttributes" %)
++**Attributes**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-10.jpeg||data-xwiki-image-style-alignment="center"]]
--(% style="text-align: center;" %)
  [[image:image-20250318221512-11.jpeg||data-xwiki-image-style-alignment="center"]]
  Note that for brevity only the XML of the DECIMALS (relationship to a group) COLLECTION (relationship to a set of Dimensions) and OBS_CONF (relationship to the Primary Measure) are shown in full.
--===== **Notes** =====
++(% class="wikigeneratedid" id="HNotes-1" %)
++**Notes**
 . The Attribute has a relationship to either a data set of one or more Dimensions as described in Chapter 4.
 . DECIMALS has a relationship to a Group with the Id of Group.
@@ -116,22 +116,23 @@
 . COLLECTION has a relationship to a set of Dimensions – in this case this is, in reality, the series key as this has been converted from version 2.0. It is possible for an attribute to have a relationship with just one, a few, or all of the Dimensions.
 . OBS_CONF has a relationship with the Primary Measure.
--===== **Measure** =====
++(% class="wikigeneratedid" id="HMeasure" %)
++**Measure**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-12.jpeg||data-xwiki-image-style-alignment="center"]]
  A typical database schema that can be set up using this DSD is shown below.
--===== **Series Key Table** =====
++(% class="wikigeneratedid" id="HSeriesKeyTable" %)
++**Series Key Table**
--(% style="text-align: center;" %)
  [[image:image-20250318221902-34.png||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
  **{{id name="image_50"/}}Figure 50: Relational Table Structure created from a DSD Dimensions and Related Attributes**
--===== **Group “Group” Table** =====
++(% class="wikigeneratedid" id="HGroup201CGroup201DTable" %)
++**Group “Group” Table**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-13.jpeg]]** **
@@ -139,9 +139,9 @@
  (% style="text-align: center;" %)
  **{{id name="image_51"/}}Figure 51: Relational Table Structure created from a DSD Group and Related Attributes**
--===== **Observation** =====
++(% class="wikigeneratedid" id="HObservation" %)
++**Observation**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-14.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -149,7 +149,6 @@
  === 16.4.2 Load Database ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-15.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -157,15 +157,14 @@
  The following is an extract from the SDMX dataset.
--(% style="text-align: center;" %)
  [[image:image-20250319123255-1.png||data-xwiki-image-style-alignment="center"]]
--(% style="text-align: center;" %)
  [[image:image-20250319123339-2.png||data-xwiki-image-style-alignment="center"]]
  Note that this is a sub set of the actual data message.
--===== **Notes** =====
++(% class="wikigeneratedid" id="HNotes-2" %)
++**Notes**
 . The reference to the DSD is given in the message header and given a local structureId
 . The local structureId (DSD) is referenced from the DataSet
@@ -175,17 +175,18 @@
  The diagrams below show the database content based on the schema created.
--===== **Series Key Table** =====
++(% class="wikigeneratedid" id="HSeriesKeyTable-1" %)
++**Series Key Table**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-16.jpeg||data-xwiki-image-style-alignment="center"]]
--===== **Group Table** =====
++(% class="wikigeneratedid" id="HGroupTable" %)
++**Group Table**
--(% style="text-align: center;" %)
  [[image:image-20250318221512-17.jpeg||data-xwiki-image-style-alignment="center"]]
--===== **Observation Table** =====
++(% class="wikigeneratedid" id="HObservationTable" %)
++**Observation Table**
  (% style="text-align: center;" %)
  [[image:image-20250318221512-18.jpeg]]** **
@@ -196,7 +196,6 @@
  The following registration is submitted to the Registry.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-19.jpeg||data-xwiki-image-style-alignment="center"]]
  The SDMX web service of the database can now be searched for in the Registry.
@@ -203,7 +203,6 @@
  === 16.4.4 Retrieve and Visualise Category Scheme and Dataflows ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-20.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -213,10 +213,8 @@
  [[http:~~/~~/ ~[ws-entry-point~]/categoryscheme/ECB/SDW_ECONOMIC_CONCEPTS/1.0?references=parents>>http:// [ws-entry-point]/categoryscheme/ECB/SDW_ECONOMIC_CONCEPTS/1.0?references=parents||rel="noopener noreferrer" target="_blank"]]
--(% style="text-align: center;" %)
  [[image:image-20250318221512-21.jpeg||data-xwiki-image-style-alignment="center"]]
--(% style="text-align: center;" %)
  [[image:image-20250318221512-22.jpeg||data-xwiki-image-style-alignment="center"]]
  The application will now retrieve the Dataflows in the list of Categorisations – in this case there is only one – effective exchange rates.
@@ -225,7 +225,6 @@
  [[http:~~/~~/ ~[ws-entry-point~]/dataflow/ECB/2034482/1.0>>http:// [ws-entry-point]/dataflow/ECB/2034482/1.0||rel="noopener noreferrer" target="_blank"]]
--(% style="text-align: center;" %)
  [[image:image-20250318221512-23.jpeg||data-xwiki-image-style-alignment="center"]]
  The actual choreography of the query application is dependent upon the implementation but the following is assumed for this example.
@@ -238,10 +238,10 @@
  [[http:~~/~~/~[ws-entry-point~]/dataflow/ECB/2034482/1.0?references=provisionagreement>>http://[ws-entry-point]/dataflow/ECB/2034482/1.0?references=provisionagreement||rel="noopener noreferrer" target="_blank"]]
--(% style="text-align: center;" %)
  [[image:image-20250318221512-24.jpeg||data-xwiki-image-style-alignment="center"]]
--===== **Notes** =====
++(% class="wikigeneratedid" id="HNotes-3" %)
++**Notes**
  ~1. The <URN> contains the same identification information as the individual attributes in the <Ref> tag in the context of the <Dataflow> and <Structure> tags (the <Dataflow> identifies that the <Structure> is a DataStructure).
@@ -249,23 +249,21 @@
  [[http:~~/~~/~[ws-entry-point~]/dataprovider/ECB>>http://[ws-entry-point]/dataprovider/ECB||rel="noopener noreferrer" target="_blank"]]
--===== **Notes** =====
++(% class="wikigeneratedid" id="HNotes-4" %)
++**Notes**
  ~1. As an Agency can have only one Data Provider Scheme and this has a mandatory Id of DATA_PROVIDER_SCHEME and must be version 1.0 there is no need to specify any other parameters to the REST query.
  This returns the following as part of the Data Provider Scheme.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-25.jpeg||data-xwiki-image-style-alignment="center"]]
  The following Registry query will return the REST data source referenced in a Registration for the Provision Agreement
--(% style="text-align: center;" %)
  [[image:image-20250318221512-26.jpeg||data-xwiki-image-style-alignment="center"]]
  The response from this query is:
--(% style="text-align: center;" %)
  [[image:image-20250318221512-27.jpeg||data-xwiki-image-style-alignment="center"]]
  The query application now has the details of the web service to call when making a query, and it also knows that the query expected uses the REST interface.
@@ -276,7 +276,6 @@
  The application can now visualize the information it has retrieved from the Registry in a way that is meaningful to the user. An example visualization is shown below.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-28.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -286,12 +286,10 @@
  === 16.4.5 Build Dimension Selection and Logical Query ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-29.jpeg||data-xwiki-image-style-alignment="center"]]
  Clicking on the Dataflow will cause the query application to query the Registry for the DSD associated to the Dataflow. The Dimensions and the relevant codes associated with the Dimensions are then shown for detailed data selections.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-30.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)
@@ -301,12 +301,12 @@
  === 16.4.6 Query Database ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-31.jpeg||data-xwiki-image-style-alignment="center"]]
  The logical query of the user is converted to an SDMX query and sent to the Data Web Service of the data publisher. The Data Web Service may also query the Metadata Repository in order to return both data and related metadata to the user application. The Data Web Service may also require structural metadata (DSD and MSD and related Code Lists and Concepts) from the structural metadata repository, in this example this an SDMX Registry.
--===== **Data Query Logical Selections** =====
++(% class="wikigeneratedid" id="HDataQueryLogicalSelections" %)
++**Data Query Logical Selections**
  (% style="width:551.446px" %)
  |(% style="width:261px" %)**Dimension**|(% style="width:286px" %)**Values**
@@ -317,18 +317,17 @@
  |(% style="width:261px" %)EXR_SUFFIX|(% style="width:286px" %)A
  |(% style="width:261px" %)Time|(% style="width:286px" %)Jan 2008-Dec 2010
--===== **REST Query** =====
++(% class="wikigeneratedid" id="HRESTQuery" %)
++**REST Query**
  [[http:~~/~~/ws-entry-point/data/2034482/M.Z26+Z28.EUR+CZK+DKK.DFC0+EN00+A /ECB?startPeriod=2008-01&endPeriod=2010-12&detail=dataonly>>http://ws-entry-point/data/2034482/M.Z26+Z28.EUR+CZK+DKK.DFC0+EN00+A /ECB?startPeriod=2008-01&endPeriod=2010-12&detail=dataonly||rel="noopener noreferrer" target="_blank"]]
  === 16.4.7 Visualise the Resultant Data Set ===
--(% style="text-align: center;" %)
  [[image:image-20250318221512-32.jpeg||data-xwiki-image-style-alignment="center" height="310" width="346"]]
  The returned data and metadata can be presented to the user in many ways – tables, graphs, charts, maps etc. SDMX does not support visualization directly but it is possible to define roles for a Dimension and Attribute in the DSD. For instance, it is possible to assign a role of “Geography” or “Title” or “Measure Unit” etc. which will aid a data visualization service to present this information in a meaningful way to the user.
--(% style="text-align: center;" %)
  [[image:image-20250318221512-33.jpeg||data-xwiki-image-style-alignment="center"]]
  (% style="text-align: center;" %)