Skip to Content
Last modified by Artur on 2025/09/10 11:19
From version 2.1
edited by Helena
on 2025/05/23 17:04
Change comment: There is no comment for this version
To version 1.2
edited by Artur
on 2025/05/20 14:28
Change comment: Update document after refactoring.

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.helena
1 +xwiki:XWiki.arturkryazhev
Content
... ... @@ -4,40 +4,42 @@
4 4  
5 5  == 8.1 Scope of this Chapter ==
6 6  
7 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_4c1c6e5debb65e85.png||data-xwiki-image-style-alignment="center" height="411" width="679"]]
7 +(% style="text-align: center;" %)
8 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_4c1c6e5debb65e85.png||height="411" width="679"]]
8 8  
9 -In this guide it has been assumed that the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] (e.g. [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]], [[MSD>>doc:sdmx:Glossary.Metadata structure definition.WebHome]], [[Code List>>doc:sdmx:Glossary.Code list.WebHome]], [[Concept Scheme>>doc:sdmx:Glossary.Concept scheme.WebHome]]) is available to processing applications such as a data extract function of a database or a data visualization function of a website. For many [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] processing applications these [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] must be made available “on demand”.
10 +In this guide it has been assumed that the structural metadata (e.g. DSD, MSD, Code List, Concept Scheme) is available to processing applications such as a data extract function of a database or a data visualization function of a website. For many SDMX processing applications these structural metadata must be made available “on demand”.
10 10  
11 -This Chapter explains the role and functions of an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] and how the Registry is used both as a storage, retrieval, and maintenance repository for [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], and as a mechanism for discovering data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] including the automated “pull” scenario of an automated data reporting system .
12 +This Chapter explains the role and functions of an SDMX Registry and how the Registry is used both as a storage, retrieval, and maintenance repository for SDMX structural metadata, and as a mechanism for discovering data and reference metadata including the automated “pull” scenario of an automated data reporting system .
12 12  
13 13  == 8.2 The Need for an SDMX Registry ==
14 14  
15 -First, it should be stated that it is not necessary to operate or have access to an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] in order to use [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]. Many use cases do, however, require access to [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] but these can be retrieved from any web service that can respond to a query for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] (either the REST query or the “Structure Where” query) - this service does not need to be an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]].
16 +First, it should be stated that it is not necessary to operate or have access to an SDMX Registry in order to use SDMX. Many use cases do, however, require access to structural metadata but these can be retrieved from any web service that can respond to a query for structural metadata (either the REST query or the “Structure Where” query) - this service does not need to be an SDMX Registry.
16 16  
17 -Whilst both an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] and a non-Registry web service use exactly the same query mechanism, a Registry differs from non-Registry web service in three important areas:
18 +Whilst both an SDMX Registry and a non-Registry web service use exactly the same query mechanism, a Registry differs from non-Registry web service in three important areas:
18 18  
19 -1. The Registry offers a maintenance service for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] – this is the “Repository” function of the Registry.
20 +1. The Registry offers a maintenance service for structural metadata – this is the “Repository” function of the Registry.
20 20  1. The Registry offers a data and metadata Registration service
21 -1. The Registry offers a [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] and [[notification>>doc:sdmx:Glossary.Notification.WebHome]] service
22 +1. The Registry offers a subscription and notification service
22 22  
23 23  The functions of these services are covered in this Chapter.
24 24  
25 -Software tools that support [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] services are becoming more functional and have reached a point where it is extremely easy to install and operate an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]. This can be used to support a community of users or be used solely within a single organization in order to have a central repository for [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] which can be maintained and retrieved.
26 +Software tools that support SDMX Registry services are becoming more functional and have reached a point where it is extremely easy to install and operate an SDMX Registry. This can be used to support a community of users or be used solely within a single organization in order to have a central repository for SDMX structural metadata which can be maintained and retrieved.
26 26  
27 27  == 8.3 Objective of a Registry ==
28 28  
29 -The objective of the [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]/repository is, in broad terms, to allow organisations to publish statistical data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] in known formats such that interested third parties can discover these data and interpret them accurately and correctly. The mechanism for doing this is twofold:
30 +The objective of the SDMX registry/repository is, in broad terms, to allow organisations to publish statistical data and reference metadata in known formats such that interested third parties can discover these data and interpret them accurately and correctly. The mechanism for doing this is twofold:
30 30  
31 -1. To maintain and publish [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] that describes the structure and valid content of data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sources such as databases, [[metadata repositories>>doc:sdmx:Glossary.Metadata repository.WebHome]], [[data sets>>doc:sdmx:Glossary.Data set.WebHome]], [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]]. This [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] enables software applications to understand and to interpret the data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] in these sources.
32 -1. To enable applications, organisations, and individuals to share and to discover data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]]. This facilitates data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] dissemination by implementing the data sharing vision of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]].
32 +1. To maintain and publish structural metadata that describes the structure and valid content of data and reference metadata sources such as databases, metadata repositories, data sets, metadata sets. This structural metadata enables software applications to understand and to interpret the data and reference metadata in these sources.
33 +1. To enable applications, organisations, and individuals to share and to discover data and reference metadata. This facilitates data and reference metadata dissemination by implementing the data sharing vision of SDMX.
33 33  
34 34  == 8.4 SDMX Registry/Repository Architecture ==
35 35  
36 36  === 8.4.1 Architectural Schematic ===
37 37  
38 -The architecture of the [[SDMX registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]/repository is a layered architecture that is founded by a [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] repository which supports a provisioning [[metadata repository>>doc:sdmx:Glossary.Metadata repository.WebHome]] which supports the registry services. These are all supported by the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] schemas. Applications can be built on top of these services which support the reporting, storage, retrieval, and dissemination aspects of the statistical lifecycle as well as the maintenance of the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] required to drive these applications.
39 +The architecture of the SDMX registry/repository is a layered architecture that is founded by a structural metadata repository which supports a provisioning metadata repository which supports the registry services. These are all supported by the SDMX-ML schemas. Applications can be built on top of these services which support the reporting, storage, retrieval, and dissemination aspects of the statistical lifecycle as well as the maintenance of the structural metadata required to drive these applications.
39 39  
40 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_811113a255dfd7e.jpg||data-xwiki-image-style-alignment="center" height="325" width="554"]]
41 +(% style="text-align: center;" %)
42 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_811113a255dfd7e.jpg||height="325" width="554"]]
41 41  
42 42  (% style="text-align: center;" %)
43 43  **{{id name="image_19"/}}Figure 19: Schematic of the Registry Content and Services**
... ... @@ -44,18 +44,18 @@
44 44  
45 45  === 8.4.2 Structural Metadata Repository ===
46 46  
47 -The basic layer is that of a [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] service which supports the lifecycle of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] such as [[Maintenance Agencies>>doc:sdmx:Glossary.Maintenance agency.WebHome]], [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]], [[Metadata Structure Definitions>>doc:sdmx:Glossary.Metadata structure definition.WebHome]], [[Provision Agreements>>doc:sdmx:Glossary.Provision agreement.WebHome]], Processes etc. This layer is supported by the Structure Maintenance and Query Service.
49 +The basic layer is that of a structural metadata service which supports the lifecycle of SDMX structural metadata artefacts such as Maintenance Agencies, Data Structure Definitions, Metadata Structure Definitions, Provision Agreements, Processes etc. This layer is supported by the Structure Maintenance and Query Service.
48 48  
49 -Note that the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Submit Structure Request message supports all of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] structural [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]]. The only Registry [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]] that are not supported by the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Submit Structure Request are:
51 +Note that the SDMX-ML Submit Structure Request message supports all of the SDMX structural artefacts. The only Registry artefacts that are not supported by the SDMX-ML Submit Structure Request are:
50 50  
51 51  * Registration of data and metadata sources
52 -* [[Subscription>>doc:sdmx:Glossary.Subscription.WebHome]] and [[Notification>>doc:sdmx:Glossary.Notification.WebHome]]
54 +* Subscription and Notification
53 53  
54 -Separate registry-based messages are defined to support these [[artefacts>>doc:sdmx:Glossary.Artefact.WebHome]].
56 +Separate registry-based messages are defined to support these artefacts.
55 55  
56 56  === 8.4.3 Provisioning Metadata Repository ===
57 57  
58 -The function of this repository is to support the registration of various types of data-store which model [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant databases or files, and to link to these data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] sources. These links can be specified for a [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]], for a specific data or [[metadata flow>>doc:sdmx:Glossary.Metadataflow.WebHome]]. In the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] model this is called the [[Provision Agreement>>doc:sdmx:Glossary.Provision agreement.WebHome]].
60 +The function of this repository is to support the registration of various types of data-store which model SDMX-conformant databases or files, and to link to these data and reference metadata sources. These links can be specified for a data provider, for a specific data or metadata flow. In the SDMX model this is called the Provision Agreement.
59 59  
60 60  This layer is supported by the Data and Metadata Registration Service.
61 61  
... ... @@ -63,45 +63,48 @@
63 63  
64 64  === 8.5.1 Structure Maintenance and Query Service ===
65 65  
66 -A Registry offers a web service for the maintenance of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. Whilst this service must comply with the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Registry Interface, it is common for such registries to also provide a graphical user interface for the submission and maintenance of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] structures.
68 +A Registry offers a web service for the maintenance of structural metadata. Whilst this service must comply with the SDMX-ML Registry Interface, it is common for such registries to also provide a graphical user interface for the submission and maintenance of SDMX structures.
67 67  
68 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_1441cca2307b86ac.jpg||data-xwiki-image-style-alignment="center" height="279" width="576"]]
70 +(% style="text-align: center;" %)
71 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_1441cca2307b86ac.jpg||height="279" width="576"]]
69 69  
70 70  (% style="text-align: center;" %)
71 71  **{{id name="image_20"/}}Figure 20: Example Graphical User Interface to an SDMX Registry **
72 72  
73 -The Registry must also offer a query service for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]].
76 +The Registry must also offer a query service for structural metadata.
74 74  
75 75  === 8.5.2 Registration Service ===
76 76  
77 -The Registration service allows a data or metadata publisher (in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] terminology this is a [[Data Provider>>doc:sdmx:Glossary.Data provider.WebHome]]) to publish that data or metadata exist. As the Registration is connected to a [[Provision Agreement>>doc:sdmx:Glossary.Provision agreement.WebHome]] which itself is connected to a [[Data Provider>>doc:sdmx:Glossary.Data provider.WebHome]] and Data or [[Metadata Flow>>doc:sdmx:Glossary.Metadataflow.WebHome]], the exact type of data or metadata registered is known, and also its structure (as the Data or [[Metadata Flow>>doc:sdmx:Glossary.Metadataflow.WebHome]] references the [[DSD>>doc:sdmx:Glossary.Data structure definition.WebHome]] or [[MSD>>doc:sdmx:Glossary.Metadata structure definition.WebHome]]).
80 +The Registration service allows a data or metadata publisher (in SDMX terminology this is a Data Provider) to publish that data or metadata exist. As the Registration is connected to a Provision Agreement which itself is connected to a Data Provider and Data or Metadata Flow, the exact type of data or metadata registered is known, and also its structure (as the Data or Metadata Flow references the DSD or MSD).
78 78  
79 -The Registration must include a URL of either a file of [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] containing a data or [[metadata set>>doc:sdmx:Glossary.Metadata set.WebHome]], or a URL of a web service which can be queried to obtain the data.
82 +The Registration must include a URL of either a file of SDMX-ML containing a data or metadata set, or a URL of a web service which can be queried to obtain the data.
80 80  
81 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_541961e6279194a1.jpg||data-xwiki-image-style-alignment="center" height="446" width="451"]]
84 +(% style="text-align: center;" %)
85 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_541961e6279194a1.jpg||height="446" width="451"]]
82 82  
83 83  (% style="text-align: center;" %)
84 84  **{{id name="image_21"/}}Figure 21: Schematic of Registered Data and Metadata Sources in the SDMX-IM**
85 85  
86 -The Registration can be used both for [[data consumers>>doc:sdmx:Glossary.Data consumer.WebHome]] to find data (this is described below in “Data and Metadata Discovery”) and for automated data reporting systems to harvest the data (the “pull” scenario).
90 +The Registration can be used both for data consumers to find data (this is described below in “Data and Metadata Discovery”) and for automated data reporting systems to harvest the data (the “pull” scenario).
87 87  
88 88  === 8.5.3 Subscription and Notification Service ===
89 89  
90 -This [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] service allows a user to monitor Registry events. An event is something that has changed in the Registry. The [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] can be specified for specific objects such as a new registration of data or a change to a [[code list>>doc:sdmx:Glossary.Code list.WebHome]], or it can be specified for many objects such as any change to a specific type of structure or all registrations. A [[subscription>>doc:sdmx:Glossary.Subscription.WebHome]] can ask for [[notification>>doc:sdmx:Glossary.Notification.WebHome]] by e-mail or [[notification>>doc:sdmx:Glossary.Notification.WebHome]] to a URL (this will be a web service that can act upon the [[notification>>doc:sdmx:Glossary.Notification.WebHome]] such as the automated update of [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] supporting a web dissemination service or the “pull” mechanism for data reporting).
94 +This subscription service allows a user to monitor Registry events. An event is something that has changed in the Registry. The subscription can be specified for specific objects such as a new registration of data or a change to a code list, or it can be specified for many objects such as any change to a specific type of structure or all registrations. A subscription can ask for notification by e-mail or notification to a URL (this will be a web service that can act upon the notification such as the automated update of structural metadata supporting a web dissemination service or the “pull” mechanism for data reporting).
91 91  
92 -When a monitored event is triggered the [[notification>>doc:sdmx:Glossary.Notification.WebHome]] service notifies interested parties of the Registry content that has changed. The [[notification>>doc:sdmx:Glossary.Notification.WebHome]] can be user (email) or application (URL).
96 +When a monitored event is triggered the notification service notifies interested parties of the Registry content that has changed. The notification can be user (email) or application (URL).
93 93  
94 94  == 8.6 Data and Metadata Discovery ==
95 95  
96 96  === 8.6.1 Choreography ===
97 97  
98 -Discovering published data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] involves interaction with the Registry to:
102 +Discovering published data and reference metadata involves interaction with the Registry to:
99 99  
100 -* optionally (but usually) browsing a subject matter domain [[category scheme>>doc:sdmx:Glossary.Category scheme.WebHome]] to find [[Dataflow>>doc:sdmx:Glossary.Dataflow.WebHome]] Definitions (and hence [[Data Structure Definitions>>doc:sdmx:Glossary.Data structure definition.WebHome]]) and [[Metadataflows>>doc:sdmx:Glossary.Metadataflow.WebHome]] (and hence [[Metadata Structure Definitions>>doc:sdmx:Glossary.Metadata structure definition.WebHome]]) which structure the type of data and/or [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] being sought
101 -* build a query, in terms of the selected [[Data Structure Definition>>doc:sdmx:Glossary.Data structure definition.WebHome]] or [[Metadata Structure Definition>>doc:sdmx:Glossary.Metadata structure definition.WebHome]], which specifies what data are required and submitting this to a service that can query an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] which will return a list of (URLs of) data and [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] files and databases which satisfy the query
102 -* processing the query result set and retrieving data and/or [[reference metadata>>doc:sdmx:Glossary.Reference metadata.WebHome]] from the supplied URLs
104 +* optionally (but usually) browsing a subject matter domain category scheme to find Dataflow Definitions (and hence Data Structure Definitions) and Metadataflows (and hence Metadata Structure Definitions) which structure the type of data and/or reference metadata being sought
105 +* build a query, in terms of the selected Data Structure Definition or Metadata Structure Definition, which specifies what data are required and submitting this to a service that can query an SDMX Registry which will return a list of (URLs of) data and reference metadata files and databases which satisfy the query
106 +* processing the query result set and retrieving data and/or reference metadata from the supplied URLs
103 103  
104 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_51f6d84398cd8ba3.jpg||data-xwiki-image-style-alignment="center" height="453" width="510"]]
108 +(% style="text-align: center;" %)
109 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_51f6d84398cd8ba3.jpg||height="453" width="510"]]
105 105  
106 106  (% style="text-align: center;" %)
107 107  **{{id name="image_22"/}}Figure 22: Schematic of Data and Metadata Discovery and Query in the SDMX-IM**
... ... @@ -112,11 +112,13 @@
112 112  
113 113  == 8.7 Patterns for Data and Metadata Exchange ==
114 114  
115 -[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifies three basic process patterns and two modes (push and pull) regarding the exchange of statistical data and metadata.
120 +SDMX identifies three basic process patterns and two modes (push and pull) regarding the exchange of statistical data and metadata.
116 116  
117 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_b247734721100020.png||data-xwiki-image-style-alignment="center" height="547" width="680"]]
122 +(% style="text-align: center;" %)
123 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_b247734721100020.png||height="547" width="680"]]
118 118  
119 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_2a1b6d63f93d6579.png||data-xwiki-image-style-alignment="center" height="300" width="680"]]
125 +(% style="text-align: center;" %)
126 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_2a1b6d63f93d6579.png||height="300" width="680"]]
120 120  
121 121  (% style="text-align: center;" %)
122 122  **{{id name="image_23"/}}Figure 23: The Three Basic Patterns of Data and Reference Metadata Exchange**
... ... @@ -123,38 +123,39 @@
123 123  
124 124  In the //push// mode the data/metadata are sent by the reporter to the collector.
125 125  
126 -In the //pull// mode the [[data consumer>>doc:sdmx:Glossary.Data consumer.WebHome]] retrieves the data from the reporter's web server. The data may be made available for download in an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant file, or they may be retrieved from a database in response to an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-conformant query, via a web service running on the reporter's server. In both cases, the data are made available to any organisation requiring them, in formats which ensure that data are consistently described by appropriate metadata, whose meaning is common to all parties in the exchange.
133 +In the //pull// mode the data consumer retrieves the data from the reporter's web server. The data may be made available for download in an SDMX-conformant file, or they may be retrieved from a database in response to an SDMX-conformant query, via a web service running on the reporter's server. In both cases, the data are made available to any organisation requiring them, in formats which ensure that data are consistently described by appropriate metadata, whose meaning is common to all parties in the exchange.
127 127  
128 -Data sharing using the pull mode is well adapted to the database-driven and data [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] architectures. Both architectures provide the best benefits for the data producers because they can lessen the burden of publishing the data to multiple counterparties.
135 +Data sharing using the pull mode is well adapted to the database-driven and data hub architectures. Both architectures provide the best benefits for the data producers because they can lessen the burden of publishing the data to multiple counterparties.
129 129  
130 -In both architectures, it is necessary to implement a [[notification>>doc:sdmx:Glossary.Notification.WebHome]] mechanism, providing provisioning metadata in order to alert collecting organisations that data and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] are made available by [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]], details about the online mechanism for getting data (for example, a queryable online database or a simple URL) and (% style="color:#e74c3c" %)constraints(%%) regarding the allowable content of the [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] that will be provided.
137 +In both architectures, it is necessary to implement a notification mechanism, providing provisioning metadata in order to alert collecting organisations that data and metadata sets are made available by data providers, details about the online mechanism for getting data (for example, a queryable online database or a simple URL) and constraints regarding the allowable content of the data sets that will be provided.
131 131  
132 -At the heart of a data-sharing architecture there is often an [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]]. This is a central location where structural and provisioning metadata can be found. In fact all the users/applications that need to access data can query the registry in order to know what [[data sets>>doc:sdmx:Glossary.Data set.WebHome]] and [[metadata sets>>doc:sdmx:Glossary.Metadata set.WebHome]] are available from [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]], and how to access them.
139 +At the heart of a data-sharing architecture there is often an SDMX Registry. This is a central location where structural and provisioning metadata can be found. In fact all the users/applications that need to access data can query the registry in order to know what data sets and metadata sets are available from data providers, and how to access them.
133 133  
134 134  === 8.7.1 The Database-driven Architecture ===
135 135  
136 -The database-driven architecture is implemented by those collecting organisations that periodically need to fetch the data and to load them in their database. In general a batch process is used in order to automate the flow in which a whole or a partial [[dataset>>doc:sdmx:Glossary.Data set.WebHome]], including incremental updating, is used.
143 +The database-driven architecture is implemented by those collecting organisations that periodically need to fetch the data and to load them in their database. In general a batch process is used in order to automate the flow in which a whole or a partial dataset, including incremental updating, is used.
137 137  
138 138  From the data management point of view, the pull approach within a database-driven architecture includes the following steps:
139 139  
140 -1) when new data are available, the [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]] should:
147 +1) when new data are available, the data provider should:
141 141  
142 -a) create an [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] file containing the new data set
149 +a) create an SDMX-ML file containing the new data set
143 143  
144 144  __**or**__
145 145  
146 -b) provide a web service (WS) that builds [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] messages upon request.
153 +b) provide a web service (WS) that builds SDMX-ML messages upon request.
147 147  
148 -In both cases a [[provision agreement>>doc:sdmx:Glossary.Provision agreement.WebHome]] must be in place.
155 +In both cases a provision agreement must be in place.
149 149  
150 -2) the data collector Pull Requestor is notified of the new registration. Note that it is common for a Registry to provide an RSS mechanism, though this is nit a formal part of the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] standard. The data collector then:
157 +2) the data collector Pull Requestor is notified of the new registration. Note that it is common for a Registry to provide an RSS mechanism, though this is nit a formal part of the SDMX standard. The data collector then:
151 151  
152 -a) retrieves the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] file from the specified URL, if it resides in a URL,
159 +a) retrieves the SDMX-ML file from the specified URL, if it resides in a URL,
153 153  __**or**__
154 154  
155 -b) uses the Query Message or REST Query included in the feed to query the [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]] web service, if the data are prepared by the [[data provider>>doc:sdmx:Glossary.Data provider.WebHome]] web service.
162 +b) uses the Query Message or REST Query included in the feed to query the data provider web service, if the data are prepared by the data provider web service.
156 156  
157 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_ba1159b7bd2e98c4.png||data-xwiki-image-style-alignment="center" height="410" width="469"]]
164 +(% style="text-align: center;" %)
165 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_ba1159b7bd2e98c4.png||height="410" width="469"]]
158 158  
159 159  (% style="text-align: center;" %)
160 160  **{{id name="image_24"/}}Figure 24: Database-driven Architecture**
... ... @@ -161,28 +161,29 @@
161 161  
162 162  === 8.7.2 The Data Hub Architecture ===
163 163  
164 -The data [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] architecture consists of an accessible system providing involved actors with the following services:
172 +The data hub architecture consists of an accessible system providing involved actors with the following services:
165 165  
166 -* [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]] can:
174 +* data providers can:
167 167  
168 -*
169 -** notify the [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] of new sets of data and corresponding [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] ([[measures>>doc:sdmx:Glossary.Measure.WebHome]], [[dimension>>doc:sdmx:Glossary.Dimension.WebHome]], [[code lists>>doc:sdmx:Glossary.Code list.WebHome]], etc.);
176 +*
177 +** notify the hub of new sets of data and corresponding structural metadata (measures, dimension, code lists, etc.);
170 170  ** make data available directly from their systems through a querying system.
171 171  
172 172  * data users can:
173 173  
174 -*
175 -** browse the [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] to define a [[dataset>>doc:sdmx:Glossary.Data set.WebHome]] of interest via the above [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]];
176 -** retrieve the [[dataset>>doc:sdmx:Glossary.Data set.WebHome]] from the [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]].
182 +*
183 +** browse the hub to define a dataset of interest via the above structural metadata;
184 +** retrieve the dataset from the data providers.
177 177  
178 -From the data management point of view, the [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] is also based on agreed hypercubes or [[datasets>>doc:sdmx:Glossary.Data set.WebHome]], but here the hypercubes or [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] are not sent to the central system. Instead the following process operates:
186 +From the data management point of view, the hub is also based on agreed hypercubes or datasets, but here the hypercubes or datasets are not sent to the central system. Instead the following process operates:
179 179  
180 -1. a user identifies a [[dataset>>doc:sdmx:Glossary.Data set.WebHome]] through the web interface of the central [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] using the [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]], and requests it;
181 -1. the central [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] translates the user request in one or more queries and sends them to the related [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]]’ systems;
182 -1. [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]]’ systems process the query and send the result to the central [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] in a standard format;
183 -1. the central [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] puts together all the results originated by all interested [[data providers>>doc:sdmx:Glossary.Data provider.WebHome]]’ systems and presents them in a human readable format.
188 +1. a user identifies a dataset through the web interface of the central hub using the structural metadata, and requests it;
189 +1. the central hub translates the user request in one or more queries and sends them to the related data providers’ systems;
190 +1. data providers’ systems process the query and send the result to the central hub in a standard format;
191 +1. the central hub puts together all the results originated by all interested data providers’ systems and presents them in a human readable format.
184 184  
185 -[[image:SDMX_2-1_User_Guide_draft_0-1_html_1c9984b4c53f1c50.png||data-xwiki-image-style-alignment="center" height="335" width="383"]]
193 +(% style="text-align: center;" %)
194 +[[image:SDMX_2-1_User_Guide_draft_0-1_html_1c9984b4c53f1c50.png||height="335" width="383"]]
186 186  
187 187  (% style="text-align: center;" %)
188 188  **{{id name="image_25"/}}Figure 25: Data Hub Architecture**
... ... @@ -189,24 +189,24 @@
189 189  
190 190  === 8.7.3 Data Producer Architectures ===
191 191  
192 -In order to implement an [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] IT architecture for data-sharing using the pull mode, several steps must be accomplished by a data producer and several questions must be considered:
201 +In order to implement an SDMX IT architecture for data-sharing using the pull mode, several steps must be accomplished by a data producer and several questions must be considered:
193 193  
194 -1. which [[statistical domains>>doc:sdmx:Glossary.Statistical subject-matter domain.WebHome]] are involved and where are the data currently stored?
195 -1. which [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] are involved, and where are they currently stored?
196 -1. what is the business process behind the [[data flow>>doc:sdmx:Glossary.Dataflow.WebHome]] involved in the exercise?
197 -1. will the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] data producer architecture be part of a data warehouse architecture, of a data [[hub>>doc:sdmx:Glossary.Hub (dissemination architecture).WebHome]] architecture or of both?
203 +1. which statistical domains are involved and where are the data currently stored?
204 +1. which structural metadata are involved, and where are they currently stored?
205 +1. what is the business process behind the data flow involved in the exercise?
206 +1. will the SDMX data producer architecture be part of a data warehouse architecture, of a data hub architecture or of both?
198 198  
199 -Generally data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] that will be involved in the new [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] information system are stored either in databases or in files. The two cases lead to different architectural approaches:
208 +Generally data and structural metadata that will be involved in the new SDMX information system are stored either in databases or in files. The two cases lead to different architectural approaches:
200 200  
201 -* a. data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] are already stored in a database and it is necessary to build suitable software interfaces in order to make the system “[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-compliant”.
202 -* b. a separate special-purpose database is set up to store data and [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]]. This database will be designed with the main aim of being part of an SDMXcompliant system. In this case the database can be modelled using the [[SDMX Information Model>>doc:sdmx:Glossary.SDMX Information Model.WebHome]]. An [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] is, of course, such a database.
210 +* a. data and structural metadata are already stored in a database and it is necessary to build suitable software interfaces in order to make the system “SDMX-compliant”.
211 +* b. a separate special-purpose database is set up to store data and structural metadata. This database will be designed with the main aim of being part of an SDMXcompliant system. In this case the database can be modelled using the SDMX Information Model. An SDMX Registry is, of course, such a database.
203 203  
204 204  Both cases make it possible to:
205 205  
206 -* extract [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] files from the database that will be made available to be pulled by data collectors;
215 +* extract SDMX-ML files from the database that will be made available to be pulled by data collectors;
207 207  * allow the database to be queried directly through a web service.
208 208  
209 -Whichever type of data producer architecture is involved, a mapping process between [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]] may be necessary, as explained in 5.3.
218 +Whichever type of data producer architecture is involved, a mapping process between structural metadata may be necessary, as explained in 5.3.
210 210  
211 211  == 8.8 Registry Interfaces ==
212 212  
... ... @@ -215,26 +215,26 @@
215 215  The Registry Interfaces are:
216 216  
217 217  * Notify Registry Event
218 -* Submit [[Subscription>>doc:sdmx:Glossary.Subscription.WebHome]] Request
219 -* Submit [[Subscription>>doc:sdmx:Glossary.Subscription.WebHome]] Response
227 +* Submit Subscription Request
228 +* Submit Subscription Response
220 220  * Submit Registration Request
221 221  * Submit Registration Response
222 222  * Query Registration Request
223 223  * Query Registration Response
224 -* Query [[Subscription>>doc:sdmx:Glossary.Subscription.WebHome]] Request
225 -* Query [[Subscription>>doc:sdmx:Glossary.Subscription.WebHome]] Response
233 +* Query Subscription Request
234 +* Query Subscription Response
226 226  * Submit Structure Request
227 227  * Submit Structure Response
228 228  
229 -Applications communicate with the Registry using either the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] Registry Interface message (which contains an XML element that defines the actual interface) or individual messages – one for each interface.
238 +Applications communicate with the Registry using either the SDMX-ML Registry Interface message (which contains an XML element that defines the actual interface) or individual messages – one for each interface.
230 230  
231 231  In more technical terms the Registry interfaces are invoked in one of two ways:
232 232  
233 -1. The interface is the name of the root node of the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] document.
234 -1. The interface is invoked as a child element of the RegistryInterface message where the RegistryInterface is the root node of the [[SDMX-ML>>doc:sdmx:Glossary.SDMX-ML.WebHome]] document.
242 +1. The interface is the name of the root node of the SDMX-ML document.
243 +1. The interface is invoked as a child element of the RegistryInterface message where the RegistryInterface is the root node of the SDMX-ML document.
235 235  
236 -In addition to these interfaces the Registry must support a mechanism for querying for [[structural metadata>>doc:sdmx:Glossary.Structural metadata.WebHome]].
245 +In addition to these interfaces the Registry must support a mechanism for querying for structural metadata.
237 237  
238 -All these interactions with the Registry – with the exception of Notify Registry Event – are designed in pairs. The first document (the one which invokes the [[SDMX Registry>>doc:sdmx:Glossary.SDMX Registry.WebHome]] Interface) is a “Request” document. The message returned by the interface is a “Response” document.
247 +All these interactions with the Registry – with the exception of Notify Registry Event – are designed in pairs. The first document (the one which invokes the SDMX Registry Interface) is a “Request” document. The message returned by the interface is a “Response” document.
239 239  
240 -It should be noted that all interactions are assumed to be synchronous, with the exception of Notify Registry Event. This document is sent by the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]]-Registry to all subscribers whenever an even occurs to which any users have subscribed. Thus, it does not conform to the request-response pattern, because it is inherently asynchronous.
249 +It should be noted that all interactions are assumed to be synchronous, with the exception of Notify Registry Event. This document is sent by the SDMX-Registry to all subscribers whenever an even occurs to which any users have subscribed. Thus, it does not conform to the request-response pattern, because it is inherently asynchronous.