1 Introduction
1.1 Scope of the User Guide
This Guide covers all of the major use cases for SDMX from simple data reporting to a completely self-updating web dissemination system of data and related reference metadata. The use cases show how best to take advantage of the SDMX Information Model in statistical systems in order to manage data and metadata reporting, data and metadata storage and retrieval, and data and metadata dissemination.
The principal intention is help organizations and individuals to determine how best to use SDMX in order to help them to improve the statistical production process. In order to achieve this objective, examples are taken from real implementation scenarios that enable the reader to understand the scope of the SDMX standards and guidelines in terms of the activities required in order to collect, process, and publish statistical data and reference metadata.
The guide is concerned principally with the reporting/collection, processing, and dissemination of aggregated statistical data. Clearly a large part of the SDMX standard is technical in nature but the audience for this guide is anyone who is involved in the processes that eventually result in the publishing of statistical data either internally within the organization or externally to interested parties. Whether the dissemination is internal or external, this is achieved more and more using web-based technologies, for which SDMX is ideally suited.
The SDMX Information Model is pivotal to implementation of SDMX and systems built with knowledge of this Information Model are easy to maintain, enable data and metadata sharing both between internal systems and with the systems used by the organizations that collect data and metadata, and enable access directly the databases and metadata repositories of data and metadata producers.
There is a growing availability of software tools, many of them free of charge or open source, that support many of the aspects of SDMX. Use of these tools or of the underlying components of the tools can reduce dramatically the cost, internal resources, and the elapsed time of the development of a new system or the integration of SDMX into an existing system.
1.2 Structure of the User Guide
Chapter Content 1. Introduction Objective, Scope, and Structure of the Guide 2. What is SDMX Background, sponsors, users, use cases, industry sectors. Brief overview of the technical and content standards, tools, where to find more information and help. 3. Scenario, Use Cases, and Example This is based on the SDMX Information Model. The chapter relates the Information Model to the real activities of reporting, processing, and dissemination of statistics. 4. Data and Metadata Creation and Reporting Explanation of the structural components of a Data Structure Definition and a Metadata Structure Definition, and of the Data Set and Metadata Set. How these are used in data and metadata reporting scenarios. 5. Data Bases and SDMX Explanation of the relationship between the tables in a database and a Data Structure Definition and how the DSD can be used to create these tables. Explanation on how to open a database to SDMX web services. 6. Data and Structure Query The scope of the data query and the map to the Information Model. Scope of the REST and SOAP queries. Useful tips on what type of queries to support. 7. Metadata Repository and Linking Data to Metadata Typical requirements for metadata (quality frameworks, linking to disseminated data).
Architecture for a metadata repository to enable data and metadata to be combined in a dissemination environment.
8. SDMX Registry Role of the Registry in statistical data and metadata reporting and dissemination systems.
Difference between the content and functions of a Registry and a non-registry based structural metadata repository.
Content and role of the Registry in terms of:- Structural metadata maintenance and query
- Registration of data and metadata sources
- SDMX Registry Services
- Web services that can make use of SDMX Registry Services
9. Architecture for an SDMX System Brings all of the components together in an overall architecture comprising: - Data and metadata persistence and interfaces
- Server side middle tier brokering of requests for data and metadata, data loading, validation, transformation
- Client side tier of:
- Structural metadata maintenance
- Data query and visualization
- Validation, transformation
10. Community Management The community may be at the level of an organization or in the context of the wider community of organizations. Topics are: - Role of a Global Registry o Maintenance agency maintenance
- Common concepts
- Community structural metadata
- Maintenance of Dimension and Attribute roles
- Hosting of a shared Registry
- Data provider maintenance