10 Community Management
- Contents
10.1 Scope of this Chapter
Many organizations have a community of data reporters, data sharing agencies, or data consumers. This places responsibilities upon the organization in terms of publishing and dissemination of structural metadata, setting up hubs for data sharing, administration of maintenance agencies, usernames etc.
These aspects are dealt with in this Chapter.
10.2 Maintenance Agency Maintenance
All structural metadata in SDMX is owned and maintained by a maintenance agency (Agency identified by agencyID in the schemas). It is vital to the integrity of the structural metadata that there are no conflicts in agencyID.
The maintenance of maintenance agencies in SDMX is a devolved function. Any organization registered as a maintenance agency can itself set up and maintain its own maintenance agencies. However, in order for this devolved system to work there must be a “top-level” maintenance agency list (called an Agency Scheme in SDMX) which is itself maintained by the recognized top-level Agency. This Agency is SDMX. Any organization registered in the SDMX Agency Scheme can itself maintain its own Agency Scheme of sub-agencies. Only one such Agency Scheme is allowed for any recognized Agency. With the exception of the “SDMX” Agency, a recognized Agency must itself be registered in a “parent” Agency Scheme.
The following rules are a more formal definition of the way the SDMX agency system works:
- Agencies are maintained in an Agency Scheme.
- The maintenance agency of the Agency Scheme must be registered as an Agency in a (different) Agency Scheme (the “parent” Agency Scheme).
- The “top-level” agency is SDMX and this agency scheme is maintained by SDMX.
- Agencies registered in the top-level scheme can themselves maintain a single Agency Scheme. SDMX is an agency in the SDMX agency scheme, thus allowing SDMX be a maintenance agency. Agencies in this (child) scheme can themselves maintain a single Agency Scheme and so on.
- The Agency Scheme cannot be versioned and so takes a default version number of 1.0, and it cannot be made “final”.
- There can be only one Agency Scheme maintained by any one Agency. It has a fixed Id of AGENCIES.
- The format of the agency identifier is agencyId.agencyID etc. The top-level agency in this identification mechanism is the agency registered in the SDMX agency scheme. In other words, SDMX is not a part of the hierarchical ID structure for agencies.
- SDMX is, itself, a maintenance agency.
This supports a hierarchical structure of agencyID.
An example is shown below.
Figure 29:Example of Hierarchic Structure of Agencies
Each agency is identified by its full hierarchy excluding SDMX.
The XML representing this structure is shown below.
Figure 30: Example Agency Schemes Showing a Hierarchy
Example of Structure Definitions:
Figure 31: Example Showing Use of Agency Identifiers
Each of these maintenance agencies has an identical Codelist with the Id CL_BOP. However, each is uniquely identified by means of the hierarchic agency structure.
Clearly, in order for such a system to work there must be a mechanism that enables a user or organisation to discover the full “list” of maintenance agencies or at least the Agency Scheme in which an organisation is registered. In order for this to be possible all Agency Schemes must be made known to a Global SDMX Registry and either maintained in that Registry or maintained and made available from a metadata source that is referenced from an entry in the Global Registry. For example, the Agency Schemes shown in Figure 30 could be “registered” in the Global Registry as follows:
Figure 32: Example XML Showing External References
10.3 Dissemination of Structural Metadata
When structural metadata are disseminated or exchanged it is important that any consuming application has access to all of the structural metadata that is referenced from structures such as a DSD or MSD (i.e. in these cases the Concept Schemes, Code Lists, Category Schemes etc. that are “used” in the DSD or the MSD). These structures can be embedded as complete structures in the file that is disseminated, or they can be embedded as “stubs” that contain the reference from where the structures can be retrieved. This referencing mechanism is achieved using one of the SDMX-ML attributes:
- structureURL
- serviceURL
These attributes are available on every structure that is maintained e.g. it is at the level of the “maintained object” such as Code List, Data Structure Definition. In addition to the use of these attributes the attribute isExternalReference should be set to “true” indicating that the full definition of the structure is not available and must be retrieved from one of the reference attributes structureURL or serviceURL.
It is therefore the responsibility of the agency maintaining these structures that the structures can be accessed using the URL. If the URL cannot be used to retrieve the structure then it is possible that applications using the structures will not be able to process the information properly. It follows that the content of structureURL and serviceURL should be deemed to be resolvable in the long term.
The use of or a link to a shared community SDMX Registry is recommended for organizations that wish to store such structures. For example, the shared community registry can be used as the repository for the referenced structures.
Note that any structural metadata submitted to an SDMX Registry (SubmitStructureRequest) must have resolvable references to all of the structures cross referenced in the submitted structure. If this is not the case then the submission may be rejected.
10.4 Maintenance of Community Concept Roles
10.4.1 Overview
The DSD Components of Dimension and Attribute can play a specific role in the DSD and it is important to some applications that this role is specified. For instance, the following roles are some examples:
Frequency – in a data set the content of this Component contains information on the frequency of the observation values
Geography - in a data set the content of this Component contains information on the geographic location of the observation values
Unit of Measure - in a data set the content of this Component contains information on the unit of measure of the observation values
In order for these roles to be extensible and also to enable user communities to maintain community-specific roles, the roles are maintained in a controlled vocabulary which is implemented in SDMX as Concepts in a Concept Scheme.
It is possible to specify zero or more concept roles for a Dimension, Measure Dimension and Data Attribute (but not the ReportingYearStartDay). The Time Dimension, Primary Measure, and the Attribute ReportingYearStartDay have explicitly defined roles and cannot be further specified with additional concept roles.
10.4.2 Maintaining and Using Concept Roles
The mechanism for maintaining and using concept roles is described in the SDMX Standards Section 6: Technical Notes. It is the responsibility of Agencies to ensure their community knows which concepts in which concept schemes play a “role” and the significance and interpretation of this role. In other words, such concepts must be known by applications, there is no technical mechanism that can inform an application on how to process such a “role”.
Clearly, Agencies defining DSDs can use Concepts from any Concept Scheme, even a Concept Scheme not maintained by this Agency. It is therefore important that any Concept Scheme that is referenced from the DSD must be available from a known URL to any application that needs to process the DSD, or be embedded in the file of structural metadata (this is no different from any other cross referenced structure as described in 10.3 above). Therefore, if an Agency decides to maintain its own Concept Scheme of concept roles it needs to consider whether this scheme is “public” (i.e. it is required in the DSDs disseminated by the organisation) and, if so, ensure that it is made available.
10.5 Hosting of a shared registry
It is clear from the responsibilities of an Agency that there is a need to ensure structural metadata maintained by that Agency, and which is to be used outside of the organization that is the Agency, is made available. As more and more organizations use SDMX and as more Agencies are created, it is clear that a central repository of structural metadata will benefit the “community” of the Agency. Many of these communities will exist already, as they will pre-date the existence of. The SDMX standard formalizes the way a community exchanges and shares data and reference metadata, and as more SDMX communities are created then the greater is the need for control over the maintenance and sharing of the structural metadata.
Community “Agencies” should consider taking on the role of hosting a shared SDMX Registry for the community. Clearly, there will be many of these shared registries and it may be necessary that these registries can be linked in a “federation” so that common structural metadata is accessible. This is especially true of the Agency Schemes that will exist: it will only be possible to validate an agencyID if the Agency is known and this may involve knowledge of the full hierarchy of Agencies which will be need to be built from the Agencies maintained in separate Agency Schemes. Furthermore, it is probable that many Code Lists and Concept Schemes will not be disseminated with DSDs and MSDs, but rather be referenced from the structure file disseminated. This type of federated architecture is best supported by an SDMX Registry, as this is the one of the key roles of an SDMX Registry.
10.6 Data Provider Maintenance
An organization that collects data or reference metadata from other organizations (these are known as Data Providers in SDMX) must maintain a DataProvider Scheme. This is self evident as the collecting organization must know from which reporting organization data or reference metadata is received. Clearly, an organization can collect data or reference metadata in an SDMX format without having an SDMX DataProvider Scheme. Nevertheless, data collecting organizations which adopt SDMX are encouraged to maintain an SDMX DataProvider Scheme, or to be able to disseminate such a scheme from their own internal scheme.
If the collecting organization wishes to adopt the “pull” method of data/metadata reporting, or wishes to host an SDMX Registry where its community can publish the existence of data and reference metadata sources (by means of a data/metadata Registration), then it is mandatory to have a DataProvider Scheme. The relationship between the fundamental structures supporting the “pull” mechanism and data/metadata discovery is shown in the diagram below.
Figure 33: SDMX structures required for the “pull” reporting mechanism or for data/metadata discovery