Changes for page Guidelines for SDMX Data Structure Definitions

Summary

• Page properties (1 modified, 0 added, 0 removed)
• Attachments (0 modified, 7 added, 0 removed)
  • 1768470533088-795.png
  • 1768470575978-226.png
  • 1768470596130-305.png
  • 1768470611326-907.png
  • 1768470626558-321.png
  • 1768470646456-652.png
  • 1768470678965-391.png

Details

Page properties

Content

...	...	@@ -296,15 +296,14 @@
296	296
297	297	**Table 6. Data structuring approaches by role in data exchange**
298	298
299		-|**Role in data exchange**|**Pure vs. composite concepts approach**
300		-|**Data provider**|(((
	299	+|(% style="width:215px" %)**Role in data exchange**|(% style="width:1400px" %)**Pure vs. composite concepts approach**
	300	+|(% style="width:215px" %)**Data provider**|(% style="width:1400px" %)(((
301	301	If the composition of the concepts in the data provider's production system largely differs from the one in the DSD, mapping it to a few composite concepts may be more complex than mapping it to many pure concepts. (Mapping to just one mixed concept is straightforward, though.) This is due to the need to decompose and recombine concepts in case of a “mixed concepts” DSD. If the data provider’s internal data structure is very granular or very similar to the DSD, it does not make a huge difference if the concepts in that DSD are pure or not.
302		-
303	303	For a “final” data provider disseminating data to the public, the flexibility offered by a pure data structure in terms of defining different output formats may be beneficial.
304	304	)))
305		-|**Data collector**|Defining constraints for data validation is more complex for a highdimensional, pure DSD. But such a DSD provides more flexibility in terms of consumption and reuse, i.e. mapping to the data collector’s internal data model mapping easier.
306		-|**DSD maintenance**|Pure concepts usually have shorter, less complex code lists and are thus easier to maintain. In contrast, the maintenance of constraints, hierarchical code lists, and derived, composite concepts (e.g. for dissemination) requires more effort.
307		-|**End user (“the public”)**|Consumption and reuse are more flexible in a pure data structure, but it is more difficult to identify observation keys that actually have data because of the created sparseness. (Constraints may help in this respect.) Frequent occurrences of “non applicable” values may also make data usage cumbersome.
	304	+|(% style="width:215px" %)**Data collector**|(% style="width:1400px" %)Defining constraints for data validation is more complex for a highdimensional, pure DSD. But such a DSD provides more flexibility in terms of consumption and reuse, i.e. mapping to the data collector’s internal data model mapping easier.
	305	+|(% style="width:215px" %)**DSD maintenance**|(% style="width:1400px" %)Pure concepts usually have shorter, less complex code lists and are thus easier to maintain. In contrast, the maintenance of constraints, hierarchical code lists, and derived, composite concepts (e.g. for dissemination) requires more effort.
	306	+|(% style="width:215px" %)**End user (“the public”)**|(% style="width:1400px" %)Consumption and reuse are more flexible in a pure data structure, but it is more difficult to identify observation keys that actually have data because of the created sparseness. (Constraints may help in this respect.) Frequent occurrences of “non applicable” values may also make data usage cumbersome.
308	308
309	309	== 4.2 Number and relations of DSDs ==
310	310
...	...	@@ -326,36 +326,22 @@
326	326
327	327	**Table 7. Data structuring approaches by level of data exchange**
328	328
329		-|**Level of data exchange**|**Data structuring approa one DSD**|(% colspan="2" %)(((
330		-**ch**
331		-
332		-**master + satellite DSDs**
333		-)))|**multiple, indep. DSDs**
	328	+|(% colspan="1" rowspan="2" %)**Level of data exchange**|(% colspan="4" rowspan="1" %)**Data structuring approach**
	329	+|**one DSD**|(% colspan="2" %)**master + satellite DSDs**|**multiple, indep. DSDs**
334	334	|**within organization**|(((
335		-best for single-domain, single-purpose can be created on the
336		-
337		-fly from structured databases
	331	+best for single-domain, single-purpose can be created on the fly from structured databases
338	338	)))|(% colspan="2" %)use if harmonization is important in covered domains or purposes or if such a set of DSDs is already available at international level|easier to do than master + satellite approach each domain/purpose can maintain DSDs independently can be created on the fly from structured databases
339	339	|**between national organizations**|(% colspan="4" %)the same applies as to the “within organization” scenario
340		-|**Level of data exchange**|(% colspan="3" %)(((
341		-**Data structuring approach**
342		-
343		-**one DSD master + satellite DSDs**
344		-)))|**multiple, indep. DSDs**
345	345	|**between int. organization and national organizations**|(% colspan="2" %)best for single domain, single purpose scenarios that are usually rather restricted with very clear specification of what needs to be exchanged|preferable over multiDSD approach in case of multi-domain and/or multi-purpose scenarios with highly correlated data flows for maintenance reasons|(((
346		-for multi-domain and/or multipurpose scenarios; only recommended if overlap of domains/purposes is minor (e.g. just w.r.t. cross-domain concepts)
347		-
348		-equivalent to multiple “one DSD” solutions, one for each domain / purpose
	335	+for multi-domain and/or multipurpose scenarios; only recommended if overlap of domains/purposes is minor (e.g. just w.r.t. cross-domain concepts) equivalent to multiple “one DSD” solutions, one for each domain / purpose
349	349	)))
350	350	|**between international organizations**|(% colspan="3" %)comparable to “national to international” scenario|
351	351	|**dissemination to public**|(% colspan="2" %)for single-domain, single-purpose cases in more complex cases this may be the preferable approach for data discovery tools (one data structure to find and access all data)|(% colspan="2" %)(((
352	352	in multi-purpose or –domain scenarios:
353	353
354		-if it is relevant for the public to see the relationship between the data structures: use master + satellites approach
355		-
356		-otherwise the multi-DSD option is preferable, although with the highest possible degree of re-use of code lists and concepts
357		-
358		-in both cases: important to include only concepts, code lists, and codes actually available / used by the data
	341	+* if it is relevant for the public to see the relationship between the data structures: use master + satellites approach
	342	+* otherwise the multi-DSD option is preferable, although with the highest possible degree of re-use of code lists and concepts
	343	+* in both cases: important to include only concepts, code lists, and codes actually available / used by the data
359	359	)))
360	360
361	361	In general, finding the “perfect” data structure is less important for bilateral data exchange. Independent, custom-tailored DSDs may do the job quite well, as harmonization and standardization are typically not of high importance. If the data exchange is just a part of a more comprehensive scenario (e.g. multi-purpose, multi-domain, gateway, or data-sharing scenarios), a master DSD with satellite DSDs is preferable.
...	...	@@ -364,20 +364,17 @@
364	364
365	365	**Table 8. Data structuring approaches by role in data exchange**
366	366
367		-|**Role in data exchange**|**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
368		-|**Data provider**|It is easier to set up a data submission process against a single DSD (= less initial costs) than against multiple DSDs.
369		-|**Data collector**|(((
	352	+|(% style="width:216px" %)**Role in data exchange**|(% style="width:1399px" %)**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
	353	+|(% style="width:216px" %)**Data provider**|(% style="width:1399px" %)It is easier to set up a data submission process against a single DSD (= less initial costs) than against multiple DSDs.
	354	+|(% style="width:216px" %)**Data collector**|(% style="width:1399px" %)(((
370	370	Data validation is easier with DSDs that only cover what needs to be collected. This is achieved via constraints in the master + satellites approach or via tailor-made independent DSDs. If a single DSD is used in a multi-domain or –purpose scenario, necessary constraints can be specified in the data flow definition or data provision agreement.
371		-
372	372	Further processing of collected data is more flexible and easier if relations are transparent and code lists are shared as in the one DSD or master + satellite DSDs approaches. The “shared context” created through the master DSD increases harmonization and standardization and this way facilitates combined usage of data.
373	373	)))
374		-|**Role in data exchange**|**One DSD vs. master + satellite DSDs vs. multiple, indep. DSDs**
375		-|**DSD maintenance**|(((
	358	+|(% style="width:216px" %)**DSD maintenance**|(% style="width:1399px" %)(((
376	376	The complexity and initial costs for developing and maintaining master + satellite DSDs are higher than for independent DSDs as this involves managing constraints and managing impacts of changes in shared code lists to all DSDs.
377		-
378	378	In the multiple independent DSDs approach, development and maintenance efforts may be distributed. This can be seen as an advantage, but on the other hand requires coordination in case the DSDs are only partially independent (i.e. share some code lists).
379	379	)))
380		-|**End user (“the public”)**|For data discovery and retrieval the user needs to know what data is actually available (instead of what might be collected/disseminated with a certain data structure). This means that the potential sparseness should be hidden from the user. A reduced DSD derived from the data structure used in the background is more useful in most cases. Whether this is done via one DSD and constraints, master + satellite DSDs, or independent DSDs does not matter that much for the user.
	362	+|(% style="width:216px" %)**End user (“the public”)**|(% style="width:1399px" %)For data discovery and retrieval the user needs to know what data is actually available (instead of what might be collected/disseminated with a certain data structure). This means that the potential sparseness should be hidden from the user. A reduced DSD derived from the data structure used in the background is more useful in most cases. Whether this is done via one DSD and constraints, master + satellite DSDs, or independent DSDs does not matter that much for the user.
381	381
382	382	= 5 MINIMUM STRUCTURAL AND SEMANTIC REQUIREMENTS =
383	383
...	...	@@ -407,19 +407,19 @@
407	407
408	408	**Table 9. Minimum requirements for DSDs~*~***
409	409
410		-|**Question**|**Concept**|**COG**|**Code list**|**Time series Cross-section**
411		-|Where?|reference area|X|revision|mand. attribute or dimension
412		-|What?|“indicator”|-|domain|one or multiple dimensions
413		-|How?|unit of measure|X|development|mand. attribute or dimension
414		-|How?|unit multiplier|X|available|mandatory attribute
415		-|How?|decimals|X|available|mandatory attribute
416		-|How?|//adjustment//|X|development|mand. att. not relevant
417		-|When?|time period|X|format|dimension mand. att.
418		-|When?|time format|X|available|mandatory attribute
419		-|When?|time period – collection|X|development|mand. att. cond. att.
420		-|When?|data update – last update|X|time stamp|mandatory attribute
421		-|How often?|//frequency//|X|available|mand. att. or not relevant
422		-|(% colspan="2" %)How much? observation value|-|numeric|dimension measure
	392	+|(% style="width:205px" %)**Question**|(% style="width:272px" %)**Concept**|(% style="width:178px" %)**COG**|(% style="width:270px" %)**Code list**|(% style="width:690px" %)**Time series Cross-section**
	393	+|(% style="width:205px" %)Where?|(% style="width:272px" %)reference area|(% style="width:178px" %)X|(% style="width:270px" %)revision|(% style="width:690px" %)mand. attribute or dimension
	394	+|(% style="width:205px" %)What?|(% style="width:272px" %)“indicator”|(% style="width:178px" %)-|(% style="width:270px" %)domain|(% style="width:690px" %)one or multiple dimensions
	395	+|(% style="width:205px" %)How?|(% style="width:272px" %)unit of measure|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. attribute or dimension
	396	+|(% style="width:205px" %)How?|(% style="width:272px" %)unit multiplier|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
	397	+|(% style="width:205px" %)How?|(% style="width:272px" %)decimals|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
	398	+|(% style="width:205px" %)How?|(% style="width:272px" %)//adjustment//|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. att. not relevant
	399	+|(% style="width:205px" %)When?|(% style="width:272px" %)time period|(% style="width:178px" %)X|(% style="width:270px" %)format|(% style="width:690px" %)dimension mand. att.
	400	+|(% style="width:205px" %)When?|(% style="width:272px" %)time format|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mandatory attribute
	401	+|(% style="width:205px" %)When?|(% style="width:272px" %)time period – collection|(% style="width:178px" %)X|(% style="width:270px" %)development|(% style="width:690px" %)mand. att. cond. att.
	402	+|(% style="width:205px" %)When?|(% style="width:272px" %)data update – last update|(% style="width:178px" %)X|(% style="width:270px" %)time stamp|(% style="width:690px" %)mandatory attribute
	403	+|(% style="width:205px" %)How often?|(% style="width:272px" %)//frequency//|(% style="width:178px" %)X|(% style="width:270px" %)available|(% style="width:690px" %)mand. att. or not relevant
	404	+|(% colspan="2" style="width:477px" %)How much? observation value|(% style="width:178px" %)-|(% style="width:270px" %)numeric|(% style="width:690px" %)dimension measure
423	423
424	424	~*~*Concepts in //italics// are only relevant for time series DSDs. An “X” in the COG column means the concept is defined in the COG. Code list “development” means that the SWG will develop a code list to be recommended in the COG; “revision” means that the code list is recommended by the COG and under revision by the SWG; “format” means that a format is defined by another concept; “text”, “time stamp”, and “numeric” provide data types used for uncoded concepts.
425	425
...	...	@@ -427,25 +427,19 @@
427	427
428	428	**Table 10. Suggested additional concepts for certain scenarios~*~***
429	429
430		-|**Question**|**Concept**|**COG**|**Code list**|**TS CS**|**Scenario**
	412	+|**Question**|**Concept**|**COG**|**Code list**|**TS**|**CS**|**Scenario**
431	431	|Who?|compiling agency|X|development|(((
432		-conditional conditional
433		-
434		- (sibling) (obs. level)
435		-)))|data provider different from data compiler
	414	+conditional (sibling)
	415	+)))|conditional (obs. level)|data provider different from data compiler
436	436	|Who?|(((
437		-confidentiality
438		-
439		-status – observation
440		-)))|X|available|mandatory (obs. level)|except dissemination
441		-|How?|observation status|X|available|conditional (obs. level)|except orig. collection
	417	+confidentiality status – observation
	418	+)))|X|available|(% colspan="2" rowspan="1" %)mandatory (obs. level)|except dissemination
	419	+|How?|observation status|X|available|(% colspan="2" rowspan="1" %)conditional (obs. level)|except orig. collection
442	442	|How much?|(((
443		-//observation pre-//
	421	+//observation pre-break value//
	422	+)))|-|numeric|cond. (obs.)|not relevant|except orig. collection
	423	+|What and how?|//time series title//|X|text|cond. (TS)|not relevant|dissemination
444	444
445		-//break value//
446		-)))|-|numeric|cond. (obs.) not relevant|except orig. collection
447		-|What and how?|//time series title//|X|text|cond. (TS) not relevant|dissemination
448		-
449	449	~** The legend of Table 9 applies to Table 10 as well. The suggested attachment level of attributes (if any) is provided in parentheses in the TS (time series) or CS (cross-section) columns. In case an attribute does not vary at that level in a certain use case, it should be attached at the highest possible level.
450	450
451	451	== 5.2 Attribute attachment levels and definition of groups ==
...	...	@@ -467,10 +467,8 @@
467	467	* //ID//: a unique identifier of the message
468	468	* //Test//: a Boolean attribute that indicates whether the message is for test purposes or not
469	469	* //Prepared//: the date the message was prepared
470		-* //Sender//: the identification of the organization that is transmitting the message
	446	+* //Sender//: the identification of the organization that is transmitting the message (recommended: code from the agency code list in the SDMX COG)
471	471
472		-(recommended: code from the agency code list in the SDMX COG)
473		-
474	474	From a business perspective, the inclusion of the //Name// element is highly recommended, as it can help to understand the purpose of the exchange message. Other header elements such as //Receiver// are optional.
475	475
476	476	= 6 STEP-BY-STEP GUIDE =
...	...	@@ -481,13 +481,15 @@
481	481
482	482	Figure 1 provides an overview of the overall process. As a first step, the context of the data exchange(s) that should be covered by the DSD(s) is defined in terms of purpose, domains, level of exchange, type of data, type of recipient, role of in data exchange, process pattern, and GSBPM phase (see Figure 2). Since reusing existing artefacts is one of the guiding principles, the second step identifies existing DSDs that may be reused (see Figure 3). In case relevant DSDs are available, their suitability in the present context is evaluated in step 3. Aspects to be taken into account are concept coverage, concept roles, attribute attachment levels, and code lists (see Figure 4). Step 4 is subject to the outcome of step 3. In case of a favorable assessment, the DSDs are simply reused. If the DSDs are partly suitable, modified versions can be derived. See section 2. for a summary of possible DSD modification scenarios. If the DSDs are not suitable or if no relevant DSDs are available at all, new DSDs will be defined as described in section 3. Finally, supporting artefacts such as data flow definitions and data provision agreements are defined (see Figure 5).
483	483
	458	+(% class="wikigeneratedid" %)
	459	+[[image:1768470533088-795.png]]
484	484
485	485	(% class="wikigeneratedid" id="HFigure1.OverviewoftheDSDdesignprocess" %)
486	486	Figure 1. Overview of the DSD design process
487	487
488		-
489	489	Figure 2 summarizes the characteristics of the data exchange context that is defined in step 1. These characteristics affect the decision on the data structuring approach that is part of the process of defining the concepts of a new DSD (step 4.3. in Figure 1; see Figure 7 in section 2.).
490	490
	466	+[[image:1768470575978-226.png]]
491	491
492	492	(% class="wikigeneratedid" id="HFigure2.Characteristicsofdataexchangecontext" %)
493	493	Figure 2. Characteristics of data exchange context
...	...	@@ -494,20 +494,23 @@
494	494
495	495	Figure 3 recaps the priorities given to different types of existing DSDs when searching for candidates for reuse in step 2. Global DSDs maintained by the SDMX consortium are ranked the highest. They can be found via the Global SDMX Registry.
496	496
	473	+(% class="wikigeneratedid" %)
	474	+[[image:1768470596130-305.png]]
497	497
498	498	(% class="wikigeneratedid" id="HFigure3.PriorityrankingofexistingDSDsforreuse" %)
499	499	Figure 3. Priority ranking of existing DSDs for reuse
500	500
501		-
502	502	Figure 4 summarizes the aspects to be considered in the assessment of the suitability of existing DSDs in step 3. For a detailed description of the cases of partial unsuitability see section 2.1. above.
503	503
	481	+(% class="wikigeneratedid" %)
	482	+[[image:1768470626558-321.png]]
504	504
505	505	(% class="wikigeneratedid" id="HFigure4.AspectsofDSDsuitability" %)
506	506	Figure 4. Aspects of DSD suitability
507	507
508		-
509	509	Figure 5 lists the most relevant artefacts required in addition to a DSD, its concept scheme, and code lists.
510	510
	489	+[[image:1768470646456-652.png]]
511	511
512	512	Figure 5. Supporting artefacts
513	513
...	...	@@ -515,8 +515,10 @@
515	515
516	516	Figure 6 briefly recapitulates the actions that can be taken to overcome partial unsuitability of DSDs. As far as possible, existing artefacts should be reused in this case. This means that even if a DSD cannot be reused as a whole, concepts and code lists from that DSD can be included in the new DSD by reference.
517	517
518		-**Figure 6. DSD modification scenarios**
	497	+[[image:1768470678965-391.png]]
519	519
	499	+Figure 6. DSD modification scenarios
	500	+
520	520	== 6.3 Defining new DSDs ==
521	521
522	522	In case no (suitable) DSD is available, the actual process of specifying a new DSD is started. Figure 7 depicts this process (step 4.3. in Figure 1). It encompasses the specification of concepts, code lists, and data formats. All three specification steps include the identification of already existing artefacts that could be reused or modified to satisfy the requirements at hand and the definition of new artefacts in case no suitable artefacts are detected. Several iterations of steps 1 (specification of concepts; see Figure 8) and 2 (specification of code lists; see Figure°13) may be necessary, including revisions of the decision concerning the data structuring approach. Finally all artefacts defined in the previous steps are put together into a DSD.

1768470533088-795.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+48.6 KB

Content

1768470575978-226.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+95.8 KB

Content

1768470596130-305.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+44.4 KB

Content

1768470611326-907.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+51.5 KB

Content

1768470626558-321.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+51.5 KB

Content

1768470646456-652.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+32.7 KB

Content

1768470678965-391.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+xwiki:XWiki.helena

Size

...	...	@@ -1,0 +1,1 @@
	1	+117.9 KB

Content