Skip to Content

Changes for page 13 Structure Mapping

Last modified by Artur on 2025/09/10 11:19
From version 4.5
edited by Helena
on 2025/06/16 14:41
Change comment: There is no comment for this version
To version 4.2
edited by Helena
on 2025/06/16 14:21
Change comment: There is no comment for this version

Summary

Details

Page properties
Tags
... ... @@ -1,1 +1,0 @@
1 -Component|Data set|Dimension|Observation value|Series key|Statistical data and metadata exchange
Content
... ... @@ -4,94 +4,97 @@
4 4  
5 5  == 13.1 Introduction ==
6 6  
7 -The purpose of [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] structure mapping is to transform [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] from one dimensionality to another. In practice, this means that the input and output [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] conform to different Data Structure Definition.
7 +The purpose of SDMX structure mapping is to transform datasets from one dimensionality to another. In practice, this means that the input and output datasets conform to different Data Structure Definition.
8 8  
9 -Structure mapping does not alter the [[observation values>>doc:sdmx:Glossary.Observation value.WebHome]] and is not intended to perform any aggregations or calculations.
9 +Structure mapping does not alter the observation values and is not intended to perform any aggregations or calculations.
10 10  
11 -An input series (% style="color:#e74c3c" %)maps(%%) to:
11 +An input series maps to:
12 12  
13 13  1. Exactly one output series; or
14 -1. Multiple output series with different [[Series Keys>>doc:sdmx:Glossary.Series key.WebHome]], but the same [[observation values>>doc:sdmx:Glossary.Observation value.WebHome]]; or
15 -1. Zero output series where no source rule matches the input [[Component>>doc:sdmx:Glossary.Component.WebHome]] values.
14 +1. Multiple output series with different Series Keys, but the same observation values; or
15 +1. Zero output series where no source rule matches the input Component values.
16 16  
17 17  Typical use cases include:
18 18  
19 19  * Transforming received data into a common internal structure;
20 20  * Transforming reported data into the data collector's preferred structure;
21 -* Transforming unidimensional [[datasets>>doc:sdmx:Glossary.Data set.WebHome]]{{footnote}}Unidimensional datasets are those with a single 'indicator' or 'series code' dimension.{{/footnote}} to multi-dimensional; and
22 -* Transforming internal [[datasets>>doc:sdmx:Glossary.Data set.WebHome]] with a complex structure to a simpler structure with fewer [[dimensions>>doc:sdmx:Glossary.Dimension.WebHome]] suitable for dissemination.
21 +* Transforming unidimensional datasets{{footnote}}Unidimensional datasets are those with a single 'indicator' or 'series code' dimension.{{/footnote}} to multi-dimensional; and
22 +* Transforming internal datasets with a complex structure to a simpler structure with fewer dimensions suitable for dissemination.
23 23  
24 24  == 13.2 1-1 structure maps ==
25 25  
26 -1-1 (pronounced 'one to one') mappings support the simple use case where the value of a [[Component>>doc:sdmx:Glossary.Component.WebHome]] in the source structure is translated to a different value in the target, usually where different classification schemes are used for the same Concept.
26 +1-1 (pronounced 'one to one') mappings support the simple use case where the value of a Component in the source structure is translated to a different value in the target, usually where different classification schemes are used for the same Concept.
27 27  
28 -In the example below, ISO 2-character country [[codes>>doc:sdmx:Glossary.Code.WebHome]] are (% style="color:#e74c3c" %)mapped(%%) to their ISO 3character equivalent.
28 +In the example below, ISO 2-character country codes are mapped to their ISO 3character equivalent.
29 29  
30 -(% style="width:585.294px" %)
31 -|(% style="width:173px" %)**Country**|(% style="width:180px" %)**Alpha-2 code**|(% style="width:229px" %)**Alpha-3 code**
32 -|(% style="width:173px" %)Afghanistan|(% style="width:180px" %)AF|(% style="width:229px" %)AFG
33 -|(% style="width:173px" %)Albania|(% style="width:180px" %)AL|(% style="width:229px" %)ALB
34 -|(% style="width:173px" %)Algeria|(% style="width:180px" %)DZ|(% style="width:229px" %)DZA
35 -|(% style="width:173px" %)American Samoa|(% style="width:180px" %)AS|(% style="width:229px" %)ASM
36 -|(% style="width:173px" %)Andorra|(% style="width:180px" %)AD|(% style="width:229px" %)AND
37 -|(% style="width:173px" %)etc…|(% style="width:180px" %) |(% style="width:229px" %)
30 +|Country|Alpha-2 code|Alpha-3 code
31 +|Afghanistan|AF|AFG
32 +|Albania|AL|ALB
33 +|Algeria|DZ|DZA
34 +|American Samoa|AS|ASM
35 +|Andorra|AD|AND
36 +|etc…| |
38 38  
39 -Different source values can also (% style="color:#e74c3c" %)map(%%) to the same target value, for example when deriving regions from country [[codes>>doc:sdmx:Glossary.Code.WebHome]].
38 +Different source values can also map to the same target value, for example when deriving regions from country codes.
40 40  
41 -(% style="width:490.294px" %)
42 -|(% style="width:260px" %)Source Component: REF_AREA|(% style="width:227px" %)Target Component: REGION
43 -|(% style="width:260px" %)FR|(% style="width:227px" %)EUR
44 -|(% style="width:260px" %)DE|(% style="width:227px" %)EUR
45 -|(% style="width:260px" %)IT|(% style="width:227px" %)EUR
46 -|(% style="width:260px" %)ES|(% style="width:227px" %)EUR
47 -|(% style="width:260px" %)BE|(% style="width:227px" %)EUR
40 +|Source Component: REF_AREA|Target Component: REGION
41 +|FR|EUR
42 +|DE|EUR
43 +|IT|EUR
44 +|ES|EUR
45 +|BE|EUR
48 48  
49 49  == 13.3 N-n structure maps ==
50 50  
51 -N-n (pronounced 'N to N') mappings describe rules where a specified combination of values in multiple source [[Components>>doc:sdmx:Glossary.Component.WebHome]] (% style="color:#e74c3c" %)map(%%) to specified values in one or more target [[Components>>doc:sdmx:Glossary.Component.WebHome]]. For example, when mapping a partial [[Series Key>>doc:sdmx:Glossary.Series key.WebHome]] from a highly multidimensional cube (like Balance of Payments) to a single 'Indicator' [[Dimension>>doc:sdmx:Glossary.Dimension.WebHome]] in a target Data Structure.
49 +N-n (pronounced 'N to N') mappings describe rules where a specified combination of values in multiple source Components map to specified values in one or more target Components. For example, when mapping a partial Series Key from a highly multidimensional cube (like Balance of Payments) to a single 'Indicator' Dimension in a target Data Structure.
52 52  
53 53  Example:
54 54  
55 -(% style="width:964.294px" %)
56 -|(% style="width:65px" %)Rule|(% style="width:519px" %)Source|(% style="width:378px" %)Target
57 -|(% style="width:65px" %)1|(% style="width:519px" %)(((
53 +|Rule|Source|Target
54 +|1|(((
58 58  If
59 -FREQUENCY=A; and
60 -ADJUSTMENT=N; and
61 -MATURITY=L.
62 -)))|(% style="width:378px" %)(((
56 +
57 +FREQUENCY=A; and ADJUSTMENT=N; and MATURITY=L.
58 +)))|(((
63 63  Set
60 +
64 64  INDICATOR=A_N_L
65 65  )))
66 -|(% style="width:65px" %)2|(% style="width:519px" %)(((
63 +|2|(((
67 67  If
68 -FREQUENCY=M; and
69 -ADJUSTMENT=S_A1; and
70 -MATURITY=TY12.
71 -)))|(% style="width:378px" %)(((
65 +
66 +FREQUENCY=M; and ADJUSTMENT=S_A1; and MATURITY=TY12.
67 +)))|(((
72 72  Set
69 +
73 73  INDICATOR=MON_SAX_12
74 74  )))
75 75  
76 76  N-n rules can also set values for multiple source Components.
77 77  
78 -(% style="width:965.294px" %)
79 -|(% style="width:73px" %)Rule|(% style="width:506px" %)Source|(% style="width:383px" %)Target
80 -|(% style="width:73px" %)1|(% style="width:506px" %)(((
75 +|Rule|Source|Target
76 +|1|(((
81 81  If
78 +
82 82  FREQUENCY=A; and ADJUSTMENT=N; and MATURITY=L.
83 -)))|(% style="width:383px" %)(((
80 +)))|(((
84 84  Set
82 +
85 85  INDICATOR=A_N_L, STATUS=QXR15,
84 +
86 86  NOTE="Unadjusted".
87 87  )))
88 -|(% style="width:73px" %)2|(% style="width:506px" %)(((
87 +|2|(((
89 89  If
89 +
90 90  FREQUENCY=M; and ADJUSTMENT=S_A1; and MATURITY=TY12.
91 -)))|(% style="width:383px" %)(((
91 +)))|(((
92 92  Set
93 +
93 93  INDICATOR=MON_SAX_12,
95 +
94 94  STATUS=MPM12,
97 +
95 95  NOTE="Seasonally Adjusted"
96 96  )))
97 97  
... ... @@ -101,90 +101,93 @@
101 101  
102 102  A simple example mapping a source dataset with a single dimension to one with multiple dimensions is shown below:
103 103  
104 -(% style="width:972.294px" %)
105 -|(% style="width:257px" %)Source|(% style="width:315px" %)Target|(% style="width:397px" %)Output Series Key
106 -|(% style="width:257px" %)SERIES_CODE=XMAN_Z_21|(% style="width:315px" %)(((
107 +|Source|Target|Output Series Key
108 +|SERIES_CODE=XMAN_Z_21|(((
107 107  Dimensions
110 +
108 108  INDICATOR=XM
109 109  FREQ=A
110 110  ADJUSTMENT=N
111 111  Attributes
112 112  UNIT_MEASURE=_Z
116 +
113 113  COMP_ORG=21
114 -)))|(% style="width:397px" %)XM:A:N
115 -|(% style="width:257px" %)(((
118 +)))|XM:A:N
119 +|(((
116 116  SERIES_CODE=XMAN_Z_34
117 117  
118 118  
119 -)))|(% style="width:315px" %)(((
123 +)))|(((
120 120  Dimensions
125 +
121 121  INDICATOR=XM
127 +
122 122  FREQ=A
129 +
123 123  ADJUSTMENT=N
131 +
124 124  Attributes
133 +
125 125  UNIT_MEASURE=_Z
135 +
126 126  COMP_ORG=34
127 -)))|(% style="width:397px" %)XM:A:N
137 +)))|XM:A:N
128 128  
129 -The above behaviour can be okay if the series XMAN_Z_21 contains observations for different periods of time then the series XMAN_Z_34. If however both series contain observations for the same point in time, the output for this mapping will be two observations with the same [[series key>>doc:sdmx:Glossary.Series key.WebHome]], for the same period in time.
139 +The above behaviour can be okay if the series XMAN_Z_21 contains observations for different periods of time then the series XMAN_Z_34. If however both series contain observations for the same point in time, the output for this mapping will be two observations with the same series key, for the same period in time.
130 130  
131 131  == 13.5 Representation maps ==
132 132  
133 -[[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Maps(%%) replace the [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 Codelist (% style="color:#e74c3c" %)Maps(%%) and are used describe explicit mappings between source and target [[Component>>doc:sdmx:Glossary.Component.WebHome]] values.
143 +Representation Maps replace the SDMX 2.1 Codelist Maps and are used describe explicit mappings between source and target Component values.
134 134  
135 -The source and target of a [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Map(%%) can reference any of the following:
145 +The source and target of a Representation Map can reference any of the following:
136 136  
137 137  1. Codelist
138 138  1. Free Text (restricted by type, e.g String, Integer, Boolean)
139 139  1. Valuelist
140 140  
141 -A [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Map(%%) mapping ISO 2-character to ISO 3-character Codelists would take the following form:
151 +A Representation Map mapping ISO 2-character to ISO 3-character Codelists would take the following form:
142 142  
143 -(% style="width:356.294px" %)
144 -|(% style="width:167px" %)CL_ISO_ALPHA2|(% style="width:186px" %)CL_ISO_ALPHA3
145 -|(% style="width:167px" %)AF|(% style="width:186px" %)AFG
146 -|(% style="width:167px" %)AL|(% style="width:186px" %)ALB
147 -|(% style="width:167px" %)DZ|(% style="width:186px" %)DZA
148 -|(% style="width:167px" %)AS|(% style="width:186px" %)ASM
149 -|(% style="width:167px" %)AD|(% style="width:186px" %)AND
150 -|(% style="width:167px" %)etc…|(% style="width:186px" %)
153 +|CL_ISO_ALPHA2|CL_ISO_ALPHA3
154 +|AF|AFG
155 +|AL|ALB
156 +|DZ|DZA
157 +|AS|ASM
158 +|AD|AND
159 +|etc…|
151 151  
152 -A [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Map(%%) mapping free text country names to an ISO 2-character Codelist could be similarly described:
161 +A Representation Map mapping free text country names to an ISO 2-character Codelist could be similarly described:
153 153  
154 -(% style="width:364.294px" %)
155 -|(% style="width:169px" %)Text|(% style="width:192px" %)CL_ISO_ALPHA2
156 -|(% style="width:169px" %)"Germany"|(% style="width:192px" %)DE
157 -|(% style="width:169px" %)"France"|(% style="width:192px" %)FR
158 -|(% style="width:169px" %)"United Kingdom"|(% style="width:192px" %)GB
159 -|(% style="width:169px" %)"Great Britain"|(% style="width:192px" %)GB
160 -|(% style="width:169px" %)"Ireland"|(% style="width:192px" %)IE
161 -|(% style="width:169px" %)"Eire"|(% style="width:192px" %)IE
162 -|(% style="width:169px" %)etc…|(% style="width:192px" %)
163 +|Text|CL_ISO_ALPHA2
164 +|"Germany"|DE
165 +|"France"|FR
166 +|"United Kingdom"|GB
167 +|"Great Britain"|GB
168 +|"Ireland"|IE
169 +|"Eire"|IE
170 +|etc…|
163 163  
164 -Valuelists, introduced in [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 3.0, are equivalent to Codelists but allow the maintenance of non-[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifiers. Importantly, their IDs do not need to conform to IDType, but as a consequence are not Identifiable.
172 +Valuelists, introduced in SDMX 3.0, are equivalent to Codelists but allow the maintenance of non-SDMX identifiers. Importantly, their IDs do not need to conform to IDType, but as a consequence are not Identifiable.
165 165  
166 -When used in [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Maps(%%), Valuelists allow Non-[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifiers containing characters like £, $, % to be (% style="color:#e74c3c" %)mapped(%%) to [[Code>>doc:sdmx:Glossary.Code.WebHome]] IDs, or [[Codes>>doc:sdmx:Glossary.Code.WebHome]] (% style="color:#e74c3c" %)mapped(%%) to non-[[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] identifiers.
174 +When used in Representation Maps, Valuelists allow Non-SDMX identifiers containing characters like £, $, % to be mapped to Code IDs, or Codes mapped to non-SDMX identifiers.
167 167  
168 168  In common with Codelists, each item in a Valuelist has a multilingual name giving it a human-readable label and an optional description. For example:
169 169  
170 -(% style="width:435.294px" %)
171 -|(% style="width:126px" %)Value|(% style="width:133px" %)Locale|(% style="width:173px" %)Name
172 -|(% style="width:126px" %)$|(% style="width:133px" %)en|(% style="width:173px" %)United States Dollar
173 -|(% style="width:126px" %)%|(% style="width:133px" %)En|(% style="width:173px" %)Percentage
174 -|(% style="width:126px" %) |(% style="width:133px" %)fr|(% style="width:173px" %)Pourcentage
178 +|Value|Locale|Name
179 +|$|en|United States Dollar
180 +|%|En|Percentage
181 +| |fr|Pourcentage
175 175  
176 -Other characteristics of [[Representation>>doc:sdmx:Glossary.Representation.WebHome]] (% style="color:#e74c3c" %)Maps(%%):
183 +Other characteristics of Representation Maps:
177 177  
178 -* Support the (% style="color:#e74c3c" %)mapping(%%) of multiple source [[Component>>doc:sdmx:Glossary.Component.WebHome]] values to multiple Target [[Component>>doc:sdmx:Glossary.Component.WebHome]] values as described in section 13.3 on n-to-n mappings; this covers also the case of (% style="color:#e74c3c" %)mapping(%%) an [[Attribute>>doc:sdmx:Glossary.Attribute.WebHome]] with an array [[representation>>doc:sdmx:Glossary.Representation.WebHome]] to (% style="color:#e74c3c" %)map(%%) combinations of values to a single target value;
179 -* Allow source or target mappings for an Item to be optional allowing rules such as 'A (% style="color:#e74c3c" %)maps(%%) to nothing' or 'nothing (% style="color:#e74c3c" %)maps(%%) to A'; and
180 -* Support for (% style="color:#e74c3c" %)mapping(%%) rules where regular expressions or substrings are used to match source [[Component>>doc:sdmx:Glossary.Component.WebHome]] values. Refer to section 13.6 for more on this topic.
185 +* Support the mapping of multiple source Component values to multiple Target Component values as described in section 13.3 on n-to-n mappings; this covers also the case of mapping an Attribute with an array representation to map combinations of values to a single target value;
186 +* Allow source or target mappings for an Item to be optional allowing rules such as 'A maps to nothing' or 'nothing maps to A'; and
187 +* Support for mapping rules where regular expressions or substrings are used to match source Component values. Refer to section 13.6 for more on this topic.
188 +*1. Regular expression and substring rules
181 181  
182 -== 13.6 Regular expression and substring rules ==
190 +It is common for classifications to contain meanings within the identifier, for example the code Id 'XULADS' may refer to a particular seasonality because it starts with the letters XU.
183 183  
184 -It is common for classifications to contain meanings within the identifier, for example the [[code>>doc:sdmx:Glossary.Code.WebHome]] Id 'XULADS' may refer to a particular seasonality because it starts with the letters XU.
192 +With SDMX 2.1 each code that starts with XU had to be individually mapped to the same seasonality, and additional mappings added when new Codes were added to the Codelists. This led to many hundreds or thousands of mappings which can be more efficiently summarised in a single conceptual rule:
185 185  
186 -With [[SDMX>>doc:sdmx:Glossary.Statistical data and metadata exchange.WebHome]] 2.1 each [[code>>doc:sdmx:Glossary.Code.WebHome]] that starts with XU had to be individually (% style="color:#e74c3c" %)mapped(%%) to the same seasonality, and additional mappings added when new [[Codes>>doc:sdmx:Glossary.Code.WebHome]] were added to the Codelists. This led to many hundreds or thousands of mappings which can be more efficiently summarised in a single conceptual rule:
187 -
188 188  //If starts with 'XU' map to 'Y'//
189 189  
190 190  These rules are described using either regular expressions, or substrings for simpler use cases.
... ... @@ -269,7 +269,7 @@
269 269  
270 270  1. The output frequency determines the output date format, but the default output can be redefined using a Frequency Format mapping to force explicit rules on how the output time period is formatted.
271 271  1. To support the use case of changing frequency the structure map can optionally provide a start of year attribute, which defines the year start date in MM-DD format. For example: YearStart=04-01.
272 -11.
278 +11.
273 273  111. Pattern based dates
274 274  
275 275  Date and time formats are specified by date and time pattern strings based on Java's Simple Date Format. Within date and time pattern strings, unquoted letters from 'A' to 'Z' and from 'a' to 'z' are interpreted as pattern letters representing the components of a date or time string. Text can be quoted using single quotes (') to avoid interpretation. "''" represents a single quote. All other characters are not interpreted; they're simply copied into the output string during formatting or matched against the input string during parsing.
... ... @@ -513,8 +513,8 @@
513 513  
514 514  **Note**: The key order is NOT based on the Dimension order of the DSD, as the mapping needs to be resilient to the DSD changing.
515 515  
516 -1.
517 -11.
522 +1.
523 +11.
518 518  111. Mapping other data types to Code Id
519 519  
520 520  In the case where the incoming data type is not a string and not a code identifier i.e. the source Dimension is of type Integer and the target is Codelist. This is supported by the RepresentationMap. The RepresentationMap source can reference a Codelist, Valuelist, or be free text, the free text can include regular expressions.