Interoperability Basis Platform

= 1. Problem statement =

In many cases, data that are exchanged in SDMX data messages do not relate to the calendar year. However, many statistical system implementations require that data are mapped to and stored as the real calendar.

8

9

This guideline provides recommendations for the following four use cases of such non-calendar year data:

10

11

1. Reporting year is equal to the calendar year

12

1. Reporting year starts on the first day of a month different to January

13

Example: school year starting on the 1^^st^^ of September, fiscal year starting on the 1^^st^^ of July

14

1. Reporting year starts on a given day in the year

15

Example: tax year starting on the 5^^th^^ of April

16

1. Reporting year **ends** on a given day in the year

17

Example: fiscal year ending on the 30^^th^^ of June, equivalent to fiscal year starting on the 1^^st^^ of July Y-1

18

19

It is also required to take into account reporting frequencies other than annual (e.g. quarterly, monthly). The use cases lined out above can be visualised graphically. Note that, in this example, all time spans would use the year 2015 (“2015” or “2015-A1”) as the reference year in reporting:

20

21

[[image:1768553211015-575.png]]

22

23

Assuming that the time dimension has the ID TIME_PERIOD, case 1 could use TIME_PERIOD=”2015”. All other cases would need to use the notation TIME_PERIOD=”2015-A1”. For the rest of the document we use the more general notation “2015-A1”.{{footnote}}The YYYY format is reserved for Gregorian years, i.e. from January 1 to December 31. For reporting years that are not Gregorian years, the format YYYY-A1 (e.g. 2016-A1) must be used for the time dimension.{{/footnote}}

24

25

For quarterly or monthly data, the first period of the reporting year would also need to be read as relative to the start of the reporting year. Here are examples for case 2 if the data is quarterly (2a) or monthly (2b):

26

27

[[image:1768553250696-532.png]]

28

29

Note that in case 2a, the third quarter of reporting year 2015 (TIME_PERIOD=2015-Q3) is covering data from March until June 2016. In case 2b, the seventh month of reporting year 2015 (TIME_PERIOD=2015-M07) is covering data for January 2016.

30

31

It is clear in the graphical view that time series under case 2b should not be directly compared with time series under case 4. If it is required to compare figures of the same time period, the 2015 observation of the type 2b series is compared with the 2016 figure of the type 4 series. Comparing a 2a series with others from the example may be more complicated, because exact alignment of the timelines might not be possible without additional data. It is possible to estimate aligned timelines by doing a time transformation using formulas; for instance, shifting the time series to comply with the calendar year. The methodological aspects of time transformation are not part of these guidelines. For coding the results of such transformations, please refer to the guidelines on coding time transformations in SDMX{{footnote}}https://sdmx.org/wp-content/uploads/SWG_TimeTransformation_V1.0.docx{{/footnote}}.

32

33

= 2. SDMX Concepts for Non-Calendar Year Series =

34

35

== //Start and end day coding// ==

36

37

In SDMX messages, the time period concept (concept ID TIME_PERIOD) specifies the **reporting period**. This reporting period, as outlined above, is in many cases not aligned with the calendar year.

38

39

To specify the calendar period that a reporting period is covering, the SDMX technical standard already defines an attribute “reporting year start” on series level with format xs:gMonthDay{{footnote}}https://sdmx.org/wp-content/uploads/SDMX_2-1-1-SECTION_6_TechnicalNotes-march-2013.pdf, row 658.{{/footnote}}. It gives a day and month when the reporting year starts. It is optional and if not provided the default is 1^^st^^ January. Using this attribute can cover cases 1, 2 and 3 from above. When added to a series, the attribute will specify on which day of the calendar year the reporting year starts.

40

41

This is not sufficient to cover case 4, because in this case the reporting year 2015 starts in 2014. In order to specify this case, a different attribute “reporting year end” should be used. It is also on series level with format xs:gMonthDay and gives a day and month when the reporting year ends. It is optional and if not provided the default is 31^^st^^ December.

42

43

The technical note (revision 1.0) suggests that the attribute ID for reporting year start day to be REPORTING_YEAR_START_DAY. However, to maintain EDI compliance, it is suggested to use the IDs REPYEARSTART and REPYEAREND when defining DSDs.

44

45

For any given series either start or end day should be used, but not both. This also implies that a reporting year will always have a duration of one year. In case reporting years do not last exactly a year, refer to the ISO time interval specification described in the next chapter. It is also strongly recommended not to use 29^^th^^ February as either start or end date of the reporting year, since that might lead to undefined situations.

46

47

== //ISO 8601 time interval// ==

48

49

As outlined above, in some cases reporting periods might not have the same duration as calendar periods. A crop year in agriculture may only last for a couple of months. To specify these periods on a more granular level, the SDMX technical standard suggests using the time intervals as defined by ISO 8601. ISO specifies four ways to express a time interval{{footnote}}http://www.iso.org/iso/home/standards/iso8601.htm and

50

https://sdmx.org/wp-content/uploads/SDMX_2-1-1-SECTION_6_TechnicalNotes-march-2013.pdf, row 749.{{/footnote}}:

51

52

1. Start and end, such as "2007-03-01T13:00:00Z/2008-05-11T15:30:00Z"

53

1. Start and duration, such as "2007-03-01T13:00:00Z/P1Y2M10DT2H30M"

54

1. Duration and end, such as "P1Y2M10DT2H30M/2008-05-11T15:30:00Z"

55

1. Duration only, such as "P1Y2M10DT2H30M", with additional context information

56

57

An attribute time range (ID:TIME_RANGE) is suggested at the observation level to specify further in which particular time range a specific observation was collected. The series will still specify when the reporting period starts or ends for the whole series and each observation can have a specific time range within that reporting period. It is suggested for that case to use only option 1, the start and end date, and not the other options in SDMX context.

58

59

= 3. Example 1: Based on National Accounts data exchange =

60

61

Note that the SDMX fragments shown in the examples below are not syntactically correct. They serve the purpose of explaining the issue and show the coding in a readable way (pseudo code).

62

63

== //Current NA definition// ==

64

65

In the National Accounts Data Structure Definitions, fiscal year reporting was initially covered with the “reference period detail” attribute. This attribute is coded and only covers cases where a fiscal year starts on the first day of a given month. For other cases, a code for “other definition” was introduced.

66

67

The code list for “reference period detail” previously used in the National Accounts DSDs is as follows:

68

69

|(% style="width:168px" %)**Code**|(% style="width:2050px" %)**Description**

70

|(% style="width:168px" %)C|(% style="width:2050px" %)Calendar year

71

|(% style="width:168px" %)F_O|(% style="width:2050px" %)Fiscal year (other definition)

72

|(% style="width:168px" %)F02|(% style="width:2050px" %)Fiscal year starting in February

73

|(% style="width:168px" %)F03|(% style="width:2050px" %)Fiscal year starting in March

74

|(% style="width:168px" %)F04|(% style="width:2050px" %)Fiscal year starting in April

75

|(% style="width:168px" %)F05|(% style="width:2050px" %)Fiscal year starting in May

76

|(% style="width:168px" %)F06|(% style="width:2050px" %)Fiscal year starting in June

77

|(% style="width:168px" %)F07|(% style="width:2050px" %)Fiscal year starting in July

78

|(% style="width:168px" %)F08|(% style="width:2050px" %)Fiscal year starting in August

79

|(% style="width:168px" %)F09|(% style="width:2050px" %)Fiscal year starting in September

80

|(% style="width:168px" %)F10|(% style="width:2050px" %)Fiscal year starting in October

81

|(% style="width:168px" %)F11|(% style="width:2050px" %)Fiscal year starting in November

82

|(% style="width:168px" %)F12|(% style="width:2050px" %)Fiscal year starting in December

83

84

Following this code list, the cases lined out in the problem statement could be coded as such:

85

86

1) Reporting year is equal to the calendar year

87

88

<na_:Series **REF_PERIOD_DETAIL="C" **STO="B1G" REF_AREA="LU" FREQ="Q">

89

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

</na_:Series>

REF_PERIOD_DETAIL="C" à reporting year 1995 starts 1^^st^^ January 1995

94

95

2) Reporting year starts on the first day of a month different to January

96

Example: fiscal year starting on the 1^^st^^ of July

97

98

<na_:Series **REF_PERIOD_DETAIL="F07" **STO="B1G" REF_AREA="LU" FREQ="Q">

99

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

</na_:Series>

REF_PERIOD_DETAIL="F07" à reporting year 1995 starts 1^^st^^ July 1995 and Q1 projected on the calendar year goes from: 07-09/1995

104

105

3) Reporting year starts on a given day in the year

106

Example: tax year starting on the 5^^th^^ of April

107

108

<na_:Series **REF_PERIOD_DETAIL="F04" **STO="B1G" REF_AREA="LU" FREQ="Q">

109

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

</na_:Series>

REF_PERIOD_DETAIL="F04" à reporting year 1995 would start 1^^st^^ April 1995. The case is not solved because we do not know anymore that it should be on the 5^^th^^. Another option would be to use the “F_O” code:

114

115

<na_:Series **REF_PERIOD_DETAIL="F_O" **STO="B1G" REF_AREA="LU" FREQ="Q">

116

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

</na_:Series>

REF_PERIOD_DETAIL="F_O" à reporting year 1995 has another definition. The case is only partly solved because we do not know the exact date. Additional metadata are needed and the reporting period cannot directly be parsed by a system.

121

122

4) Reporting year **ends** on a given day in the year

123

Example: fiscal year ending on the 30^^th^^ of June, equivalent to fiscal year starting on the 1^^st^^ of July Y-1

124

125

<na_:Series **REF_PERIOD_DETAIL="??" **STO="B1G" REF_AREA="LU" FREQ="Q">

126

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

</na_:Series>

REF_PERIOD_DETAIL="??" à the case is not covered. The code “F_O” could be used with the same problems as lined above for case 3.

131

132

== //Recommendations based on the use cases from National Accounts// ==

133

134

Using the attributes as suggested above, all cases outlined in the National Account examples can be fully covered:

135

136

1) Reporting year is equal to the calendar year

137

138

<na_:Series **REF_PERIOD_DETAIL="C" REPYEARSTART="~-~-01-01"// //**STO="B1G" REF_AREA="LU" FREQ="Q">

139

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

140

</na_:Series>

141

142

1995-Q1 is the same as the calendar definition: from 1^^st^^ January 1995 until 31^^st^^ March 1995. In this case, the attribute REPYEARSTART may be omitted, since it expressed the default value of 1^^st^^ January.

143

144

2) Reporting year starts on the first day of a month different to January

145

Example: fiscal year starting on the 1^^st^^ of July

146

147

<na_:Series **REF_PERIOD_DETAIL="F07" REPYEARSTART="~-~-07-01" **STO="B1G" REF_AREA="LU" FREQ="Q">

148

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

149

</na_:Series>

150

151

Reporting period 1995-Q1 lasts in that case from 1^^st^^ July 1995 until 30^^th^^ September 1995.

152

153

3) Reporting year starts on a given day in the year

154

Example: tax year starting on the 5^^th^^ of April

155

156

<na_:Series **REF_PERIOD_DETAIL="F04" REPYEARSTART="~-~-04-05" **STO="B1G" REF_AREA="LU" FREQ="Q">

157

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

158

</na_:Series>

159

160

Reporting period 1995-Q1 lasts in that case from 5^^th^^ April 1995 until 4^^th^^ July 1995.

161

162

4) Reporting year **ends** on a given day in the year

163

Example: fiscal year ending on the 30^^th^^ of June, equivalent to fiscal year starting on the 1^^st^^ of July Y-1

164

165

<na_:Series **REF_PERIOD_DETAIL="??" REPYEAREND="~-~-06-30" **STO="B1G" REF_AREA="LU" FREQ="Q">

166

<na_:Obs OBS_VALUE="44" TIME_PERIOD="**1995**-Q1"/>

167

</na_:Series>

168

169

Reporting period 1995-Q1 lasts in that case from 1^^st^^ July 1994 until 30^^th^^ September 1994

170

171

= 4. Example 2: Based on Agriculture data exchange =

172

173

== //Problem statement// ==

174

175

The problem statement explains some issues faced in agriculture statistics when reporting on non-calendar year observations. The examples are somewhat simplified to focus on the issue at hand.

176

177

In agriculture there are usually three numbers associated with crop production: the planted area, the harvested area and the production in quantity or value. From this, yield may be derived in order to perform economic calculations and use it for other purposes.

178

179

Let us take the example of wheat. In most cases an area is planted in year n and it is harvested in year n+1. As can be seen in the examples below about wheat growing periods in several regions in the world, the reference period usually refers to the year of the harvest (year n+1). From these observations, different indicators may be derived and the relation be made that the planted area of year n is to the production of year n+1.

180

181

To be able to do this without mistakes when looking at the three datasets mentioned (harvested area, planted area, production), the start date, the end date and the year reported by the country are required for each dataset for comparability reasons. The Start date and end date of the observation will allow the data to be linked to perform calculations and analysis. The reported year is important to determine how the country reports and publishes its data and to compare and explain differences between the published data of each country and, for instance, data published by an international organisation.

182

183

Also, the growing periods for wheat may change according to the weather. Sometimes the planting and harvesting may happen during the same calendar year and sometimes not. The growing period might be a little shorter or longer. It would be good to know for each year according to the length of the growing period how the country codes the data: when the season goes beyond 31 Dec, they might use year n or year n+1.

184

185

In many countries, data are published with the year written as periods (e.g. 2009-10 for annual data about the 2009-2010 period). This notation is also commonly used in education statistics, but is not part of the standard observational time-period format defined in SDMX.

186

187

== //Proof of concept DSD// ==

188

189

Note that the DSD proposal below does not constitute a real usable DSD for agriculture statistics. It is heavily simplified to only include concepts that are relevant for the purpose of these guidelines. Also the SDMX fragments shown in the example are not syntactically correct. They just serve the purpose of explaining the issue and show the coding in a readable way (pseudo code).

190

191

|**Concept ID**|**Description**|**Role**|**Code List / Format**

192

|**FREQ**|Frequency|Dimension|A (annual), S (Half-yearly)

193

|**REF_AREA**|Reference area|Dimension|IN (India), CN (China)

194

|**CROP**|Crop|Dimension|W (Wheat)

195

| |Crop

196

(alternative coding)|Dimension|W (Total Wheat), WW (winter wheat), SW (summer wheat)

197

|**ACTIVITY**|Activity|Dimension|P (planting), H (harvesting)

198

|**REPYEARSTART**|Reference year start|Attribute

199

(optional, series)|xs:gMonthDay

200

|**REPYEAREND**|Reference year end|Attribute

201

(optional, series)|xs:gMonthDay

202

203

== //Usage of guidelines// ==

=== India ===

Planting: wheat takes place in October to December (e.g. 2010)

208

209

Harvest: during the months of February and May (e.g. 2011).

210

211

Assumption: India attributes both activities to “crop year” 2011, which goes from Oct 2010-May 2011

**Example data:**

Oct-Dec 2010: 100 units planted

216

217

Feb-May 2011: 100 units harvested

218

219

Oct-Dec 2011: 200 units planted

220

221

Feb-May 2012: 200 units harvested

**Coding:**

Obs TIME_PERIOD=2011-A1 VALUE=100

227

Obs TIME_PERIOD=2012-A1 VALUE=200

Obs TIME_PERIOD=2011-A1 VALUE=100

232

Obs TIME_PERIOD=2012-A1 VALUE=200

233

234

235

The end of the reporting year is the end of May (attribute REPYEAREND). By definition, the year 2011 would thus go from 1^^st^^ June 2010 – 31^^st^^ May 2011. The real crop year is shorter, but since it is the same for every year and always within that time range, this information is sufficient to be available as reference metadata.

=== China ===

Winter wheat is planted from mid-September through October and harvested from mid-May through June. Summer wheat is planted from mid-March through April and harvested from mid-July to mid-August.

240

241

Assumption: China attributes both activities to a “crop year”, which goes from September of the previous until May of the current year.

242

243

**Sample Data for China**

244

245

(% style="width:918.446px" %)

246

|(% style="width:121px" %)**Crop year**|(% style="width:165px" %)**Season**|(% style="width:191px" %)**Calendar period**|(% style="width:438px" %)**Units**

247

|(% style="width:121px" %)**2011**|(% style="width:165px" %)Winter|(% style="width:191px" %)Sep-Oct 2010|(% style="width:438px" %)100 planted

248

|(% style="width:121px" %) |(% style="width:165px" %) |(% style="width:191px" %)May-Jun 2011|(% style="width:438px" %)100 harvested

249

|(% style="width:121px" %) |(% style="width:165px" %)Summer|(% style="width:191px" %)Mar-Apr 2011|(% style="width:438px" %)200 planted

250

|(% style="width:121px" %) |(% style="width:165px" %) |(% style="width:191px" %)Jul-Aug 2011|(% style="width:438px" %)200 harvested

251

|(% style="width:121px" %)**2012**|(% style="width:165px" %)Winter|(% style="width:191px" %)Sep-Oct 2011|(% style="width:438px" %)120 planted

252

|(% style="width:121px" %) |(% style="width:165px" %) |(% style="width:191px" %)May-Jun 2011|(% style="width:438px" %)120 harvested

253

|(% style="width:121px" %) |(% style="width:165px" %)Summer|(% style="width:191px" %)Mar-Apr 2012|(% style="width:438px" %)220 planted

254

|(% style="width:121px" %) |(% style="width:165px" %) |(% style="width:191px" %)Jul-Aug 2012|(% style="width:438px" %)220 harvested

255

256

== //Recommendations based on the Use Cases from Agriculture// ==

257

258

For coding the SDMX message, the frequency of the data is half-yearly, since we have two seasons per year. The reporting year end (31^^st^^ August) corresponds to the whole reporting year, ranging from 1^^st^^ September Y-1 until 31^^st^^ August Y. The exact duration of the season would, in this example, only be available in metadata.

259

260

261

Obs TIME_PERIOD=2011-S1 VALUE=100

262

Obs TIME_PERIOD=2011-S2 VALUE=200

263

Obs TIME_PERIOD=2012-S1 VALUE=120

264

Obs TIME_PERIOD=2012-S2 VALUE=220

Obs TIME_PERIOD=2011-S1 VALUE=100

269

Obs TIME_PERIOD=2011-S2 VALUE=200

270

Obs TIME_PERIOD=2012-S1 VALUE=120

271

Obs TIME_PERIOD=2012-S2 VALUE=220

272

273

274

To calculate the total annual planting of crop, the half-yearly series are added up to annual series (2011=2011-S1+2011-S2 à P: 100+200=300; H: 100+200=300).

275

276

277

Obs TIME_PERIOD=2011-A1 VALUE=300

278

Obs TIME_PERIOD=2012-A1 VALUE=340

279

280

Obs TIME_PERIOD=2011-A1 VALUE=300

281

Obs TIME_PERIOD=2012-A1 VALUE=340

282

283

284

If the alternative coding of Winter Wheat (WW) and Summer Wheat (SW) is used, the frequency becomes annual and values can also be added up separately:

285

286

287

Obs TIME_PERIOD=2011-A1 VALUE=100

288

Obs TIME_PERIOD=2012-A1 VALUE=120

Obs TIME_PERIOD=2011-A1 VALUE=100

293

Obs TIME_PERIOD=2012-A1 VALUE=120

Obs TIME_PERIOD=2011-A1 VALUE=200

298

Obs TIME_PERIOD=2012-A1 VALUE=220

Obs TIME_PERIOD=2011-A1 VALUE=200

303

Obs TIME_PERIOD=2012-A1 VALUE=220

304

305

306

For the total annual planting of crop, the winter and summer wheat can be added (W=WW+SW) and lead to the same result as above:

307

308

309

Obs TIME_PERIOD=2011-A1 VALUE=300

310

Obs TIME_PERIOD=2012-A1 VALUE=340

311

312

Obs TIME_PERIOD=2011-A1 VALUE=300

313

Obs TIME_PERIOD=2012-A1 VALUE=340

314

315

=== Date range coding example ===

316

317

As it can be seen in the examples above, the exact period when the actual planting / harvesting took place can usually be expressed as reference metadata. However, when adding up figures from different reporting year definitions (e.g. adding India and China for 2011), one needs to pay attention to the different definitions. There is no implicit SDMX solution, because SDMX only expresses the data but does not contain the methodology of how data are added for different reporting periods. If more detailed information is needed; for instance, an algorithm to attribute crop production to exact comparable periods, the exact time ranges need to be known as input to the estimation formulas.

318

319

To take the examples of India from above, slightly modified:

320

321

1st Oct- 27th Dec 2010: 100 units planted

322

323

5th Feb- 17th May 2011: 100 units harvested

324

325

15th Oct- 2nd Jan 2012: 200 units planted

326

327

7th Feb-5th May 2012: 200 units harvested

328

329

It can be coded as follows, to give the data user the maximum information for further processing:

330

331

332

Obs TIME_PERIOD=2011-A1 VALUE=100 TIME_RANGE=2010-10-01/2010-12-27

333

Obs TIME_PERIOD=2012-A1 VALUE=200 TIME_RANGE=2011-10-15/2012-01-02

Obs TIME_PERIOD=2011-A1 VALUE=100 TIME_RANGE=2011-02-05/2011-05-17

338

Obs TIME_PERIOD=2012-A1 VALUE=200 TIME_RANGE=2012-02-07/2012-05-05

339

340

341

Care needs to be taken that the time range for each observation is within the period covered by the reporting period projected on the calendar. For instance, a harvesting range lasting from the beginning of March until the end of June would be invalid, because the reporting period ends at the end of May. In case harvesting would last until June, the reporting year end would need to be changed accordingly.

----

Wiki source code of Guidelines on Non-Calendar Year Reporting of Data