How to query large datasets with data services?
The CompactData method has a maximum number of time series that it will return. The GetMaxSeriesInResult method returns the maximum number of time series that can be returned by CompactData. The CompactData will return a truncated response if the query result is greater than the GetMaxSeriesInResult.
The following steps should help avoid this
scenario:
Step 1: Determine the number of possible time series that will be returned by your query. This can be calculated by multiplying the number of elements included in the query for each of the dimensions in the dataset being queried.
In order to get information about elements in a dimension, the DataStructure method returns the structure of the dataset including elements of the dimensions. The CodeList method will return list of elements in a particular dimension.
For example: In the query, the Country dimension includes 135 elements; Indicator dimension includes 20 elements; and Sector has 4 elements. The maximum possible number of time series returned will be 10,800 (= 135 x 20 x 4).
Step 2: If you determined in Step 1, that the maximum number is greater than GetMaxSeriesInResult, consider breaking your query into multiple queries. One strategy would be to loop through one or more of the dimensions to greatly reduce the possible maximum number of time series that can be returned.
Using the same example as above, if you loop through each element of Sector, you will have 4 queries; each will have 2700 (=135 x 20 x 1) as the maximum possible number of time series it could return.
NOTE: This limit applies to data retrieval irrespective of the format (XML or JSON), method (SOAP or REST) or the SDMX standard (2.0 or 2.1).