Platform Usage Limits
FINRA reserves the right to limit the number of requests made by an API Client, the number of records returned in a response or the size of a response payload, as well as other parameters as necessary to ensure the reliability, performance, and integrity of the API platform.
The API platform enforces the following throttling limits:
- Sychronous requests: 20 requests per second per IP address.
- Asynchronous requests (authenticated): 20 requests per hour per dataset per API account.
- Asynchronous requests (anonymous): 4000 requests per hour per dataset across the platform.
Responsible Use Guidelines
Maintaining a reliable, performant API platform is a joint responsibility of the FINRA API team and all API users. Adhering to the following guidelines will help ensure the integrity of the platform:
- Only access as much data as needed to accomplish your objective or use case. Use compare filters and the other POST filter parameters to reduce the amount of data returned in a response whenever possible.
- Utilize a GET request only when trying to sample a small number of records using the limit parameter. Otherwise, the POST request should be used in order to filter the data on each request. NOTE: This guideline does not apply when using asynchronous requests and polling for the async result or accessing the async response file, which should utilize GET requests.
- Understand the data you are trying to access. In many cases, the data is static after a period of time and there is no need to repeatedly download historical data. For example, most of the data available via the Market Transparency API is generally static after one year. Therefore adjust your access patterns to avoid downloading historical data over one year old on a recurring basis.
- Develop sound algorithms that access available data in smaller traunches rather than simply accessing as many records as possible on each requests. Each dataset offers a number of fields that can be used in filters such as trade dates, week and month start dates, tier levels, CRD numbers, etc.
- Utilize asynchronous requests whenever accessing a large number of records as this will allow you to obtain more records per API request while helping to protect the platform.
- Utilize asynchronous requests in combination with the available historical datasets (e.g. weeklySummaryHistorical).
- Brute force algorithms that simply run 24*7*365 and continuously access the same data are never appropriate and are subject to being blocked.
Response Size Limits
The API platform enforces two limits regarding the maximum amount of data returned by an API request:
- Maximum Record Limit: the maximum number of records returned by an API request.
- Maximum Payload Size: the maximum payload size returned in the response payload.
- Whichever limit is reached first will govern the amount of data returned.
The API platform limits the number of records that can be returned from a individual API request:
- Synchronous requests will return a maximum of 5000 records for any one API request
- Asynchronous requests will return a maximum or 100,000 records for any one API request.
NOTE : The API record limit defaults to 1000 unless a larger number is specified using the limit query string (GET) or request parameter (POST).
Payload Size Limit
The API platform limits the amount of data that can be returned in the body of an individual API request:
- Synchronous requests will return a maximum of 3MB of data in the response body of any one API request.
- Asynchronous requests can return any amount of data and are not limited to the 3MB response body size.
Working with Large Datasets
When working with large datasets that contain more records than the maximum record limits (5,000 synchronous /100,000 asynchronous) the API user must design an algorithm to access the data in traunches that honor the platform usage limits.
The key request parameters that faciliate accessing data in traunches include:
- Limit parameter: Number of records to return. The default value (if limit parameter is not provided on a request) is 1000.
- Offset parameter: Record number to start with (exclusive). For example, if the offset is 0 and the limit is 20, then records 1 to 20 are returned for a total of 20 records. If the offset is 10 and the limit is 10, then records 11 to 20 are returned.
The offset parameter has a maximum value of 500,000. The effect of this limit is that a maximum of 505,000 records can be accessed synchronously, and a maximum of 600,000 records can be accessed asynchronously, without the use of additional filters. For example, the weeklySummary dataset contains more than 20 million records. Because of the 500,000 offset limit it is not possible to access all 20M+ records simply by increasing the offset parameter until all data is accessed.
Instead the API user would need to utilize filters in conjunction with the limit and offset parameters to access the entire range of records in the dataset. In the case of the weeklySummary dataset, for example, the API user might filter the dataset by the weekStartDate and tierIdentifier fields to reduce the overall size of the data and then page through that traunch of data using the limit and offset parameters as needed. The data access algorithm can then step through the available weekStartDate and tierIdentifier values to access the entire dataset in a manner consistent with the platform usage limits.
There are a number of response headers that can be used to page through larger datasets in compliance with the response size and offset limits, including:
- Record-Total response header: Total records found at the time of the request. Use this value when paging through large datasets to determine when all data has been retrieved.
- Record-Offset response header: Set to match the offset value provided with the request.
- Record-Limit response header: Set to match the limit value provided with the request or the platform maximum limit, whichever is smaller.
- Record-Max-Limit response header: Returns the current maximum number of records that will be returned by the API platform. Use to build a flexible integration that can adapt to future record limit changes.
- Response-Payload_Max_Size response header: Returns the current maximum payload size (in MB) returned by the API platform. Use to build a flexible integration that can adapt to future payload response size changes. Applies only to synchronous API requests.
- To maximize the number of records returned in a response minimize the number of fields you return and use the text/plain Accept header on your requests whenever possible (e.g. the Market Transparency datasets support text/plain).
- API clients should count the number of records returned in each response in order to ensure all expected data is returned, and to adapt future requests if less data than expected was returned because the record and/or the response payload size limits are reached.