Learn what API paging limits are, how Intrinio uses paging limits, and how you can write code to retrieve all pages.
If you're a developer using Intrinio's API, you will have noticed that for many data types, the results are limited to a specific number per page. This article answers these questions about API paging limits:
- What are API paging limits?
- What are Intrinio's paging limits?
- How can you write code to get all of the pages?
What are API paging limits?
When you make an API call in your code, you are requesting data. Paging limits put a threshold on the amount of data you can get in a single API call. For example, you can use this API call to request Apple's stock price history from Intrinio:
https://api-v2.intrinio.com/securities/AAPL/prices
There are more than 30 years of daily price history for Apple. In order for our API to provide timely and efficient service to every user, we will only return 100 price entries at a time. This results in multiple "pages" of data, allowing developers to request the data one page at a time instead of all at once.
What are Intrinio's paging limits?
Intrinio's API documentation shows three different paging limits depending on the data type you want to access:
- Historical Data: 1,000 results per page
- File Downloads to CSV: 10,000 results per page
- All Others: 100 results per page
As you can see, most endpoints are limited to 100 results per page. Historical data allows for 1,000 results per page because the size of each entry is relatively small. Historical data is the quickest way to get historical values for the data point you need. CSV downloads allow for 10,000 results per page. However, because CSV downloads are for human use, they are limited to 1 download per second.
How can you write code to get all of the pages?
Every coding language is different and within each coding language there are different packages and techniques for pulling in multiple pages. Additionally, different APIs use different methods for paging. Intercom, for example, uses a "scroll parameter" and Stripe uses a "has more" parameter. For Stripe, if has_more = TRUE, there are more pages and the developer can get the next one.
Intrinio provides the total number of pages as part of the initial API call response. This means that you can write a loop that will continue to make API calls until you reach the total number of pages. If you loop through multiple pages, each page will count as a separate API call.
In whatever programming language you use, you can use this initial API call to determine how many pages there are. Then you need to make calls that include ?next_page=, with a page number at the end, until you hit that number:
https://api-v2.intrinio.com/securities/AAPL/prices?next_page=[next_page_value]
Usually, making each subsequent API call is handled with a "for" loop. It can also be done with a "while" loop. The logic is that "while" the page number is less than or equal to the total number of pages, repeat the API call and increase the page number by 1. If you add the resulting data from each API call to the data you have, you will end up with each page of data.