Below you will find pages that utilize the taxonomy term “Pagination”
A Tiny ETL Binary Competes With curl, jq and SQLite in a Cron Job, So Build It That Small
The real competitor is a shell script in a crontab, and it usually looks like this:
curl -s "https://api.example.com/v1/launches?limit=100" \
| jq -r '.data[] | [.id, .name, .net] | @csv' \
| sqlite3 -csv launches.db ".import /dev/stdin launches"
It works on the day you write it. Then the API answers 429 and curl pipes an error page into jq. Or the API has a second page, and the script never asks for it. When the job dies halfway, the rerun inserts the same rows again or trips over the primary key, depending on how the table was made. A field that starts arriving as a string goes unnoticed until a chart looks wrong. Retries, backoff, pagination, incremental state, idempotent writes and schema drift: that’s the list, and shell scripts get every item on it wrong in predictable ways. (To be fair, curl --retry covers the first two.)
Querying USAspending for Federal Contract Awards With a POST Search and No API Key
USAspending publishes every federal award the United States government makes, and the API needs no key, no registration and no authorization header. That combination is rare enough on government data that it is worth saying twice.
The thing that stops most people is the shape of the request. The interesting endpoints are POST, the body is a nested object, and a wrong field name gets you a 400 with little explanation. Here is the working version.
Reading the Launch Library API for Rocket Launch Schedules Without a Key
The Space Devs run Launch Library, a REST API covering orbital and suborbital launches worldwide, past and upcoming. No key, no signup, and a forward schedule that currently runs to 369 launches. If you need a dated calendar of something genuinely interesting, this is one of the easiest open APIs to consume.
The Endpoints
Base URL carries the version in the path:
https://ll.thespacedevs.com/2.3.0/
The three that matter:
launches/upcoming/ everything scheduled ahead
launches/previous/ everything flown
launches/ both
events/upcoming/ spacewalks, dockings, test fires
A minimal call:
Pagination Strategies for Large Datasets and Why Offset Pagination Fails
Offset pagination — the pattern where a consumer requests a page by specifying how many records to skip — is the default choice for most APIs because it maps naturally to SQL’s LIMIT and OFFSET clauses and allows consumers to request any page directly by number. It is also the pagination strategy that fails most visibly at scale and produces the most confusing behavior when underlying data changes between page requests.