OpenSearch clusters are capable of ingesting massive amounts of data through a number of different HTTP APIs. How to use the ingestion APIs efficiently however, we leave for our users to figure out. Turns out, sizing bulk requests is actually kind of hard. Or: sometimes ingestion will succeed at first, and then within a few minutes OpenSearch complains that there are too many requests. Usability of the ingestion APIs is… lacking.
A while back OpenSearch Project core members suggested to introduce a streaming flavor to ingestion, in an attempt to provide an easy-to-use, efficient and modern mechanism to bringing data into OpenSearch clusters. And indeed we’ve started down that road. This talk will cover what we have tried, retried and retired, to finally find the way to make it work.
