Agent Ingestion Usage in OpenSearch Survey Results

Tue, Aug 23, 2022 · Joshua Bright

First, a huge thank you to all of you who responded to the survey. Understanding how you use agents in your ingestion pipelines helps us prioritize use cases that deliver the most value to the community.

In total, 67 individuals responded to the survey run in July 2022. Following are the results:

  • ~54% of participants said that they still use Beats in their client ingestion pipeline (down from ~66% in June 2021).
    • Of the participants who said that they still use Beats:
      • ~52% are not planning to move off of Beats.
      • ~23% plan to move off in the next 12 months.
      • ~25% are waiting for a feature/solution before moving off.
  • ~46% of participants do not use Beats in their client ingestion pipeline.

The survey also asked the community which agents and modules are most popular in their environments. Below are the results from those who use Beats agents in their client ingestion environment.

Agent/Module Number Who Use
Filebeat w/ Logstash 19
Metricbeat w/ system 16
Filebeat w/ Apache 16
Winlogbeat w/ security 15
Auditbeat w/ auditd 11
Filebeat w/ *SQL (all SQL logs) 10
Filebeat w/ Netflow 10
Filebeat w/ Nginx 10
Auditbeat w/ file integrity 8
Auditbeat w/ system 8
Metricbeat w/ http 6
Metricbeat w/ Kafka 6
Metricbeat w/ *SQL (all SQL logs) 6
Filebeat w/ Cisco 6
Filebeat w/ Kafka 6
Heartbeat 6
Metricbeat w/ Nginx 5
Filebeat w/ IIS 5
Packetbeat 4
Metricbeat w/ IIS 3
Functionbeat 3
Journalbeat 2
Filebeat w/ HAProxy 1
Fortinet 1
Checkpoint 1

So what did we learn?

Thanks to the survey response, the community now has a better understanding of how Beats usage is trending. Last year, 66% of community members were using Beats, which dropped to 54% this year. If all things go as expected with planned migrations in the commuity, Beats usage will drop to 42% in 2023. For those who are still using Beats, the most popular agents are Filebeat, Metricbeat, Winlogbeat, and Auditbeat.

For the ~28% of the community who have no plan to stop using Beats into OpenSearch via Logstash using the OpenSearch Output plugin, users should be aware that Elastic Common Schema (ECS) compatibility mode is turned on by default in Logstash 8.0. If community members encounter ECS compatibility errors, they should disable ECS in their pipeline.