Benchmarking Elasticsearch with Rally: how to start

Rally (the esrally command) is Elastic’s official tool for Elasticsearch performance testing. It runs a repeatable scenario: it feeds the cluster with data, sends queries and measures throughput and response times. That lets you judge a configuration change, a version upgrade or new hardware by numbers rather than by a feeling that it “seems faster”. Below we explain why to benchmark, how to prepare a test and how to read the report.

Why test a cluster that works?

A cluster that responds quickly today may struggle tomorrow because the data has grown, query patterns have changed or the team has added a new log source. A benchmark answers questions that production monitoring alone cannot settle:

• how many documents per second the cluster accepts before indexing starts to lag;
• how query times change after altering the number of shards, the mapping or the disk type;
• whether an Elasticsearch upgrade makes results worse for your workload;
• how many nodes, and what hardware, you need for the planned data growth.

We usually run this test before a production Elastic platform goes live and before any major change to a cluster that collects logs or data for visualisation.

Core Rally concepts

• Track describes a scenario: which data goes into which indices, which queries are sent and in what order. Rally ships with ready-made tracks, including geonames, http_logs, nyc_taxis and pmc. You can also write your own track from a sample of real data.
• Challenge is a variant of execution within a track, for example indexing only, or indexing and querying at the same time.
• Car is the Elasticsearch configuration that Rally starts when it provisions nodes itself. You do not use it when testing an existing cluster.
• Pipeline defines where Rally gets Elasticsearch from. To test your own cluster, use the benchmark-only pipeline: Rally installs nothing and only generates load.
• Race is a single run of a test. Each race gets an ID, so you can compare it with another later.

How to prepare the test

1. Set up a separate environment. The test generates heavy traffic, so do not run it against a production cluster that serves users. The Rally documentation recommends a dedicated environment with no other traffic, because otherwise the result is skewed.
2. Run Rally on its own machine. The load generator should not compete for CPU with the Elasticsearch nodes. Rally runs on Linux and macOS, and it can test clusters running on other systems.
3. Choose a track that matches your workload. For logs, http_logs works well; for full-text search, pmc; for numeric aggregations, nyc_taxis. If your data is different, build a custom track, because a ready-made scenario only tells you how the cluster handles that scenario.
4. Record the baseline. Elasticsearch version, number of nodes, heap size, disk type, number of shards and replicas. Without this, a month from now you will not know what you are comparing.
5. Change one thing at a time. If you change the shard count and the heap size together, you will not know which one improved the result.

Running a test on an existing cluster

Installation is a single command (it needs Python, git and a JDK suitable for Elasticsearch, with JAVA_HOME pointing to the JDK):

pip3 install esrally

esrally list tracks shows the available tracks. It is a good idea to make the first run with --test-mode, which uses a small part of the data and lets you check that the connection and configuration work:

esrally race --track=http_logs --pipeline=benchmark-only \
  --target-hosts=10.5.5.10:9200,10.5.5.11:9200,10.5.5.12:9200 \
  --test-mode

Only when the trial run succeeds do you start the real benchmark without that option. If the cluster has security enabled, use HTTPS and credentials:

esrally race --track=http_logs --pipeline=benchmark-only \
  --target-hosts=10.5.5.10:9200,10.5.5.11:9200,10.5.5.12:9200 \
  --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'<password>'"

Do not leave the password in your shell history or in scripts in a repository. For tests, create a separate user with limited permissions. The full list of options is in the Rally documentation (Command Line Reference).

How to read the results

At the end Rally prints a report with a dozen sections. Four groups of metrics matter most often.

Metric	What it tells you
Throughput	How many operations per second the cluster performed. Rally reports the minimum, median and maximum. For indexing, this is documents per second.
Service time	The time from sending a request to receiving the response. It shows how fast the cluster handles a single query.
Latency	Service time plus the time a request waits in the queue. It differs from service time when you set a target throughput and the cluster cannot keep up.
Error rate	The share of operations that ended in an error. A result with errors is worthless until you understand their cause.

Look at percentiles, not only the median. The median (50th percentile) shows the typical query, while the 99th percentile shows the small share that users experience as “the system froze”. The report also includes merge, refresh and garbage collector times, plus heap and disk size. Rising GC time at constant throughput usually means the heap is too small or there are too many shards.

A single run says little. Repeat the test at least three times and compare the spread. If results differ by more than about 10 percent with no change to the cluster, the environment is too noisy to detect small differences.

Comparing runs

Rally stores results and lets you compare two runs. esrally list races lists them, and you compare them like this:

esrally compare --baseline=<race-id-before> --contender=<race-id-after>

The report shows the difference for each metric, so you can see at once whether a change helped indexing at the expense of queries, or the other way round. For longer work, keep results in a separate Elasticsearch cluster (a metrics store) configured in ~/.rally/rally.ini. The history of measurements can then be viewed on a Kibana dashboard, and results do not disappear with the test machine.

What a benchmark will not tell you

A Rally result describes the scenario you ran. It does not replace a test with real data and real queries, and a ready-made track does not account for traffic from your dashboards or alert rules. That is why, for larger clusters, we build a custom track from a sample of the client’s data and let it drive sizing decisions. When you plan a new cluster or a migration, we help as part of our Elastic deployment and maintenance.