Elastic log storage costs: LogsDB, pattern_text and tiers

Logs grow faster than budgets. A few months after deploying Elastic, a team usually finds that the biggest cost is not the licences but the disks and nodes holding data older than a week, which hardly anyone looks at. Elastic has several mechanisms that can reduce this without cutting retention. Below we describe four, in the order in which we usually check them, and we mark which ones need an upgrade to 9.x.

1. LogsDB: an index mode for logs

LogsDB is an index mode designed for log data. The Elastic documentation says that in benchmarks it reduces storage by up to 60%, with an impact of the order of 10 to 20% on indexing performance. The result depends on the data and the version, so treat these figures as an upper bound to check, not a promise.

An important detail for companies after an upgrade: in Elastic Stack 9.0 and later LogsDB is the default for new logs-*-* data streams, but in a cluster upgraded from 8.x it applies only if no such data stream existed at the time of the upgrade. Existing data streams from 8.x (including those of integrations and APM) do not switch by themselves. If you upgraded a cluster from 8.x, you may be paying for logs without this saving.

You enable it by setting index.mode to logsdb in an index template, or in the @custom component for an integration:

PUT _component_template/logs-system.syslog@custom
{
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  }
}

The change takes effect at the next rollover, so existing indices stay as they were. The documentation warns that a PUT request overwrites an existing template, so retrieve the current configuration first. The component name above is an example: check the right name for your integration.

2. pattern_text for messages

The pattern_text field splits repetitive messages into a template and variables. In Elastic Stack it has had GA status since version 9.3, and Elastic claims savings of up to 50% of space on message fields. We described the limitations (no span queries, a constant score for phrases, one value per document) in the article on Elastic 9.3, together with a way to test it on a data sample. It is worth running that test together with LogsDB, because the two mechanisms work on different parts of a document.

3. Data tiers and searchable snapshots

Logs have a simple profile: the newest data is read often, older data rarely. Elastic reflects this in data tiers:

Tier	Data	What the documentation says
Hot	newest, read most often	fast disks, required
Warm	the last few weeks, read less often	weaker hardware than hot, replicas recommended
Cold	rarely searched	fully mounted searchable snapshots, no replicas, disk use about 50% lower
Frozen	almost never read, never updated	partially mounted snapshots, up to 20 times the capacity of warm, slower searches

Movement between tiers is managed by ILM or by data stream lifecycle. An example policy in which data moves to the cold tier after 30 days, to frozen after 90, and is deleted after a year:

PUT _ilm/policy/logs-retention
{
  "policy": {
    "phases": {
      "hot": {
        "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "repo-logs" } }
      },
      "frozen": {
        "min_age": "90d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "repo-logs" } }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

The thresholds of 30, 90 and 365 days are only an example: choose them according to how long you really search your logs and what your retention requirements are. Searchable snapshots need a snapshot repository, for example in object storage. Check that your subscription covers this feature before you base an architecture on it.

From version 9.5, data stream lifecycle can move indices to the frozen tier using the frozen_after setting, which has GA status. This is a simpler alternative to a full ILM policy if you do not need its detailed phases.

4. Metrics and the columnar log mode

In version 9.5 Elastic announces an ES95 codec for metrics, which is meant to bring storage down to about 3 bytes per sample, around 20% less than before. It also describes a columnar mode (Columnar Mode) with a profile for logs. The columnar mode has technical preview status, so it is not yet suitable for production data, but it is a reason to follow later releases. We describe the metrics details in the article on Elastic 9.4 for metrics.

Downsampling of metrics and ILM

If you keep metrics with downsampling as well as logs, remember the change from version 9.4: the downsampling action in ILM no longer runs a force merge by default. If your policy relied on it, add a force merge action or set force_merge_index to true. Without it, indices after downsampling may take up more space than your capacity plan assumes, and you will see the difference only after a few weeks.

How to approach a cost review

Remember that the savings from several mechanisms do not simply add up. The vendor’s results concern specific data sets, and the effect of LogsDB, pattern_text and moving data to cheaper tiers depends on what your logs look like. That is why we measure each step separately, on the same sample of data, before we change the whole cluster.

1. Measure what takes up space. Check the size of indices by log source, for example with GET _cat/indices?v&h=index,docs.count,store.size&s=store.size:desc. Usually a few sources account for most of the data.
2. Check whether LogsDB is active. This is the first step after upgrading to 9.x.
3. Review retention. Do all your logs have to be searchable on hot for 30 days? Usually not.
4. Move older data to cold and frozen, if your subscription allows it.
5. Reduce noise at the source. Debug logs from production and duplicates cost as much as important events.

You can estimate the order of magnitude of Elastic licence and cloud costs in the Elastic Stack cost calculator. We carry out a storage cost review on your cluster as part of log management.