feat: add JSON log handler for Logstash / Elasticsearch ingestion

[tsu-shiuan]

Summary

Adds Zexbox.Logging.install_json_handler!/1 (and the underlying Zexbox.Logging.JsonHandler). Swaps the default :logger handler's formatter for a JSON one wrapping LoggerJSON.Formatters.Basic. Every log event becomes a single JSON line so multi-line content (Elixir struct inspections, multi-line SQL, stack traces) collapses into one Elasticsearch document at the ingest layer instead of fanning out into many.

This is the Phoenix / Elixir equivalent of opsbox's JsonFormatter for Ruby.

Why

Today filebeat ships every line of every container as its own ES document. Phoenix apps emit multi-line plain text (default Logger.bare_format), so a single struct inspection or SQL INSERT … VALUES (…), (…), … statement materialises as dozens of indexed docs. Today's production analysis showed live-api (the syndicated-api in our index) emitting ~2.9M docs/day from this fan-out alone, and data-stories ~580k/day from per-line %Phoenix.Socket{} struct dumps.

The Ruby pipeline doesn't have this problem because opsbox's JsonFormatter already emits one JSON line per event, and Logstash's if [kubernetes][container][name] == "rails" block JSON-parses each line into structured [log] fields. This PR brings that same pattern to Phoenix.

Usage

In config/runtime.exs of the consuming Phoenix app:

if config_env() == :prod do
  Zexbox.Logging.install_json_handler!()
end

Output (one line per event):

{"time":"2026-05-02T01:23:45.678Z","severity":"info","message":"...","metadata":{...}}

Optional metadata allow-list:

Zexbox.Logging.install_json_handler!(metadata: [:request_id, :trace_id, :user_id])

Companion changes (not in this PR)

End-to-end the pipeline needs two more changes, each independently safe:

1. kubernetes-infra Logstash config — extend the existing kubernetes.container.name == "rails" block in 02_config_maps.yml.j2:103-155 to also match the phoenix and elixir container names. The existing JSON parser handles the shape this handler emits.
2. Per-app rollout — 3-line runtime.exs change. skip_on_invalid_json => true is already set in the Logstash json filter, so a Phoenix container still emitting plain text (pre-rollout) is a safe no-op.

The intended sequence: ship this gem → release → ship the kubernetes-infra change → roll Phoenix apps one at a time. Each phase is reversible.

Changes

• lib/zexbox/logging/json_handler.ex — new module, single public function install!/1.
• lib/zexbox/logging.ex — exposes install_json_handler!/1 as a delegate, mirroring the existing attach_controller_logs!/0 pattern.
• mix.exs — adds {:logger_json, "~> 7.0"} and {:jason, "~> 1.4"} as runtime deps; bumps :elixir constraint from ~> 1.14 to ~> 1.15 to match logger_json 7.x's requirement. Of the consuming Phoenix apps, only narrator is still on ~> 1.14 in its mix.exs and can bump trivially; live-api, data-stories, brand_service, zappi-api are already on 1.18+ and data-api on 1.15.
• mix.lock — locks logger_json 7.0.4.
• README.md — documents the new ### JSON-formatted logs for Logstash / Elasticsearch section.
• CHANGELOG.md — Unreleased entry; left version bump to whoever cuts the release.

Tests

test/zexbox/logging/json_handler_test.exs — 6 cases covering formatter swap, default options, metadata allow-list, redactor passthrough, idempotency, parent-module delegate. The setup block snapshots and restores the default handler config so the tests don't bleed into other suites.

Local run: 37 tests, 0 failures across the full zexbox suite.

Risk

Low. New code path with no implicit activation — apps opt in via the explicit install_json_handler!() call. The Elixir constraint bump is the only consumer-visible change in this PR; nothing existing in zexbox's API breaks.

Claude, but not flawed

[brendon9x]

Top-level question: should this wrapper exist at all?

The biggest piece of feedback isn't about the implementation — it's about whether the install_json_handler!/1 API needs to exist in the first place.

The Elixir ecosystem norm is declarative config, not runtime installers. Phoenix, Ecto, Plug, Telemetry all document a config snippet to paste; none of them ship a Foo.install!/1 wrapper. logger_json itself tells users to write this in config/runtime.exs:

config :logger, :default_handler,
  formatter: {LoggerJSON.Formatters.Basic, %{}}

That is the entire setup. Concrete downsides of the runtime-call approach in this PR vs. pure config:

1. Late activation. :logger is configured before the application starts. Anything logged during app boot — supervisor start, dependency init, early telemetry — goes out in the old text format until install_json_handler!() runs. Pure config makes JSON active from the very first log line.
2. Hidden seam. A new engineer looking at config/runtime.exs to understand why logs look the way they do sees nothing JSON-related. They have to know to grep application code for install_json_handler.
3. API surface for two lines of config. Both options (:metadata, :redactors) are passthroughs into a map that would otherwise live in config. The wrapper adds a maintenance burden without abstracting anything.

The legitimate counter-argument is "we want one place to set Zappi-wide defaults so apps don't drift." That's real, but it's better served by either:

• A documented config snippet in the Zexbox README with the standardised defaults, or
• Exposing Zexbox.Logging.json_formatter_config/1 that returns the {LoggerJSON.Formatters.Basic, %{...}} tuple to splat into config — keeping the configuration declarative while still centralising defaults.

Recommendation: drop install!/1 and the delegate. Replace with a README section showing the exact config :logger, :default_handler, ... line plus the standardised redactors/metadata list. If centralised defaults are wanted, expose them as a function that returns config, not one that mutates :logger at runtime.

The remaining two notes assume the wrapper survives this discussion.

---

The ! is wrong for the return type

lib/zexbox/logging/json_handler.ex declares:

@spec install!(keyword()) :: :ok | {:error, term()}

and the moduledoc explicitly documents the {:error, reason} return. In Elixir, ! means "raises on failure." A function that returns {:error, _} must not be named with a bang. Same applies to the defdelegate ... as: :install! in lib/zexbox/logging.ex.

Pick one:

• Drop the bang: install/1 :: :ok | {:error, term()}, or
• Keep the bang and actually raise on {:error, reason} (e.g. wrap the :logger.update_handler_config/3 call in a case that raises a RuntimeError with the reason).

---

Don't pin :jason as a direct dep

mix.exs adds {:jason, "~> 1.4"} alongside {:logger_json, "~> 7.0"}. Per the logger_json docs:

By default, LoggerJSON is using Jason as the JSON encoder. If you use Elixir 1.18 or later, you can use the built-in JSON module as the encoder. To do this, you need to set the :encoder option in your config.exs file. This setting is only available at compile-time:

config :logger_json, encoder: JSON

Pinning :jason in zexbox pushes the encoder choice down onto every consuming app, even those on Elixir 1.18+ that would prefer the stdlib JSON module. That's exactly the kind of constraint a base library shouldn't impose.

Recommendation: drop the :jason line from mix.exs. Let consumers pick their encoder via config :logger_json, encoder: ….

[brendon9x]

As per comment. I don't think this needs to exist other than docs.

Maybe an Igniter task but that feels like overkill.

[Hermanlangner]

As per comment. I don't think this needs to exist other than docs.

Maybe an Igniter task but that feels like overkill.

An igniter task sounds like a good starting point for having the utility of "Set up my application for observability". It should be easy enough to add with a ton of benefits