Use a Verbose Local Log and a Filtered Forwarding Path

A single logging path that tries to do forensics and alerting together forces tradeoffs: either you log too much and choke your aggregation platform, or you log too little and miss the trail you need when something fails.

Dual-sink logging splits the problem. Write everything you need locally—full request bodies, timing details, state transitions, even noise that only matters during a dig. Forward a filtered, structured subset to your centralized system where costs accumulate by volume. Each sink serves a different consumer: the local logs answer “what was this thing doing?” The forwarded events answer “should I wake someone up?”

Different consumers need different data shapes

Local logs are cheap to make noisy. Disk I/O on a single machine is fast; the friction is human: you have to know to look there, connect to that host, find the right window, and read a wall of text. The cost is latency and attention. You can afford debug-level detail because you’re only paying for it once, and you only read it if something is wrong.

Centralized logs are the opposite. Every byte forwarded costs pipeline throughput, storage, and query time. A 1 MB/s flood of verbose logs from twenty hosts becomes 20 MB/s ingest, plus replication, compaction, retention costs. The same message repeated across ten instances wastes ten times the money. Centralized systems want structure—key-value pairs, not prose—and they want restraint.

The bridge is filtering. Local logs capture the raw material. Forwarding rules decide what matters to the central system: errors, state changes, resource exhaustion, action confirmations. Everything else stays local, waiting to be useful if you need to drill into that specific host.

Loss and buffering behavior are not symmetric

Losing a local log message means a forensic gap on one machine. You might not know it happened until someone pulls logs days later and finds a discontinuity. The impact is typically delayed—you’re not alerting on it in real time.

Losing a forwarded message is usually worse: it’s an alert you never saw, a metric you never recorded, a resource spike nobody caught. This asymmetry shapes your buffering strategy.

For local logs, write-through or async-with-a-modest-queue usually works. The host has memory and disk. A kernel buffer, an application queue, and a rotating file set absorb most bursts without losing data to normal congestion. Catastrophic loss (disk full, process crash) is acceptable because the next rotation or restart clears the problem.

For forwarding, assume the network or the receiver will be unavailable. Every time you forward an event, you need to know it arrived or you need to retry. The standard move is a durable queue—a local file buffer, a ring buffer with recovery, or a message broker commitment before the event moves from local log to wire. The cost is that forwarded logs lag behind local logs, sometimes by seconds or minutes. That lag is real and needs to be visible in your dashboards: if your centralized system hasn’t seen an event yet, you can’t alert on it yet.

Redaction before forwarding is your trust boundary

The local log contains everything: SQL query parameters, request headers, retry counts, internal cache state. Some of that is sensitive—passwords in debug output, API keys in config dumps, user data that should not leave the host. You cannot forward it safely.

Redaction at the source—when the message is created—is fragile. Developers forget, or they add a new field that should have been redacted but isn’t. Redaction at the forwarder is more reliable: every message gets checked before it leaves the host. The cost is CPU and latency (you’re scanning bytes) but the payoff is a clear trust boundary.

Common patterns: regex patterns for known secrets (API key formats, SSNs), field whitelisting (forward only approved keys), sampling (forward 5% of debug logs, 100% of errors). A field whitelist is the strongest: you explicitly decide what values from your local log are safe to send. Everything else stays home.

Redaction is part of your operational contract. Document which fields are forwarded, which are redacted, and why. If someone looks at your central logs and expects a field that isn’t there, they need to know to check the local log instead.

Test both paths as independent systems

Dual-sink logging introduces testing surface. A bug in forwarding logic doesn’t show up in local logs; it shows up as missing events in the central system. A bug in local logging might leave you blind during the one moment you really need those logs.

Three test cases cover the essentials:

1. Generate an event at the source; verify it appears in the local log. This is a unit test: does the logging call itself work? Does buffering flush? Does the file rotate correctly? No network involved.

2. Generate an event; verify both the local log and a downstream system see it. Spin up a test receiver (syslog listener, HTTP endpoint, Kafka consumer), send events, check both places. Test normal operation and test that the forwarder retries after a network partition.

3. Generate an event containing sensitive data; verify the local log has the full value, but the forwarded event has it redacted or removed. This is your trust boundary test. Send a message with an API key, credentials, or PII. Assert local = full data, forwarded = sanitized.

Automation: These tests should run in your test suite. Mocking is acceptable for local tests (mock the file I/O, assert the call), but forwarding tests need to exercise the actual forwarder code against a test receiver.

Validation checklist

• [ ] Local logs are written synchronously or with a bounded queue; no events drop under normal load
• [ ] Forwarding uses durable buffering; check that a connection failure doesn’t lose events
• [ ] Redaction rules are applied before forwarding; sensitive fields don’t appear in central logs
• [ ] Forwarder lag is documented and visible to ops (a dashboard gauge showing queue depth, or a timestamp comparison)
• [ ] Local log rotation or archival prevents disk exhaustion (explicit size limit, retention policy)
• [ ] Filtering logic is testable; add a test case that verifies which messages forward and which don’t
• [ ] Ops can manually query the local log from any host without knowing a central system exists
• [ ] Forwarding can be toggled or disabled per-host or per-level without redeploying

Tradeoffs and when this breaks

Dual-sink logging works when you have machine-local storage and can accept forwarding lag. It breaks if you’re logging from a container with an ephemeral filesystem: once the container is gone, local logs are gone. In that case, sidecar patterns (shared volume, log router in the same pod) or direct streaming (no local buffer) replace the dual-sink model.

It also breaks if you log in an environment without persistent identity per machine—if container IDs are random and you can’t correlate logs across reboots, your local logs become orphaned noise. You need either stable hostnames or a mechanism to query logs by process ID and short-term context.

The filtering step adds complexity. If your filtering rules are buggy, you either under-forward (missing alerts) or over-forward (wasting money). Keep the forwarder logic simple and test it hard.

Dual-sink is not a replacement for good structured logging. If your local logs are freeform prose, they’re still hard to parse when you need them. The pattern works best when both sinks use consistent formats: structured fields locally, a subset forwarded.

Grounded takeaway

Logging is not one problem; it’s at least two. Local logs support deep debugging on known hosts. Centralized logs support alerting and correlation across a fleet. Trying to optimize for both in a single path usually means compromising on either cost or visibility. A dual-sink design acknowledges that the two consumers have different needs. Write verbosely where it’s cheap, forward selectively where it matters, and test both paths independently.