Audit Log System Design#

Every production system needs audit logs. They answer the fundamental question: who did what, when, and where? Whether you are building for SOC 2 compliance, debugging production incidents, or satisfying legal discovery requests, a well-designed audit log system is non-negotiable.

What to Log#

An audit log entry must capture five dimensions:

Additionally, capture:

• Before/after state — for update operations, store the diff or full snapshots
• Request context — correlation ID, session ID, user agent
• Result — success or failure, error codes

What NOT to Log#

• Passwords, tokens, or secrets (even hashed)
• Full credit card numbers or SSNs
• PHI (Protected Health Information) unless the log system itself is HIPAA-compliant

Redact sensitive fields before writing. Use allowlists rather than denylists — explicitly define what gets logged.

Immutable Append-Only Storage#

Audit logs must be immutable. No one — not even administrators — should be able to modify or delete entries. This is the cornerstone of trustworthy auditing.

Storage Options#

• Append-only databases — Amazon QLDB (purpose-built for ledger workloads)
• Object storage with locks — S3 with Object Lock in compliance mode
• Write-once tables — PostgreSQL with row-level security and no UPDATE/DELETE grants
• Dedicated log stores — Elasticsearch with ILM (Index Lifecycle Management) policies that prevent deletion during retention windows

Schema Design#

CREATE TABLE audit_logs (
  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  timestamp    TIMESTAMPTZ NOT NULL DEFAULT now(),
  actor_id     TEXT NOT NULL,
  actor_type   TEXT NOT NULL,  -- 'user', 'service', 'system'
  action       TEXT NOT NULL,
  resource_type TEXT NOT NULL,
  resource_id  TEXT NOT NULL,
  metadata     JSONB,
  before_state JSONB,
  after_state  JSONB,
  source_ip    INET,
  correlation_id UUID,
  checksum     TEXT NOT NULL
);

Use partitioning by month or week for large-scale deployments.

Event Sourcing for Audits#

Event sourcing naturally produces an audit trail. Every state change is stored as an immutable event. The current state is derived by replaying events.

Event 1: InvoiceCreated { id: "inv_123", amount: 500, customer: "acme" }
Event 2: InvoiceUpdated { id: "inv_123", amount: 550 }
Event 3: InvoicePaid    { id: "inv_123", paid_at: "2026-03-28" }

This approach gives you a complete history with zero additional effort. However, it comes with trade-offs:

• Read complexity — reconstructing current state requires replaying events (mitigated with snapshots)
• Schema evolution — event schemas must be versioned and forward-compatible
• Storage growth — events accumulate indefinitely

Event sourcing works best when audit is a primary requirement, not a bolt-on feature.

Structured Logging#

Audit logs must be machine-parseable. Use structured formats (JSON) rather than free-text log lines.

{
  "timestamp": "2026-03-28T14:32:01.442Z",
  "level": "audit",
  "actor": { "id": "usr_92x", "type": "user", "email": "jane@acme.com" },
  "action": "invoice.delete",
  "resource": { "type": "invoice", "id": "inv_8xk2p" },
  "result": "success",
  "source_ip": "192.168.1.42",
  "correlation_id": "req_abc123"
}

Naming Conventions#

Use a consistent resource.action naming scheme:

• invoice.created, invoice.updated, invoice.deleted
• user.login, user.login_failed, user.password_changed
• permission.granted, permission.revoked

This makes filtering and alerting trivial.

Compliance Requirements#

SOC 2#

SOC 2 Type II requires demonstrating that audit logging is continuous and tamper-resistant. Key controls:

• All access to sensitive data is logged
• Logs are retained for a defined period (typically 1 year)
• Log integrity is verifiable
• Access to logs is restricted and itself audited

HIPAA#

For healthcare applications, HIPAA mandates:

• Logging all access to PHI
• Tracking disclosures to third parties
• Retaining logs for 6 years minimum
• Regular review of audit logs by a compliance officer

GDPR#

GDPR adds complexity — audit logs containing personal data are themselves subject to data protection rules. You may need to pseudonymize actor identities and support erasure requests with careful carve-outs for legal obligations.

Querying and Searching Logs#

Audit logs are useless if you cannot search them efficiently. Design for these query patterns:

• By actor — "Show me everything user X did in the last 30 days"
• By resource — "Show me all changes to invoice #123"
• By time range — "What happened between 2 AM and 4 AM on March 15?"
• By action type — "Show all deletion events across the system"

Technology Choices#

For most systems, start with PostgreSQL. Migrate to Elasticsearch when query complexity or volume demands it.

Indexing Strategy#

At minimum, index on:

• actor_id + timestamp
• resource_type + resource_id + timestamp
• action + timestamp
• correlation_id

Retention Policies#

Not all logs need to live forever. Define retention tiers:

Automate transitions with lifecycle policies. Never manually delete audit logs — use automated, policy-driven expiration.

Tamper-Proof Hash Chains#

To prove logs have not been altered, chain entries using cryptographic hashes:

Entry N:
  data: { ... }
  previous_hash: "sha256_of_entry_N-1"
  hash: sha256(data + previous_hash)

This creates a blockchain-like structure where modifying any entry breaks the chain. Periodically anchor the chain head to an external timestamping service or a public blockchain for independent verification.

Verification#

Run a verification job nightly:

1. Read entries sequentially
2. Recompute each hash from data + previous hash
3. Compare with stored hash
4. Alert on any mismatch

Architecture Overview#

Application → Audit SDK → Message Queue → Audit Writer → Storage
                                              ↓
                                        Hash Chain Computer
                                              ↓
                                        Search Index (ES)

Key principles:

• Decouple writing from the application — use async messaging so audit logging never blocks business logic
• Separate read and write paths — write to append-only storage, replicate to search index for queries
• Encrypt at rest and in transit — audit logs often contain sensitive metadata

Tools and Platforms#

• AWS CloudTrail — managed audit logging for AWS API calls
• Elastic SIEM — search and alerting on audit data
• Panther — cloud-native SIEM with compliance packs
• Teleport — infrastructure access audit logging
• OpenTelemetry — can be extended for audit event collection

Key Takeaways#

1. Capture who, what, when, where, and on what for every significant action
2. Store logs in append-only, immutable storage
3. Use hash chains to prove tamper resistance
4. Design for compliance from day one — retrofitting audit logs is painful
5. Index for the query patterns your compliance and security teams actually need

A well-designed audit log system is not just a compliance checkbox. It is the foundation of trust, accountability, and operational visibility.

---

Want to explore more system design topics? Visit codelit.io for interactive guides and tools.

This is article #217 in the Codelit engineering blog series.