Skip to content

Caddy access logs broken on Kubernetes due to Supervisord stdout prefix #41876

@tomgrossman

Description

@tomgrossman

Description

When running Appsmith on Kubernetes (fat container with Supervisord), Caddy access logs are unusable for structured log ingestion because Supervisord prepends editor stdout | to every line.

Caddy outputs clean JSON:

{"level":"info","ts":1780555371,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"POST","uri":"/api/v1/actions/execute","headers":{"X-Goog-Authenticated-User-Email":["user@example.com"],"Referer":["https://app.example.com/app/my-app/page?branch=main"]}}}

But in the container's stdout (which is what Kubernetes log collectors pick up), it becomes:

editor stdout | {"level":"info","ts":1780555371,...}

This breaks auto-detection of JSON logs by FluentBit/Fluentd/GKE's managed logging agent. The logs land as unstructured textPayload strings instead of queryable jsonPayload objects. Key fields like the authenticated user email, the Appsmith page/action executed, HTTP status, and response duration are buried and require regex extraction to use.

Expected Behavior

Caddy access logs should land as structured JSON in Kubernetes log collectors (Cloud Logging, Datadog, ELK, etc.).

Suggested Fix

Add an environment variable (e.g. APPSMITH_CADDY_LOG_OUTPUT) that controls the Caddy log output destination in caddy-reconfigure.mjs. Default to stdout for backwards compatibility, but allow setting it to file /proc/1/fd/1 to bypass Supervisord's stdout capture and write directly to the container's stdout file descriptor.

In caddy-reconfigure.mjs, the change would be:

// Current (hardcoded):
log {
  output stdout
}

// Proposed:
log {
  output ${process.env.APPSMITH_CADDY_LOG_OUTPUT || "stdout"}
}

Setting APPSMITH_CADDY_LOG_OUTPUT=file /proc/1/fd/1 would bypass Supervisord entirely, and Kubernetes log collectors would see clean JSON.

Environment

  • Appsmith CE v2.1 (fat container)
  • Kubernetes: GKE Autopilot
  • Helm chart: official Appsmith chart v3.8.1
  • Log collector: GKE managed FluentBit (not customizable on Autopilot)

Workaround

Currently we use Log Analytics SQL queries to strip the prefix and parse the JSON manually:

SELECT
  JSON_VALUE(SUBSTR(text_payload, 17), '$.request.headers.Cf-Access-Authenticated-User-Email[0]') AS user_email,
  JSON_VALUE(SUBSTR(text_payload, 17), '$.request.headers.Referer[0]') AS appsmith_page
FROM logs
WHERE text_payload LIKE 'editor stdout%'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions