Skip to content

docker backend: ES readiness doesn't check cluster health, UpOrReuse doesn't handle a paused container #81

Description

@ikeikeikeike

Surfaced by the wave-2 retrospective /code-review sweep of PR #6 (docker backend for all 4 plugins). Two docker-backend readiness/reuse precision gaps (config-drift detection on reuse is already tracked separately as #77):

1. dockerReadyCheck for Elasticsearch treats any HTTP 200 from GET / as ready

plugins/engine/elasticsearch/docker.go's dockerReadyCheck gates readiness on nothing but a plain HTTP 200 from the ES root endpoint — no cluster-health or shard-recovery check, unlike the nix-backend ReadyCheck in elasticsearch.go, which polls _cluster/health?wait_for_status=yellow. ES's root endpoint answers as soon as the HTTP layer is bound, before shard/index recovery of any pre-existing on-disk data has finished. If a container is recreated against a datadir that still holds indexed data from a prior run (e.g. via UpOrReuse's stopped-container branch, or after a crash), a caller that starts querying immediately after bough reports "ready" can hit search_phase_execution_exception / cluster-block errors against indices that exist on disk but aren't yet recovered.

2. UpOrReuse treats a paused container as "already up, reuse it"

pkg/dockerutil/container.go's UpOrReuse() checks only info.State.Running, but Docker keeps Running=true for a paused container too (Paused is a separate, additive flag). A manually-docker paused bough container is reused as-is rather than recreated; the subsequent exec-based readiness probes (mysqladmin/pg_isready/redis-cli/HTTP) then fail on every tick — Docker rejects exec create against a paused container — until bough create times out, instead of bough noticing the container is unusable and cleanly recreating it the way it already does for a merely-stopped container.

Files

  • plugins/engine/elasticsearch/docker.go (dockerReadyCheck)
  • pkg/dockerutil/container.go (UpOrReuse)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions