Skip to content

Timeout or Race condition resulting in Cannot run all required containers #114

@sumdog

Description

@sumdog

In our continuous integration environment, I've been seeing errors like the following a lot:

com.example.iw.MigrationSpecs *** ABORTED *** (21 seconds, 591 milliseconds)
  java.lang.RuntimeException: Cannot run all required containers
  at com.whisk.docker.DockerKit.startAllOrFail(DockerKit.scala:59)
  at com.whisk.docker.DockerKit.startAllOrFail$(DockerKit.scala:43)
  at com.example.iw.MigrationSpecs.startAllOrFail(MigrationSpecs.scala:9)
  at com.whisk.docker.scalatest.DockerTestKit.beforeAll(DockerTestKit.scala:21)
  at com.whisk.docker.scalatest.DockerTestKit.beforeAll$(DockerTestKit.scala:19)
  at com.example.iw.MigrationSpecs.beforeAll(MigrationSpecs.scala:9)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)

The test kit I'm using is pretty simple too:

import com.whisk.docker.{DockerContainer, DockerKit, DockerReadyChecker}

import scala.concurrent.Await
import scala.concurrent.duration._

trait DatabaseDockerContainers extends DockerKit {

  val mariaDBContainer = DockerContainer("mariadb:5.5").
    withEnv("MYSQL_DATABASE=dev","MYSQL_USER=dev", "MYSQL_PASSWORD=dev", "MYSQL_ROOT_PASSWORD=dev").
    withPorts(3306 -> None).
    withReadyChecker(DockerReadyChecker.LogLineContains("ready for connections"))

  val cassandraContainer = DockerContainer("cassandra:3").
    withPorts(9042 -> None).
    withReadyChecker(DockerReadyChecker.LogLineContains("Starting listening for CQL clients on"))

  def containerIP(c : DockerContainer) = Await.result(c.getIpAddresses(), 20 second).head

  abstract override def dockerContainers : List[DockerContainer] =
    mariaDBContainer :: cassandraContainer :: super.dockerContainers

}

And my Spces2 is built like so:

import com.whisk.docker.impl.dockerjava.DockerKitDockerJava
import com.whisk.docker.scalatest.DockerTestKit
import org.flywaydb.core.internal.exception.FlywaySqlException
import org.scalatest.{FreeSpec, Matchers}

class MigrationSpecs extends FreeSpec with Matchers with DockerTestKit with DatabaseDockerContainers with DockerKitDockerJava {

  "MySQL migrations" - {

I suspect that DockerKit is failing out too early when waiting for the containers to become alive. I'm going to turn logging up and see if I can reproduce/extract more information. Is there any way to expose the internal timeout and make it configurable?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions