How to cache Downloading Forge installer and Running Forge installer using Docker multi-stage builds?

[solzard]

I am writing a Dockerfile for martinohanlon/mcpi: Minecraft: Pi Edition API Python Library to teach children Python through manipulating Minecraft. mcpi library requires Minecraft 1.12.2, Forge 14.23.5.2859, and RaspberryJamMod.

Here's my custom Dockerfile:

• mods_installer stage downloads RaspberryJamMod.
• itzg/minecraft-server:java8-multiarch stage copies the mod from mods_installer and build a Minecraft world.

# Dockerfile
FROM busybox:latest AS mods_installer

WORKDIR /mods
ARG MODS_URL=https://github.com/arpruss/raspberryjammod/releases/download/0.94/mods.zip
RUN <<EOF
    wget ${MODS_URL} --output-document ./mods.zip
    unzip -p ./mods.zip 1.12.2/RaspberryJamMod.jar > ./RaspberryJamMod.jar
    rm ./mods.zip
EOF

FROM itzg/minecraft-server:java8-multiarch

WORKDIR /data
COPY --link --from=mods_installer /mods ./mods/

EXPOSE 25565/tcp 4711/tcp

compose.yaml:

• Accept environment variables for host ports (MCPI_PORT, RCON_PORT) and SEED

# compose.yaml
services:
  minecraft-server:
    build:
      context: .
      dockerfile: Dockerfile

    container_name: "mcpi-server"

    ports:
      - "${MCPI_PORT:-25565}:25565"
      - "${RCON_PORT:-4711}:4711"

    volumes:
      - type: volume
        source: mcpi-world
        target: /data

    environment:
      EULA: "TRUE"
      TYPE: "FORGE"
      VERSION: "1.12.2"
      FORGEVERSION: "14.23.5.2859"

      MODE: "creative"

      SEED: "${SEED:-}"

      RCON_CMDS_STARTUP: |-
        time set day
        weather clear
        gamerule doDaylightCycle false
        gamerule doWeatherCycle false

    tty: true
    stdin_open: true
    restart: unless-stopped

volumes:
  mcpi-world:

When I build the image, Docker caches work well and ends instantly (<2s).

> docker compose up --detach --build
[+] Building 0.0s (11/11) FINISHED                                         
 => [internal] load build definition from Dockerfile                  0.0s
 => => transferring dockerfile: 508B                                  0.0s
 => [internal] load .dockerignore                                     0.0s
 => => transferring context: 2B                                       0.0s
 => [internal] load metadata for docker.io/itzg/minecraft-server:jav  0.0s
 => [internal] load metadata for docker.io/library/busybox:latest     0.0s
 => [mods_installer 1/3] FROM docker.io/library/busybox:latest        0.0s
 => [stage-1 1/3] FROM docker.io/itzg/minecraft-server:java8-multiar  0.0s
 => CACHED [stage-1 2/3] WORKDIR /data                                0.0s
 => CACHED [mods_installer 2/3] WORKDIR /mods                         0.0s
 => CACHED [mods_installer 3/3] RUN <<EOF (wget https://github.com/a  0.0s
 => CACHED [stage-1 3/3] COPY --link --from=mods_installer /mods ./m  0.0s
 => exporting to image                                                0.0s
 => => exporting layers                                               0.0s
 => => writing image sha256:e8df177e417d1dc027401da8207eb00772e90396  0.0s
 => => naming to docker.io/library/minecraft-minecraft-server         0.0s
[+] Running 2/2
 ✔ Volume "minecraft_mcpi-world"  Created                             0.0s 
 ✔ Container mcpi-server          Started                             1.6s 

However, after building the image, mc-image-helper (Downloading Forge installer and Running Forge installer) takes 2-3 minutes.

# Docker container log
2023-05-06 15:50:23 [init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x 3 1000 1000 4096 May  6 06:50 /data'
2023-05-06 15:50:23 [init] Resolved version given 1.12.2 into 1.12.2 and major version 1.12
2023-05-06 15:50:23 [init] Starting RCON commands
2023-05-06 15:50:23 [init] Rcon cmds functionality enabled
2023-05-06 15:50:23 [init] Resolving type given FORGE
2023-05-06 15:50:24 [mc-image-helper] 06:50:24.217 INFO  : Downloading Forge installer 14.23.5.2859 for Minecraft 1.12.2
2023-05-06 15:50:45 [mc-image-helper] 06:50:45.329 INFO  : Running Forge installer. This might take a while...
2023-05-06 15:53:10 [init] Creating server properties in /data/server.properties
...
2023-05-06 15:53:23 [Rcon loop] No addition rcon commands are given, stopping rcon cmd service

Is it possible to cache the mc-image-helper part by adding more layers of itzg/minecraft-server stage in the Dockerfile? Alternative solutions are also welcome!

[itzg]

I generally don't recommend building a Dockerfile upon something other than an OS/language base image; however, given how incredibly slow RaspberryPi's are to run, the caching strategy is very useful.

Since /data is a volume and volumes are not persisted in build images, you have to get creative with preparing, persisting, and using the content. Also, no need to re-invent the wheel -- use MODPACK to grab the mods zip.

Let me know if you have any questions about why/what I did in the following example:

FROM itzg/minecraft-server:java8-multiarch AS prep

ARG MODPACK=https://github.com/arpruss/raspberryjammod/releases/download/0.94/mods.zip
# NOTE: didn't resolve when named VERSION
ARG MC_VERSION=1.12.2
ARG FORGE_VERSION=14.23.5.2859

# SETUP_ONLY: go through the motions of processing TYPE, versions, MODPACK, etc
# FORGE_VERSION: important to pin the Forge version so that any drift with the latest version won't escape the cache
# cp: since /data is transient, being a volume, grab into a non-volume path
RUN TYPE=FORGE VERSION=$MC_VERSION FORGE_VERSION=$FORGE_VERSION \
    SETUP_ONLY=true \
    EULA=true \
    MODPACK=$MODPACK \
    /start \
    && cp -r /data /prepped

FROM itzg/minecraft-server:java8-multiarch

# These should match the same in prep RUN above
ARG MC_VERSION=1.12.2
ARG FORGE_VERSION=14.23.5.2859

ENV TYPE=FORGE VERSION=$MC_VERSION FORGE_VERSION=$FORGE_VERSION

COPY --from=prep /prepped /prepped

# NOTE:
# - the final dot in /prepped/. is important, since the prepped content includes
#   hidden, dot-prefixed manifest files.
# - use no-clobber so that prep can be safely run every time
ENTRYPOINT sudo -u minecraft cp -r --no-clobber /prepped/. /data \
    && /start

[solzard]

Thank you for your fantastic help, @itzg!!

Just one troubleshooting.
I tried to convert

ENTRYPOINT sudo -u minecraft cp -r --no-clobber /prepped/. /data && /start

to json style to comply with hadolint DL3025 rule.

However,

ENTRYPOINT ["sudo", "-u", "minecraft", "cp", "-r", "--no-clobber", "/prepped/.", "/data", "&&", "/start"]

fails and its log says cp: target '/start' is not a directory.
None of ";", "/start", "&&", "sh", "/start", "&&", "bash", "/start", "&&", "/usr/bin/sh", "/start", "&&", or "usr/bin/bash", "/start" worked.

Do you know why I could not convert /start part to the json style?

[itzg]

That lint rule is not accounting for shell form

https://docs.docker.com/engine/reference/builder/#entrypoint

While I agree that shell form should be avoided, it is necessary to quickly hack in the capability you needed. You could wrap that up in your own entrypoint script to do it right.