level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

[kinslayer1982]

Describe the bug

Starting the container with matrix notifications enabled reults in multiple log entries with level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n".
After these lines the usual info (version, schedule) gets printed.
Notifications on updated images works, but this results in the container being considered unhealthy.

Steps to reproduce

1. Create Watchtower container using no shoutrrr notifications.
2. Check logs: no errors and container healthy
3. Add shoutrrr matrix notifications
4. check logs: mentioned errors and container unhealthy

Expected behavior

Get a healthy container without errors as everything seems to be working as intended.

Screenshots

No response

Environment

• Platform Debian 12
• Architecture amd64
• Docker Version Standalone 25.0.0

Your logs

time="2024-01-24T16:17:52+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:17:53+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:17:54+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:17:54+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:17:55+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:17:57+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:18:01+01:00" level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n"

time="2024-01-24T16:18:10+01:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."

time="2024-01-24T16:18:11+01:00" level=debug msg="Making sure everything is sane before starting"

time="2024-01-24T16:18:11+01:00" level=debug msg="Retrieving running, stopped and exited containers"

time="2024-01-24T16:18:11+01:00" level=debug msg="There are no additional watchtower containers"

time="2024-01-24T16:18:11+01:00" level=debug msg="Watchtower HTTP API skipped."

time="2024-01-24T16:18:11+01:00" level=info msg="Watchtower 1.7.1" notify=no

time="2024-01-24T16:18:11+01:00" level=info msg="Using notifications: matrix" notify=no

time="2024-01-24T16:18:11+01:00" level=info msg="Checking all containers (except explicitly disabled with label)" notify=no

time="2024-01-24T16:18:11+01:00" level=info msg="Scheduling first run: 2024-01-25 16:18:11 +0100 CET" notify=no

time="2024-01-24T16:18:11+01:00" level=info msg="Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds" notify=no

Additional context

No response

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[kinslayer1982]

Ok, I think there's two different things here.

Duplicating my watchtower to a Synology DS1621+ running DSM 7.2 and Docker 20.10.23 there are no level=fatal msg="Failed to initialize Shoutrrr notifications: error initializing router services: failed to log in: Too Many Requests\n" messages.
Duplicating it to a Raspberry Pi 4 running Debian 11 and Docker 24.0.7 the same error messages occur.

In all cases the containers are considered unhealthy/problematic/call-it-what-you-want. I think that is because the whole output of watchtower is sent to stderr...

[tilllt]

I can confirm this issue for v1.7.1 and it persists in the combination of matrix, unhealthy containers and shoutrrr enabled. How can we fix this?

Apparently when dealing with an unhealthy container, watchtower / shoutrrr enters a state which itself turns the watchtower container to be seen as "unhealthy" although it keeps operating normally.

I tried to add the SHOUTRRR_DEBUG env but I still cannot see what the problem is or how to further debug.

In our case this leads to notification spam by netdata, reporting watchtower as unhealthy.

As the OP of this issue, I also have

"failed to log in: Too Many Requests\n"

In stdout of the watchtower containers. So apparently there is something going on with watchtower / shoutrrr bombing the matrix API, getting rate limited. I have configured report templates which are correctly send to the matrix bot, after a watchtower check was running, so I assume this part is cause by the unhealthy containers and something watchtower tries to do with them and shoutrrr wants to report about.

How can I further look into this, it's really annoying.