k8s-dns issue around mirrorbrain-db

[rgaudin]

the cronjob mb-update-hashes is not able to complete anymore. All of its runs fail like:

Hashing '/var/www/download.kiwix.org/zim/ted/ted_mul_potential_2025-02.zim'... done.
File 'zim/ted/ted_mul_potential_2025-02.zim' not in database. Not on mirrors yet? Will be inserted.
Hashing '/var/www/download.kiwix.org/zim/ted/ted_mul_poverty_2025-02.zim'... done.
File 'zim/ted/ted_mul_poverty_2025-02.zim' not in database. Not on mirrors yet? Will be inserted.
Hashing '/var/www/download.kiwix.org/zim/ted/ted_mul_public-health_2025-02.zim'... done.
File 'zim/ted/ted_mul_public-health_2025-02.zim' not in database. Not on mirrors yet? Will be inserted.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/mb/hashes.py", line 114, in check_db
    c = conn.mycursor
        ^^^^^^^^^^^^^
AttributeError: 'Conn' object has no attribute 'mycursor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/sqlobject/postgres/pgconnection.py", line 221, in makeConnection
    conn = self.module.connect(**self.dsn_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not translate host name "mirrorbrain-db-service" to address: Name or service not known


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mb", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/mb/mb.py", line 2054, in main
    r = mirrordoctor.main()
        ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cmdln.py", line 262, in main
    return self.cmd(args)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cmdln.py", line 285, in cmd
    retval = self.onecmd(argv)
             ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cmdln.py", line 423, in onecmd
    return self._dispatch_cmd(handler, argv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cmdln.py", line 1124, in _dispatch_cmd
    return handler(argv[0], opts, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/mb/mb.py", line 1341, in do_makehashes
    hasheable.check_db(
  File "/usr/local/lib/python3.11/dist-packages/mb/hashes.py", line 116, in check_db
    c = conn.Hash._connection.getConnection().cursor()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlobject/dbconnection.py", line 351, in getConnection
    conn = self.makeConnection()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/sqlobject/postgres/pgconnection.py", line 223, in makeConnection
    raise dberrors.OperationalError(
sqlobject.dberrors.OperationalError: could not translate host name "mirrorbrain-db-service" to address: Name or service not known
used connection string 'dbname=mirrorbrain user=mirrorbrain host=mirrorbrain-db-service port=5432'
stream closed EOF for zim/mb-update-hashes-manual-zdf-wjwt2 (mirrorbrain)

This mirrorbrain-db-service is provided by k8s and is one of its main goal. Not being able to resolve is very concerning.

Given I found this only for this task, I looked into mirrorbrain-db and could not find any issue. Other services using it (mirrorbrain-web mostly) are not complaining. Given its RAM request was low compared to usage, I significantly increased it and restarted.
The following run (manually triggered) ran for more than 5mn while others typically fail around 1/1.5mn but it's a single occurrence so it might be random.

[rgaudin]

The following run succeeded. Let's see of the next one goes (it runs every hour at :10)

[rgaudin]

Hasn't occurred since. Closing for now.

[benoit74]

Happened again 20h ago (same job, same error) but then next job was OK ...