backend: re-add kafka startup check

[weeco]

Re-adding the kafka startup check.

Success:

{"level":"info","ts":"2025-03-14T16:41:18.609Z","msg":"started Redpanda Console","version":"development","built_at":"<not set>"}
{"level":"info","ts":"2025-03-14T16:41:18.609Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"}
{"level":"info","ts":"2025-03-14T16:41:18.633Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":1,"failed_clusters":0}
{"level":"info","ts":"2025-03-14T16:41:18.633Z","msg":"successfully create Kafka connect service"}
{"level":"info","ts":"2025-03-14T16:41:18.643Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"info","ts":"2025-03-14T16:41:18.673Z","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":0,"controller_id":0,"kafka_version":"unknown custom version at least v0.11.0"}
{"level":"info","ts":"2025-03-14T16:41:18.709Z","msg":"no static files will be served as serving the frontend has been disabled"}
{"level":"info","ts":"2025-03-14T16:41:18.709Z","msg":"Server listening on address","address":"[::]:9090","port":9090}

Failure:

{"level":"info","ts":"2025-03-14T16:40:27.064Z","msg":"started Redpanda Console","version":"development","built_at":"<not set>"}
{"level":"info","ts":"2025-03-14T16:40:27.064Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"}
{"level":"info","ts":"2025-03-14T16:40:27.095Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":1,"failed_clusters":0}
{"level":"info","ts":"2025-03-14T16:40:27.096Z","msg":"successfully create Kafka connect service"}
{"level":"info","ts":"2025-03-14T16:40:27.106Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"error","ts":"2025-03-14T16:40:27.122Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"seed_0","err":"SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"warn","ts":"2025-03-14T16:40:27.122Z","msg":"failed to test Kafka connection, going to retry in 1s","remaining_retries":5}
{"level":"info","ts":"2025-03-14T16:40:28.124Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"error","ts":"2025-03-14T16:40:28.174Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"seed_0","err":"SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"warn","ts":"2025-03-14T16:40:28.174Z","msg":"failed to test Kafka connection, going to retry in 2s","remaining_retries":4}
{"level":"info","ts":"2025-03-14T16:40:30.177Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"error","ts":"2025-03-14T16:40:30.224Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"seed_0","err":"SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"warn","ts":"2025-03-14T16:40:30.225Z","msg":"failed to test Kafka connection, going to retry in 4s","remaining_retries":3}
{"level":"info","ts":"2025-03-14T16:40:34.226Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"error","ts":"2025-03-14T16:40:34.275Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"seed_0","err":"SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"warn","ts":"2025-03-14T16:40:34.275Z","msg":"failed to test Kafka connection, going to retry in 8s","remaining_retries":2}
{"level":"info","ts":"2025-03-14T16:40:42.277Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":["localhost:19092"]}
{"level":"error","ts":"2025-03-14T16:40:42.314Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"seed_0","err":"SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"warn","ts":"2025-03-14T16:40:42.314Z","msg":"failed to test Kafka connection, going to retry in 16s","remaining_retries":1}
{"level":"fatal","ts":"2025-03-14T16:40:58.316Z","msg":"failed to start console service","error":"failed to test kafka connection: failed to request metadata: SASL authentication failed: security: Invalid credentials: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}

[bojand]

Ideally I think this could be based on cfg.Kafka.Startup settings (ie doing the math to determine the max it could take)... with some absolute maximum value . that is we take minimum of "calculated based on settings" and "default absolute max"