regular stability issues i belive are cuased by this error in spectator server

[Nubston]

spectator-1 | fail: Metadata[0]
spectator-1 | [user:4] Failed to invoke hub method: GetChangesSince(10070345)
spectator-1 | MySqlConnector.MySqlException (0x80004005): Table 'osu_api.beatmap_sync' doesn't exist

im currently using the latest commit of g0v0 server not the offciall docker image so this may have something to do with it as im launching form the included dockerfile for the main server

[Nubston]

slight update this seemingly only happens if paused for to long or if someone is spectating in a multiplayer lobby but it also seems to hard kill the mp lobby when it happens(need to test this latter more to verify)

[Nubston]

little more testign it seems to be possibly related to these random disconnect as it dosent throught he sql erroe for a little bti but then does if someoen is actively in a game

nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:52 +0000] "GET /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 0 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "GET /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 3 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "POST /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 0 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" spectator-1 | info: ConcurrentConnectionLimiter[0] spectator-1 | [user:4] [connection:R4QL9ETZvZUpjxVvAgrraA] [hub:MetadataHub] subsequent connection from same client instance, registering spectator-1 | info: Metadata[0] spectator-1 | [user:4] Connected spectator-1 | info: Metadata[0] spectator-1 | [user:4] Cleaning up state on connect spectator-1 | info: Metadata[0] spectator-1 | [user:4] State cleanup completed osu_api_server_osurx | 2026-03-30 00:20:53 [INFO] | UserOnlineSubscriber | Received user online status update for user_id: 4 osu_api_server_osurx | 2026-03-30 00:20:53 [INFO] | UserOnlineSubscriber | Received user online status update for user_id: 4 nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "GET /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 30 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "POST /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 0 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "GET /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 37 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" nginx-1 | 172.18.0.1 - - [30/Mar/2026:00:20:53 +0000] "POST /signalr/metadata?id=7ZTQFKIbazTQ_KJNqXP1Vw HTTP/1.0" 200 0 "-" "Microsoft SignalR/10.0 (10.0.5+a612c2a1056fe3265387ae3ff7c94eba1505caf9; Linux; .NET; .NET 8.0.25)" "213.226.66.58" spectator-1 | fail: Metadata[0] spectator-1 | [user:4] Failed to invoke hub method: GetChangesSince(10070345) spectator-1 | MySqlConnector.MySqlException (0x80004005): Table 'osu_api.beatmap_sync' doesn't exist spectator-1 | at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 885 spectator-1 | at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 37 spectator-1 | at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130 spectator-1 | at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlDataReader.cs:line 482
spectator-1 | at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/Core/CommandExecutor.cs:line 56
spectator-1 | at MySqlConnector.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlCommand.cs:line 357
spectator-1 | at MySqlConnector.MySqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlCommand.cs:line 350
spectator-1 | at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in //Dapper/SqlMapper.Async.cs:line 434
spectator-1 | at osu.Server.Spectator.Database.DatabaseAccess.GetChangedBeatmapSetsAsync(DateTimeOffset after) in /app/Database/DatabaseAccess.cs:line 678
spectator-1 | at osu.Server.Spectator.Hubs.Metadata.MetadataHub.GetChangesSince(Int32 queueId) in /app/Hubs/Metadata/MetadataHub.cs:line 115
spectator-1 | at lambda_method245(Closure, Object)
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments) spectator-1 | at osu.Server.Spectator.ClientVersionChecker.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/ClientVersionChecker.cs:line 45
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next)
spectator-1 | at osu.Server.Spectator.ConcurrentConnectionLimiter.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/ConcurrentConnectionLimiter.cs:line 106 spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next)
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) spectator-1 | at osu.Server.Spectator.LoggingHubFilter.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/LoggingHubFilter.cs:line 30
spectator-1 | fail: Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher[8]
spectator-1 | Failed to invoke hub method 'GetChangesSince'.
spectator-1 | MySqlConnector.MySqlException (0x80004005): Table 'osu_api.beatmap_sync' doesn't exist
spectator-1 | at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/Core/ServerSession.cs:line 885
spectator-1 | at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in //src/MySqlConnector/Core/ResultSet.cs:line 37
spectator-1 | at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in //src/MySqlConnector/MySqlDataReader.cs:line 130
spectator-1 | at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 482 spectator-1 | at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56 spectator-1 | at MySqlConnector.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 357 spectator-1 | at MySqlConnector.MySqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 350 spectator-1 | at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 434 spectator-1 | at osu.Server.Spectator.Database.DatabaseAccess.GetChangedBeatmapSetsAsync(DateTimeOffset after) in /app/Database/DatabaseAccess.cs:line 678 spectator-1 | at osu.Server.Spectator.Hubs.Metadata.MetadataHub.GetChangesSince(Int32 queueId) in /app/Hubs/Metadata/MetadataHub.cs:line 115 spectator-1 | at lambda_method245(Closure, Object) spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments)
spectator-1 | at osu.Server.Spectator.ClientVersionChecker.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/ClientVersionChecker.cs:line 45 spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next)
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) spectator-1 | at osu.Server.Spectator.ConcurrentConnectionLimiter.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/ConcurrentConnectionLimiter.cs:line 106
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next)
spectator-1 | at osu.Server.Spectator.LoggingHubFilter.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) in /app/LoggingHubFilter.cs:line 30 spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next)
spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.HubFilterFactory.InvokeMethodAsync(HubInvocationContext invocationContext, Func2 next) spectator-1 | at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.g__ExecuteInvocation|18_0(DefaultHubDispatcher1 dispatcher, ObjectMethodExecutor methodExecutor, THub hub, Object[] arguments, AsyncServiceScope scope, IHubActivator1 hubActivator, HubConnectionContext connection, HubMethodInvocationMessage hubMethodInvocationMessage,`

here is my log from moment i paused this game about 10-15 seconds after pausing but maybe unrelated to pausing may just be related to been in a map

[Yomorei]

first off, your mysql volume is old/out of sync, migrations also did not apply, and your current main has a model/schema mismatch and is expect a table your db never actually got.

So what I suggest doing first is running these cmds:

docker exec -it osu_api_server_osurx sh
uv run --no-sync g0v0-migrate current
uv run --no-sync g0v0-migrate upgrade-all

then verify the table exists:
docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;"

So if that still shows no beatmap_sync, then i'd strongly suspect latest main added the model but your DB schema never got a matching migration.

[GooGuJiang]

Can you check whether the beatmap_sync table has been created in the database?

[Nubston]

first off, your mysql volume is old/out of sync, migrations also did not apply, and your current main has a model/schema mismatch and is expect a table your db never actually got.

So what I suggest doing first is running these cmds:

docker exec -it osu_api_server_osurx sh uv run --no-sync g0v0-migrate current uv run --no-sync g0v0-migrate upgrade-all

then verify the table exists: docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;"

So if that still shows no beatmap_sync, then i'd strongly suspect latest main added the model but your DB schema never got a matching migration.

this seemed to fix it but im surprised it wasn't updated when i first ran it tonight as i nuked the entire server and its persistent files

[Nubston]

okay no it hasnet serve still spitting the error

[Yomorei]

first off, your mysql volume is old/out of sync, migrations also did not apply, and your current main has a model/schema mismatch and is expect a table your db never actually got.
So what I suggest doing first is running these cmds:
docker exec -it osu_api_server_osurx sh uv run --no-sync g0v0-migrate current uv run --no-sync g0v0-migrate upgrade-all
then verify the table exists: docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;"
So if that still shows no beatmap_sync, then i'd strongly suspect latest main added the model but your DB schema never got a matching migration.

this seemed to fix it but im surprised it wasn't updated when i first ran it tonight as i nuked the entire server and its persistent files

Good

[Nubston]

⋊> gortsu@osu ⋊> ~/g0v0-server on main ◦ sudo docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;" 02:16:07
Enter password:
+--------------+
| version_num |
+--------------+
| c5472f592d13 |

is output from mariadb/mysql spectator server still throwing same error

[Yomorei]

⋊> gortsu@osu ⋊> ~/g0v0-server on main ◦ sudo docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;" 02:16:07 Enter password: +--------------+ | version_num | +--------------+ | c5472f592d13 |

is output from mariadb/mysql spectator server still throwing same error

just run this real quick:
sudo docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmapsync'; SHOW TABLES LIKE 'beatmap_sync';"

if beatmapsync exists and beatmap_sync does not, then the mismatch is for sure confirmed.

a fast temp workaround to test it could be:
CREATE OR REPLACE VIEW beatmap_sync AS
SELECT beatmapset_id, beatmaps, beatmap_status, consecutive_no_change, next_sync_time, updated_at
FROM beatmapsync;

if that makes spectator stop erroring, then you have basically just proven th root cause is version/schema skew between spec and the serve db schema lol, which I believe is still case.

Now, a real fix would be use a matching server + spec pair OR build spec from the matching source/revision, ORRR pin both to the same known-good release instead of mixing local main with the prebuild spec image lmfao.

[Nubston]

also seeing this just before the instability errors seems the server isnt ready for the reconnect when it happens earlie or its making to many connection and kicking the client

spectator-1 | info: ConcurrentConnectionLimiter[0]
spectator-1 | [user:4] [connection:D2vw3Ar63XPfl4XYnDzFVQ] [hub:MultiplayerHub] subsequent connection from same client instance, registering
spectator-1 | info: Multiplayer[0]
spectator-1 | [user:4] Connected
spectator-1 | info: Multiplayer[0]
spectator-1 | [user:4] Cleaning up state on connect

unless im misunderstanding this

[Yomorei]

also seeing this just before the instability errors seems the server isnt ready for the reconnect when it happens earlie or its making to many connection and kicking the client

spectator-1 | info: ConcurrentConnectionLimiter[0] spectator-1 | [user:4] [connection:D2vw3Ar63XPfl4XYnDzFVQ] [hub:MultiplayerHub] subsequent connection from same client instance, registering spectator-1 | info: Multiplayer[0] spectator-1 | [user:4] Connected spectator-1 | info: Multiplayer[0] spectator-1 | [user:4] Cleaning up state on connect

unless im misunderstanding this

looks like a bad reconnect handling / duped-connection flow lol.

[Nubston]

⋊> gortsu@osu ⋊> ~/g0v0-server on main ◦ sudo docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmap_sync'; SELECT * FROM alembic_version;" 02:16:07 Enter password: +--------------+ | version_num | +--------------+ | c5472f592d13 |
is output from mariadb/mysql spectator server still throwing same error

just run this real quick: sudo docker exec -it osu_api_mysql_osurx mysql -u root -p -e "USE osu_api; SHOW TABLES LIKE 'beatmapsync'; SHOW TABLES LIKE 'beatmap_sync';"

if beatmapsync exists and beatmap_sync does not, then the mismatch is for sure confirmed.

a fast temp workaround to test it could be: CREATE OR REPLACE VIEW beatmap_sync AS SELECT beatmapset_id, beatmaps, beatmap_status, consecutive_no_change, next_sync_time, updated_at FROM beatmapsync;

if that makes spectator stop erroring, then you have basically just proven th root cause is version/schema skew between spec and the serve db schema lol, which I believe is still case.

Now, a real fix would be use a matching server + spec pair OR build spec from the matching source/revision, ORRR pin both to the same known-good release instead of mixing local main with the prebuild spec image lmfao.

it is indeed the one with no underscore

[MingxuanGame]

Maybe the spectator-server needs to be synced with upstream.
beatmap_sync is a typo. The correct spelling is beatmapsync so sorry.

[Nubston]

I feel beatmap_sync is better for readability so I'd personally suggest converting to that and any others that may e just all rolled Into one long word

[Nubston]

also seems spectator serve may need some serious looking into nothing change between today and yesterday and connection instability is now turning into you've been logged out due to multiple connections