Some repos stay in the indexation queue and are never indexed

[gaeljw]

Hello folks,

I don't have any relevant logs or a reproduction case. Just opening a discussion in case someone face the same issue.

I just noticed that among our ~700 repos, about ~100 are in the "indexation queue" and have not been indexed for more than a month. See screenshot below:

What is surprising is that some repos are correctly indexed.

I was wondering if there could be some kind of phenomenon where if the queue is long enough, some repos have chance to never be picked up?

If you have any idea of things to troubleshoot this, I'll be happy to give it a try.

Version: 4.6.3

[gaeljw]

For the record, restarting Sourcebot didn't help. I'll try to reduce reindexIntervalMs so that maybe it gives more time to process the queue?

[gaeljw]

Changing reindexIntervalMs doesn't help.

---

I cleared Redis data in doubt, no change, still these repos hanging and not being indexed.

---

When looking in Postgres DB, I see the repos are "updated" but not "indexed":

select "id", "createdAt" , "updatedAt" , "indexedAt" , "repoIndexingStatus" 
from sourcebot.public."Repo" where "repoIndexingStatus" != 'INDEXED';

[gaeljw]

Updating the status to "NEW" made the repos being picked up and indexed.

update sourcebot.public."Repo" set "repoIndexingStatus" = 'NEW' where "repoIndexingStatus" = 'IN_INDEX_QUEUE';

Weird because just after that, a bunch of repos went again in status IN_INDEX_QUEUE but now they were progressively picked up.

[msukkari]

A repo gets marked as IN_INDEX_QUEUE after being picked up for indexing and a index job for that repo is added to redis. Explanation of the flow: https://demo.sourcebot.dev/~/chat/cmea5bi2g000nny7qi6i5acuy

If it's not being picked up this could be due to a couple reasons:

• The index job was never actually added to the redis (unlikely)
• The index job was added to the redis queue but was lost (ex. redis was shutdown or wiped). In this case, the repo will never be scheduled for indexing because it will remain in the IN_INDEX_QUEUE

We should probably make this more resilient. Created an issue to track this: #462

Please keep us updated if this continue to occur

[gaeljw]

That's what I started to suspect: some kind of inconsistency between the state in Postgres and in-memory/Redis state.

I'll let you know if this happens again.