Embedding service crashes with 500 error

[TheWojtek]

Which version of Nextcloud are you using?

32.0.2

Which version of PHP context_chat are you using?

5.1.0

Which version of backend context_chat are you using?

5.1.0

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

Safari, irrelevant in this case

Nextcloud deployment method?

bare metal

Describe the Bug

I have a seemingly working ("The Context Chat Backend app is installed and responsive") setup with 20699 (content provider: files__default) files eligible for indexing, yet "Queued content update actions: 0" and "Queued file system events: 0". "The initial indexing is still running" was displayed for past 4 days (NC AI workers set up properly), so I decided for a manual push:

root@nextcloud:/var/www/nextcloud# sudo -u www-data php occ context_chat:scan thewojtek -vvv
[thewojtek] Scanned Sources/Part I/source_1.pdf
[thewojtek] Scanned Sources/Part I/source2.pdf

In LangRopeService.php line 168:

  [RuntimeException]
  Error received from Context Chat Backend (ExApp) with status code 500: Embedding Request Error: Error: failed to connect to the embedding service


Exception trace:
  at /var/www/nextcloud/apps/context_chat/lib/Service/LangRopeService.php:168
 OCA\ContextChat\Service\LangRopeService->requestToExApp() at /var/www/nextcloud/apps/context_chat/lib/Service/LangRopeService.php:304
 OCA\ContextChat\Service\LangRopeService->indexSources() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:90
 OCA\ContextChat\Service\ScanService->scanDirectory() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:95
 OCA\ContextChat\Service\ScanService->scanDirectory() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:95
 OCA\ContextChat\Service\ScanService->scanDirectory() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:95
 OCA\ContextChat\Service\ScanService->scanDirectory() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:95
 OCA\ContextChat\Service\ScanService->scanDirectory() at /var/www/nextcloud/apps/context_chat/lib/Service/ScanService.php:42
 OCA\ContextChat\Service\ScanService->scanUserFiles() at /var/www/nextcloud/apps/context_chat/lib/Command/ScanFiles.php:50
 OCA\ContextChat\Command\ScanFiles->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:326
 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1078
 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:324
 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:175
 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:187
 OC\Console\Application->run() at /var/www/nextcloud/console.php:90
 require_once() at /var/www/nextcloud/occ:33

The container (running on Docker Socket Proxy) seems to be running OK (I manage containers with Portainer). GPU enabled (Tesla P4), host network, the container log looking like:

    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/context_chat_backend/utils.py", line 74, in exception_wrap
    resconn.send({ 'value': fun(*args, **kwargs), 'error': None })
                            ^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/chain/ingest/injest.py", line 202, in embed_sources
    return _process_sources(vectordb, config, sources_filtered)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/chain/ingest/injest.py", line 157, in _process_sources
    added_source_ids, retry_source_ids = vectordb.add_indocuments(indocuments)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments
    chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 279, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts
    embeddings = self.embedding_function.embed_documents(texts_)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 99, in embed_documents
    return self._get_embedding(texts)  # pyright: ignore[reportReturnType]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 76, in _get_embedding
    raise EmbeddingException('Error: failed to connect to the embedding service') from e
context_chat_backend.types.EmbeddingException: Error: failed to connect to the embedding service
2025-12-07T11:36:10+0000: [ERROR|controller]: Error occurred in an embedding request: /loadSources:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 105, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 385, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 286, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/starlette/concurrency.py", line 32, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/to_thread.py", line 61, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 2525, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/controller.py", line 192, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/controller.py", line 385, in _
    loaded_sources, not_added_sources = exec_in_proc(
                                        ^^^^^^^^^^^^^
  File "/app/context_chat_backend/utils.py", line 97, in exec_in_proc
    raise result['error']
context_chat_backend.types.EmbeddingException: Error: failed to connect to the embedding service
2025-12-07T11:36:10+0000: [ERROR|utils]: Failed request (500): Embedding Request Error: Error: failed to connect to the embedding service
INFO:     127.0.0.1:32918 - "PUT /loadSources HTTP/1.1" 500 Internal Server Error
INFO:     127.0.0.1:58766 - "POST /countIndexedDocuments HTTP/1.1" 200 OK

"Connection refused" is gets me confused here.

Container environment setup:

AA_DOCKER_ENV	1
AA_VERSION	32.0.0
APP_DISPLAY_NAME	Context Chat Backend
APP_HOST	127.0.0.1
APP_ID	context_chat_backend
APP_PERSISTENT_STORAGE	/nc_app_context_chat_backend_data
APP_PORT	23000
APP_SECRET	***
APP_VERSION	5.1.0
CCB_DB_NAME	ccb
CCB_DB_PASS	ccbpass
CCB_DB_USER	ccbuser
COMPUTE_DEVICE	CUDA
CUDA_VERSION	12.2.2
DEBIAN_FRONTEND	dialog
LD_LIBRARY_PATH	/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_VERSION	2.19.3-1
NEXTCLOUD_URL	https://cloud.example.com
NV_CUDA_COMPAT_PACKAGE	cuda-compat-12-2
NV_CUDA_CUDART_VERSION	12.2.140-1
NV_CUDA_LIB_VERSION	12.2.2-1
NV_LIBCUBLAS_PACKAGE	libcublas-12-2=12.2.5.6-1
NV_LIBCUBLAS_PACKAGE_NAME	libcublas-12-2
NV_LIBCUBLAS_VERSION	12.2.5.6-1
NV_LIBCUSPARSE_VERSION	12.1.2.141-1
NV_LIBNCCL_PACKAGE	libnccl2=2.19.3-1+cuda12.2
NV_LIBNCCL_PACKAGE_NAME	libnccl2
NV_LIBNCCL_PACKAGE_VERSION	2.19.3-1
NV_LIBNPP_PACKAGE	libnpp-12-2=12.2.1.4-1
NV_LIBNPP_VERSION	12.2.1.4-1
NV_NVTX_VERSION	12.2.140-1
NVARCH	x86_64
NVIDIA_DRIVER_CAPABILITIES	compute,utility
NVIDIA_PRODUCT_NAME	CUDA
NVIDIA_REQUIRE_CUDA	cuda>=12.2 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NVIDIA_VISIBLE_DEVICES	all
PATH	/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

To Reproduce

New bare metal installation, NC Context Chat backend, assistant, Context chat installed in order.

Please note, the PHP logs display there is a hardcoded 1MB limitation to the ingest ([ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"12190317429","maxSize":"104857600") which makes indexing of meaningful documents rather impossible. I am OK with a size limit for a proof of concept, however we do work with significantly larger documents from our clients.

To summarize:
500 error?
Connection refused error?

PHP logs (Warning these might contain sensitive information)

{"reqId":"bmmjgO3pHQyuc4kbs4QE","level":3,"time":"December 07, 2025 11:33:31","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error received from Context Chat Backend (ExApp)","userAgent":"--","version":"32.0.2.2","data":{"code":"500","response":"{"error":"Embedding Request Error: Error: failed to connect to the embedding service"}","route":"/loadSources","method":"PUT","retry":""}}
{"reqId":"OoUVyXo33NMV5191M61Y","level":3,"time":"December 07, 2025 11:35:11","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OoUVyXo33NMV5191M61Y","level":3,"time":"December 07, 2025 11:35:11","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OoUVyXo33NMV5191M61Y","level":3,"time":"December 07, 2025 11:35:11","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OoUVyXo33NMV5191M61Y","level":3,"time":"December 07, 2025 11:35:11","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"685777223","maxSize":"104857600","nodeId":"662073","path":"/Wojtek/files/Archiwum/DOK. PROJ 17.08.2023.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"2886794224","maxSize":"104857600","nodeId":"662070","path":"/Wojtek/files/Archiwum/Radosna.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"3956351966","maxSize":"104857600","nodeId":"662074","path":"/Wojtek/files/Archiwum/SWZ_TOM III_PFU.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"3270675044","maxSize":"104857600","nodeId":"676105","path":"/Wojtek/files/Archiwum/source1/O.Sz.D-3.2410.1.2023_S11_1z3.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"4135660429","maxSize":"104857600","nodeId":"676098","path":"/Wojtek/files/Archiwum/source1/O.Sz.D-3.2410.1.2023_S11_2z3.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"3677610256","maxSize":"104857600","nodeId":"676100","path":"/Wojtek/files/Archiwum/source1/O.Sz.D-3.2410.1.2023_S11_3z3.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"2665180193","maxSize":"104857600","nodeId":"676101","path":"/Wojtek/files/Archiwum/source1/source5.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"12190317429","maxSize":"104857600","nodeId":"676106","path":"/Wojtek/files/Archiwum/source1/Tom_III_OPZ_O.Sz.D-3.2421.38.2023.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"2873269067","maxSize":"104857600","nodeId":"676104","path":"/Wojtek/files/Archiwum/source1/u015a.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:35:55","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"[ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"2672322381","maxSize":"104857600","nodeId":"676107","path":"/Wojtek/files/Archiwum/source1/ste015a.zip"}}
{"reqId":"liZKvZBxBQrPDSmBsdG8","level":3,"time":"December 07, 2025 11:36:10","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error received from Context Chat Backend (ExApp)","userAgent":"--","version":"32.0.2.2","data":{"code":"500","response":"{"error":"Embedding Request Error: Error: failed to connect to the embedding service"}","route":"/loadSources","method":"PUT","retry":""}}
{"reqId":"OSdMsS6otKX7SXUi4IOR","level":3,"time":"December 07, 2025 11:45:03","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OSdMsS6otKX7SXUi4IOR","level":3,"time":"December 07, 2025 11:45:03","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OSdMsS6otKX7SXUi4IOR","level":3,"time":"December 07, 2025 11:45:03","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OSdMsS6otKX7SXUi4IOR","level":3,"time":"December 07, 2025 11:45:03","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"9AVp467mWCtHo6DzEYHn","level":3,"time":"December 07, 2025 11:50:09","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"9AVp467mWCtHo6DzEYHn","level":3,"time":"December 07, 2025 11:50:09","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"9AVp467mWCtHo6DzEYHn","level":3,"time":"December 07, 2025 11:50:09","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"9AVp467mWCtHo6DzEYHn","level":3,"time":"December 07, 2025 11:50:09","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OVKUYOeBJoeL49twVjN3","level":3,"time":"December 07, 2025 12:00:04","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OVKUYOeBJoeL49twVjN3","level":3,"time":"December 07, 2025 12:00:04","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\FileSystemListenerJob 378872 ended","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OVKUYOeBJoeL49twVjN3","level":3,"time":"December 07, 2025 12:00:04","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 started","userAgent":"--","version":"32.0.2.2","data":[]}
{"reqId":"OVKUYOeBJoeL49twVjN3","level":3,"time":"December 07, 2025 12:00:04","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Background jobs: OCA\ContextChat\BackgroundJobs\ActionJob 378873 ended","userAgent":"--","version":"32.0.2.2","data":[]}

Ex-App logs (Warning these might contain sensitive information)

{"timestamp": "2025-12-07T11:35:56.052579+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "Embedding sources:", "filename": "injest.py", "function": "embed_sources", "line": 184, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_ids": ["files__default: 676110 (Sources/Part I/source_1.pdf)", "files__default: 676111 (Sources/Part I/source_2.pdf)"], "invalid_source_ids": [], "not_allowed_file_ids": [], "len(source_ids)": 2, "len(total_source_ids)": 2}
{"timestamp": "2025-12-07T11:35:57.136105+00:00", "level": "DEBUG", "logger": "ccb.utils", "message": "check_sources took 5.38ms", "filename": "utils.py", "function": "wrapper", "line": 120, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0"}
{"timestamp": "2025-12-07T11:35:57.136428+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "db filter source results", "filename": "injest.py", "function": "_process_sources", "line": 114, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "len(existing_sources)": 0, "existing_sources": [], "len(filtered_sources)": 2, "filtered_sources": ["UploadFile(filename='files__default: 676110', size=1592341, headers=Headers({'userids': 'LISTOFUSERIDS', 'title': 'Sources/Part I/source_1.pdf', 'type': 'application/pdf', 'modified': '1691660442', 'provider': 'files__default', 'content-disposition': 'form-data; name="sources"; filename="files__default: 676110"', 'content-length': '1592341', 'content-type': 'application/octet-stream'}))", "UploadFile(filename='files__default: 676111', size=605545, headers=Headers({'userids':LISTOFUSERIDS', 'title': 'Sources/Part I/source_2.pdf', 'type': 'application/pdf', 'modified': '1692178908', 'provider': 'files__default', 'content-disposition': 'form-data; name="sources"; filename="files__default: 676111"', 'content-length': '605545', 'content-type': 'application/octet-stream'}))"]}
{"timestamp": "2025-12-07T11:35:57.136786+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "Filtered sources:", "filename": "injest.py", "function": "_process_sources", "line": 144, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_ids": ["files__default: 676110", "files__default: 676111"]}
{"timestamp": "2025-12-07T11:35:57.137008+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "processing source", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 57, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676110"}
{"timestamp": "2025-12-07T11:35:58.097260+00:00", "level": "INFO", "logger": "ccb.controller", "message": "Currently indexing 2 documents (filename, size): ", "filename": "controller.py", "function": "background_thread_task", "line": 130, "thread_name": "Thread-1 (background_thread_task)", "pid": 32, "version": "5.1.0", "_indexing": {"files__default: 676110": 1592341, "files__default: 676111": 605545}}
{"timestamp": "2025-12-07T11:36:08.098373+00:00", "level": "INFO", "logger": "ccb.controller", "message": "Currently indexing 2 documents (filename, size): ", "filename": "controller.py", "function": "background_thread_task", "line": 130, "thread_name": "Thread-1 (background_thread_task)", "pid": 32, "version": "5.1.0", "_indexing": {"files__default: 676110": 1592341, "files__default: 676111": 605545}}
{"timestamp": "2025-12-07T11:36:09.539994+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "decoded non empty source", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 77, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676110"}
{"timestamp": "2025-12-07T11:36:09.554835+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "split document into chunks", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 88, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676110", "len(split_docs)": 286}
{"timestamp": "2025-12-07T11:36:09.555103+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "processing source", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 57, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676111"}
{"timestamp": "2025-12-07T11:36:09.856371+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "decoded non empty source", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 77, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676111"}
{"timestamp": "2025-12-07T11:36:09.856942+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "split document into chunks", "filename": "injest.py", "function": "_sources_to_indocuments", "line": 88, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "source_id": "files__default: 676111", "len(split_docs)": 5}
{"timestamp": "2025-12-07T11:36:09.857242+00:00", "level": "DEBUG", "logger": "ccb.injest", "message": "Converted all sources to documents", "filename": "injest.py", "function": "_process_sources", "line": 150, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0"}
{"timestamp": "2025-12-07T11:36:09.857735+00:00", "level": "INFO", "logger": "ccb.nextwork_em", "message": "Sending embedding request for 286 chunks of the following sizes (total: 392561):", "filename": "network_em.py", "function": "_get_embedding", "line": 44, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "lengths": [932, 1970, 1949, 352, 1964, 1916, 450, 1973, 1944, 448, 1418, 1848, 1851, 1957, 545, 1940, 765, 1930, 1863, 1946, 798, 1924, 1936, 1810, 1255, 1590, 1924, 658, 1833, 1986, 173, 1954, 244, 1690, 1974, 278, 1854, 1942, 559, 1962, 627, 1989, 332, 1963, 1596, 1990, 450, 1963, 603, 1931, 1031, 1962, 306, 1939, 760, 1996, 622, 1927, 922, 1981, 805, 1943, 305, 1930, 826, 1984, 820, 1960, 720, 1975, 618, 725, 1983, 249, 1981, 458, 1955, 1014, 1982, 1001, 1940, 814, 1920, 345, 1638, 1963, 1073, 1976, 698, 1964, 269, 1953, 427, 1991, 627, 1950, 299, 1984, 284, 1972, 608, 1943, 572, 1771, 1165, 1951, 277, 1993, 408, 1981, 288, 1872, 1993, 599, 1965, 595, 1924, 308, 1942, 665, 1915, 880, 1945, 512, 1933, 569, 1945, 803, 1978, 932, 1933, 711, 1932, 412, 1921, 391, 1922, 413, 1983, 900, 1985, 286, 1995, 463, 1967, 338, 1951, 1154, 1785, 1946, 1117, 1917, 1315, 1978, 1433, 1959, 1180, 1969, 809, 545, 1928, 1312, 1945, 1382, 1945, 1322, 1964, 1439, 1965, 1140, 1996, 313, 1957, 857, 1956, 969, 1968, 1274, 1912, 1009, 1954, 1096, 1945, 1015, 1957, 975, 1981, 1252, 1926, 1333, 1981, 1081, 1981, 403, 1977, 1131, 1932, 929, 1931, 979, 1964, 889, 1686, 1967, 816, 1920, 1303, 1927, 1253, 1938, 738, 1985, 301, 1980, 721, 1975, 906, 1819, 1917, 1247, 1941, 852, 1958, 648, 1960, 441, 1938, 808, 1935, 538, 1924, 816, 1947, 733, 1961, 878, 1977, 624, 1944, 260, 1951, 360, 1990, 300, 1982, 1018, 1943, 1676, 1979, 1123, 1756, 1913, 1539, 1968, 401, 1978, 819, 1986, 608, 1964, 917, 1918, 715, 1918, 789, 1950, 512, 1988, 603, 1960, 747, 1940, 892, 1979, 334, 1949, 268, 588, 1863, 1960, 465, 1946, 897, 1941, 517, 746]}
{"timestamp": "2025-12-07T11:36:09.954636+00:00", "level": "ERROR", "logger": "ccb.nextwork_em", "message": "Error connecting to the embedding server, check if it is running and the logs", "filename": "network_em.py", "function": "_get_embedding", "line": 72, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions\n yield\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request\n resp = self._pool.handle_request(req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request\n raise exc from None\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request\n response = connection.handle_request(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request\n raise exc\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request\n stream = self._connect(request)\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect\n stream = self._network_backend.connect_tcp(**kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp\n with map_exceptions(exc_map):\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions\n raise to_exc(exc) from exc\nhttpcore.ConnectError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding\n response = client.post(\n ^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post\n return self.request(\n ^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request\n return self.send(request, auth=auth, follow_redirects=follow_redirects)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send\n response = self._send_handling_auth(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth\n response = self._send_handling_redirects(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects\n response = self._send_single_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request\n response = transport.handle_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request\n with map_httpcore_exceptions():\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: [Errno 111] Connection refused"}
{"timestamp": "2025-12-07T11:36:09.961573+00:00", "level": "ERROR", "logger": "ccb.vectordb", "message": "Error adding documents to vectordb", "filename": "pgvector.py", "function": "add_indocuments", "line": 179, "thread_name": "AnyIO worker thread", "pid": 140, "version": "5.1.0", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions\n yield\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request\n resp = self._pool.handle_request(req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request\n raise exc from None\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request\n response = connection.handle_request(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request\n raise exc\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request\n stream = self._connect(request)\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect\n stream = self._network_backend.connect_tcp(**kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp\n with map_exceptions(exc_map):\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions\n raise to_exc(exc) from exc\nhttpcore.ConnectError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding\n response = client.post(\n ^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post\n return self.request(\n ^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request\n return self.send(request, auth=auth, follow_redirects=follow_redirects)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send\n response = self._send_handling_auth(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth\n response = self._send_handling_redirects(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects\n response = self._send_single_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request\n response = transport.handle_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request\n with map_httpcore_exceptions():\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpx/transports/default.py", line 118, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments\n chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 279, in add_documents\n return self.add_texts(texts, metadatas, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts\n embeddings = self.embedding_function.embed_documents(texts)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/network_em.py", line 99, in embed_documents\n return self._get_embedding(texts) # pyright: ignore[reportReturnType]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/network_em.py", line 76, in _get_embedding\n raise EmbeddingException('Error: failed to connect to the embedding service') from e\ncontext_chat_backend.types.EmbeddingException: Error: failed to connect to the embedding service", "source_id": "files__default: 676110"}
{"timestamp": "2025-12-07T11:36:10.017122+00:00", "level": "ERROR", "logger": "ccb.utils", "message": "original traceback: Traceback (most recent call last):\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions\n yield\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request\n resp = self._pool.handle_request(req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request\n raise exc from None\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request\n response = connection.handle_request(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request\n raise exc\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request\n stream = self._connect(request)\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect\n stream = self._network_backend.connect_tcp(**kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp\n with map_exceptions(exc_map):\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions\n raise to_exc(exc) from exc\nhttpcore.ConnectError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding\n response = client.post(\n ^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post\n return self.request(\n ^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request\n return self.send(request, auth=auth, follow_redirects=follow_redirects)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send\n response = self._send_handling_auth(\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth\n response = self._send_handling_redirects(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects\n response = self._send_single_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request\n response = transport.handle_request(request)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request\n with map_httpcore_exceptions():\n File "/usr/lib/python3.11/contextlib.py", line 158, in exit\n self.gen.throw(typ, value, traceback)\n File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/app/context_chat_backend/utils.py", line 74, in exception_wrap\n resconn.send({ 'value': fun(*args, **kwargs), 'error': None })\n ^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/chain/ingest/injest.py", line 202, in embed_sources\n return _process_sources(vectordb, config, sources_filtered)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/chain/ingest/injest.py", line 157, in process_sources\n added_source_ids, retry_source_ids = vectordb.add_indocuments(indocuments)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments\n chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 279, in add_documents\n return self.add_texts(texts, metadatas, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts\n embeddings = self.embedding_function.embed_documents(texts)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/context_chat_backend/network_em.py", line 99, in embed_documents\n return self._get_embedding(texts) # pyright: ignore[reportReturnType]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n

Server logs (if applicable)

No response

[kyteinsky]

hello,

Please note, the PHP logs display there is a hardcoded 1MB limitation to the ingest ([ScanService] File too large to index","userAgent":"--","version":"32.0.2.2","data":{"nodeSize":"12190317429","maxSize":"104857600") which makes indexing of meaningful documents rather impossible.

the limit is 100MB so the majority of the docs should be covered I suppose. The file where this log happened is a zip file so that wouldn't be indexed directly anyway.

looking at the errors, it seems the internal embedding server failed to start.

1. Would you mind sending the start of the docker container's logs? docker logs -f nc_app_context_chat_backend |& head -n 500
2. does your cpu support AVX? cat /proc/cpuinfo | grep avx

"Connection refused" is gets me confused here.

it's the connection of the main app to the internal embedding server, which is not working.

[kyteinsky]

I've removed the APP_SECRET and modified the NEXTCLOUD_URL env var values. Please take care not to post these, they can have security implications.

[TheWojtek]

@kyteinsky Oh yes, 100MB, got dizzy with the zeroes ;) Right, it's a ZIP so non-indexable. Let's forget this one.

flags		: fpu vme (...) avx (...)

So this is OK. Now for the logs:

docker logs -f nc_app_context_chat_backend |& head -n 500
/usr/lib/python3/dist-packages/supervisor/options.py:13: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Starting PostgreSQL...
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start.... done
server started
Waiting for PostgreSQL to start...
PostgreSQL is up and running.
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
ALTER ROLE
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
NOTICE:  extension "vector" already exists, skipping
CREATE EXTENSION
Running repairs...
No repairs are required.
/usr/lib/python3/dist-packages/supervisor/options.py:13: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Starting PostgreSQL...
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start.... done
server started
Waiting for PostgreSQL to start...
PostgreSQL is up and running.
CCB_DB_URL is already set. Skipping database setup.
Running repairs...
No repairs are required.
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
App config:
{
  "debug": true,
  "uvicorn_log_level": "info",
  "disable_aaa": false,
  "httpx_verify_ssl": true,
  "use_colors": true,
  "uvicorn_workers": 1,
  "embedding_chunk_size": 2000,
  "doc_parser_worker_limit": 10,
  "vectordb": [
    "pgvector",
    {}
  ],
  "embedding": {
    "protocol": "http",
    "host": "localhost",
    "port": 5000,
    "workers": 1,
    "offload_after_mins": 15,
    "request_timeout": 1800,
    "llama": {
      "model": "multilingual-e5-large-instruct-q6_k.gguf",
      "n_batch": 16,
      "n_ctx": 8192,
      "n_gpu_layers": -1
    }
  },
  "llm": [
    "nc_texttotext",
    {}
  ]
}
INFO:     Started server process [32]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:23000 (Press CTRL+C to quit)
INFO:     127.0.0.1:51756 - "GET /heartbeat HTTP/1.1" 200 OK
INFO:     127.0.0.1:51756 - "POST /init HTTP/1.1" 200 OK
Fetching 5 files: 100%|██████████| 5/5 [00:00<00:00, 75.37it/s]
INFO:     127.0.0.1:57150 - "GET /heartbeat HTTP/1.1" 200 OK
INFO:     127.0.0.1:57150 - "PUT /enabled?enabled=1 HTTP/1.1" 200 OK
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:57160 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:43128 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:50364 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:36352 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
INFO:     127.0.0.1:36494 - "GET /enabled HTTP/1.1" 200 OK
INFO:     127.0.0.1:36494 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:56070 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:46608 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:46610 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:60710 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     164.40.242.202:0 - "GET /downloadLogs HTTP/1.1" 200 OK
INFO:     127.0.0.1:54700 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:42356 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:47176 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:55688 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
/usr/lib/python3/dist-packages/supervisor/options.py:13: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Starting PostgreSQL...
sudo: unable to resolve host context_chat_backend: Temporary failure in name resolution
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start.... done
server started
Waiting for PostgreSQL to start...
PostgreSQL is up and running.
CCB_DB_URL is already set. Skipping database setup.
Running repairs...
No repairs are required.
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
App config:
{
  "debug": true,
  "uvicorn_log_level": "info",
  "disable_aaa": false,
  "httpx_verify_ssl": true,
  "use_colors": true,
  "uvicorn_workers": 1,
  "embedding_chunk_size": 2000,
  "doc_parser_worker_limit": 10,
  "vectordb": [
    "pgvector",
    {}
  ],
  "embedding": {
    "protocol": "http",
    "host": "localhost",
    "port": 5000,
    "workers": 1,
    "offload_after_mins": 15,
    "request_timeout": 1800,
    "llama": {
      "model": "multilingual-e5-large-instruct-q6_k.gguf",
      "n_batch": 16,
      "n_ctx": 8192,
      "n_gpu_layers": -1
    }
  },
  "llm": [
    "nc_texttotext",
    {}
  ]
}
INFO:     Started server process [32]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:23000 (Press CTRL+C to quit)
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:38322 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:34696 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:53198 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
INFO:     127.0.0.1:35364 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:60466 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
INFO:     127.0.0.1:36218 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:38182 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
INFO:     127.0.0.1:38186 - "POST /countIndexedDocuments HTTP/1.1" 200 OK
2025-12-07T11:33:31+0000: [ERROR|network_em]: Error connecting to the embedding server, check if it is running and the logs
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused
2025-12-07T11:33:31+0000: [ERROR|pgvector]: Error adding documents to vectordb
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments
    chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 279, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts
    embeddings = self.embedding_function.embed_documents(texts_)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 99, in embed_documents
    return self._get_embedding(texts)  # pyright: ignore[reportReturnType]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 76, in _get_embedding
    raise EmbeddingException('Error: failed to connect to the embedding service') from e
context_chat_backend.types.EmbeddingException: Error: failed to connect to the embedding service
2025-12-07T11:33:31+0000: [ERROR|utils]: original traceback: Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc

The unable to resolve host context_chat_backend: Temporary failure in name resolution looks suspicious here, doesn't it? Tried adding this as 127.0.0.1 to machine (not docker) hosts file, to no avail. I know this may be simple, but I can't get my head over it.

[kyteinsky]

Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").
Embedder config:
{
  "protocol": "http",
  "host": "localhost",
  "port": 5000,
  "workers": 1,
  "offload_after_mins": 15,
  "request_timeout": 1800,
  "llama": {
    "model": "multilingual-e5-large-instruct-q6_k.gguf",
    "n_batch": 16,
    "n_ctx": 8192,
    "n_gpu_layers": -1
  }
}
INFO:     127.0.0.1:57160 - "GET /enabled HTTP/1.1" 200 OK
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    yes
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P4, compute capability 6.1, VMM: yes
Config file already exists in the persistent storage ("/nc_app_context_chat_backend_data/config.yaml").

^ these logs are from the embedding server, which uses llama.cpp. It crashes, restarts and crashes over and over again as seen from the incomplete repeating logs. The reason might be that RAM is not enough or the GPU is acting up, or something else. We use CUDA 12.2 right now so that should support Tesla P4.
I'm not quite sure of the exact reason. Are you able to run llama.cpp or ollama independently on the same system and GPU?

The unable to resolve host context_chat_backend: Temporary failure in name resolution looks suspicious here, doesn't it?

this is harmless, it comes from sudo not being able to resolve "context_chat_backend" host.

[TheWojtek]

^ these logs are from the embedding server, which uses llama.cpp. It crashes, restarts and crashes over and over again as seen from the incomplete repeating logs. The reason might be that RAM is not enough or the GPU is acting up, or something else. We use CUDA 12.2 right now so that should support Tesla P4. I'm not quite sure of the exact reason. Are you able to run llama.cpp or ollama independently on the same system and GPU?

Yupp, no problems with Gemma:

ollama list
NAME                     ID              SIZE      MODIFIED
embeddinggemma:latest    85462619ee72    621 MB    2 weeks ago
gemma3:latest            a2af6cc3eb7f    3.3 GB    3 weeks ago

ollama run gemma3 "Can you introduce yourself?"
Hello there! I'm Gemma, a large language model created by the Gemma team at Google DeepMind. I'm an open-weights model, which means I’m widely available for public
use!

I can take text and images as inputs and generate text-based responses.

It's nice to meet you!

You can learn more about me on the Gemma project page: [https://ai.google.com/gemma](https://ai.google.com/gemma)

Works almost instantly on the GPU (this from another SSH session to the same server at the same time):

nvidia-smi
Wed Dec 10 18:38:46 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P4                       On  |   00000000:13:00.0 Off |                  Off |
| N/A   58C    P0             24W /   75W |    4763MiB /   8192MiB |      18%     Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         1147802      C   /usr/local/bin/ollama                  4760MiB |
+-----------------------------------------------------------------------------------------+

I have this Ollama service connected to NC Assistant and it sure works as "Chat with AI" in NC Assistant as well.

[TheWojtek]

New log entry after NC 32.0.3 update:

Error | core | AppConfigTypeConflictException conflict with value type from databaseError while running background job OCA\ContextChat\BackgroundJobs\IndexerJob (id: 378116, arguments: {"storageId":3,"rootId":209})

[TheWojtek]

@kyteinsky - the above error is now repeated over 4000 times, with storageIDs 3 and 15 and rootIDs 209 and 694835 respectively.
This exception is thrown by lib/private/AppConfig.php

[kyteinsky]

hi, would you have the entire json log line from "data/nextcloud.log" file for this error?
it would be nice to open a new issue about this.

[kyteinsky]

| NVIDIA-SMI 580.105.08 Driver Version: 580.105.08 CUDA Version: 13.0 |

the driver and cuda versions look suspecious though, and looks like compatibility was broken here with cuda 12.x
maybe downgrading the nvidia driver to <580 would solve this issue
ref: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id7

not sure if context_chat_backend should upgrade to 580/cuda 13 since it would then break existing setups, and also lose support for many gpus, while adding support for the newer gen which very less people have I imagine.

[TheWojtek]

hi, would you have the entire json log line from "data/nextcloud.log" file for this error? it would be nice to open a new issue about this.

Oooh, you lost me here.
Just upgraded to 32.0.5 and wiped the logs, unfortunately. The current situation is - /var/log/nextcloud.log does not show any errors, the "Assistant" administration page says:

Indexing Status

The initial indexing is still running.

The Context Chat Backend app is installed and responsive.

Content provider | Queued documents including updates | Documents in vector database -- | -- | -- files__default | 8842 (8813 new files) | (0 sent)

 

Eligible files for indexing: 19813

Queued content update actions: 0

Queued file system events: 0

What is interesting, is that the number of files eligible drops from time to time (was over 30k docs two weeks ago), but the number of docs in vector db is constantly 0.

I have the PHP App logs and Ex-App Backend logs, available although too big to post here, the Docker logs do not show any errors. Let me know how I can help.

[TheWojtek]

OK, the errors just came popping back a day later:

AppConfigTypeConflictException
conflict with value type from database
/var/www/nextcloud/lib/private/AppConfig.php
Line 368
OC\AppConfig->getTypedValue()
/var/www/nextcloud/lib/private/AppFramework/Services/AppConfig.php
Line 239
OC\AppConfig->getValueString()
/var/www/nextcloud/apps/context_chat/lib/BackgroundJobs/IndexerJob.php
Line 94
OC\AppFramework\Services\AppConfig->getAppValueString()
/var/www/nextcloud/lib/public/BackgroundJob/Job.php
Line 61
OCA\ContextChat\BackgroundJobs\IndexerJob->run()
/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php
Line 97
OCP\BackgroundJob\Job->start()
/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php
Line 84
OCP\BackgroundJob\TimedJob->start()
/var/www/nextcloud/cron.php
Line 175
OCP\BackgroundJob\TimedJob->execute()

Raw entry:

{"reqId":"fxABlGKB4BkE1CIi4kL8","level":3,"time":"2026-01-18T05:40:03+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"--","scriptName":"/var/www/nextcloud/cron.php","message":"Error while running background job OCA\\ContextChat\\BackgroundJobs\\IndexerJob (id: 381508, arguments: {\"storageId\":15,\"rootId\":766251})","userAgent":"--","version":"32.0.5.0","occ_command":["/var/www/nextcloud/cron.php"],"exception":{"Exception":"OCP\\Exceptions\\AppConfigTypeConflictException","Message":"conflict with value type from database","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/AppConfig.php","line":368,"function":"getTypedValue","class":"OC\\AppConfig","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Services/AppConfig.php","line":239,"function":"getValueString","class":"OC\\AppConfig","type":"->"},{"file":"/var/www/nextcloud/apps/context_chat/lib/BackgroundJobs/IndexerJob.php","line":94,"function":"getAppValueString","class":"OC\\AppFramework\\Services\\AppConfig","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":61,"function":"run","class":"OCA\\ContextChat\\BackgroundJobs\\IndexerJob","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":97,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":84,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":175,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/lib/private/AppConfig.php","Line":505,"message":"Error while running background job OCA\\ContextChat\\BackgroundJobs\\IndexerJob (id: 381508, arguments: {\"storageId\":15,\"rootId\":766251})","exception":[],"CustomMessage":"Error while running background job OCA\\ContextChat\\BackgroundJobs\\IndexerJob (id: 381508, arguments: {\"storageId\":15,\"rootId\":766251})"},"id":"696cc04509e31"}

I think I will just remove everything, context chat backend, the context chat, and start over. Will the "remove" button in Nextcloud apps also remove relevant entries from the database?