Failed to parse ThemeRefinementResponses from completion

[scmdavid]

Hi,

I am testing ThemeFinder 0.7.3 with LLM mistral:7b (running under Ollama). The process run until the theme refinement and got the following error:

2025-11-12 09:45:27,180 INFO: Running theme refinement on 10 responses 
2025-11-12 09:45:27,182 INFO: Running batch and run with batch size 10000 
2025-11-12 09:45:46,592 WARNING: Failed to parse ThemeRefinementResponses from completion {"responses": [{"topic": "AAA", "source_topic_count": 3}, {"topic": "Council Interaction", "source_topic_count": 1}, {"topic": "BBB", "source_topic_count": 1},]}. Got: 10 validation errors for ThemeRefinementResponses responses.0 Value error, Topic must contain a label and description separated by a colon [type=value_error, input_value={'topic': 'AAA...'source_topic_count': 3}, input_type=dict]For further information visit https://errors.pydantic.dev/2.12/v/value_error`
...
2025-11-12 09:47:44,417 INFO: Running theme target alignment on 0 themes compressing to 20 themes
2025-11-12 09:47:44,419 INFO: Running batch and run with batch size 10000
2025-11-12 09:47:44,425 INFO: Running theme mapping on 10 responses using 0 themes
....
raceback (most recent call last):
  File "./main.py", line 118, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "./share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "./share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "./share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/asyncio/base_events.py", line 719, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "./main.py", line 113, in main
    result = await find_themes(df, llm, question, system_prompt=system_prompt, target_n_themes=target_n_themes, verbose=True, concurrency=1 )
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "./themefinder/src/themefinder/tasks.py", line 106, in find_themes
    mapping_df, mapping_unprocessables = await theme_mapping(
                                         ^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "./themefinder/src/themefinder/tasks.py", line 601, in theme_mapping
    refined_themes=transpose_refined_themes(refined_themes_df).to_dict(
                   ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "./themefinder/src/themefinder/tasks.py", line 591, in transpose_refined_themes
    [refined_themes["topic"].to_numpy()], columns=refined_themes["topic_id"]
     ~~~~~~~~~~~~~~^^^^^^^^^
  File "./.venv/lib/python3.13/site-packages/pandas/core/frame.py", line 4113, in __getitem__
    indexer = self.columns.get_loc(key)
  File "./.venv/lib/python3.13/site-packages/pandas/core/indexes/range.py", line 417, in get_loc
    raise KeyError(key)
KeyError: 'topic'

[scmdavid]

The problem could be related to target_n_themes. I assigned to 30. It cased the theme_target_alignment to run but topic_id were not assign to the dataframe.