When using the scraping functionality with pagination, errors occurred in the response of the LLM.
When analyzing the problem, I understood that it was because the string 'PROMPT_PAGINATION' was being concatenated several times, which the llm did not always understand, especially if it used a local model with fewer parameters than for example ChatGPT O4. This is fixed in "Bugfix: duplicated text in pagination prompt".
So And I forked the repository hoping to fix this, and ended up doing a revision of all the files linting with autopep8, adding files like .gitignore, changes.md, .vscode/launch.json or improvements In file readme-md. I also made small modifications to the pagination and escaper prompts, to make them clearer for the LLM.
So, this is contained in the pull request #6, shared for your revision. 😸
When using the scraping functionality with pagination, errors occurred in the response of the LLM.
When analyzing the problem, I understood that it was because the string 'PROMPT_PAGINATION' was being concatenated several times, which the llm did not always understand, especially if it used a local model with fewer parameters than for example ChatGPT O4. This is fixed in "Bugfix: duplicated text in pagination prompt".
So And I forked the repository hoping to fix this, and ended up doing a revision of all the files linting with autopep8, adding files like
.gitignore,changes.md,.vscode/launch.jsonor improvements In filereadme-md. I also made small modifications to the pagination and escaper prompts, to make them clearer for the LLM.So, this is contained in the pull request #6, shared for your revision. 😸