Fixed issue in Prompt Pagination, Update to OpenAI API, and various formatting fixes

[gatuno1]

• Bugfix: duplicated text in pagination prompt.
  Removed error of adding twice the main pagination prompt text when scraping for pagination links with a LLM call.

• Bugfix: migrated to v1.0 call
  When using generic openai-compatible model, the chat completation call is using client object as recommended in v1.0 API.

• Prompt improvement:
  Modifications on scaper and pagination Prompts, to be clearer for the LLM.

• Formatting & linting on all files:
  Includes removal of unused imports, removal of old comments, added header with she-bang for python, formatted python code for better readability, corrected problems with f-strings and logging options, etc.

• Created .gitignore file with exclusions for output dir, and common cases for python, virtual environment, windows, vscode and pycharm.
  Hint from amgrbi96/AI-scraper fork.

• Added changes.md file: to maintain changes log.

• Added streamlit app debug configuration for VSCode.

• Streamlit opt-out configuration added.

• Ordered contents of requirements.txt files.

• Removed empty test.py file.

• Added links to Youtube videos for this project in README.md, as silasneo/scrape-master fork did.