Rate Limiting, Max Concurrency, Infinite Crawl & Additional Configurations#102

Open

cpdata wants to merge 14 commits intoBuilderIO:mainfrom cpdata:main

+271-34

Commits on Dec 4, 2023

Add VSCode workspace file in .gitignore
cpdata
committed
Add outputs dir to .gitignore for final outputs
cpdata
committed
Add Dynamic OutputFileName based on date-timestamp
cpdata
committed
Allow maxPagesToCrawl to be optional and infinite by setting 0 which will display the infinity symbol. Default is 50
cpdata
committed
Added maxConcurrency config to set maximum concurrent parallel requests.
cpdata
committed
Update to core.ts for maxPagesToCrawl
cpdata
committed
Addded headless mode as a configuration parameter
cpdata
committed
Added waitPerPageCrawlTimeoutRange for a random range in milliseconds between page requests to help with rate limiting
cpdata
committed
Added ts-ignore for docker config.ts to prevent VSCode from declaring missing file that isn't created until the Docker is.
cpdata
committed
Added Output Directory for all outputFileName to go into so they aren't overwritten in storage
cpdata
committed
Additions to dynamic url and match configurations in config.ts
cpdata
committed
Added waitForSelectorTimeout to README.md
cpdata
committed
Adding details to README.md and config.ts as well as extra formatting.
cpdata
committed

Commits on Dec 6, 2023

Update prettier formatting for README.md, src/config.ts, src/core.ts, and config.ts and formatting for jsdoc/typedoc as recommened by @marcelovicentegc in pull request BuilderIO#102 , added .prettie…
cpdata
committed