Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Suggestion] Scan Newegg search results instead of product pages #133

Closed
MRizkBV opened this issue Dec 30, 2020 · 0 comments
Closed

[Suggestion] Scan Newegg search results instead of product pages #133

MRizkBV opened this issue Dec 30, 2020 · 0 comments

Comments

@MRizkBV
Copy link
Contributor

MRizkBV commented Dec 30, 2020

Newegg is starting to be a little harder to scrape as they use captcha more often now. The current method scrapes a ton of pages, one per product and keeps cycling. This not only spams newegg more often but also takes long as it cycles back to the first link.

A fix would be to scan search results pages instead. I am not able to imagine how such a thing could be done but maybe this could be possible.
Example link: https://www.newegg.com/p/pl?d=rtx+3080&N=8000%20100007709%20601357282&isdeptsrh=1

How would the bot know if an item is in stock?

  1. Each item has a unique and static identifier which is basically its Item number on Newegg
  2. Newegg does not always show an add to cart button on the search result page but it is safe to assume an item is in stock if it shows a price instead of out of stock. Alternative method is to reverse the logic and do not consider an item in stock if "out of stock" is found in "#item_cell_14-126-453_1_0 > div > div.item-info > p" for example.
  3. If the item is in stock, the bot will read the name, check the price and pull the link and send it to the alerter specified

I tried to implement such a system myself but so far I couldn't. It would be amazing if someone else with the knowledge could do it.

@MRizkBV MRizkBV closed this as completed Jan 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant