Language for Scraper

[TrystonMinsquero]

What Language should we use for web scraping data?

Currently, we are using Python for scraping data, but as we are essentially making the scraper from scratch, this would be the best time to change the language. I'd like to hear @hochladen's and @AdamMcAdamson's take on this as well as anyone else's. If you are suggesting a different language, please give concrete advantages and disadvantages it has over python.

My Take

I recognize that I am biased as I personally like Python and have made web scraping projects in the past with Python so I'm comfortable with it. So I'm gonna try to give some pros and cons on why I think we should still use it. I'm sure I'm missing some things so feel but these were the biggest things that came to thought.

Some pros

• Python is easy to pick up/learn
  • This allows for more people to contribute easier
  • Even programmers who never coded in python could still understand it
• It's extremely popular
  • This makes it so more people are willing to contribute
• Forces readability
• Very easy to write, so makes for fast development

Some Cons

• It's slow as hell
• no intro classes taught at UTD are in Python
• no curly braces ;(
• C'mon man it's Python!

[jpahm]

I agree on the points provided, though I'd like to offer an argument in favor of using TypeScript instead of Python for this.

Pros

• Easy to learn, already used in the API so it doesn't require learning an entirely new language
• Compatible with all web libraries that support JS, in particular Selenium which I believe will be a very useful tool for a more powerful scraper
• Makes it extremely simple to work with JSON and various other common scraping data types
• Has strong support for strict-typing, forcing cleaner and more stable code, as well as easier debugging
• Can sometimes encourage cleaner code than Python
• Easy to make tasks cleanly multithreaded

Cons

• Can sometimes encourage worse code than Python
• Requires a Node environment instead of just the Python libraries
• It's still JavaScript, but with types sprinkled on top

These things being said, the main point I'd like to bring up is Selenium. Python, JS, and various other languages have full Selenium libraries at our disposal, and I think we should use it regardless of whichever language we choose. Combining Selenium with web requests will allow us to automate sophisticated traversal of web-pages, enabling us to scrape for data that would otherwise be very difficult to access. For instance, Selenium makes it easy to parse a page's HTML, find a specific form element, and then interact with it whether via clicking, typing, or some other means. This would make our scraping much more powerful and ease the development process.

[AdamMcAdamson]

Personally, I am sold on Selenium. It seems like an all around win.

As far as the particular language, I'm willing to go either way with it, but I think @hochladen's points are convincing enough for me to lean towards TypeScript.

[CharlieMahana]

Is everyone in agreement with using TypeScript instead of Python for writing the scraper per @hochladen's response above?

[AdamMcAdamson]

To add to the discussion, on further inspection with Selenium, I find that Python looks much cleaner to me, but it shouldn't be too much of an issue. The toolset is so powerful that the code need not be remotely convoluted.

[jpahm]

As it seems there's not currently much opposition to this, I'll start doing some research and planning as far as how to start developing it. In the meantime, we should all talk together here in the coming days to figure out what/how we want to scrape. I think the data scraped by the existing scraper is a good baseline, but w/ the added power of Selenium we should be able to pull more.

[AdamMcAdamson]

I am in the process of developing an example scraper in python which scrapes from the UT Dallas College Scheduler in Orion. Since it's incomplete and facing several issues I'm not sure it's worth a pull request at the moment.. I'm hoping to have it able to scrape a full subject's worth of courses by EOD Wednesday.

*I have yet to be rate limited by the College Scheduler.
**If you are having issues with sending keys to React elements, try using ActionChains.

[jpahm]

Thanks for the notes regarding College Scheduler and ActionChains. I'm writing up a prototype Coursebook scraper in the meantime.

[jpahm]

I have ~250 lines of a prototype TS/Selenium coursebook scraper written up, should be finished with it by EOD tomorrow. Seems promising so far, I'll make a repo for it and share it here when it's ready.