Skip to content

Cannot scrape full article text of Google News RSS Feeds #125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Kate-Actuary-Viola opened this issue Mar 16, 2024 · 4 comments
Open

Cannot scrape full article text of Google News RSS Feeds #125

Kate-Actuary-Viola opened this issue Mar 16, 2024 · 4 comments

Comments

@Kate-Actuary-Viola
Copy link

How can I scrape the full article text of Google News feeds. When I insert the link, I get a cookie-reminder-text for every article instead of the article text. Is there a way to automatically reply to the cookie selector when parsing?

@pictuga
Copy link
Owner

pictuga commented Mar 17, 2024

Can you share the link of the feed?

@pictuga
Copy link
Owner

pictuga commented Mar 17, 2024

Google seems to replace the original links with some redirects that create this problem...

@Kate-Actuary-Viola
Copy link
Author

Exactly, thats the issue. Mostly only news articles that show up on google news are publicly available for some time and then hidden behind a paywall. Thats why full text view is much more interesting for those feeds than other feeds of newspapers. Is there any way around it? Is it possible to use the redirect link? I am reading only about python scripts as an alternative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants