Skip to content

Speedup Sciebo link finding #121

@D-VR

Description

@D-VR

Currently the following code (around line 1130), may check a lot of duplicate links (up to 6x or higher! in one of my courses), leading to significant slowdowns due to unnecessary get requests. Adding some simple caching can improve this.

# https://rwth-aachen.sciebo.de/s/XXX
        if self.config.get("used_modules", {}).get("url", {}).get("sciebo", {}):
            sciebo_links = re.findall(
                "https://rwth-aachen.sciebo.de/s/[a-zA-Z0-9-]+", text
            )

            for vid in sciebo_links:
                response = self.session.get(vid)
                soup = bs(response.text, features="html.parser")
                url = soup.find("input", {"name": "downloadURL"})
                filename = soup.find("input", {"name": "filename"})
                if url and filename:
                    parent_node.add_child(
                        filename["value"], url["value"], "Sciebo file", url=url["value"]
                    )

I will create something when I have time

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions