docs: Add guide about integrating Stagehand #1290
Conversation
|
I had to use |
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Outdated
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/browser_classes.py
Outdated
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Show resolved
Hide resolved
docs/guides/code_examples/playwright_crawler_stagehand/support_classes.py
Show resolved
Hide resolved
There was a problem hiding this comment.
Good job Max!
The integration itself is not as easy as I expected. Maybe this could show us the direction in which we could improve/simplify the browsers/Playwright-related interface.
And/or we could introduce a dedicated crawler to this directly in Crawlee, something like PlaywrightStagehandCrawler. Then the guide could focus solely on its usage, showing how to use AI-based selectors for web scraping.
Let's further discuss it with @B4nan and maybe @janbuchar once they're back from their vacations.
| @@ -0,0 +1,66 @@ | |||
| --- | |||
There was a problem hiding this comment.
I suppose we can't use "Run on Apify" for these examples as it contains more than 1 file.
There was a problem hiding this comment.
That's right.
And in one file, it would look very cumbersome
I think the integration comes out more complicated because of the current I hope that they will improve their API and then the guide can be simplified |
|
|
||
| self._total_opened_pages += 1 | ||
|
|
||
| # Wrap StagehandPage to provide Playwright Page interface |
There was a problem hiding this comment.
This comment seems inaccurate
| pw_page = page._page # noqa: SLF001 | ||
|
|
||
| # Handle page close event | ||
| pw_page.on(event='close', f=self._on_page_close) | ||
|
|
||
| # Update internal state | ||
| self._pages.append(pw_page) | ||
| self._last_page_opened_at = datetime.now(timezone.utc) | ||
|
|
||
| self._total_opened_pages += 1 |
There was a problem hiding this comment.
There is quite a bit of code copied over from PlaywrightBrowserController, isn't it? Any chance we could improve the PlaywrightBrowserController internal API so that integrating libraries that extend Playwright is easier?
There was a problem hiding this comment.
Perhaps context creation should be put into a separate public method. As well as updating states. That would make the same thing a bit cleaner.
But I would say that the main problem with this integration is that you have to do for example, this - pw_page = page._page.
docs/guides/code_examples/playwright_crawler_stagehand/stagehand_run.py
Outdated
Show resolved
Hide resolved
…nd_run.py Co-authored-by: Jan Buchar <[email protected]>
Co-authored-by: Vlada Dusek <[email protected]>
Co-authored-by: Vlada Dusek <[email protected]>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/apify/crawlee-python). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4yMy4yIiwidXBkYXRlZEluVmVyIjoiNDEuMjMuMiIsInRhcmdldEJyYW5jaCI6Im1hc3RlciIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Description
stagehand-pythonv.0.4.0Issues