Support for sending traffic via HTTP proxies

[kirillklimenko]

Please add --proxy parameter to CLI interface (https://playwright.dev/python/docs/network#http-proxy):

cli.py

def proxy_option(fn):
    click.option("--proxy", help="HTTP Proxy to use")(fn)
    return fn


# other options
@proxy_option
def shot(...):
    ...


def _browser_context(
    p,
    auth,
    interactive=False,
    devtools=False,
    retina=False,
    browser="chromium",
    user_agent=None,
    proxy=None,
    timeout=None,
    reduced_motion=False,
):
    browser_kwargs = dict(headless=not interactive, devtools=devtools, proxy=proxy)
    if browser == "chromium":
        browser_obj = p.chromium.launch(**browser_kwargs)
    elif browser == "firefox":
        browser_obj = p.firefox.launch(**browser_kwargs)
    elif browser == "webkit":
        browser_obj = p.webkit.launch(**browser_kwargs)
    else:
        browser_kwargs["channel"] = browser
        browser_obj = p.chromium.launch(**browser_kwargs)
    context_args = {}
    if auth:
        context_args["storage_state"] = json.load(auth)
    if retina:
        context_args["device_scale_factor"] = 2
    if reduced_motion:
        context_args["reduced_motion"] = "reduce"
    if user_agent is not None:
        context_args["user_agent"] = user_agent
    if proxy is not None:
        context_args["proxy"] = proxy
    context = browser_obj.new_context(**context_args)
    if timeout:
        context.set_default_timeout(timeout)
    return context, browser_obj

[simonw]

I'm happy to add this but I'm not sure how best to test this. Do you have any guidance on an easy way to test that this proxy option is working? Maybe an existing proxy I can run this through, or a proxy server that's easy to get running on macOS?

[kirillklimenko]

Sure, you can find free proxies on https://free-proxy-list.net/ and test the parameter with them.

[manugarri]

@simonw is there any plan to add this? seems like an easy PR that would add a ton of functionality

[simonw]

I’d accept a PR for this!

…

On Sat, Sep 14, 2024 at 21:14 Manuel Garrido ***@***.***> wrote: @simonw <https://github.com/simonw> is there any plan to add this? seems like an easy PR that would add a ton of functionality — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACK7ZGW3DPL6ZDTIJ7YNDZWSKLXAVCNFSM6AAAAABOHECW2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGEZTMMRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[simonw]

I built a prototype of this in a branch: feaefdf

Used like this:

shot-scraper har https://doge.gov/savings -o savings.zip --wait 2000 \
  --proxy-server $PROXY_SERVER --proxy-username $PROXY_USERNAME --proxy-password $PROXY_PASSWORD

Still needs more testing, and needs to be added to a bunch of other commands, plus documentation.

[simonw]

One thing I haven't quite figured out is the difference between http:// and https:// proxies and if that matters.

I've been testing with a --proxy-server address of 'https://brd.superproxy.io:33335'.

Also I had to do this to get anything to work:

shot-scraper/shot_scraper/cli.py

Lines 452 to 460 in feaefdf

	if proxy_server:
	proxy = {"server": proxy_server}
	if proxy_username:
	proxy["username"] = proxy_username
	if proxy_password:
	proxy["password"] = proxy_password
	context_args["proxy"] = proxy
	# Not sure why I needed this, figure that out!
	context_args["ignore_https_errors"] = True

I'd like to understand if that ignore_https_errors thing is needed.

[simonw]

Tried this for the first time here: simonw/scrape-doge-gov@7aca228

• Cloudflare is preventing this site from being scraped from GitHub Actions scrape-doge-gov#1

[simonw]

I am going to add environment, variable support here, so you can set SHOT_SCRAPER_PROXY_SERVER once and then forget about it.