Headless Chrome updates

[gadenbuie]

I'm opening this issue mostly to stash some things I've learned while debugging #170.

Chrome's headless mode was previously an entirely separate web browser from headful Chrome. Last year, Chrome introduced a new headless mode that uses the same browser implementation with the UI turned off, introduced as --headless=new. Old headless mode was previously available as --headless=old or --headless.

Chrome is now in the process of transitioning the default value of --headless to --headless=new.

Note: Passing the --headless command-line flag without an explicit value still activates the old Headless mode, but we intend to change this default. We intend to remove the old Headless from the Chrome binary and stop supporting this mode in Puppeteer in 2024. Simultaneously, we'll make old Headless mode available as a standalone binary for users who can't upgrade yet.

I haven't found an official statement saying that this change happened, but clearly --headless is now --headless=new. At some point in the future, --headless=old will be removed and users will be expected to use a separate chrome-headless-shell binary, which is already available.

launch_chrome(), by way of launch_chrome_impl(), forces --headless

chromote/R/chrome.R

Lines 175 to 180 in f030285

	args = c(
	"--headless",
	paste0("--remote-debugging-port=", port),
	paste0("--remote-allow-origins=http://127.0.0.1:", port),
	args
	),

but we could ease the transition by allowing users to set this value themselves. We could:

1. Move --headless into default_chrome_args(), but this makes it easy to accidentally forget to include the --headless flag. OTOH, that might be desirable given that it is possible to use the Chrome DevTools Protocol to drive headful Chrome.

2. Add --headless when launching Chrome only if --headless or --headless=* are not included in args.

We should also test that chromote is compatible with chrome-headless-shell.

[gadenbuie]

Download chrome-headless-shell from https://googlechromelabs.github.io/chrome-for-testing/#stable

The current easiest way to get chrome-headless-shell is to use pupeteer and npx. Store the downloaded binary somewhere stable using --path, like ~/.browsers:

npx @puppeteer/browsers install chrome-headless-shell --path ~/.browsers

Downloading chrome-headless-shell 130.0.6684.0 - 86.8 MB [====================] 100% 0.0s 
[email protected] /Users/garrick/.browsers/chrome-headless-shell/mac_arm-130.0.6684.0/chrome-headless-shell-mac-arm64/chrome-headless-shell

This outputs the path to the binary, which you can then use in the CHROMOTE_CHROME envvar:

Sys.setenv(CHROMOTE_CHROME = "/Users/garrick/.browsers/chrome-headless-shell/mac_arm-130.0.6684.0/chrome-headless-shell-mac-arm64/chrome-headless-shell")

At that point, chromote will use the chrome-headles-shell, i.e. --headless=old at a stable version.

[hadley]

Is there some reason that chromote can't use the new headless mode? What am I missing?

[gadenbuie]

chromote can use the new headless mode. I think Chrome intended for the new headless mode to be a drop-in replacement for the old headless mode but that hasn't been the case. There are many instances of differences and broken behavior between the two modes, the biggest of which is that you cannot simultaneously open headless and headfull Chrome instances on Windows.

We'll benefit both from giving Chrome some time to work out these issues and from any testing with the new headless mode that we or chromote users can do in the mean time.

[gadenbuie]

Old headless mode was removed: https://developer.chrome.com/blog/removing-headless-old-from-chrome

[llrs-roche]

Probably related to this, on updated windows, I see some errors pointing to old headless mode (when testing with Chrome open):

packageVersion("chromote")
[1] '0.3.1'
system.time({devtools::test_active_file()})  #Testing a package test file that takes too long
Error in with_random_port(launch_chrome_impl, path = path, args = args) : 
  Cannot find an available port. Please try again.
Caused by error in `startup()`:
! Failed to start chrome. Error:
Old Headless mode has been removed from the Chrome binary. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).

I had to use options(chromote.headless = "new") (From a comment) in order to check the file.

System info

EditionWindows 11 Enterprise
Version23H2
Installed on‎04/‎10/‎2024
OS build22631.4602
ExperienceWindows Feature Experience Pack 1000.22700.1055.0
Chrome: Version 132.0.6834.83 (Official Build) (64-bit)

library(chromote)
b <- ChromoteSession$new()
Error in `with_random_port()`:
! Cannot find an available port. Please try again.
Caused by error in `startup()`:
! Failed to start chrome. Error:
Old Headless mode has been removed from the Chrome binary. Please use the new Headless mode (https://developer.chrome.com/docs/chromium/new-headless) or the chrome-headless-shell which is a standalone implementation of the old Headless mode (https://developer.chrome.com/blog/chrome-headless-shell).
Run `rlang::last_trace()` to see where the error occurred.
options(chromote.headless = "new")
library(chromote)
b <- ChromoteSession$new() # Opened a new Chrome window

Thanks for sharing the lessons learned.

[gadenbuie]

@llrs-roche #187 describes the rest of the context. In short, v132 and later of Chrome no longer bundle old headless mode. You can use that option or update to the dev version of chromote.

[tdhock]

hi, I'm using chromote for testing on GitHub Actions, and I'm having the same isssue, Old Headless mode has been removed from the Chrome binary https://github.com/animint/animint2/actions/runs/12946454044/job/36111017361

I understand that this is already resolved if I can use the dev version of library(chromote) from this github repo.
For my github action I believe we are using the CRAN version. Do you plan to submit an update to CRAN soon? or should we try to install dev chromote from github, in our github action?

[gadenbuie]

@tdhock I'm planning on sending chromote to CRAN today. You can follow that progress in #195.

[tdhock]

great thanks
in the meantime, I confirm that adding "Remotes: rstudio/chromote" to DESCRIPTION fixes this issue.