markup5ever non-deterministic generated.rs

[bmwiedemann]

Originally filed in lycheeverse/lychee#1632 - check out the details there.

Please ensure that markup5ever's generated.rs is created in a deterministic way, to allow for reproducible builds.

[Ygg01]

This will be a bit more tricky than advertised.

What seems to be the source of non-determinism is the phf turning https://github.com/servo/html5ever/blob/main/markup5ever/entities.rs into an perfect hash map.

html5ever/markup5ever/build.rs

Line 36 in 0e4e125

named_entities_to_phf(&Path::new(&env::var("OUT_DIR").unwrap()).join("named_entities.rs"));

The problem I see is, I don't know where the non-determinism is coming from. It could be as simple as changing HashMap into TreeMap:

html5ever/markup5ever/build.rs

Lines 82 to 88 in 0e4e125

	let mut entities: HashMap<&str, (u32, u32)> = entities::NAMED_ENTITIES
	.iter()
	.map(|(name, cp1, cp2)| {
	assert!(name.starts_with('&'));
	(&name[1..], (*cp1, *cp2))
	})
	.collect();

or as hard as getting phf or string_cache_codegen to emit stable perfect hash map.

[bmwiedemann]

Rust HashMaps have random iteration order. osa1/tiny#438 solved that with an indexmap. Though it would be nice, if rust had some builtin way for that.

[nicoburns]

Rust HashMaps have random iteration order

Yeah, there's a strong chance that it's that HashMap causing this non-determinism.

though it would be nice, if rust had some builtin way for that.

There is std::collectionsBTreeMap would likely work well in this case.

[Ygg01]

that HashMap causing this non-determinism.

The big thing is how do we repro this non-determinism? Without a test, it's going to be hard to not have a regression somewhere along the line.

[bmwiedemann]

In openSUSE, I build our packages with

osc checkout openSUSE:Factory/lowfi && cd $_
for N in 1 2 ; do
  osc build --debuginfo --vm-type=kvm --clean --noservice --keep-pkgs=RPMS.$N --release=1.1 standard
done

but it will probably be easier for you with some cargo build of https://github.com/talwat/lowfi or slower https://github.com/lycheeverse/lychee

[bmwiedemann]

So when can we expect a new version with servo/string-cache#290 integrated? With nushell the third affected package is just coming to openSUSE:Factory

[Ygg01]

I don't know the process, I can ask around Zulip for deets.