generated.rs is not deterministic

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that
our lychee-0.18.1 package does produce different build results in every clean build because generated.rs varies randomly.

https://rb.zq1.de/other/lychee-compare.out

Here is an extract of the '7u8' part of the diff:

-pub const ATOM_LOCALNAME__61_6C_74_74_65_78_74 : LocalName = LocalName :: pack_inline (8392569456448790784u64 , 7u8) ;
-pub const ATOM_LOCALNAME__73_75_6D_6D_61_72_79 : LocalName = LocalName :: pack_inline (8751164148550038272u64 , 7u8) ;
-pub const ATOM_LOCALNAME__70_61_74_74_65_72_6E : LocalName = LocalName :: pack_inline (7958535042360242176u64 , 7u8) ;
+pub const ATOM_LOCALNAME__63_68_61_72_73_65_74 : LocalName = LocalName :: pack_inline (8387236816145113856u64 , 7u8) ;
+pub const ATOM_LOCALNAME__74_61_72_67_65_74_58 : LocalName = LocalName :: pack_inline (6373830867611120640u64 , 7u8) ;
+pub const ATOM_LOCALNAME__64_65_74_61_69_6C_73 : LocalName = LocalName :: pack_inline (8317138479132009472u64 , 7u8) ;
+pub const ATOM_LOCALNAME__6C_6F_61_64_69_6E_67 : LocalName = LocalName :: pack_inline (7453010313197087744u64 , 7u8) ;
+pub const ATOM_LOCALNAME__72_65_73_74_61_72_74 : LocalName = LocalName :: pack_inline (8390876208524784128u64 , 7u8) ;
+pub const ATOM_LOCALNAME__64_65_73_63_65_6E_74 : LocalName = LocalName :: pack_inline (8389754633650004992u64 , 7u8) ;
+pub const ATOM_LOCALNAME__61_72_63_74_61_6E_68 : LocalName = LocalName :: pack_inline (7525059179895218432u64 , 7u8) ;
+pub const ATOM_LOCALNAME__66_65_42_6C_65_6E_64 : LocalName = LocalName :: pack_inline (7236833166875780608u64 , 7u8) ;
+pub const ATOM_LOCALNAME__6B_65_72_6E_69_6E_67 : LocalName = LocalName :: pack_inline (7453010356431317760u64 , 7u8) ;
+pub const ATOM_LOCALNAME__74_61_72_67_65_74_59 : LocalName = LocalName :: pack_inline (6445888461649048576u64 , 7u8) ;
 pub const ATOM_LOCALNAME__76_69_65_77_62_6F_78 : LocalName = LocalName :: pack_inline (8678263271908275712u64 , 7u8) ;
+pub const ATOM_LOCALNAME__63_6C_6F_73_75_72_65 : LocalName = LocalName :: pack_inline (7310034283826799360u64 , 7u8) ;
+pub const ATOM_LOCALNAME__66_65_46_75_6E_63_52 : LocalName = LocalName :: pack_inline (5936710183775725056u64 , 7u8) ;
+pub const ATOM_LOCALNAME__65_6C_6C_69_70_73_65 : LocalName = LocalName :: pack_inline (7310310218245367040u64 , 7u8) ;
+pub const ATOM_LOCALNAME__6F_6E_6B_65_79_75_70 : LocalName = LocalName :: pack_inline (8103516581024132864u64 , 7u8) ;
+pub const ATOM_LOCALNAME__73_65_63_74_69_6F_6E : LocalName = LocalName :: pack_inline (7957695015191671552u64 , 7u8) ;
+pub const ATOM_LOCALNAME__68_65_61_64_65_72_73 : LocalName = LocalName :: pack_inline (8318822943511898112u64 , 7u8) ;
+pub const ATOM_LOCALNAME__61_72_63_73_69_6E_68 : LocalName = LocalName :: pack_inline (7525067971693273344u64 , 7u8) ;
-pub const ATOM_LOCALNAME__66_65_46_75_6E_63_42 : LocalName = LocalName :: pack_inline (4783788679168878080u64 , 7u8) ;
-pub const ATOM_LOCALNAME__70_6F_6C_79_67_6F_6E : LocalName = LocalName :: pack_inline (7957692837794902016u64 , 7u8) ;
-pub const ATOM_LOCALNAME__66_65_66_75_6E_63_62 : LocalName = LocalName :: pack_inline (7089631688919442944u64 , 7u8) ;
+pub const ATOM_LOCALNAME__6C_6F_67_62_61_73_65 : LocalName = LocalName :: pack_inline (7310293695422491648u64 , 7u8) ;
+pub const ATOM_LOCALNAME__63_68_65_63_6B_65_64 : LocalName = LocalName :: pack_inline (7234306451087844096u64 , 7u8) ;
+pub const ATOM_LOCALNAME__63_61_70_74_69_6F_6E : LocalName = LocalName :: pack_inline (7957695015409509120u64 , 7u8) ;
-pub const ATOM_LOCALNAME__74_61_72_67_65_74_58 : LocalName = LocalName :: pack_inline (6373830867611120640u64 , 7u8) ;
+pub const ATOM_LOCALNAME__6D_66_65_6E_63_65_64 : LocalName = LocalName :: pack_inline (7234297702239333632u64 , 7u8) ;
+pub const ATOM_LOCALNAME__63_6F_6D_70_61_63_74 : LocalName = LocalName :: pack_inline (8386654066594243328u64 , 7u8) ;
+pub const ATOM_LOCALNAME__69_6E_76_65_72_73_65 : LocalName = LocalName :: pack_inline (7310312400256657664u64 , 7u8) ;
+pub const ATOM_LOCALNAME__74_61_72_67_65_74_78 : LocalName = LocalName :: pack_inline (8679673876824814592u64 , 7u8) ;
+pub const ATOM_LOCALNAME__66_65_66_6C_6F_6F_64 : LocalName = LocalName :: pack_inline (7237125637572748800u64 , 7u8) ;

Version 0.18.0 was still marked as fully deterministic in my tests, so this might be a recent regression. But it might also be that I tested it less thoroughly (without debuginfo enabled)

[mre]

That's strange. We didn't make a lot of changes in 0.18.1:
lychee-v0.18.0...lychee-v0.18.1

The issue likely stems from the markup5ever crate's code generation process.

The non-deterministic behavior seems to arise from a hash table or similar data structure being used during code generation, and the implementation isn't providing stable ordering or values across builds.

Looking at their version history, I can't see anything out of the ordinary right away: https://github.com/servo/html5ever/commits/main/markup5ever

But I think this part is the non-deterministic part: https://github.com/servo/html5ever/blob/0e4e125b8672abad4ff14d1e0043d3be9305af6a/markup5ever/build.rs#L32
It's probably just a matter of sorting the local names before writing them into generated.rs?

You might want to file an issue with the markup5ever project about the non-deterministic builds.

[bmwiedemann]

There was an update to servo/html5ever / string_cache_codegen that fixed the issue. Will this change appear here automatically or do some Cargo.lock files need updating?

[mre]

We can check. Do you know if servo cut a new release of html5ever already? We merged some dependency updates yesterday.

[mre]

Seems like the last dependency updates didn't contain html5ever 0.29.2.

#1653

[bmwiedemann]

They did not tag anything since 9 months :-(

[mre]

@bmwiedemann, the latest batch of updates now contains html5ever 0.30.0.
#1656