Detect 2019 modules and allow to install 2019, skipping dmg install for now #392

lacostej · 2019-11-25T15:40:06Z

Definitively WIP with Windows not supported at all.

The approach is to use the modules.asset files to detect Playback Engines. @niezbop any better idea on how to support this?

I will also need some help to test this on Windows at some point. Maybe @BastianBlokland would like to test it on Linux?

BastianBlokland · 2019-11-25T15:43:09Z

Thanks, yeah should be able to test it out in a few hours.

BastianBlokland · 2019-11-25T18:43:17Z

Tried out the branch however there are a few issues still.

In line 261 of installation.rb it calls IvyPlaybackEngineUtils.list_module_configs and the line after it aborts if no package as found. Currently that doesn't go into the modules.asset path. Also 2 lines below that it tries to get the version from the ivy file.

If i change to ModulePlaybackEngineUtils.list_module_configs and hardcode the version then it work perfectly.

So i guess the biggest question remaining is how to get the version number.

BastianBlokland · 2019-11-25T18:54:41Z

Did a grep for it and only get binary files 😢

/opt/unity-editor-2019.2.13f1:  grep -R "2019.2.13f1" *
Binary file Editor/Unity matches
Binary file Editor/Data/Resources/unity_builtin_extra matches
Binary file Editor/Data/Resources/unity editor resources matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/SpriteAtlasDatabase.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/SceneVisibilityState.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/ScriptMapper matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/InspectorExpandedItems.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/AnnotationManager matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/BuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/assetDatabase3 matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/MonoManager.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.2d-3.2.3/EditorUserBuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/SpriteAtlasDatabase.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/SceneVisibilityState.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/ScriptMapper matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/InspectorExpandedItems.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/AnnotationManager matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/BuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/assetDatabase3 matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/MonoManager.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.lightweight-6.9.1/EditorUserBuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/SpriteAtlasDatabase.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/SceneVisibilityState.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/ScriptMapper matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/InspectorExpandedItems.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/AnnotationManager matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/BuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/assetDatabase3 matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/MonoManager.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.hd-6.9.0-preview/EditorUserBuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/SpriteAtlasDatabase.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/SceneVisibilityState.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/ScriptMapper matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/InspectorExpandedItems.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/AnnotationManager matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/BuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/assetDatabase3 matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/MonoManager.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3d-3.1.4/EditorUserBuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/SpriteAtlasDatabase.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/SceneVisibilityState.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/ScriptMapper matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/InspectorExpandedItems.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/AnnotationManager matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/BuildSettings.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/assetDatabase3 matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/MonoManager.asset matches
Binary file Editor/Data/Resources/PackageManager/ProjectTemplates/libcache/com.unity.template.3dextra-3.2.4/EditorUserBuildSettings.asset matches
Binary file Editor/Data/Tools/WebExtract matches
Binary file Editor/Data/Tools/binary2text matches
Binary file Editor/Data/PlaybackEngines/LinuxStandaloneSupport/Variations/linux64_headless_nondevelopment_mono/LinuxPlayer matches
Binary file Editor/Data/PlaybackEngines/LinuxStandaloneSupport/Variations/linux64_withgfx_development_mono/LinuxPlayer matches
Binary file Editor/Data/PlaybackEngines/LinuxStandaloneSupport/Variations/linux64_withgfx_nondevelopment_mono/LinuxPlayer matches
Binary file Editor/Data/PlaybackEngines/LinuxStandaloneSupport/Variations/linux64_headless_development_mono/LinuxPlayer matches
Binary file Editor/Data/Managed/UnityEditor.Graphs.dll matches
Binary file Editor/Data/UnityExtensions/Unity/UnityVR/Editor/UnityEditor.VR.dll matches
Binary file Editor/BugReporter/UnityBugReporter matches

niezbop · 2019-11-26T11:37:15Z

@lacostej I think that this is a solid alternative to the ivy file since it seems to be as present as it used to be so can rely on this. However, this is super lackluster and does not support a ton of the modules. For instance:

$ u3d available -u 2019.2.12f1 -p
Version 2019.2.12f1: https://download.unity3d.com/download_unity/b1a7e1fb4fa5/
Packages:
 - Unity
 - Mono
 - Visualstudio
 - Android
 - Ios
 - Appletv
 - Linux
 - Mac-il2cpp
 - Webgl
 - Windows-mono
 - Facebook-games
 - Lumin
 - Documentation
 - Language-ja
 - Language-ko
 - Language-zh-hans
 - Language-zh-hant
 - Android-sdk-ndk-tools
 - Android-sdk-platform-tools
 - Android-sdk-build-tools
 - Android-sdk-platforms
 - Android-ndk
 - Android-open-jdk

Many of those would not be covered by the modules.asset file, for instance all NDK/SDK tools and platforms, which are rather sizeable and large to download.

So I think that if we want to address this issue correctly we need to tackle those as well. Languages could be fairly easy: Dir.glob('Localization/*').map {|path| File.basename path }.map {|path| "Localization-#{path.gsub('.po', '')}" }, while the other may be a bit trickier.

niezbop · 2019-11-26T11:41:50Z

@BastianBlokland Thanks a bunch for your help on this!

I think that we shouldn't be too afraid of looking into binaries. We already do this for Windows to get the version number from the exe:

u3d/lib/u3d/installation.rb

Lines 327 to 362 in e980ddd

    
           def string_file_info(info, path) 
        
             require "Win32API" 
        
             get_file_version_info_size = Win32API.new('version.dll', 'GetFileVersionInfoSize', 'PP', 'L') 
        
             get_file_version_info = Win32API.new('version.dll', 'GetFileVersionInfo', 'PIIP', 'I') 
        
             ver_query_value = Win32API.new('version.dll', 'VerQueryValue', 'PPPP', 'I') 
        
             rtl_move_memory = Win32API.new('kernel32.dll', 'RtlMoveMemory', 'PLL', 'I') 
        
             file = path.tr("/", "\\") 
        
             buf = [0].pack('L') 
        
             version_size = get_file_version_info_size.call(file + "\0", buf) 
        
             raise Exception if version_size.zero? # TODO: use GetLastError 
        
             version_info = 0.chr * version_size 
        
             version_ok = get_file_version_info.call(file, 0, version_size, version_info) 
        
             raise Exception if version_ok.zero? # TODO: use GetLastError 
        
             # hardcoding lang codepage 
        
             struct_path = "\\StringFileInfo\\040904b0\\#{info}" 
        
             addr = [0].pack('L') 
        
             size = [0].pack('L') 
        
             query_ok = ver_query_value.call(version_info, struct_path + "\0", addr, size) 
        
             raise Exception if query_ok.zero? 
        
             raddr = addr.unpack('L')[0] 
        
             rsize = size.unpack('L')[0] 
        
             info = Array.new(rsize, 0).pack('L*') 
        
             rtl_move_memory.call(info, raddr, info.length) 
        
             info.strip 
        
           rescue StandardError => e 
        
             UI.verbose("Failure to find '#{info}' under '#{path}': #{e}") 
        
             UI.verbose(e.backtrace) 
        
             nil 
        
           end

So I believe that we should try looking into this.

BastianBlokland · 2019-11-26T18:11:21Z

Did some digging on the Unity executable, unfortunately all the occurrences seem to be in the rodata section of the elf file:

(Numbers of the left are byte offsets into the file)

 grep -abPo '.{0,50}2019.2.13f1.{0,50}' Unity
  87694023: tingsm_EditorVersionm_EditorVersionWithRevision2019.2.13f1 (e20f6c7e5017)UI/UI-DefaultETC1.shaderUI/UI-Def
  87896891: cesResource previews-buildBuiltinPreviewsUnity 2019.2.13f11.0.0-alpha1.0.0-alpha.11.0.0-alpha-11.0.0-alp
  88997909: lized file version. File: "%s". Expected version: 2019.2.13f1. Actual version: %s.Unable to load the number of
  91410862: torRandSameAsRandNvalue >= 30.0fvalue <= 40.0f2019.2.13f1_e20f6c7e5017value >= 30value < 40value > 30va
  93488722: ainCurlExecutor::SubmitCurlRequestUnityPlayer/2019.2.13f1 (UnityWebRequest/1.0, libcurl/7.52.0-DEV)U
  93682693: ke Gecko) Chrome/37.0.2062.94 Safari/537.36 Unity/2019.2.13f1 (unity3d.com;launcheronUnityEventhttp://editor

I guess those are all just constants they use in their source code:

readelf -p .rodata Unity | grep 2019.2
  [12eaf9]  2019.2.13f1 (e20f6c7e5017)
  [160367]  Unity 2019.2.13f1
  [18a5d1]  UNITY_2019_2_OR_NEWER
  [26d008]  Invalid serialized file version. File: "%s". Expected version: 2019.2.13f1. Actual version: %s.
  [2be6bb]  /Localization/2019.2.13
  [4ba1e0]  2019.2.13f1_e20f6c7e5017
  [6b5678]  UnityPlayer/2019.2.13f1 (UnityWebRequest/1.0, libcurl/7.52.0-DEV)
  [6e4bc8]  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36 Unity/2019.2.13f1 (unity3d.com;
  [7e1dc8]  2019.2/release

Not sure how we would access those in any sane way.

BastianBlokland · 2019-11-26T19:05:29Z

Kinda crazy but the Editor/Data/Resources/unity_builtin_extra file might be useable.

Seems to be present across a bunch of unity versions and has the version number at a fixed offset into the file:

od -j 20 -N 11 -a unity-editor-2017.4.10f1/Editor/Data/Resources/unity_builtin_extra
0000024   2   0   1   7   .   4   .   1   0   f   1

od -j 20 -N 10 -a unity-editor-2018.2.5f1/Editor/Data/Resources/unity_builtin_extra
0000024   2   0   1   8   .   2   .   5   f   1

od -j 20 -N 10 -a unity-editor-2019.1.0f2/Editor/Data/Resources/unity_builtin_extra
0000024   2   0   1   9   .   1   .   0   f   2

bdominguez · 2019-12-18T13:30:22Z

Any updates on this?

lacostej · 2020-02-18T15:54:59Z

Any updates on this?

got delayed, and January was mostly offtime. Resuming this now.

lacostej · 2020-02-18T21:05:47Z

@lacostej I think that this is a solid alternative to the ivy file since it seems to be as present as it used to be so can rely on this. However, this is super lackluster and does not support a ton of the modules. For instance:
$ u3d available -u 2019.2.12f1 -p
Version 2019.2.12f1: https://download.unity3d.com/download_unity/b1a7e1fb4fa5/
Packages:
 - Unity
 - Mono
 - Visualstudio
 - Android
 - Ios
 - Appletv
 - Linux
 - Mac-il2cpp
 - Webgl
 - Windows-mono
 - Facebook-games
 - Lumin
 - Documentation
 - Language-ja
 - Language-ko
 - Language-zh-hans
 - Language-zh-hant
 - Android-sdk-ndk-tools
 - Android-sdk-platform-tools
 - Android-sdk-build-tools
 - Android-sdk-platforms
 - Android-ndk
 - Android-open-jdk
Many of those would not be covered by the modules.asset file, for instance all NDK/SDK tools and platforms, which are rather sizeable and large to download.

So I think that if we want to address this issue correctly we need to tackle those as well. Languages could be fairly easy: Dir.glob('Localization/*').map {|path| File.basename path }.map {|path| "Localization-#{path.gsub('.po', '')}" }, while the other may be a bit trickier.

According to https://forum.unity.com/threads/manual-install-sdk-ndk-tools-in-unity-2019-1-0f2.663907/ NDK goes under PlaybackEngines as well.

… now)

lacostej · 2020-02-25T21:14:12Z

This doesn't solve the reinstalll of the modules yet, but it at least makes installing 2019 functional. Let's use it and release a new version with this, and I'll implement the #394 support after that. WDYT?

bdominguez · 2020-02-26T08:05:51Z

In our case we are interested in the original purpose of this pull request: to be able to correctly detect the installed modules.

We're wasting a lot of time doing unnecessary reinstallations through Jenkins.

I agree with the approach as long as it does not take too long to solve the main problem.

lacostej · 2020-02-26T09:21:19Z

@bdominguez I heard you! Just a bit too tight for me to deal with this this week. It's next in pipeline.

niezbop

Minor nitpick, but it looks good if the objective is to bring things up for Unity 2019

niezbop · 2020-02-26T09:23:23Z

lib/u3d.rb

+    deprecated = {
+      PlaybackEngineUtils: IvyPlaybackEngineUtils,
+      INIParser: INIModulesParser
+    }


I would move this out of the method, maybe to a constant like DEPRECATED_CLASSES or something

Will do in a future cleanup.

lacostej force-pushed the feature/detect_2019_modules branch from acc6228 to a6d6269 Compare February 18, 2020 17:33

lacostej changed the title ~~WIP detect 2019 modules~~ Detect 2019 modules Feb 18, 2020

lacostej added 6 commits February 25, 2020 22:10

WIP detect 2019 modules

18a7393

Support reading version from unity_builtin_extra (on Linux only right…

770f289

… now)

Find a clean way to implement deprecation

4d966c0

Retrofit backward compatibility of INIParser into new design

b96965d

Support downloading/installing dmg modules

ffc0e69

Slightly refactor return

20b780a

lacostej force-pushed the feature/detect_2019_modules branch from 69fc3c2 to 20b780a Compare February 25, 2020 21:11

lacostej changed the title ~~Detect 2019 modules~~ Detect 2019 modules and allow to install 2019, skipping dmg install for now Feb 25, 2020

lacostej requested a review from niezbop February 25, 2020 21:13

niezbop approved these changes Feb 26, 2020

View reviewed changes

lacostej merged commit a46fc31 into DragonBox:master Feb 26, 2020

lacostej deleted the feature/detect_2019_modules branch February 26, 2020 11:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect 2019 modules and allow to install 2019, skipping dmg install for now #392

Detect 2019 modules and allow to install 2019, skipping dmg install for now #392

lacostej commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019 •

edited

Loading

niezbop commented Nov 26, 2019

niezbop commented Nov 26, 2019 •

edited

Loading

BastianBlokland commented Nov 26, 2019 •

edited

Loading

BastianBlokland commented Nov 26, 2019

bdominguez commented Dec 18, 2019

lacostej commented Feb 18, 2020

lacostej commented Feb 18, 2020

lacostej commented Feb 25, 2020

bdominguez commented Feb 26, 2020

lacostej commented Feb 26, 2020

niezbop left a comment

niezbop Feb 26, 2020

lacostej Feb 26, 2020

Detect 2019 modules and allow to install 2019, skipping dmg install for now #392

Detect 2019 modules and allow to install 2019, skipping dmg install for now #392

Conversation

lacostej commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019

BastianBlokland commented Nov 25, 2019 • edited Loading

niezbop commented Nov 26, 2019

niezbop commented Nov 26, 2019 • edited Loading

BastianBlokland commented Nov 26, 2019 • edited Loading

BastianBlokland commented Nov 26, 2019

bdominguez commented Dec 18, 2019

lacostej commented Feb 18, 2020

lacostej commented Feb 18, 2020

lacostej commented Feb 25, 2020

bdominguez commented Feb 26, 2020

lacostej commented Feb 26, 2020

niezbop left a comment

Choose a reason for hiding this comment

niezbop Feb 26, 2020

Choose a reason for hiding this comment

lacostej Feb 26, 2020

Choose a reason for hiding this comment

BastianBlokland commented Nov 25, 2019 •

edited

Loading

niezbop commented Nov 26, 2019 •

edited

Loading

BastianBlokland commented Nov 26, 2019 •

edited

Loading