Fixes FP rounding issue and adds tests

[jsturtevant]

fixes #86

The issue was caused because cranelift compilation assume SSE2 even for the where as rust compiler doesn't include these instructions by default. This means when transitioning through wasmtime libcalls the parameters are lost since wasm is using SSE2 instructions and wasmtime isn't. The more advance SSE instructions require a separate pr in HL core that is needed to enable them.

[ludfjig]

Nice catch 👍 LGTM. Did you check that the tests fail without the -soft-float,+sse,+sse2?

[ludfjig]

Btw, I am not sure if you already have, but we should double check that we set all required cpu control registers for enabling sse,sse2. I know x86_64 assumes sse,sse2 are part base ISA, but still might need some setup. In addition, we should make sure these are reset between guest functions call to avoid any information leaks (if they aren't already).

[jsturtevant]

Nice catch 👍 LGTM. Did you check that the tests fail without the -soft-float,+sse,+sse2?

Yes. Also tested this against the sample in #86 (comment)

Btw, I am not sure if you already have, but we should double check that we set all required cpu control registers for enabling sse,sse2. I know x86_64 assumes sse,sse2 are part base ISA, but still might need some setup. In addition, we should make sure these are reset between guest functions call to avoid any information leaks (if they aren't already).

These seem to be enabled in KVM (I check the mxcsr register). I haven't verified on Windows/MSVH. Maybe the best thing do here it push hyperlight-dev/hyperlight#686 forward as well.

Good call on clearing registers which will need to happen in HL, will work on a change for that.