Skip to content

int precision problem in H3Index.SetIndexDigit #1

@oboka

Description

@oboka

I was using this port and comparing results with the original C version, and I found an error in some results. After some debugging I finally found that, in some cases, the int precision was not enough in this function H3Index.SetIndexDigit.

After changing it to this it works perfectly now:

    public void SetIndexDigit(int res, int digit)
    {
		ulong p0a = ((ulong)Constants.MAX_H3_RES - (ulong)res);
		ulong p0b = (p0a * H3_PER_DIGIT_OFFSET);
		ulong p1 = ((ulong)H3_DIGIT_MASK << (int)p0b);
		ulong p2 = (((ulong)Constants.MAX_H3_RES - (ulong)res) * (ulong)H3_PER_DIGIT_OFFSET);

		Value = (Value & (ulong)~p1) | ((ulong)digit << (int)p2);
    }

Thanks for your port! Nice work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions