Skip to content

Methodology docs: per-game aggregation example arithmetic is incorrect #15

@rbeauchamp

Description

@rbeauchamp

The ARC-AGI-3 scoring methodology page appears to have an internal arithmetic inconsistency in the per-game aggregation example.

Docs page:

Current content:

  • Levels 1-3 are described as 0.25 each
  • Levels 4-7 are 0
  • The formula line then shows (0.25x1 + 0.25x2 + 0.5x3 + 0x4 + 0x5 + 0x6 + 0x7) / (1+2+3+4+5+6+7) = 0.01289 (1.29%)

Why this is inconsistent:

  1. The bullet list says the first three level scores are all 0.25, but the formula uses 0.5 for the third weighted term.
  2. The stated result 0.01289 matches neither interpretation.

Checked arithmetic:

  • If levels 1-3 are all 0.25, then the weighted average is:
    • (0.25*1 + 0.25*2 + 0.25*3) / 28 = 1.5 / 28 = 0.0535714286 (about 5.36%)
  • If the intended formula was 0.25, 0.25, 0.5, then:
    • (0.25*1 + 0.25*2 + 0.5*3) / 28 = 2.25 / 28 = 0.0803571429 (about 8.04%)

So the example result should be corrected, and either the bullet list or the formula line should be made consistent with the other.

This looks like a docs/example bug rather than a bug in the scoring rules themselves.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions