Base Score - Wrong scale

[jmonteroers]

First of all, thank you for providing such an exciting utility.

In the code, the base score is assumed to be in the logit scale (for example, when defining tree_leafs in XGBScorecardConstructor.construct_scorecard). However, the following MRE shows that the base score seems to be instead in the probability scale:

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from xgboost import DMatrix
from xbooster.constructor import XGBScorecardConstructor

import numpy as np
from scipy.special import expit


data = load_breast_cancer()
X, y = data.data, data.target

# Build and fit classifier with only two trees
model = xgb.XGBClassifier(n_estimators=2, eval_metric='logloss')
model.fit(X, y)


# Retrieve base score using constructor class
scorecard_constructor = XGBScorecardConstructor(
    model, X, y
)
bscore = scorecard_constructor.base_score

# Return scores using individual trees
individual_preds = [tree.predict(DMatrix(X), output_margin=True) for tree in model.get_booster()]
# hacky way to get base score
# base_score_matrix = (base score + score_T1)+ (base_score + score_T2) - (base_score + score_T1 + score_T2) = base_score 
# NOTE: this must be in the logit scale
base_score_matrix = sum(individual_preds) - model.predict(X, output_margin=True)
# Transform into probability scale
base_score_matrix = expit(base_score_matrix)

# Assertion
check_arr = np.isclose(base_score_matrix, bscore)
assert np.all(check_arr), "Base score not in probability scale"

The trick is to use two trees only to have the base score repeated twice in their individual predictions, and then substract the actual prediction in the raw score scale (logit) to identify the actual raw score being used, again in the logit scale. Then I compare it against the base score used in the XGBScorecardConstructor class. They only match after transforming the former into the probability scale, that is why the assertion does not fail. I would advise to add this idea as a test to the test suite at some point.

Happy to help if this indeed needs some fixing 😄

[jmonteroers]

Further evidence that the base score is in the probability scale, in this demo the base score is defined as 0.5, and has to be transformed back into the logit scale to be used as a base score.

[deburky]

Hi @jmonteroers, thanks a ton for the clear MRE and the "two-tree trick." 🙌

I remember having spent some time verifying this and testing XGBoost examples with base score and a logit. The way it is currently implemented in the constructor matches the logic of predict_proba method of XGBClassifier.

Below I provide an example:

Details

import json
from pprint import pprint

import numpy as np
import xgboost as xgb
from scipy.special import logit
from sklearn.datasets import load_breast_cancer
from xgboost import plot_tree
from xbooster.constructor import XGBScorecardConstructor


def extract_model_param(booster, param):
    """
    Extracts a specific parameter from the XGBoost model configuration.

    Args:
        param (str): The name of the parameter to extract.

    Returns:
        float: The extracted parameter value.
    """
    config = json.loads(booster.save_config())
    if param == "base_score":
        return float(config["learner"]["learner_model_param"][param])
    return float(config["learner"]["gradient_booster"]["tree_train_param"][param])


# Load data
data = load_breast_cancer()
X, y = data.data, data.target
n_rounds = 2

# Fit model
model = xgb.XGBClassifier(n_estimators=n_rounds, max_depth=2, eval_metric="logloss")
model.fit(X, y)

# Get booster object
booster = model.get_booster()
base_score = extract_model_param(booster, "base_score")

# Plot a tree
tree_idx = 0
plot_tree(booster, num_trees=tree_idx)
# Print dump from tree 1
pprint(booster[tree_idx].get_dump())


scorecard_constructor = XGBScorecardConstructor(model, X, y)
scorecard_constructor.base_score

# Build scorecard
scorecard_constructor.construct_scorecard().query(f"Tree == {tree_idx}").iloc[0]

The results in the scorecard table will match both the tree leaf values in the plot and booster dump.

What is happening

• XGBoost stores base_score in probability space (e.g., 0.5).
• The internal additive model operates in margin space. So when we use base_score as a margin we do:
  base_margin = logit(base_score).

Your “two-tree trick” isolates that margin: summing per-tree raw outputs (margins) and subtracting the full raw prediction leaves the base margin; applying expit puts it back into the probability space, which then matches the constructor’s base_score.

Within xbooster we compute per-tree deltas against a fixed base and sum them. That’s equivalent to the iterative “update base_margin every round” style often shown in examples; it’s just more convenient for extracting per-tree contributions.

For completeness, here’s the tiny helper we use to mirror get_leafs’ per-tree-delta logic:

Details

import numpy as np
import xgboost as xgb
from scipy.special import logit, expit


def calculate_results_xbooster(booster, X, base_score, logit_scale=True):
    """
    Fixed base (prob or margin depending on `logit_scale`), per-tree raw deltas, sum -> final margin.
    """
    n_rounds = booster.num_boosted_rounds()
    base = logit(base_score) if logit_scale else base_score  # convert prob -> margin if needed
    base_vec = np.full((X.shape[0],), base)
    dm = xgb.DMatrix(X, base_margin=base_vec)

    per_tree = []
    for i in range(n_rounds):
        pred = booster.predict(dm, iteration_range=(i, i + 1), output_margin=True)
        per_tree.append((pred - base_vec).ravel())  # tree contribution in raw space

    return base_vec + np.column_stack(per_tree).sum(axis=1) if per_tree else base_vec


results_proba_scale = calculate_results_xbooster(booster, X, base_score, logit_scale=True)
results_logit_scale = calculate_results_xbooster(booster, X, base_score, logit_scale=False)

print("predict_proba:", model.predict_proba(X)[:, 1][:5])
print("internal (logit):", expit(results_logit_scale)[:5])
print("internal (proba):", expit(results_proba_scale)[:5])
# predict_proba: [0.48015887 0.3213339  0.3213339  0.6082139  0.48015887]
# internal (logit): [0.50891299 0.34692429 0.34692429 0.63526533 0.50891299]
# internal (proba): [0.48015882 0.32133387 0.32133387 0.60821386 0.48015882]

I’ll refine the docs to make this distinction clearer for a user. If you’d like to contribute the "two-tree" test you suggested, that would be fantastic, otherwise I'll try to add it in upcoming releases.

If you spot anything inconsistent with the above, I’m happy to iterate, thanks again for the sharp deep-dive and the neat test idea!