Minimal Pairs Causal Experiment Report

Gemini 2.5 Flash (gdmorig) Preference Decomposition Across Numbers & Names Domains

📌 Executive Summary & Key Results

This report presents the causal evaluation of Gemini 2.5 Flash on Minimal Pairs datasets constructed via K-Means Non-Target Feature Space Stratification.

1. Main Result: Measured Minimal Pair Effects vs. Logistic Regression Predictions

The figure below compares the measured causal win rates (from Gemini 2.5 Flash minimal pair evaluations, 25 pairs × ~50 prompts × 2 presentation orderings = 2,500 total evals per feature) against the predictions of the learned Logistic Regression model fit on the original dataset (as in Figure 1 of the main paper).

Main Figure: Measured vs Predicted Deltas
Figure 1: Comparison of Measured Minimal Pair Causal Effects (Blue) vs Logistic Regression Predictions (Red) for Numbers and Names (K-Means Stratified minimal pairs).
📊 Goodness of Fit & Model Calibration Note:
The Logistic Regression model (fit on observational pairs_circle data) accurately predicts minimal pair causal win rates in experimental settings:
  • Numbers Domain: Mean Absolute Error (MAE) = 0.051, Pearson r = 0.941.
  • Names Domain (K-Means): Mean Absolute Error (MAE) = 0.078, Pearson r = 0.884.
This close correspondence confirms that linear feature preferences accurately capture model behavior across both observational and counterfactual minimal pair evaluations.
2. Interactive Minimal Pair Inspector Tables

Inspect all 25 minimal pairs, model choices, logistic predictions, and complete non-target feature balances across target features for Numbers and Names (K-Means Stratified minimal pairs):

📄 Numbers Domain Minimal Pairs (K-Means)
📄 Names Domain Minimal Pairs (K-Means)
3. Minimal Pair Generation Logic & Entity Pool Explainer

💡 Non-Target Feature Space Stratified Pairing Algorithm & Literature Context

Standard minimal pair selection can suffer from spatial clustering bias—where all 25 pairs for a feature are drawn from a single narrow sub-region of entity space (e.g. all sum_digits pairs being near centennial boundaries like 297 vs 301). Non-target feature space stratification extends classical non-parametric causal matching literature (e.g. Rosenbaum & Rubin, 1983; Stuart, 2010) by replacing local greedy selection with uniform spatial coverage across non-target covariate space:

  1. Entity Candidate Pool:
    • Numbers Domain: All 1,000 integers between 0 and 999 (11 features computed for each).
    • Names Domain: Top 2,000 candidate names by Census 2020 frequency (with decade popularity features computed from SSA 1960–2020 data; 13–14 features computed for each).
    • Note on Zero Decade Popularities (frac_XXXX = 0): When a candidate name does not appear in a given SSA decade top-1,000 table (1960, 1980, 2000, or 2020), its popularity feature frac_XXXX is assigned 0.0. Because the minimum recorded popularity for names present on the SSA tables is extremely small (ranging from ≈ 1.38 × 10-5 or 0.00138% of births in 1960 to ≈ 5.89 × 10-5 in 2020), assigning 0.0 serves as a close numerical approximation for unlisted rare names rather than introducing missing NaN values into covariate matching and logistic regression modeling.
  2. Target Feature Separation: To ensure paired entities (e1, e2) differ substantially on the target feature, the candidate entity pool is partitioned into two distinct target pools:
    • Continuous Features: High Target (top quartile, > 75th percentile) vs. Low Target (bottom quartile, < 25th percentile).
    • Binary Features: True vs. False target value pools.
  3. Non-Target Feature Scaling & K-Means Clustering: All non-target features are min-max scaled to [0, 1]D. Candidate cross-pool pairs (e1 ∈ High, e2 ∈ Low) are clustered into K=25 clusters using their midpoints in non-target feature space.
  4. Confound Distance Minimization & Set-Based Deduplication: Within each cluster k ∈ {1, ..., 25}, candidate pairs are evaluated by Euclidean distance √(∑ (fi(e1) - fi(e2))2). To enforce entity uniqueness across all 25 pairs, a global used_entities tracking set is maintained. The algorithm greedily selects the pair (e1, e2) minimizing distance such that e1 ∉ used_entities and e2 ∉ used_entities, ensuring exactly 25 × 2 = 50 unique entities per target feature.
4. Alternate Pairing Methods & Multi-Method Comparison

Below is a direct comparison of causal effect sizes across different pair selection algorithms (K-Means Stratified vs. K-Means accounting for gender vs. Greedy vs. Greedy with Deduplication vs. Logistic Regression Predictions):

Multi-Method Comparison Figure
Figure 2: Multi-method comparison across Numbers and Names domains showing K-Means, K-Means (accounting for gender), Greedy, Greedy (with Deduplication), and Logistic Regression Predictions.

🔍 Explanations of Alternate Pairing Strategies

  • Greedy: Nearest-neighbor matching in non-target feature space across target-split pools (High vs Low / True vs False). Evaluates all candidate pairs across the two target pools and greedily selects the 25 pairs that minimize Euclidean distance in non-target feature space, allowing entity reuse across pairs. Minimal pairs generated this way can suffer from spatial clustering bias (drawn from narrow sub-regions of entity space).
  • Greedy (with Deduplication): Nearest-neighbor matching across target-split pools enforcing entity uniqueness (50 distinct entities per feature). Eliminates entity reuse bias but retains spatial clustering bias.
  • K-Means (accounting for gender): Evaluates K-Means non-target space stratification by explicitly adding female_frac to the 12 regression features (13 features total). In our initial setup, random pairing within a gender-balanced name pool yielded same-gender pairs ~50% of the time, but non-target feature matching shifted this probability toward opposite-gender pairs (e.g. for starting_vowel, which skews ~65% female). Adding gender to non-target covariate matching explicitly tests and controls for this feature-induced gender mismatch.

📄 Inspectable Alternate Dataset Tables

📄 Alternate Dataset: Numbers — Greedy
📄 Alternate Dataset: Numbers — Greedy (with Deduplication)
📄 Alternate Dataset: Names — Greedy
📄 Alternate Dataset: Names — Greedy (with Deduplication)
📄 Alternate Dataset: Names — K-Means (Accounting for Gender)
5. Detailed Raw Data Tables & Model Coefficients
📄 Table A: Logistic Regression Fit Parameters & Odds Ratios

Numbers Domain Logistic Regression (Fit on pairs_circle, N=20000 rows)

Coefficient (Beta) Std Error z-score p-value Odds Ratio (exp_beta)
subtr_score_higher 4.6123 0.1308 35.2557 0.0000 100.7148
subtr_score_has_repeat_digits 0.0705 0.0695 1.0141 0.3105 1.0730
subtr_score_palindrome 1.1492 0.0996 11.5426 0.0000 3.1556
subtr_score_prime 1.6587 0.0735 22.5671 0.0000 5.2523
subtr_score_higher_sum_digits -1.4365 0.1434 -10.0174 0.0000 0.2377
subtr_score_asc_digit_order 0.4982 0.0699 7.1312 0.0000 1.6458
subtr_score_desc_digit_order -0.4336 0.0673 -6.4408 0.0000 0.6482
subtr_score_higher_order_mag -2.3320 0.2052 -11.3637 0.0000 0.0971
subtr_score_mult_2 0.0342 0.0558 0.6132 0.5397 1.0348
subtr_score_mult_5 0.6604 0.0802 8.2395 0.0000 1.9356
subtr_score_mult_10 0.1661 0.1109 1.4980 0.1341 1.1807
ent1_is_first_presented -1.9182 0.1574 -12.1852 0.0000 0.1469

Names Domain Logistic Regression (Fit on pairs_circle, N=20000 rows)

Coefficient (Beta) Std Error z-score p-value Odds Ratio (exp_beta)
subtr_score_first_alph 0.5843 0.0391 14.9459 0.0000 1.7938
subtr_score_higher_length -0.3205 0.1615 -1.9849 0.0472 0.7258
subtr_score_higher_vowel_frac 0.8737 0.1274 6.8555 0.0000 2.3958
subtr_score_higher_unique_char_frac -0.5521 0.1300 -4.2453 0.0000 0.5757
subtr_score_starting_vowel 0.2941 0.0908 3.2398 0.0012 1.3420
subtr_score_ending_vowel 1.0348 0.0704 14.6992 0.0000 2.8146
subtr_score_higher_syllable_heur -0.4271 0.1714 -2.4919 0.0127 0.6524
subtr_score_higher_bouba_frac 1.6951 0.1119 15.1460 0.0000 5.4470
subtr_score_higher_rare_letter_frac 4.0042 0.2899 13.8124 0.0000 54.8255
subtr_score_higher_frac_1960 -1.9725 0.3441 -5.7319 0.0000 0.1391
subtr_score_higher_frac_1980 1.6718 0.1982 8.4368 0.0000 5.3218
subtr_score_higher_frac_2000 1.2461 0.1557 8.0017 0.0000 3.4766
subtr_score_higher_frac_2020 6.8005 0.2110 32.2269 0.0000 898.3331
subtr_score_higher_census_freq 0.2194 0.4509 0.4866 0.6266 1.2454
ent1_is_first_presented -7.4784 0.2263 -33.0532 0.0000 0.0006
📄 Table B: Numbers Domain — Multi-Method Causal Effect Sizes & Predictions
target_feature Effect_KMeans Effect_Predicted Effect_Orig Effect_Dedup
asc_digit_order -0.1260 +0.1654 -0.1216 -0.1292
desc_digit_order +0.0668 +0.0617 +0.1892 +0.1920
has_repeat_digits +0.0580 +0.1476 +0.0162 +0.0000
is_mult_10 +0.3148 +0.1454 +0.3508 +0.3576
is_mult_2 -0.0004 +0.0683 -0.0318 -0.0334
is_mult_5 +0.0912 +0.1502 +0.0128 +0.0180
is_palindrome +0.1639 +0.2440 +0.2043 +0.2015
is_prime +0.1092 +0.2537 +0.1900 +0.1556
sum_digits -0.0424 +0.0665 -0.3596 -0.2552
value +0.2440 +0.3690 +0.1927 +0.1671
📄 Table C: Names Domain — Feature Set Causal Effect Sizes & Predictions
target_feature Effect_DefaultKMeans Pred_DefaultKMeans Effect_RegFeats Pred_RegFeats Effect_RegFeatsGender Pred_RegFeatsGender
bouba_frac +0.0717 +0.0868 +0.1427 +0.1282 +0.1431 +0.1239
census_freq +0.0504 +0.0354 +0.1494 +0.0830 +0.1404 +0.0682
ending_vowel -0.0990 +0.1192 +0.0765 +0.1445 -0.0084 +0.1466
female_frac +0.1821 +0.0544 NaN NaN +0.2091 +0.0482
frac_1960 NaN NaN -0.0707 +0.0578 -0.0408 +0.0517
frac_1980 NaN NaN +0.0225 +0.0148 +0.1360 +0.0677
frac_2000 NaN NaN +0.1005 +0.0778 +0.1729 +0.0982
frac_2020 +0.3585 +0.1893 +0.2254 +0.2073 +0.2533 +0.2014
length -0.1207 +0.0076 -0.1140 -0.0266 +0.0172 +0.0292
starting_vowel +0.1848 +0.1045 -0.0901 +0.0799 +0.1736 +0.0761
syllable_count +0.1493 -0.0187 +0.1087 -0.0353 +0.0892 -0.0561
unique_char_frac +0.0300 -0.0224 -0.0374 +0.0083 -0.0066 -0.0211
vowel_frac +0.0595 +0.1486 +0.0642 +0.1402 +0.0965 +0.1359