gdmorig) Preference Decomposition Across Numbers & Names DomainsThis report presents the causal evaluation of Gemini 2.5 Flash on Minimal Pairs datasets constructed via K-Means Non-Target Feature Space Stratification.
is_mult_10 (+31% to +35%), value (+17% to +24%), is_palindrome (+16% to +20%), is_prime (+11% to +19%).frac_2020 (+25% to +36%), female_frac (+18% to +21%), bouba_frac (+11% to +14%), census_freq (+14% to +19%).starting_vowel (since starting-vowel names skew ~65% female in Census data). To test whether this feature-induced gender mismatch affected confounding, we evaluate a variant that explicitly adds female_frac to non-target covariate matching (13 features total, reported in Section 4), demonstrating that controlling for gender resolves the starting_vowel confound.The figure below compares the measured causal win rates (from Gemini 2.5 Flash minimal pair evaluations, 25 pairs × ~50 prompts × 2 presentation orderings = 2,500 total evals per feature) against the predictions of the learned Logistic Regression model fit on the original dataset (as in Figure 1 of the main paper).
pairs_circle data) accurately predicts minimal pair causal win rates in experimental settings:
Inspect all 25 minimal pairs, model choices, logistic predictions, and complete non-target feature balances across target features for Numbers and Names (K-Means Stratified minimal pairs):
Standard minimal pair selection can suffer from spatial clustering bias—where all 25 pairs for a feature are drawn from a single narrow sub-region of entity space (e.g. all sum_digits pairs being near centennial boundaries like 297 vs 301). Non-target feature space stratification extends classical non-parametric causal matching literature (e.g. Rosenbaum & Rubin, 1983; Stuart, 2010) by replacing local greedy selection with uniform spatial coverage across non-target covariate space:
frac_XXXX = 0): When a candidate name does not appear in a given SSA decade top-1,000 table (1960, 1980, 2000, or 2020), its popularity feature frac_XXXX is assigned 0.0. Because the minimum recorded popularity for names present on the SSA tables is extremely small (ranging from ≈ 1.38 × 10-5 or 0.00138% of births in 1960 to ≈ 5.89 × 10-5 in 2020), assigning 0.0 serves as a close numerical approximation for unlisted rare names rather than introducing missing NaN values into covariate matching and logistic regression modeling.used_entities tracking set is maintained. The algorithm greedily selects the pair (e1, e2) minimizing distance such that e1 ∉ used_entities and e2 ∉ used_entities, ensuring exactly 25 × 2 = 50 unique entities per target feature.Below is a direct comparison of causal effect sizes across different pair selection algorithms (K-Means Stratified vs. K-Means accounting for gender vs. Greedy vs. Greedy with Deduplication vs. Logistic Regression Predictions):
female_frac to the 12 regression features (13 features total). In our initial setup, random pairing within a gender-balanced name pool yielded same-gender pairs ~50% of the time, but non-target feature matching shifted this probability toward opposite-gender pairs (e.g. for starting_vowel, which skews ~65% female). Adding gender to non-target covariate matching explicitly tests and controls for this feature-induced gender mismatch.| Coefficient (Beta) | Std Error | z-score | p-value | Odds Ratio (exp_beta) | |
|---|---|---|---|---|---|
| subtr_score_higher | 4.6123 | 0.1308 | 35.2557 | 0.0000 | 100.7148 |
| subtr_score_has_repeat_digits | 0.0705 | 0.0695 | 1.0141 | 0.3105 | 1.0730 |
| subtr_score_palindrome | 1.1492 | 0.0996 | 11.5426 | 0.0000 | 3.1556 |
| subtr_score_prime | 1.6587 | 0.0735 | 22.5671 | 0.0000 | 5.2523 |
| subtr_score_higher_sum_digits | -1.4365 | 0.1434 | -10.0174 | 0.0000 | 0.2377 |
| subtr_score_asc_digit_order | 0.4982 | 0.0699 | 7.1312 | 0.0000 | 1.6458 |
| subtr_score_desc_digit_order | -0.4336 | 0.0673 | -6.4408 | 0.0000 | 0.6482 |
| subtr_score_higher_order_mag | -2.3320 | 0.2052 | -11.3637 | 0.0000 | 0.0971 |
| subtr_score_mult_2 | 0.0342 | 0.0558 | 0.6132 | 0.5397 | 1.0348 |
| subtr_score_mult_5 | 0.6604 | 0.0802 | 8.2395 | 0.0000 | 1.9356 |
| subtr_score_mult_10 | 0.1661 | 0.1109 | 1.4980 | 0.1341 | 1.1807 |
| ent1_is_first_presented | -1.9182 | 0.1574 | -12.1852 | 0.0000 | 0.1469 |
| Coefficient (Beta) | Std Error | z-score | p-value | Odds Ratio (exp_beta) | |
|---|---|---|---|---|---|
| subtr_score_first_alph | 0.5843 | 0.0391 | 14.9459 | 0.0000 | 1.7938 |
| subtr_score_higher_length | -0.3205 | 0.1615 | -1.9849 | 0.0472 | 0.7258 |
| subtr_score_higher_vowel_frac | 0.8737 | 0.1274 | 6.8555 | 0.0000 | 2.3958 |
| subtr_score_higher_unique_char_frac | -0.5521 | 0.1300 | -4.2453 | 0.0000 | 0.5757 |
| subtr_score_starting_vowel | 0.2941 | 0.0908 | 3.2398 | 0.0012 | 1.3420 |
| subtr_score_ending_vowel | 1.0348 | 0.0704 | 14.6992 | 0.0000 | 2.8146 |
| subtr_score_higher_syllable_heur | -0.4271 | 0.1714 | -2.4919 | 0.0127 | 0.6524 |
| subtr_score_higher_bouba_frac | 1.6951 | 0.1119 | 15.1460 | 0.0000 | 5.4470 |
| subtr_score_higher_rare_letter_frac | 4.0042 | 0.2899 | 13.8124 | 0.0000 | 54.8255 |
| subtr_score_higher_frac_1960 | -1.9725 | 0.3441 | -5.7319 | 0.0000 | 0.1391 |
| subtr_score_higher_frac_1980 | 1.6718 | 0.1982 | 8.4368 | 0.0000 | 5.3218 |
| subtr_score_higher_frac_2000 | 1.2461 | 0.1557 | 8.0017 | 0.0000 | 3.4766 |
| subtr_score_higher_frac_2020 | 6.8005 | 0.2110 | 32.2269 | 0.0000 | 898.3331 |
| subtr_score_higher_census_freq | 0.2194 | 0.4509 | 0.4866 | 0.6266 | 1.2454 |
| ent1_is_first_presented | -7.4784 | 0.2263 | -33.0532 | 0.0000 | 0.0006 |
| target_feature | Effect_KMeans | Effect_Predicted | Effect_Orig | Effect_Dedup |
|---|---|---|---|---|
| asc_digit_order | -0.1260 | +0.1654 | -0.1216 | -0.1292 |
| desc_digit_order | +0.0668 | +0.0617 | +0.1892 | +0.1920 |
| has_repeat_digits | +0.0580 | +0.1476 | +0.0162 | +0.0000 |
| is_mult_10 | +0.3148 | +0.1454 | +0.3508 | +0.3576 |
| is_mult_2 | -0.0004 | +0.0683 | -0.0318 | -0.0334 |
| is_mult_5 | +0.0912 | +0.1502 | +0.0128 | +0.0180 |
| is_palindrome | +0.1639 | +0.2440 | +0.2043 | +0.2015 |
| is_prime | +0.1092 | +0.2537 | +0.1900 | +0.1556 |
| sum_digits | -0.0424 | +0.0665 | -0.3596 | -0.2552 |
| value | +0.2440 | +0.3690 | +0.1927 | +0.1671 |
| target_feature | Effect_DefaultKMeans | Pred_DefaultKMeans | Effect_RegFeats | Pred_RegFeats | Effect_RegFeatsGender | Pred_RegFeatsGender |
|---|---|---|---|---|---|---|
| bouba_frac | +0.0717 | +0.0868 | +0.1427 | +0.1282 | +0.1431 | +0.1239 |
| census_freq | +0.0504 | +0.0354 | +0.1494 | +0.0830 | +0.1404 | +0.0682 |
| ending_vowel | -0.0990 | +0.1192 | +0.0765 | +0.1445 | -0.0084 | +0.1466 |
| female_frac | +0.1821 | +0.0544 | NaN | NaN | +0.2091 | +0.0482 |
| frac_1960 | NaN | NaN | -0.0707 | +0.0578 | -0.0408 | +0.0517 |
| frac_1980 | NaN | NaN | +0.0225 | +0.0148 | +0.1360 | +0.0677 |
| frac_2000 | NaN | NaN | +0.1005 | +0.0778 | +0.1729 | +0.0982 |
| frac_2020 | +0.3585 | +0.1893 | +0.2254 | +0.2073 | +0.2533 | +0.2014 |
| length | -0.1207 | +0.0076 | -0.1140 | -0.0266 | +0.0172 | +0.0292 |
| starting_vowel | +0.1848 | +0.1045 | -0.0901 | +0.0799 | +0.1736 | +0.0761 |
| syllable_count | +0.1493 | -0.0187 | +0.1087 | -0.0353 | +0.0892 | -0.0561 |
| unique_char_frac | +0.0300 | -0.0224 | -0.0374 | +0.0083 | -0.0066 | -0.0211 |
| vowel_frac | +0.0595 | +0.1486 | +0.0642 | +0.1402 | +0.0965 | +0.1359 |