Study design, program evaluation, statistical analysis, AI evaluation, and working with statisticians.
The chi-square test of independence and the chi-square test of homogeneity use the same table, the same formula and the same p-value. They differ in how the data were collected and what you can conclude. How to tell which one you ran, and how to word and report it.
Two large studies of AI assistants found a solid average gain, and then found that the gain was very uneven. How to look for who benefits, and what it takes for a difference between groups to be believable.
This week's AI-evaluation preprints asked what happens when the grader changes: judge swaps that reshuffle a leaderboard, answer parsers that move accuracy 22 points, judges that agree for the wrong reasons, and calibrating judge scores to humans.
A public leaderboard ranks AI models from first to last, but many of the gaps between neighbors are smaller than the uncertainty in the scores. How Chatbot Arena builds its ranking, why it publishes a range of ranks, and how to read a leaderboard without crowning a winner.
Two statisticians analyze the same before-and-after weights and reach opposite conclusions, and both are right about what they computed. Lord's paradox, and what it means for any pre/post comparison between two groups.
Matching makes groups look alike on what you measured. Whether that makes your program comparison fair depends on what you didn't.
{{ p.excerpt }}
{{ noResultsMsg }}
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}