Kahneman-Tversky Optimization (KTO): Mathematical Foundations, Prospect Theory, and Binary Signal Alignment in Large Language Models

Post-training alignment of large language models has long relied on pairwise comparison datasets. Methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) assume access to curated pairs $(x, y_w, y_l)$, where a human or automated judge explicitly marks completion $y_w$ as superior to $y_l$ for a given prompt $x$. In production environments, however, collecting paired preferences is logistically complex, expensive, and artificial. Real-world telemetry predominantly generates unpaired binary signals: users upvote or downvote an answer, accept or reject a code suggestion, or abandon a conversational turn.

Kahneman-Tversky Optimization (KTO), introduced by Ethayarajh et al. (2024), reformulates language model alignment through the lens of behavioral economics. Rather than maximizing the log-likelihood of preference pairs under a Bradley-Terry model, KTO directly maximizes the subjective utility of individual generations based on Daniel Kahneman and Amos Tversky's prospect theory. By treating alignment as utility maximization under cognitive heuristics (including loss aversion and reference-point dependence), KTO achieves alignment parity with DPO across model scales from 1B to 30B parameters while training exclusively on binary, unpaired feedback.

1. The Preference Bottleneck and Bradley-Terry Assumptions

Standard preference alignment algorithms ground their optimization in the Bradley-Terry preference model (Bradley and Terry, 1952). Given prompt $x$ and two candidate responses $y_w, y_l$, the probability that $y_w$ is preferred to $y_l$ is formalized as:

$$p^<em>(y_w \succ y_l \mid x) = \sigma\left(r^</em>(x, y_w) - r^*(x, y_l)\right)$$

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the standard logistic function, and $r^*(x, y)$ represents the latent, ground-truth scalar reward of the completion.

In classical RLHF, a proxy reward model $r_\phi$ is parameterized to minimize negative log-likelihood on preference pairs:

$$\mathcal{L}_R(r_\phi) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)\right]$$

The language model $\pi_\theta$ is subsequently updated using policy gradient algorithms (such as PPO) against the reward model with a Kullback-Leibler (KL) divergence penalty to prevent policy collapse relative to reference model $\pi_{\text{ref}}$:

$$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta}\left[r_\phi(x, y)\right] - \beta D_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)\right)$$

DPO simplified this two-stage process by leveraging the closed-form analytical solution of the optimal policy:

$$\pi^<em>(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r^</em>(x, y)\right)$$

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r^*(x, y)\right)$ is the partition function. Substituting this formulation directly into the Bradley-Terry objective eliminates the reward model entirely:

$$\mathcal{L}_{\text{DPO}}(\pi_\theta, \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$$

While computationally efficient, DPO remains strictly tied to paired data. It cannot natively update model parameters when presented with an isolated positive example (such as an approved response) or an isolated negative example (such as a toxic hallucination) without artificially pairing it against a counterfactual baseline.

2. Prospect Theory and Human-Aware Losses (HALOs)

To understand how humans evaluate outputs without explicit comparisons, Ethayarajh et al. draw from cumulative prospect theory (Tversky and Kahneman, 1992). Prospect theory models decision-making under uncertainty, establishing three core tenets of human judgment:

1. Reference Dependence: Utility is evaluated not in terms of absolute outcomes, but as gains or losses relative to a neutral reference point $z_0$.
2. Diminishing Sensitivity: The marginal perceived value of both gains and losses decreases as their magnitude grows (concavity in the gain regime, convexity in the loss regime).
3. Loss Aversion: Losses loom larger than gains of equal objective magnitude; the value function is steeper for losses than for gains.

Tversky and Kahneman parameterized this subjective value function $v(z)$ as:

$$v(z; \lambda, \alpha, z_0) = \begin{cases} (z - z_0)^\alpha & \text{if } z \ge z_0 \\ -\lambda (z_0 - z)^\alpha & \text{if } z < z_0 \end{cases}$$

Empirical estimates across human subjects typically place $\alpha \approx 0.88$ (governing risk curvature) and $\lambda \approx 2.25$ (the loss aversion multiplier).

Defining the HALO Framework

Ethayarajh et al. generalize alignment objectives into a broader mathematical family called Human-Aware Losses (HALOs). Let the implied implicit reward of a model generation be:

$$r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

where $r_\theta(x, y)$ measures the decrease in conditional surprisal in nats when transitioning from $\pi_{\text{ref}}$ to $\pi_\theta$.

Let $Q(Y' \mid x)$ denote a reference distribution over completions, and $v: \mathbb{R} \to \mathbb{R}$ represent a monotonic value function that is concave on $(0, \infty)$. A loss function $f$ is defined as a HALO if there exist direction indicators $a_{x, y} \in \{-1, +1\}$ such that:

$$f(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{(x, y) \sim \mathcal{D}}\left[a_{x, y} v\left(r_\theta(x, y) - \mathbb{E}_{y' \sim Q}[r_\theta(x, y')]\right)\right] + C_\mathcal{D}$$

Ethayarajh et al. prove that both DPO and PPO-Clip belong to the HALO family. In DPO, the reference distribution $Q(Y' \mid x)$ places all probability mass on the dispreferred output $y_l$, setting $v(z) = \log \sigma(z)$. However, treating a single paired output as the universal reference point causes severe distortions when preferences are noisy or non-transitive.

3. Mathematical Formulation of KTO

KTO derives an alignment objective directly from prospect theory without relying on paired counterfactuals. Instead of comparing two completions $y_w$ and $y_l$ side-by-side, KTO assumes that a human evaluates an individual output $y$ relative to an expected baseline representing the entire distribution of possible model outputs for prompt $x$.

The Reference Point

Under KTO, the human reference point for prompt $x$ is defined as the expected implicit reward under the current policy distribution:

$$z_0 = \mathbb{E}_{y' \sim \pi_\theta}[r_\theta(x, y')] = \beta \sum_{y'} \pi_\theta(y' \mid x) \log \frac{\pi_\theta(y' \mid x)}{\pi_{\text{ref}}(y' \mid x)} = \beta D_{\text{KL}}\left(\pi_\theta(y' \mid x) \parallel \pi_{\text{ref}}(y' \mid x)\right)$$

The reference point is therefore the KL divergence between the current policy and the reference model. If the policy shifts its distribution broadly without improving alignment, $z_0$ rises proportionally, penalizing unconditional reward inflation.

The Logistic Value Function

To avoid numerical instability from polynomial exponents while preserving the characteristic S-curve of prospect theory, KTO replaces the power law with the logistic function $\sigma$:

$$v(x, y) = \begin{cases} \lambda_D \sigma\left(\beta \left(r_\theta(x, y) - z_0\right)\right) & \text{if } y \in \mathcal{Y}_{\text{desirable}} \\ \lambda_U \sigma\left(\beta \left(z_0 - r_\theta(x, y)\right)\right) & \text{if } y \in \mathcal{Y}_{\text{undesirable}} \end{cases}$$

Here, $\beta > 0$ dictates the risk curvature, while $\lambda_D$ and $\lambda_U$ serve as loss aversion weights for desirable and undesirable examples, respectively.

The KTO Loss Objective

The complete KTO loss function minimizes the expected shortfall below maximum subjective utility:

$$\mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{(x, y) \sim \mathcal{D}}\left[\lambda_y - v(x, y)\right]$$

Written explicitly across binary subsets, where $\lambda_y = \lambda_D$ for desirable examples and $\lambda_y = \lambda_U$ for undesirable examples:

$$\mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{(x, y) \in \mathcal{D}_{\text{desirable}}}\left[\lambda_D \left(1 - \sigma\left(\beta \left(\log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - z_0\right)\right)\right)\right] + \mathbb{E}_{(x, y) \in \mathcal{D}_{\text{undesirable}}}\left[\lambda_U \left(1 - \sigma\left(\beta \left(z_0 - \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\right)\right)\right)\right]$$

Since $1 - \sigma(u) = \sigma(-u)$, the objective can also be expressed directly as:

$$\mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{\mathcal{D}_{\text{desirable}}}\left[\lambda_D \sigma\left(-\beta \left(r_\theta(x, y) - z_0\right)\right)\right] + \mathbb{E}_{\mathcal{D}_{\text{undesirable}}}\left[\lambda_U \sigma\left(-\beta \left(z_0 - r_\theta(x, y)\right)\right)\right]$$

During backpropagation, gradients are not propagated through $z_0$; the reference term acts as a detached baseline that adaptively centers the loss saturation regime.

4. Gradient Dynamics and Saturation Control

Differentiating the KTO loss with respect to parameters $\theta$ illuminates its stability characteristics. Let $d(y) = -1$ if $y$ is desirable and $d(y) = +1$ if $y$ is undesirable. Let $z = r_\theta(x, y) - z_0$. The parameter gradient is:

$$\nabla_\theta \mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{(x, y) \sim \mathcal{D}}\left[d(y) \lambda_y \sigma(\beta z) \left(1 - \sigma(\beta z)\right) \beta \nabla_\theta \log \pi_\theta(y \mid x)\right]$$

This gradient yields several critical structural properties:

1. Directional Updates: For desirable completions ($d(y) = -1$), the negative sign increases the log-likelihood of tokens under $\pi_\theta$. For undesirable completions ($d(y) = +1$), the positive sign suppresses token log-likelihoods.
2. Gradient Vanishing at Extreme Rewards: Because the derivative contains the term $\sigma(\beta z)(1 - \sigma(\beta z))$, the update weight approaches zero as the implicit reward margin $|r_\theta(x, y) - z_0| \to \infty$.

   Gradient Weight: σ(βz)(1 - σ(βz))
        ▲
   0.25 ┼          ╭───╮
        │         ╭╯   ╰╮
        │        ╭╯     ╰╮
   0.10 ┼       ╭╯       ╰╮
        │     ╭─╯         ╰─╮
   0.00 ┴────┴───────────────┴────►
           -4    -2   0    2    4
               Margin (βz)

This saturation property protects optimization against noisy, contaminated, or extreme outliers. If a training example contains a mislabeled positive with an irreconcilably low probability, or an extreme degenerate output, its gradient contribution vanishes rather than destabilizing the policy network.

5. Practical Reference Point Estimation in Microbatches

Computing the exact reference point $z_0 = \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ on every forward pass would require generating on-policy completions from $\pi_\theta(y \mid x)$, which reintroduces the substantial compute overhead of online RL.

To circumvent this bottleneck, KTO estimates $z_0$ empirically within each training microbatch of size $m$. For a microbatch of prompt-response pairs $\{(x_1, y_1), (x_2, y_2), \dots, (x_m, y_m)\}$, the responses are cyclically shifted by index $j = (i + 1) \bmod m$ to create mismatched pairs $(x_i, y_j)$. The shared batch reference estimator is computed as:

$$\hat{z}_0 = \max\left(0, \frac{1}{m} \sum_{i=1}^m \log \frac{\pi_\theta(y_j \mid x_i)}{\pi_{\text{ref}}(y_j \mid x_i)}\right)$$

Why use mismatched pairs $(x_i, y_j)$ instead of $(x_i, y_i)$?

• Aligned dataset responses $y_i$ are selected specifically for high or low quality, creating extreme reward magnitudes that bias the expectation.
• Mismatched responses $y_j$ reflect arbitrary textual continuations for prompt $x_i$, providing an unbiased estimate of background model surprisal.
• The non-negative clamping $\max(0, \cdot)$ introduces a slight positive bias while significantly suppressing estimator variance across distributed worker nodes.

When KTO is applied immediately after supervised fine-tuning (SFT) on the same dataset, the policy distribution is already close to the reference distribution, allowing practitioners to safely set $\hat{z}_0 = 0$ without sacrificing convergence stability.

6. Theoretical Properties: Utility vs. Preference Likelihood

Ethayarajh et al. establish two formal theorems showing why optimizing prospect-theoretic utility outperforms maximizing preference likelihood.

Disconnect Between Preference Likelihood and Human Utility

In DPO and Bradley-Terry models, two reward functions $r_a^(x, y)$ and $r_b^(x, y) = r_a^*(x, y) + h(x)$ belong to the same equivalence class because any prompt-dependent offset $h(x)$ cancels out when computing pairwise margins:

$$r_b^<em>(x, y_w) - r_b^</em>(x, y_l) = (r_a^<em>(x, y_w) + h(x)) - (r_a^</em>(x, y_l) + h(x)) = r_a^<em>(x, y_w) - r_a^</em>(x, y_l)$$

Consequently, $r_a^$ and $r_b^$ induce the exact same optimal policy $\pi^*$ in Bradley-Terry modeling.

However, Ethayarajh et al. prove (Theorem 4.2) that under a nonlinear human value function $v(z)$, $r_a^$ and $r_b^$ induce completely different human utility distributions unless $h(x) = 0$. Maximizing pairwise preference likelihood does not guarantee the maximization of subjective human utility across varying prompts.

Robustness to Intransitive and Contradictory Feedback

Real-world feedback contains frequent contradictions. Consider a prompt $x$ with two valid responses $y_a$ and $y_b$. Suppose a majority $p \in (0.5, 1.0)$ of evaluators prefer $y_a \succ y_b$, while a minority $1 - p$ prefer $y_b \succ y_a$.

Ethayarajh et al. prove (Theorem 4.3) that if the reference model strongly favors the minority output such that:

$$p^{1/\beta} \pi_{\text{ref}}(y_a \mid x) < (1 - p)^{1/\beta} \pi_{\text{ref}}(y_b \mid x)$$

then the optimal DPO policy will produce the minority-preferred completion $y_b$. In contrast, under a loss-neutral value function ($\lambda_D = \lambda_U$), the optimal KTO policy is guaranteed to produce the majority-preferred completion $y_a$.

7. Hyperparameters and Implementation Guidelines

• Learning Rate: 5e-6 to 1e-5 with AdamW. KTO implicit rewards have smaller magnitudes than pairwise DPO margins, requiring 2x to 10x higher learning rates.
• Risk Curvature ($\beta$): 0.05 to 0.10 for models post-SFT; 0.10 to 0.50 when aligning base pretrained models directly.
• Loss Aversion ($\lambda_D, \lambda_U$): Scaled according to class ratio. Set such that $\frac{\lambda_D n_D}{\lambda_U n_U} \in [1.0, 1.5]$, where $n_D, n_U$ are the counts of positive and negative samples.
• Microbatch Size: $\ge 2$ (effective batch size 32 to 128) to enable cyclic permutation for the $\hat{z}_0$ reference estimator.

Handling Severe Data Imbalance

In production logs, negative signals often vastly outnumber positive signals (or vice versa). In standard DPO, an unpaired ratio requires synthesizing artificial pairs or discarding excess data. KTO handles class imbalance by scaling $\lambda_D$ and $\lambda_U$.

For example, if an application logs 1 positive thumbs-up for every 10 negative thumbs-down signals ($n_D : n_U = 1 : 10$), setting $\lambda_U = 1.0$ and $\lambda_D = 13.3$ maintains the balanced gradient condition:

$$\frac{\lambda_D n_D}{\lambda_U n_U} = \frac{13.3 \times 1}{1.0 \times 10} = 1.33 \in [1.0, 1.5]$$

Empirical evaluations show that models aligned with KTO retain full benchmark accuracy even when 90% of positive examples are randomly discarded, demonstrating resilience against asymmetric real-world telemetry.

Sources

• Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. International Conference on Machine Learning (ICML). arXiv:2402.01306
• Tversky, A. and Kahneman, D. (1992). Advances in Prospect Theory: Cumulative Representation of Uncertainty. Journal of Risk and Uncertainty, 5(4), 297–323. Springer DOI:10.1007/BF00122574
• Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS. arXiv:2305.18290
• Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS. arXiv:1706.03741
• Bradley, R. A. and Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345. JSTOR:2334029