The Score Function Estimator: Mathematical Foundations of REINFORCE, Log-Derivative Tricks, and Baseline Variance Reduction
In modern artificial intelligence, standard backpropagation relies on continuous differentiability: every operation between model parameters and the final loss must provide well-behaved analytical Jacobian matrices. However, many of the most critical optimization challenges in machine learning break this continuity. Autoregressive token generation in large language models, discrete tool invocation, programmatic compiler execution, and black-box reward environments are fundamentally non-different
1 min