How does the Proxy-KD method compare to traditional white-box knowledge distillation when transferring knowledge from bl

How Proxy‑KD Compares to Traditional White‑Box Distillation

When you want a smaller language model to learn from a big, closed-source (black‑box) teacher, the old playbook of white‑box distillation runs into a wall. Proxy‑KD was designed to climb that wall—and according to the experiments, it actually lands higher than the white‑box methods it replaces.

What Traditional White‑Box Distillation Requires

Standard knowledge distillation expects the teacher to be wide open: the student can look at the teacher’s internal parameters and, crucially, its output probabilities (often called “soft labels”). With a black‑box teacher, those probabilities are hidden; you only get a hard class label. As a result, the usual likelihood‑based tricks become unusable, and the student can only learn from one‑hot labels—a much weaker signal [6][11][8][9][7]. So when the teacher stays tight‑lipped, classic white‑box distillation struggles.

How Proxy‑KD Works Differently

Proxy‑KD cleverly bridges the gap by training a helper “proxy” model on the black‑box teacher’s outputs. This proxy can then produce the soft labels (the rich output distributions) that the black‑box teacher refuses to share [3]. During the actual student training, Proxy‑KD mixes two kinds of supervision:

• Hard‑label distillation directly from the black‑box teacher (like a basic black‑box approach).
• Soft‑label distillation from the proxy model (which mimics what a white‑box teacher would give).

By blending both signals, Proxy‑KD intentionally combines the best of both worlds: it follows the teacher’s final decisions while also using the probability‑level hints that are normally out of reach [3][4]. The goal, as the authors put it, is to “combine the benefits of both black‑box and white‑box knowledge distillation while mitigating their respective drawbacks” [2].

The Performance Showdown

Here is the headline result: Proxy‑KD does not just improve black‑box distillation—it outright surpasses traditional white‑box knowledge distillation techniques [1][5][10][12][13][14]. In other words, even when a white‑box teacher would have the advantage of full internal access, Proxy‑KD’s proxy‑based approach beats the classic white‑box methods while still respecting the black‑box restriction. This makes Proxy‑KD a powerful tool for squeezing knowledge out of closed‑source large models into a smaller, more practical student.