01 UNDER THE HOOD
Each corpus is a distribution of texts, and each text a distribution of tokens. Double slicing turns this nested optimal transport problem into many one-dimensional ones, each solved exactly by sorting.
5,000 text sequences
5,000 text sequences
The OWT benchmark uses 5,000 sequences per corpus. Scroll on to follow 6 synthetic sequences per corpus so every one stays visible.
A sequence can be a sentence, a passage, or a fixed-length chunk. For example:
Small birds fly across the quiet lake as the sun rises.
A token can be a word, part of a word, or punctuation. This token split illustrates the idea; the plotted positions are synthetic.
STEP 01 / 09
A reference corpus and a generated corpus, each a collection of text sequences. The OpenWebText benchmark uses 5,000 per side; here we follow six per corpus.
Scroll to move through the method. The picture follows your reading position.
STEP 02 / 09
The encoder maps each token to a contextual hidden state, so each text becomes a cloud of token states. One labeled cloud stands for one sequence.
Here d = 2 so the clouds can be drawn. GPT-2 Large uses d = 1,280 and Dream 7B uses d = 3,584.
STEP 03 / 09
Draw a unit direction ξ uniformly from the sphere. Every sequence in both corpora is sliced along this same direction.
SOL repeats the whole construction for many directions and averages the results.
STEP 04 / 09
Each token state moves perpendicularly onto the line through ξ. Its position along the line, ⟨ξ, hᵢ⟩, is now one number.
STEP 05 / 09
The projected values of each sequence form a one-dimensional distribution: one row per sequence, R1–R6 and G1–G6, on a shared scale.
This is the first slice. Every sequence keeps its own identity.
STEP 06 / 09
Sorting a row gives its quantile function Q: the rank of a token sets the fraction u, and its value sets the height Q(u).
Each dot lands on its sequence’s curve. The 12 curves share one set of axes.
STEP 07 / 09
A Gaussian-process draw g turns each curve into one number. This second slice leaves each corpus with six scalars, one per sequence.
μᵢ is the token distribution of sequence i; M maps it to one number.
STEP 08 / 09
The two sets of scalars form a one-dimensional optimal transport problem. It is solved exactly by sorting, in O(n log n): match the values by rank and add the squared gaps.
Matching follows rank, so R1 can meet any G sequence.
STEP 09 / 09
Repeat for many directions ξ and functions g, average the transport costs, multiply by the hidden dimension d, and take one square root.
μ and ν are the reference and generated corpora.
Lower distance. Closer distributions. SOL compares both the content of individual texts and the variety across a corpus.
Explore the math ↗02 MEASURED EXAMPLES
Three controlled comparisons: sampling temperature, context length, and the news topics a corpus covers.
Click or drag across the plot to inspect a setting. Arrow keys also work.
Download all values (CSV) ↓ · Data and source hashes (JSON) ↓ · AG News protocol and provenance ↓