Meet SOL: a sample-based distance between generated and reference text that treats each text as a distribution over token states. It works even when model likelihoods are unavailable.
01 BENCHMARK RESULTS
Evaluate language models on OpenWebText.
How to read these results
Reference = the text to match. G uses held-out GPT-2-packed OpenWebText. E uses text decoded from ELF’s training corpus. B uses held-out OpenWebText in BD3-LM’s training format, without block-edge tokens. Every row carries its reference badge, G, E or B. All references share one SOL ranking. Rows are sorted together by the selected column.
Encoder = the token representation. Dream 7B and GPT-2 Large turn text into hidden token states before SOL compares them. Select one encoder for a comparison; their raw score scales differ. Dream 7B is selected by default.
Real–real control = sampling variation. Two finite samples of real text still differ. The control gives context for G and E; the table has no real–real control for B. Lower SOL means closer to that reference; it is not an individual text quality grade.
G, E and B identify the reference corpus used for each score. Rank numbers run across all references.
Select a model for its family, perplexity, entropy, and full run details.
| RANK | MODEL / SAMPLING SETTING | FAMILY | Details | |||
|---|---|---|---|---|---|---|
Loading configurations…
Mean ± population standard deviation across 9 direction/GP seed pairs. Compare SOL within the same reference and encoder; scales differ across encoders. Sampling budgets and training setups vary. Snapshot: 26 September 2026.
02 GET STARTED
EVALUATE
YOUR MODEL.
Bring two text corpora. Get one distributional distance.
SOL’s implementation is the sol-metric Python package on PyPI, imported as sol_metric.