Fantastic Voyage

Isaac Asimov

Year first published: 1966
Status: ©
Selection: Manual
Publisher: Bantam Dell; Random House, Inc.; Bantam Books
Books3 manifest entry: https://the-eye.eu/public/Books/Bibliotik/I/IsaacAsimov-FantasticVoyage_9780307790965.epub

How to read our results

The numbers in our experiments come from a specific measurement procedure: we take 100-token sequences throughout a book (every 10 characters, from start to end), and compute the extraction probability of the last 50 tokens (the suffix) given the first 50 tokens (the prefix). This method is called probabilistic extraction.

The claims we make about memorization are specific to this setup, including the decoding policy we use (top-k sampling with k = 40 and temperature 1). Other settings may reveal different memorization: different prompting strategies (e.g., longer prefixes or adversarial prompts), other decoding policies, or other success conditions (e.g., near-verbatim matching).

It's worth remembering that extraction is the observable signal we measure, while memorization is the underlying property of the model that we use this signal to probe. Different setups may capture different signals, but as long as the setup is valid, each one reveals some subset of what the model has memorized.

Model groups

We arrange our results into three groups of models, based on what we know about their training data.

LLMs known to be trained on Books3 LLMs we conclude were trained on (at least parts of) some books that are also contained in Books3 Not trained on whole copyrighted books

Extraction coverage by model

Extraction coverage is the fraction of the book that's memorized. More precisely, it's the share of the book's characters that fall within at least one 50-token suffix that we extract from the model (extraction probability at least 0.1%). We show coverage for each of the 14 models; longer bars mean more of Fantastic Voyage is memorized. (High coverage doesn't mean the whole book can be generated at once; it's a way to compare how much is memorized across books and models.)

LLMs known to be trained on Books3

Llama 2 7B0.00%

Llama 3 8B0.00%

Llama 3.1 8B0.00%

Llama 1 13B0.00%

Llama 2 13B0.00%

Pythia 12B0.08%

Llama 1 65B0.00%

Llama 2 70B0.00%

Llama 3 70B0.00%

Llama 3.1 70B0.00%

LLMs we conclude were trained on (at least parts of) some books that are also contained in Books3

DeepSeek v1 67B0.00%

Qwen 2.5 72B0.00%

Gemma 2 27B0.00%

Not trained on whole copyrighted books

Phi 40.00%

0%25%50%75%100%

Memorization heatmaps

Each heatmap reflects results from an experiment on one model. At each character position, the color shows the maximum extraction probability among the overlapping 50-token suffixes that span that character. Darker means higher extraction probability, on a log scale. 🔍 Hover over a heatmap to zoom in.

Memorization heatmaps for Fantastic Voyage, one per model for the 3 models where it's undisclosed if they were trained on Books3 (one row each). Each heatmap plots the maximum extraction probability at each character in the book, where the maximum is taken over all 50-token suffixes that span that character. Color is on a log scale, with white indicating no extraction, pale indicating low extraction probability, and dark meaning a high extraction probability. The horizontal axis is character position in the book.

Memorization heatmap for Fantastic Voyage, for Phi 4 (a single model, one row). Phi 4 was not trained on whole copyrighted books. The heatmap plots the maximum extraction probability at each character in the book, where the maximum is taken over all 50-token suffixes that span that character. Color is on a log scale, with white indicating no extraction, pale indicating low extraction probability, and dark meaning a high extraction probability. The horizontal axis is character position in the book.

Distributions over extraction probability

Each histogram shows the distribution of the 50-token sequences we extracted from a single model. n is the raw number of sequences we extracted from that model (for example, n = 40 means we extracted 40 sequences from it). The horizontal axis is extraction probability (log scale, as in the other plots); the vertical axis is a count — the number of extracted sequences whose probability falls in that bin — so the bar heights add up to n. Mass toward higher probability means more of the extracted sequences are highly extractable. If a model has no extractable sequences, its plot is left empty.

Extraction-probability distributions for Fantastic Voyage: a grid of histograms, one per model. Each histogram shows the distribution of the 50-token sequences extracted from that model. n is the raw number of sequences extracted from the model (e.g. n=40 means 40 sequences were extracted from it). The horizontal axis is extraction probability on a log scale; the vertical axis is a count — the number of extracted sequences whose probability falls in each bin — so the bar heights add up to n. Mass toward high probability means many of the extracted sequences are highly extractable. If a model has no extractable sequences, its plot is left empty.

← Back to all books