Corpus loading, encoding, split, windows, batches
5 of 5 functions, classes and methods here are reachable from train.main, and 0 from talk.main.
| Pipeline stage | Step | Explained on |
|---|---|---|
| Text to training windows | Text | data pipeline |
| Text to training windows | Ids | tokenization |
| Text to training windows | Windows | data pipeline |
tokenizer.pynumpy, pandastests/test_data.py, train.pytrain.pytests/test_data.py| Concept | What this file does | Lines | Status |
|---|---|---|---|
| The split | Computes it | 20, 21 | Implemented |
| Window count | Computes it | 25, 28 | Derived from implementation |
| Sliding windows | Computes it | 28, 29, 31, 32 | Implemented |
| Shuffling and batching | Computes it | 39, 41 | Derived from implementation |
Reads up to max_rows CSV rows, drops missing values, strips, joins with newlines, adds a final newline
train.maintrain.maintests.test_data.test_load_corpus_joins_rows_and_skips_missing, tests.test_data.test_load_corpus_respects_max_rows, tests.test_data.test_load_corpus_wrong_column_raisesEncodes text into a 1-D int64 array
train.maintrain.maintokenizer.Tokenizer.encodeSplits the id stream without shuffling: leading part is train, trailing part is validation
train.maintrain.maintests.test_data.test_split_is_contiguous_and_disjoint, tests.test_data.test_window_counts_for_n_3000Returns input and target pandas DataFrames of sliding windows
train.maintrain.maintests.test_data.test_batches_cover_every_window_once, tests.test_data.test_no_shuffle_keeps_order, tests.test_data.test_stride_skips_windows, tests.test_data.test_targets_are_inputs_shifted_by_one, tests.test_data.test_too_few_tokens_raises, tests.test_data.test_window_counts_for_n_3000, tests.test_data.test_window_table_matches_hand_exampledef iterate_batches(inputs, targets, batch_size, rng, shuffle = True)Lines 36-43GitHub: data.iterate_batchesYields batches as NumPy arrays; keeps a final short batch
train.maintrain.maintests.test_data.test_batches_cover_every_window_once, tests.test_data.test_no_shuffle_keeps_orderimport numpy as npimport pandas as pdfrom tokenizer import Tokenizerdef load_corpus(path, column, max_rows) -> str: df = pd.read_csv(path, nrows=max_rows) if column not in df.columns: raise ValueError(f"no column {column!r}, the file has {list(df.columns)}") texts = df[column].dropna().astype(str).str.strip() return "\n".join(texts) + "\n"# BPE tokenizer is called to turn this into a 1D array of integersdef encode_corpus(text: str, tok: Tokenizer) -> np.ndarray: return np.array(tok.encode(text), dtype=np.int64)# this cuts id stream so 0.1 keeps first 90% for training and last 10% for testingdef split_ids(ids, val_fraction): cut = int((1 - val_fraction) * len(ids)) return ids[:cut], ids[cut:]# A language model learns to predict the next token. We slide a window of length T along the ids. The input is the window, and the target is the same window shifted one token to the left.def make_windows(ids, seq_len, stride = 1): if len(ids) <= seq_len: raise ValueError(f"need more than seq_len={seq_len} tokens, got {len(ids)}") starts = np.arange(0, len(ids) - seq_len, stride) rows = starts[:, None] + np.arange(seq_len) inputs = pd.DataFrame(ids[rows]) targets = pd.DataFrame(ids[rows + 1]) return inputs, targets # Training doesn’t use all windows at once, because that would make each update slow. It also doesn’t use one window at a time, because that would make updates noisy. This function hands out batch_size windows at a time.def iterate_batches(inputs, targets, batch_size, rng, shuffle = True): order = inputs.index if shuffle: order = inputs.sample(frac=1, random_state=rng).index for i in range(0, len(order), batch_size): rows = order[i : i + batch_size] yield inputs.loc[rows].to_numpy(), targets.loc[rows].to_numpy()