{ "cells": [ { "cell_type": "markdown", "id": "f4bdb240-9099-411b-a859-daaf3269c900", "metadata": {}, "source": [ "## Implementing the BPE algorithm\n", "The aim of this exercise is to implement the BPE tokenization algorithm. As a reminder, the principle consists in gathering the words or “tokens” that appear the most times in succession.\n", "\n", "For example, if we consider the corpus containing the words in the following table (with the number of occurrences of each word):\n", "\n", "| words | occurrence |\n", "|------|-----------|\n", "| voting | 2 |\n", "| vote | 3 |\n", "| slow | 1 |\n", "| slowly | 2 |\n", "\n", "\n", "And if the initial “tokens” are the letters of the alphabet, then the prefix “vo” will initially be added to the list of sub-words (tokens), cause the bigram \"v\" \"o\" occurs 5 times (2 + 3).\n", "\n", "The steps involved in implementing the BPE algorithm are as follows:\n", "1. Download a text corpus (here a wikipedia page)\n", "2. Cut the text into words (using the “space” and “ponctuation” characters) and count the number of occurrences of each word.\n", "3. Initialize the word dictionary with the initial tokens (letters of the alphabet)\n", "4. Run BPE algorithm (learn vocabulary)\n", "5. Test token decomposition on selected sentences (apply learned rules)\n", "\n", "\n", "In the documents you will observe a lot of TODO in the code, replace it by your code.\n", "\n", "\n", "### The class we will complete\n", "\n", "\n", "At the end we will create an object (python) class as it follows :\n", "\n", "```python\n", "class Tokenizer:\n", " ''' \n", " A class for our Tokenizer\n", "\n", " Methods\n", " -------\n", " fit(text_corpus: str) : None\n", " Fit the tokenizer based on the input text corpus \n", " tokenize(text: str): List[str]\n", " Tokenize a text and return the list of token ids\n", " detokenize(tokens: List[str]): str\n", " From a list of token ids return the corresponding text\n", " '''\n", "\n", " def __init__(self, vocabulary_size: int = 500):\n", " ''' \n", " Parameters\n", " ----------\n", " vocabulary_size : int\n", " The expected size of the vocabulary\n", "\n", " '''\n", " super().__init__()\n", " self.vs = vocabulary_size\n", "\n", " def fit(self, text_corpus: str):\n", " ''' Train the tokenizer on the provided text.\n", "\n", " Parameters\n", " ----------\n", " text_corpus : str\n", " The text used to train the model\n", " '''\n", " raise NotImplementedError\n", "\n", "\n", " def tokenize(self, text: str) -> List[str]:\n", " ''' Tokenize a text.\n", "\n", " Parameters\n", " ----------\n", " text : str\n", " The text to tokenize\n", "\n", " Returns\n", " -------\n", " List[str]\n", " The list of tokens\n", " '''\n", " raise NotImplementedError\n", "\n", " def detokenize(self, tokens : List[int]) -> str:\n", " ''' Reverse the tokenization.\n", "\n", " Parameters\n", " ----------\n", " tokens : List[str]\n", " A list of token ids\n", "\n", " Returns\n", " -------\n", " str\n", " The text corresponding to tokens\n", " '''\n", " raise NotImplementedError\n", "\n", "```" ] }, { "cell_type": "markdown", "id": "47e200fe", "metadata": {}, "source": [ "## Requirement\n", "\n", "**For this lab you only need the python standard library :D**" ] }, { "cell_type": "code", "execution_count": 77, "id": "874ed1bb", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_134797/2305373172.py:6: DeprecationWarning: Importing display from IPython.core.display is deprecated since IPython 7.14, please import from IPython.display\n", " from IPython.core.display import display, HTML # just for pretty print\n" ] } ], "source": [ "import re # the regex library\n", "import json # read export json format\n", "\n", "from typing import List, Dict, Tuple # to specify the types in function def\n", "from collections import Counter # a tools to counts unique occurences\n", "from IPython.core.display import display, HTML # just for pretty print\n", "\n", "from urllib.request import urlopen, Request" ] }, { "cell_type": "markdown", "id": "ee88b07e-4de8-436e-9dc7-f405c69c9042", "metadata": {}, "source": [ "### Step 1: Download a corpus\n", "\n", "For this lab we will consider a wikipedia page in french [Grèce antique](https://fr.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&explaintext&redirects=1&titles=Gr%C3%A8ce_antique), but you are free to choose any content you want !" ] }, { "cell_type": "code", "execution_count": null, "id": "5352d48f-27ca-41ad-9265-0e128578a5ec", "metadata": {}, "outputs": [], "source": [ "url_request = 'https://fr.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&explaintext&redirects=1&titles=Gr%C3%A8ce_antique'\n", "wikipedia_request = Request(url_request)\n", "wikipedia_request.add_header(\"User-Agent\", \"Course (thomas.gerald@lisn.fr)\")\n", "raw_page = urlopen(wikipedia_request)\n", "json_page = json.load(raw_page)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "97523f67", "metadata": {}, "outputs": [], "source": [ "corpus = list(json_page['query']['pages'].values())[0]['extract']" ] }, { "cell_type": "markdown", "id": "f8b6b7a2-74b4-4ba2-aa4b-93885a286fb2", "metadata": {}, "source": [ "### Step 2: Splitting text into words (or sequence of character no containing space)\n", "\n", "To split the text into words, we'll use the following regex ```r'(\\b[^\\s]+\\b)'```. To count words, we'll use python's Counter object. \n", "1. Store each word and its number of occurrences in **count_words** using the [Counter Object](https://docs.python.org/3/library/collections.html#collections.Counter).\n", "2. Print the 10 most frequent words using the method **most_commons** of the Counter object.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6d7341ff-61bf-4a5a-9d0d-4ae8fd8797ce", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "

The most common words are :

\n", " \n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "
Word Rank Number of occurrences
de11756
la21135
et31010
des4945
les5793
à6624
le7538
en8491
qui9394
dans10364
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "\n", "word_regex = re.compile(r'(\\b[^\\s]+\\b)')\n", "words = word_regex.findall(corpus)\n", "\n", "count_words = # TODO\n", "most_commons_words = # TODO\n", "display(HTML(f\"\"\"\n", "

The most common words are :

\n", "
\n", "\\t\n", "\\t{'\\n\\t'.join([f''\n", " for i, (k, v) in enumerate(most_commons_words)])}\n", "
Word Rank Number of occurrences
{k}{i+1}{v}
\"\"\"\n", "))" ] }, { "cell_type": "markdown", "id": "7a7a38b8-97f5-4851-b642-f7b40c023491", "metadata": {}, "source": [ "### Step 3: Initialize word dictionary with initial tokens (letters of the alphabet)\n", "\n", "Create the initial vocabulary in the vocab variable, we recall that we consider characters as base tokens. How many initial tokens do you have?" ] }, { "cell_type": "code", "execution_count": null, "id": "901f7ea5-9d71-4f8c-95bf-32b098c3241c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "we have 107 initial tokens\n" ] } ], "source": [ "vocab = # TODO\n", "vocab.sort()\n", "vocab\n", "print(f\"we have {len(vocab)} initial tokens\")" ] }, { "cell_type": "markdown", "id": "efd79ab8-b1b1-4802-ac80-39b12cb5917f", "metadata": {}, "source": [ "### Step 4: Learning the tokenizer\n", "To learn the tokenizer we need several functions:\n", "1. A function to calculate the frequency of each token pair.\n", "2. A function to merge a pair\n", "\n", "Several variables will be required:\n", "1. **vocab** containing current vocabulary\n", "2. **merge_rules** containing all the merge rules (a dictionary containing as key a pair of tokens to merge and the result of the token merge). For example: {('e', 's'), 'es', ('en', 't') :'ent'}.\n", "3. **splits** A dictionary containing the current breakdown of the corpus, with the word as key and the list of “tokens” as value.\n", "\n", "#### Step 4.1: Splitting words into tokens\n", "In the first step splits contains the words broken down into characters. For instance if we consider the words \"La\", \"Grèce\", \"Antique\", \"est\" we should have into the splits variables the following structure :\n", "\n", "```python\n", "{\n", " 'La': ['L', 'a'],\n", " 'Grèce': ['G', 'r', 'è', 'c', 'e'],\n", " 'antique': ['a', 'n', 't', 'i', 'q', 'u', 'e'],\n", " 'est': ['e', 's', 't'],\n", " 'une': ['u', 'n', 'e']\n", "}\n", "```\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8357b6bd-9414-4263-9bf5-ef249614cb04", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "La -> L-a\n", "Grèce -> G-r-è-c-e\n", "antique -> a-n-t-i-q-u-e\n", "est -> e-s-t\n", "une -> u-n-e\n", "civilisation -> c-i-v-i-l-i-s-a-t-i-o-n\n", "de -> d-e\n", "l'Antiquité -> l-'-A-n-t-i-q-u-i-t-é\n", "des -> d-e-s\n", "peuples -> p-e-u-p-l-e-s\n" ] } ], "source": [ "# In the first step splits contains the words broken down into characters\n", "splits = # TODO\n", "print(\"\\n\".join([f\"{k} -> {'-'.join([v for v in splits[k]])}\" for k in list(splits.keys())[:10]]))" ] }, { "cell_type": "markdown", "id": "722df20d-fae1-41f1-96bc-fecb6b10a0ca", "metadata": {}, "source": [ "#### Step 4.2: Compute the frequency of token pairs\n", "Create a function **compute_pair_freqs**, which, given the words broken down into tokens (splits dictionary) and the frequency of the words, returns the frequency of each pair of tokens (note only successive sub-words)." ] }, { "cell_type": "code", "execution_count": null, "id": "cb995e01-74f4-432f-9ddb-5c676afc32c8", "metadata": {}, "outputs": [], "source": [ "def compute_pair_freqs(splits, word_freqs):\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": 114, "id": "d806c606-8cd3-43ff-8e14-aea74f9d4172", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " The pair of tokens ('L', 'a') occur 166 times in the corpus\n", " The pair of tokens ('G', 'r') occur 257 times in the corpus\n", " The pair of tokens ('r', 'è') occur 249 times in the corpus\n", " The pair of tokens ('è', 'c') occur 285 times in the corpus\n", " The pair of tokens ('c', 'e') occur 1191 times in the corpus\n" ] } ], "source": [ "pair_freqs = compute_pair_freqs(splits, count_words)\n", "print(\"\\n\".join([f\" The pair of tokens {k} occur {pair_freqs[k]} times in the corpus\" for k in list(pair_freqs.keys())[:5]]))" ] }, { "cell_type": "markdown", "id": "72e05d19-df89-4b0b-a2ad-ba5af727b1e5", "metadata": {}, "source": [ "#### Step 4.3: Find the most frequent pair\n", "Create a function **most_frequent(pair_freqs)** returning the most frequent pair of tokens. You can use the function `max` provided by python.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "be9bccb0-86b8-4ba8-9b15-d1f3cab5b10b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The most frequent pair is ('e', 's')\n" ] } ], "source": [ "def most_frequent(pair_freqs):\n", " raise NotImplementedError\n", "print(f\"The most frequent pair is {most_frequent(pair_freqs)}\")" ] }, { "cell_type": "markdown", "id": "aafdb8d8", "metadata": {}, "source": [ "#### Step 4.4: Merge all tokens from the retrieved pair\n", "Create a **merge_pair()** function which, given a pair, returns the new splits of the corpus." ] }, { "cell_type": "code", "execution_count": null, "id": "dd44467a-326b-499c-9995-8661fd102bec", "metadata": {}, "outputs": [], "source": [ "def merge_pair(a : str, b: str, splits : Dict[str, List[str]]) -> Dict[str, List[str]]:\n", " ''' Merge tokens where a.b\n", "\n", " Parameters\n", " ----------\n", " a : str\n", " The first token of the pair\n", " b : str\n", " The first token of the pair\n", " splits : Dict[str, List[str]\n", " The dataset considering tokenized words\n", "\n", " Returns\n", " -------\n", " Dict[str, List[str]\n", " The new split with tokens a and b merged\n", " '''\n", " raise NotImplementedError" ] }, { "cell_type": "code", "execution_count": 117, "id": "462376c4-6abc-4d4c-8bb6-a7e4d5bb96bb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['g', 'r', 'e', 'c', 'q', 'u', 'es']\n" ] } ], "source": [ "new_splits = merge_pair(*most_frequent(pair_freqs), splits)\n", "print(new_splits['grecques'])" ] }, { "cell_type": "markdown", "id": "1a9cdfeb-ade8-4ac6-9cd5-9049090ca142", "metadata": {}, "source": [ "#### Apply the algorithm until the desired vocabulary size is reached.\n", "Create a BPE object that takes as arguments a corpus, a vocabulary size and train the BPE algorithm. The algorithm stores the final vocabulary in the **vocabulary** attribute and the merge rules in **mr**. The attributes **mr** will be a list of couple of tokens \n", "```\n", "[('e', 's'),\n", " ('n', 't'),\n", " ('q', 'u'),\n", " ('r', 'e'),\n", " ('o', 'n'),\n", " ('d', 'e'),\n", " ('l', 'e'),\n", " ('t', 'i'),\n", " ('l', 'a'),\n", " ('i', 's'),\n", " ('e', 'nt'), ...\n", "]\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "c0367f37-9b40-4ae7-8ec6-ca7ab86ae687", "metadata": {}, "outputs": [], "source": [ "class BPETokenizer:\n", " ''' \n", " A class for our Tokenizer\n", "\n", " Methods\n", " -------\n", " fit(text_corpus: str) : None\n", " Fit the tokenizer based on the input text corpus \n", " tokenize(text: str): List[str]\n", " Tokenize a text and return the list of token ids\n", " detokenize(tokens: List[str]): str\n", " From a list of token ids return the corresponding text\n", " '''\n", " @staticmethod \n", " def compute_pair_freqs(splits: Dict[str, List[str]], word_freqs: Dict[str, int]) -> Dict[Tuple[str, str], int]:\n", " return compute_pair_freqs(splits, word_freqs)\n", " \n", " @staticmethod\n", " def most_frequent(pair_freqs: Dict[Tuple[str, str], int]) -> Tuple[str, str]:\n", " return most_frequent(pair_freqs)\n", " \n", " @staticmethod \n", " def merge_pair(a : str, b: str, splits : Dict[str, List[str]]) -> Dict[str, List[str]]:\n", " return merge_pair(a, b, splits)\n", " \n", "\n", "\n", " def __init__(self, vocabulary_size: int = 500):\n", " ''' \n", " Parameters\n", " ----------\n", " vocabulary_size : int\n", " The expected size of the vocabulary\n", "\n", " '''\n", " super().__init__()\n", " self.vs = vocabulary_size\n", " self.vocab = []\n", " self.wr = re.compile(r'(\\b[^\\s]+\\b)') # the regex to split onto space\n", " self.mr = [] # the merge rules in the same order as the one retrieved\n", "\n", "\n", " def fit(self, text_corpus: str):\n", " ''' Train the tokenizer on the provided text.\n", "\n", " Parameters\n", " ----------\n", " text_corpus : str\n", " The text used to train the model\n", " '''\n", " words = self.wr.findall(text_corpus)\n", "\n", " # initialize the vocabulary (using characters)\n", " self.vocab = # TODO\n", " # get the number of occurences of each words\n", " word_freqs = # TODO\n", " # tokenize each word of the dataset (according to vocab which is here the characters)\n", " splits = # TODO \n", "\n", " # while we did not reach the desired vocabulary size\n", " while len(self.vocab) < self.vs:\n", " raise NotImplementedError\n", "\n", " def tokenize(self, text: str) -> List[List[str]]:\n", " ''' Tokenize a text.\n", "\n", " Parameters\n", " ----------\n", " text : str\n", " The text to tokenize\n", "\n", " Returns\n", " -------\n", " List[str]\n", " The list of tokens\n", " '''\n", " words = self.wr.findall(text)\n", " splits = [[l for l in word] for word in words]\n", " for prefix, suffix in self.mr:\n", " raise NotImplementedError\n", " return splits\n", "\n", " def detokenize(self, tokens : List[List[str]]) -> str:\n", " ''' Reverse the tokenization.\n", "\n", " Parameters\n", " ----------\n", " tokens : List[str]\n", " A list of token ids\n", "\n", " Returns\n", " -------\n", " str\n", " The text corresponding to tokens\n", " '''\n", " return ' '.join([ \"\".join(w) for w in tokens])\n" ] }, { "cell_type": "code", "execution_count": 218, "id": "c282a2a1-f23c-4b22-b858-35a0049250c1", "metadata": { "scrolled": true }, "outputs": [], "source": [ "my_bpe = BPETokenizer(vocabulary_size=5000)\n", "my_bpe.fit(corpus)\n" ] }, { "cell_type": "code", "execution_count": 222, "id": "b225b59b-6103-4b3f-b776-ea2b71714d64", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[['Anti'], ['b', 'l', 'p', 'q', 's', 'd', 'in', 'j', 'z'], ['a', 'j', 'f', 'r', 'f', 'j', 'q', 'k'], ['grecques'], ['développée'], ['en'], ['Grèce']]\n" ] }, { "data": { "text/plain": [ "'Anti blpqsdinjz ajfrfjqk grecques développée en Grèce'" ] }, "execution_count": 222, "metadata": {}, "output_type": "execute_result" } ], "source": [ "texte = '''Anti blpqsdinjz ajfrfjqk grecques développée en Grèce '''\n", "print(my_bpe.tokenize(texte))\n", "my_bpe.detokenize(my_bpe.tokenize(texte))" ] }, { "cell_type": "markdown", "id": "a6f478a3-5f52-43c4-9b48-0e912304985a", "metadata": {}, "source": [ "#### Test by modifying parameters or corpus\n", "Test the algorithm with different hyper-parameters or data" ] }, { "cell_type": "markdown", "id": "0c11246b", "metadata": {}, "source": [ "## Using sentencepiece\n", "We're now going to use the `sentencepiece` library, which can be installed (if not already installed) with pip : \n", "\n", "`! pip install sentencepiece`\n", "\n", "To train the tokenizer, we'll use the `train` function of `SentencePieceTrainer`. " ] }, { "cell_type": "code", "execution_count": 195, "id": "2154fc4d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "sentencepiece_trainer.cc(78) LOG(INFO) Starts training with : \n", "trainer_spec {\n", " input: test-cours.txt\n", " input_format: \n", " model_prefix: m\n", " model_type: BPE\n", " vocab_size: 500\n", " self_test_sample_size: 0\n", " character_coverage: 0.9995\n", " input_sentence_size: 0\n", " shuffle_input_sentence: 1\n", " seed_sentencepiece_size: 1000000\n", " shrinking_factor: 0.75\n", " max_sentence_length: 4192\n", " num_threads: 16\n", " num_sub_iterations: 2\n", " max_sentencepiece_length: 16\n", " split_by_unicode_script: 1\n", " split_by_number: 1\n", " split_by_whitespace: 1\n", " split_digits: 0\n", " pretokenization_delimiter: \n", " treat_whitespace_as_suffix: 0\n", " allow_whitespace_only_pieces: 0\n", " required_chars: \n", " byte_fallback: 0\n", " vocabulary_output_piece_score: 1\n", " train_extremely_large_corpus: 0\n", " seed_sentencepieces_file: \n", " hard_vocab_limit: 1\n", " use_all_vocab: 0\n", " unk_id: 0\n", " bos_id: 1\n", " eos_id: 2\n", " pad_id: -1\n", " unk_piece: \n", " bos_piece: \n", " eos_piece: \n", " pad_piece: \n", " unk_surface: ⁇ \n", " enable_differential_privacy: 0\n", " differential_privacy_noise_level: 0\n", " differential_privacy_clipping_threshold: 0\n", "}\n", "normalizer_spec {\n", " name: nmt_nfkc\n", " add_dummy_prefix: 1\n", " remove_extra_whitespaces: 1\n", " escape_whitespaces: 1\n", " normalization_rule_tsv: \n", "}\n", "denormalizer_spec {}\n", "trainer_interface.cc(353) LOG(INFO) SentenceIterator is not specified. Using MultiFileSentenceIterator.\n", "trainer_interface.cc(185) LOG(INFO) Loading corpus: test-cours.txt\n", "trainer_interface.cc(409) LOG(INFO) Loaded all 329 sentences\n", "trainer_interface.cc(425) LOG(INFO) Adding meta_piece: \n", "trainer_interface.cc(425) LOG(INFO) Adding meta_piece: \n", "trainer_interface.cc(425) LOG(INFO) Adding meta_piece: \n", "trainer_interface.cc(430) LOG(INFO) Normalizing sentences...\n", "trainer_interface.cc(539) LOG(INFO) all chars count=231118\n", "trainer_interface.cc(550) LOG(INFO) Done: 99.9554% characters are covered.\n", "trainer_interface.cc(560) LOG(INFO) Alphabet size=83\n", "trainer_interface.cc(561) LOG(INFO) Final character coverage=0.999554\n", "trainer_interface.cc(592) LOG(INFO) Done! preprocessed 329 sentences.\n", "trainer_interface.cc(598) LOG(INFO) Tokenizing input sentences with whitespace: 329\n", "trainer_interface.cc(609) LOG(INFO) Done! 8287\n", "bpe_model_trainer.cc(159) LOG(INFO) Updating active symbols. max_freq=5803 min_freq=1\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=1316 size=20 all=1531 active=1447 piece=an\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=705 size=40 all=2177 active=2093 piece=▁f\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=514 size=60 all=2661 active=2577 piece=▁par\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=379 size=80 all=3146 active=3062 piece=▁pl\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=314 size=100 all=3598 active=3514 piece=▁re\n", "bpe_model_trainer.cc(159) LOG(INFO) Updating active symbols. max_freq=312 min_freq=21\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=248 size=120 all=4072 active=1455 piece===\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=203 size=140 all=4542 active=1925 piece=▁ré\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=170 size=160 all=5003 active=2386 piece=▁I\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=151 size=180 all=5263 active=2646 piece=▁peu\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=134 size=200 all=5567 active=2950 piece=▁pas\n", "bpe_model_trainer.cc(159) LOG(INFO) Updating active symbols. max_freq=134 min_freq=20\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=119 size=220 all=5785 active=1216 piece=ance\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=108 size=240 all=6002 active=1433 piece=▁puis\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=102 size=260 all=6192 active=1623 piece=ces\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=92 size=280 all=6486 active=1917 piece=▁tra\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=86 size=300 all=6726 active=2157 piece=▁clas\n", "bpe_model_trainer.cc(159) LOG(INFO) Updating active symbols. max_freq=85 min_freq=18\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=80 size=320 all=7003 active=1275 piece=iens\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=75 size=340 all=7324 active=1596 piece=ations\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=70 size=360 all=7513 active=1785 piece=nc\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=66 size=380 all=7719 active=1991 piece=▁péri\n", "bpe_model_trainer.cc(268) LOG(INFO) Added: freq=63 size=400 all=7894 active=2166 piece=tiv\n", "bpe_model_trainer.cc(159) LOG(INFO) Updating active symbols. max_freq=63 min_freq=16\n", "trainer_interface.cc(687) LOG(INFO) Saving model: m.model\n", "trainer_interface.cc(699) LOG(INFO) Saving vocabs: m.vocab\n" ] } ], "source": [ "import sentencepiece as spm\n", "\n", "with open(\"test-cours.txt\", \"w\") as f:\n", " f.write(corpus)\n", "spm.SentencePieceTrainer.train(input=\"test-cours.txt\", model_type='BPE', model_prefix='m', vocab_size=500)" ] }, { "cell_type": "markdown", "id": "0e069387", "metadata": {}, "source": [ "Looking at the “m.vocab” file, what are the differences with the vocabulary learned with your implementation? What modifications could you consider to obtain a similar vocabulary?" ] }, { "cell_type": "markdown", "id": "d7437917", "metadata": {}, "source": [ "## Detokenization and space characters\n", "\n", "Propose and implement a methods to detokenize encoded sentence (you should want to extend your vocabulary).\n", "\n", "\n", "**Hint :** One modification (or addition) in the function `fit` and One (or two) modification in the tokenize function\n" ] }, { "cell_type": "markdown", "id": "5af3c31a", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.8" } }, "nbformat": 4, "nbformat_minor": 5 }