{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Pos Tag Lstm" ], "id": "site-title" }, { "cell_type": "markdown", "id": "ff0775ef-abeb-45cb-9f07-3371ba712ff5", "metadata": {}, "source": "## A Parts of Speech Tagger with LSTM\n\nNow that we know how to use a dataloader, define flax modules and set up training pipelines, let's look at an actual example which will combine all of these and also take a peek at the LSTM class in flax.\n\nFor this task, we're going to use the `batterydata/pos_tagging` dataset from Huggingface Datasets. " }, { "cell_type": "markdown", "id": "2fd0e4b6-95eb-4d31-932b-e0dd8b4b5ac4", "metadata": {}, "source": "### Loading the dataset\n\nThis particular dataset comes with train and test splits and no validation splits. Ideally, we should have all the three splits in a model training pipeline. So one solution here is to break the train split and create train and validation splits from it." }, { "cell_type": "code", "execution_count": 1, "id": "e1844daa-87f7-4e77-97e0-07af425d668e", "metadata": {}, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "main_dataset = load_dataset(\"batterydata/pos_tagging\")\n", "train_split = main_dataset[\"train\"]\n", "test_split = main_dataset[\"test\"]" ] }, { "cell_type": "code", "execution_count": 2, "id": "788a776c-a04a-4b7f-8448-8b7d446d320e", "metadata": {}, "outputs": [], "source": [ "def make_train_validation_splits(dataset_split,\n", " validation_size = 0.2,\n", " seed: int = 42):\n", " # make a copy of the data\n", " dataset_split = dataset_split.shuffle(seed=seed)\n", " # using the train test split method to create validation set\n", " dataset_split = dataset_split.train_test_split(test_size=validation_size,\n", " shuffle=True,\n", " seed=seed)\n", " return dataset_split[\"train\"], dataset_split[\"test\"]\n", "\n", "\n", "train_split, validation_split = make_train_validation_splits(train_split)" ] }, { "cell_type": "markdown", "id": "8a90a72d-2177-42e7-a680-d106828f751d", "metadata": {}, "source": "### Preprocessing\n\nSo we're going to take a very pre-Transformer era approach to preprocessing the dataset here. That means no byte pair tokenisation, no attention masks. We simple tokenise the words, use them as features. Which, also, practically leads us to the following steps:\n\n1. Tokenise\n2. Create a vocabulary\n3. Create a word to integer mapping (a dictionary will do)\n4. Create a label to integer mapping (pos tags are labels)\n5. Add out of vocabulary (OOV) and padding tokens\n\n\nOne caveat here is that we have to use the main train split (without creating the validation split). But you probably have noticed by now that I've left a `main_dataset` variable up there. We're going to use that one." }, { "cell_type": "code", "execution_count": 3, "id": "21cd5ee1-63aa-47e3-b80c-179f25d0f237", "metadata": {}, "outputs": [], "source": [ "# a dict containing word -> idx mapping\n", "# pad token is 1 by default\n", "def map_word_to_idx(dataset_split, pad_token_idx = 1):\n", " unique_words = set()\n", " word_to_idx = dict()\n", "\n", " # add an out of vocab token\n", " oov_token = \"\"\n", " # pad token\n", " pad_token = \"\"\n", "\n", " word_to_idx[oov_token] = 0\n", " word_to_idx[pad_token] = pad_token_idx\n", "\n", " # find the unique words\n", " for data in dataset_split:\n", " words = data[\"words\"]\n", " for w in words:\n", " unique_words.add(w)\n", "\n", " # add index to them\n", " for idx, uw in enumerate(list(unique_words)):\n", " word_to_idx[uw] = idx + 2 # since oov is at 0 and pad at pad_token_idx\n", "\n", " return word_to_idx\n", "\n", "word_to_idx = map_word_to_idx(main_dataset[\"train\"])" ] }, { "cell_type": "code", "execution_count": 4, "id": "244a5556-5a24-48b6-ba07-f189d7f3ca44", "metadata": {}, "outputs": [], "source": [ "# pos tag -> idx mapping\n", "def map_label_to_idx(dataset, pad_token_idx = 1):\n", " unique_labels = set()\n", " label_to_idx = dict()\n", "\n", " # add an out of vocab token\n", " oov_token = \"\"\n", " # pad token\n", " pad_token = \"\"\n", "\n", " label_to_idx[oov_token] = 0\n", " label_to_idx[pad_token] = pad_token_idx\n", "\n", " # find the unique labels\n", " for data in dataset:\n", " labels = data[\"labels\"]\n", " for l in labels:\n", " unique_labels.add(l)\n", "\n", " # index\n", " for idx, label in enumerate(list(unique_labels)):\n", " label_to_idx[label] = idx + 2\n", "\n", " return label_to_idx\n", "\n", "label_to_idx = map_label_to_idx(main_dataset[\"train\"])" ] }, { "cell_type": "markdown", "id": "342b1baa-d1b3-4d66-a560-f40e260a7822", "metadata": {}, "source": "Perfect! Now we have everything we need to create a Dataset class and Dataloaders." }, { "cell_type": "markdown", "id": "b813852d-592e-45aa-ad22-361f50319eac", "metadata": {}, "source": "### Dataset and Dataloaders" }, { "cell_type": "code", "execution_count": 5, "id": "96baddf7-553b-428e-8c43-556fe6061815", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from torch.utils.data import Dataset\n", "\n", "\n", "\n", "class TagDataset(Dataset):\n", " def __init__(self, dataset_split,\n", " pad_token_idx,\n", " max_seq_len,\n", " word_to_idx,\n", " label_to_idx) -> None:\n", " self.dataset = dataset_split\n", " self.pad_token_idx = pad_token_idx\n", " self.max_seq_len = max_seq_len\n", " self.word_to_idx = word_to_idx\n", " self.label_to_idx = label_to_idx\n", "\n", " def __len__(self) -> int:\n", " return len(self.dataset)\n", "\n", " # use word_to_idx and label_to_idx to convert\n", " # the string sequences to int sequences\n", " def __encode(self, data_instance: dict) -> tuple:\n", " words = data_instance[\"words\"]\n", " labels = data_instance[\"labels\"]\n", "\n", " # convert to int sequences\n", " words = [self.word_to_idx.get(w, 0) for w in words]\n", " labels = [self.label_to_idx.get(l) for l in labels]\n", "\n", " return words, labels\n", "\n", " def __getitem__(self, index: int) -> tuple[np.ndarray, np.ndarray]:\n", " data = self.dataset[index]\n", " words, labels = self.__encode(data)\n", "\n", " # padding\n", " words_padded = np.ones((self.max_seq_len,), dtype=np.int32) * self.pad_token_idx\n", " words_padded[:len(words)] = words\n", "\n", " labels_padded = np.ones((self.max_seq_len,), dtype=np.int32) * self.pad_token_idx\n", " labels_padded[:len(labels)] = labels\n", "\n", " # return padded words and labels\n", " return words_padded, labels_padded" ] }, { "cell_type": "markdown", "id": "d9125168-ce26-471e-9ab9-6f8e8de3deb6", "metadata": {}, "source": "Now we need to decide on the maximum sequence length. From some searching on the dataset, I found 300 to be a good value. You can also choose something around 280. " }, { "cell_type": "code", "execution_count": 6, "id": "6b7337cb-82a3-4af9-a553-44f9179710c6", "metadata": {}, "outputs": [], "source": [ "max_seq_len = 300\n", "\n", "train_set = TagDataset(train_split, 1, max_seq_len, word_to_idx, label_to_idx)\n", "validation_set = TagDataset(validation_split, 1, max_seq_len, word_to_idx, label_to_idx)\n", "test_set = TagDataset(test_split, 1, max_seq_len, word_to_idx, label_to_idx)" ] }, { "cell_type": "code", "execution_count": 7, "id": "aaf586b3-2608-4d08-8d9a-574650382f7e", "metadata": {}, "outputs": [], "source": [ "# dataloaders\n", "import torch\n", "import jax_dataloader as jdl\n", "\n", "torch.manual_seed(2023)\n", "\n", "BATCH_SIZE = 128\n", "\n", "train_loader = jdl.DataLoader(train_set, \"pytorch\", batch_size=BATCH_SIZE, shuffle=True)\n", "val_loader = jdl.DataLoader(validation_set, \"pytorch\", batch_size=BATCH_SIZE, shuffle=False)\n", "test_loader = jdl.DataLoader(test_set, \"pytorch\", batch_size=BATCH_SIZE, shuffle=False)" ] }, { "cell_type": "markdown", "id": "677dbef2-49e8-4356-9e06-10b2f8a7bb6e", "metadata": {}, "source": "### Model Definition\n\nTime to define the model. " }, { "cell_type": "code", "execution_count": 8, "id": "aba78987-04ae-4688-866f-5b8d3796f468", "metadata": {}, "outputs": [], "source": [ "import flax.linen as nn\n", "from jax import random\n", "import jax.numpy as jnp\n", "\n", "\"\"\"\n", "class LSTMTagger\n", "\n", "vocab_size: int, size of the vocabulary\n", "embedding_dimensions: int, size of the embedding dimensions\n", "lstm_hidden_dims: int, size of the lstm hidden dimensions\n", "n_labels: int, number of labels in the dataset (the padded size, e.g. if padding length is 100, then n_labels = 100)\n", "training: bool, whether the model is in training mode or not\n", "\"\"\"\n", "\n", "\n", "class LSTMTagger(nn.Module):\n", " vocab_size: int\n", " embedding_dimensions: int\n", " lstm_hidden_dims: int\n", " n_labels: int\n", " lstm_seed: int\n", " dropout_rate: float\n", "\n", " def setup(self) -> None:\n", " # embedding layer\n", " self.embedding = nn.Embed(\n", " num_embeddings=self.vocab_size,\n", " features=self.embedding_dimensions,\n", " name=\"embedding\")\n", "\n", " # lstm layer\n", " self.lstm = nn.OptimizedLSTMCell(features=self.lstm_hidden_dims, name=\"lstm\")\n", "\n", " # dropout layer\n", " self.dropout = nn.Dropout(self.dropout_rate, deterministic=True)\n", "\n", " # dense layer\n", " self.dense = nn.Dense(features=self.n_labels, name=\"dense\")\n", "\n", " # lstm in flax: https://flax.readthedocs.io/en/latest/api_reference/flax.linen/_autosummary/flax.linen.LSTMCell.html\n", " def __call__(self, words: jnp.ndarray) -> jnp.ndarray:\n", " x = self.embedding(words)\n", "\n", " carry = self.lstm.initialize_carry(random.key(self.lstm_seed), x.shape)\n", " carry, x = self.lstm(carry=carry, inputs=x)\n", " x = self.dropout(x)\n", "\n", " x = self.dense(x)\n", " x = nn.leaky_relu(x)\n", "\n", " return x" ] }, { "cell_type": "markdown", "id": "a5f60a97-ebdc-4071-834f-31bba3d6e6e1", "metadata": {}, "source": "Let's look at the model definition a bit. We have an embedding layer to create embedding vectors, followed by an LSTM which models the text sequences, a dropout layer and finally, a dense layer to map the lstm output to labels. \n\nOne important thing to check here is how lstm works in flax. There are multiple implementations of a LSTMCell in flax. The one used here is a more optimised version. You can read more in the docs about it's implementation. In the forward pass of the model, you've to carry the initialize the carry or the state of the lstm for each batch or call and like any flax module, it requires a PRNG key. In other frameworks (e.g. Pytorch) you don't have to explcitly set the carry. \n\nAnother is the dropout layer in flax. It requires its own PRNG, which you have to supply during the model init. Let's see how it's done. " }, { "cell_type": "code", "execution_count": 9, "id": "c7abc181-341a-4c7d-b72c-f3f33639dff9", "metadata": {}, "outputs": [], "source": [ "# init the model\n", "SEED = 2023\n", "\n", "master_key = random.key(seed=SEED)\n", "master_key, model_init_key = random.split(master_key)\n", "master_key, dropout_key = random.split(master_key)\n", "model_rngs = {\"params\": model_init_key, \"dropout\": dropout_key}" ] }, { "cell_type": "code", "execution_count": 10, "id": "3c0588fa-5c6b-4c61-900e-157068b365d4", "metadata": {}, "outputs": [], "source": [ "model_config = {\n", " \"vocab_size\": len(word_to_idx),\n", " \"embedding_dimensions\": 300,\n", " \"lstm_hidden_dims\": 300,\n", " \"n_labels\": len(label_to_idx),\n", " \"lstm_seed\": 2024,\n", " \"dropout_rate\": 0.2,\n", "}\n", "\n", "model = LSTMTagger(**model_config)\n", "init_params = model.init(model_rngs, np.arange(max_seq_len))" ] }, { "cell_type": "code", "execution_count": 11, "id": "6a9110a3-c60e-44d9-9b82-bdc9b9988908", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(300, 50)\n" ] } ], "source": [ "# a sample forward pass\n", "logits = model.apply(init_params, jnp.arange(300), rngs={\"dropout\": dropout_key})\n", "print(logits.shape)" ] }, { "cell_type": "markdown", "id": "b0931465-e7a0-4d2e-a7ce-098b1d69af1f", "metadata": {}, "source": "#### Training functions" }, { "cell_type": "code", "execution_count": 12, "id": "74512e99-60a5-4fa1-9384-a048a720330d", "metadata": {}, "outputs": [], "source": [ "import optax\n", "from jax import vmap, jit\n", "\n", "# using cross entropy\n", "@jit\n", "def calculate_loss(params, words, labels):\n", " logits = model.apply(params, words, rngs={\"dropout\": dropout_key})\n", " loss = optax.softmax_cross_entropy_with_integer_labels(logits, labels)\n", " return loss.mean(axis=-1)\n", "\n", "\n", "@jit\n", "def batched_loss(params, words_batched, labels_batched):\n", " batch_loss = vmap(calculate_loss, in_axes=(None, 0, 0))(params, words_batched, labels_batched)\n", " return batch_loss.mean(axis=-1)\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "7b905f14-2e10-4398-804b-dc09b4a977cf", "metadata": {}, "outputs": [], "source": [ "from tqdm.auto import trange, tqdm\n", "from flax.training import train_state\n", "from functools import partial\n", "from jax import value_and_grad\n", "\n", "optimiser = optax.adam(learning_rate=0.001)\n", "init_state = train_state.TrainState.create(\n", " apply_fn=model.apply, # the forward function\n", " params=init_params,\n", " tx=optimiser\n", ")\n", "criterion = value_and_grad(batched_loss)" ] }, { "cell_type": "code", "execution_count": 14, "id": "3db94ddc-cca4-461c-a6af-af32686db33d", "metadata": {}, "outputs": [], "source": [ "@partial(jit, static_argnums=0)\n", "def train_step(criterion, state, words_batched, labels_batched):\n", " loss_value, grads = criterion(state.params, words_batched, labels_batched)\n", " updated_state = state.apply_gradients(grads=grads)\n", " return loss_value, updated_state\n", "\n", "\n", "@partial(jit, static_argnums=0)\n", "def validation_step(criterion, state, words_batched, labels_batched):\n", " loss_value, _ = criterion(state.params, words_batched, labels_batched)\n", " return loss_value" ] }, { "cell_type": "code", "execution_count": 15, "id": "a3f1f2a8-4371-4958-ba85-a143f5347eb1", "metadata": {}, "outputs": [], "source": [ "def train_model(state, train_loader, epochs=5, log_every_n_step=100):\n", " step_counter = 0\n", " train_losses = list()\n", " mean_validation_losses = list()\n", " train_acc = list()\n", " val_acc = list()\n", " test_acc = list()\n", "\n", " for _ in trange(epochs):\n", " for batch in train_loader:\n", " loss, state = train_step(criterion, state, *batch)\n", " step_counter += 1\n", "\n", " if step_counter % log_every_n_step == 0:\n", " # log train loss\n", " train_losses.append(loss)\n", " # run validation\n", " vlosses = list()\n", " for vbatch in val_loader:\n", " val_loss = validation_step(criterion, state, *vbatch)\n", " vlosses.append(val_loss)\n", "\n", " vlosses = jnp.array(vlosses)\n", " mean_validation_losses.append(vlosses.mean(axis=-1))\n", "\n", " print(\n", " f\"Step [{step_counter + 1}] ---- Loss/Train :: {loss} ---- Loss/Val :: {vlosses.mean(axis=-1)}\")\n", "\n", "\n", "\n", " return state, train_losses, mean_validation_losses" ] }, { "cell_type": "markdown", "id": "4fdc13fc-529d-43d4-bea0-5a5a0afedeae", "metadata": {}, "source": "#### Training" }, { "cell_type": "code", "execution_count": 16, "id": "2efb0607-f054-452f-b7c8-9ccda480554a", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ac0dd214ef3f434789cb9a53630ac698", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/5 [00:00" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABhKUlEQVR4nO3deVxUVf8H8M+wzAAiIKIsiqK5b2gkiFZYkZg+qWVpZom44G5KmlIZlSWYlpaSlhuW9XOpzDIzFZdKcQklNzQ13AG3HBSUZeb8/jgyMDAgg8AMw+f9et3Xc7n33DtnRh7m073nfo9CCCFAREREVM1ZmboDRERERBWBoYaIiIgsAkMNERERWQSGGiIiIrIIDDVERERkERhqiIiIyCIw1BAREZFFYKghIiIii2Bj6g5UFa1Wi8uXL6N27dpQKBSm7g4RERGVgRACt27dgpeXF6ysSr8WU2NCzeXLl+Ht7W3qbhAREVE5XLhwAQ0bNiy1TY0JNbVr1wYgPxQnJycT94aIiIjKIiMjA97e3rrv8dLUmFCTf8vJycmJoYaIiKiaKcvQEQ4UJiIiIovAUENEREQWgaGGiIiILEKNGVNDRESWQwiBvLw8aDQaU3eFHpC1tTVsbGwqpNwKQw0REVUrOTk5SE1NRVZWlqm7QhXEwcEBnp6eUCqVD3QehhoiIqo2tFotUlJSYG1tDS8vLyiVShZUrcaEEMjJycHVq1eRkpKC5s2b37fAXmkYaoiIqNrIycmBVquFt7c3HBwcTN0dqgD29vawtbXFuXPnkJOTAzs7u3Kfq1xxKDY2Fj4+PrCzs0NAQAD2799fYtslS5bgscceQ506dVCnTh0EBwfrtc/NzcW0adPQvn171KpVC15eXhgyZAguX76sdx4fHx8oFAq9JSYmpjzdJyKiau5B/muezE9F/XsafZY1a9YgIiICUVFROHjwIHx9fRESEoIrV64YbL9z504MGjQIO3bsQEJCAry9vdGjRw9cunQJAJCVlYWDBw9ixowZOHjwIH744QecPHkSffr0KXau999/H6mpqbplwoQJxnafiIiILJRCCCGMOSAgIACdO3fGwoULAUB3GXDChAmYPn36fY/XaDSoU6cOFi5ciCFDhhhsc+DAAfj7++PcuXNo1KgRAHmlZtKkSZg0aZIx3dXJyMiAs7Mz1Go1KwoTEVVTd+/eRUpKCpo0afJAtynIvJT272rM97dRV2pycnKQmJiI4ODgghNYWSE4OBgJCQllOkdWVhZyc3Ph6upaYhu1Wg2FQgEXFxe97TExMahbty46deqEOXPmIC8vr8RzZGdnIyMjQ28hIiKiiqdQKPDjjz+auhvGhZpr165Bo9HA3d1db7u7uzvS0tLKdI5p06bBy8tLLxgVdvfuXUybNg2DBg3SS2QTJ07E6tWrsWPHDowaNQqzZs3CG2+8UeLrREdHw9nZWbdwhm4iIjKloUOHol+/fiZ57ZUrV6Jhw4bFxqYWXeLi4sp1/tTUVDzzzDMV2+lyqNKnn2JiYrB69Wrs3LnT4GXD3NxcDBgwAEIILFq0SG9fRESEbr1Dhw5QKpUYNWoUoqOjoVKpip0rMjJS75j8WT4r3M2bwJtvAp07A2FhFX9+IiKiB7RhwwZMmDABoaGhum1z587F5s2bsW3bNt02Z2dn3bpGo4FCoSjTIF4PD4+K7XA5GXWlxs3NDdbW1khPT9fbnp6eft83NHfuXMTExGDLli3o0KFDsf35gebcuXPYunXrfe+bBQQEIC8vD2fPnjW4X6VS6WbkrtSZub/+Gli0CJgyBbh6tXJeg4iIDBMCyMw0zWLckNRS7dq1C/7+/lCpVPD09MT06dP1hlh89913aN++Pezt7VG3bl0EBwcjMzMTgHwgx9/fH7Vq1YKLiwu6deuGc+fO6Y69e/cutmzZgr59+8LDw0O3ODo6wsbGRvfz5s2b4enpiZ9++glt2rSBSqXC+fPnceDAATz99NNwc3ODs7MzgoKCcPDgQb3+F779dPbsWSgUCvzwww944okn4ODgAF9f3zIPU3kQRoUapVIJPz8/xMfH67ZptVrEx8cjMDCwxOM++ugjzJw5E5s3b8YjjzxSbH9+oDl16hS2bduGunXr3rcvSUlJsLKyQv369Y15CxVvzBjA1xe4cUMGGyIiqjpZWYCjo2mWCqpofOnSJfTq1QudO3fG33//jUWLFmHZsmX44IMPAMhbO4MGDcKwYcOQnJyMnTt34vnnn9dNFdGvXz8EBQXh8OHDSEhIQHh4uF5Bwvj4eDRo0ACtWrUqw8eZhdmzZ2Pp0qU4duwY6tevj1u3biE0NBR//vkn9u7di+bNm6NXr164detWqed66623MGXKFCQlJaFFixYYNGhQqWNhK4Qw0urVq4VKpRJxcXHi+PHjIjw8XLi4uIi0tDQhhBCvvvqqmD59uq59TEyMUCqV4rvvvhOpqam65datW0IIIXJyckSfPn1Ew4YNRVJSkl6b7OxsIYQQe/bsEfPmzRNJSUnizJkzYtWqVaJevXpiyJAhZe63Wq0WAIRarTb2Ld/fvn1CKBRCAELEx1f8+YmISAghxJ07d8Tx48fFnTt35Ibbt+XfXlMst28b1ffQ0FDRt2/fYtvffPNN0bJlS6HVanXbYmNjhaOjo9BoNCIxMVEAEGfPni127PXr1wUAsXPnzhJfd+TIkWLKlCnFtkdFRQlfX1/dzytWrBAARFJSUqnvQ6PRiNq1a4uff/5Ztw2AWL9+vRBCiJSUFAFALF26VLf/2LFjAoBITk42eM5i/66FGPP9bfSYmoEDB+Lq1at45513kJaWho4dO2Lz5s26wcPnz5/Xu/+2aNEi5OTk4IUXXtA7T1RUFN59911cunQJP/30EwCgY8eOem127NiB7t27Q6VSYfXq1Xj33XeRnZ2NJk2aYPLkyXpjZkzK3x8YNw5YuBAYPRo4fBjgo4ZERJXPwQG4fdt0r10BkpOTERgYqHd1pVu3brh9+zYuXrwIX19fPPXUU2jfvj1CQkLQo0cPvPDCC6hTpw5cXV0xdOhQhISE4Omnn0ZwcDAGDBgAT09PAHIagp9//hlr164tU1+USmWxISLp6el4++23sXPnTly5cgUajQZZWVk4f/58qecqfJ78/ly5cqVMV4zKq1wDhcePH4/x48cb3Ldz5069n0sa85LPx8cH4j73JR9++GHs3bvXmC5WvQ8+AH74ATh1CoiOBt57z9Q9IiKyfAoFUKuWqXtRqaytrbF161bs2bMHW7ZswYIFC/DWW29h3759aNKkCVasWIGJEydi8+bNWLNmDd5++21s3boVXbp0wf79+5GXl4euXbuW6bXs7e2LzaUVGhqK69ev49NPP0Xjxo2hUqkQGBiInJycUs9la2urW88/p1arNfLdG4d1piuKszPw2WdyPToaOHHCtP0hIqJqoXXr1khISND7D/zdu3ejdu3aaNiwIQAZCrp164b33nsPhw4dglKpxPr163XtO3XqhMjISOzZswft2rXDt99+C0A+9dS7d29YW1uXu3+7d+/GxIkT0atXL7Rt2xYqlQrXrl0r9/kqE0NNRXr+eeB//wNyc4FRoyp0ZDwREVV/arUaSUlJekt4eDguXLiACRMm4MSJE9iwYQOioqIQEREBKysr7Nu3D7NmzcJff/2F8+fP44cffsDVq1fRunVrpKSkIDIyEgkJCTh37hy2bNmCU6dOoXXr1gCAn376yeC0Q8Zo3rw5vv76ayQnJ2Pfvn0YPHgw7O3tK+LjqHCcpbsiKRRyXM327cDvvwNxcaxdQ0REOjt37kSnTp30tg0fPhybNm3C1KlT4evrC1dXVwwfPhxvv/02AMDJyQm///475s+fj4yMDDRu3Bgff/wxnnnmGaSnp+PEiRNYuXIlrl+/Dk9PT4wbNw6jRo3CmTNncPr0aYSEhDxQn5ctW4bw8HA8/PDD8Pb2xqxZszDFTJ/2NXrup+qqSud++vhj+Xi3q6u8DVWvXuW+HhFRDcG5n8ruk08+wbZt27Bp0yZTd+W+TDL3E5XRa6+xdg0REZlUw4YNERkZaepuVCmGmspgYwN8+aW8HfXVV/J2FBERURUaMGAAHnvsMVN3o0ox1FQWf39g7Fi5Pno0cPeuaftDRERk4RhqKtOHHwKengW1a4iIiKjSMNRUJtauISIiqjIMNZWtf3+gd29Zu2b0aNauISIiqiQMNRUgL09mFoPya9c4OAC7dgErV1Zp34iIiGoKhpoHdPw4EBgop34qkY9PwVxQU6YAZlpemoiIqDpjqHlAR44Af/0lxwQfPFhKw/zaNdevs3YNERFRJWCoeUADBgAvvABoNEBoKJCdXUJDW1vgiy/k7aiVK4EdO6q0n0RERBXFx8cH8+fPN3U3imGoeUAKBfD553ImhKNHC+4yGRQQwNo1REQ11NChQ9GvXz+TvPbKlSvx6KOPon379hg9erTBNl9//bVZz8BdFgw1FaBePWDxYrk+ezawf38pjfNr1/zzDxATUyX9IyKimm3Dhg3o06cPhg8fjtWrV+POnTvF2qxYsQJ9+vSBm5ubCXpYMRhqKsjzzwMvvwxotfI2lIHfF4m1a4iIKowQQGamaZaKrNCxa9cu+Pv7Q6VSwdPTE9OnT0deXp5u/3fffYf27dvD3t4edevWRXBwMDIzMwHImb/9/f1Rq1YtuLi4oFu3bjh37pzu2Lt372LLli3o06cPXnnlFdy5cwfff/+93uunpKRg586dGD58OM6cOYO+ffvC3d0djo6O6Ny5M7Zt21Zxb7YSMdRUoM8+Azw8ZE55551SGubXrsnJYe0aIqIHkJUFODqaZsnKqpj3cOnSJfTq1QudO3fG33//jUWLFmHZsmX44N5jtampqRg0aBCGDRuG5ORk7Ny5E88//zyEEMjLy0O/fv0QFBSEw4cPIyEhAeHh4VAoFLrzx8fHo0GDBmjVqhXc3NzQt29fLF++XK8PcXFxaNiwIXr06IHbt2+jV69eiI+Px6FDh9CzZ088++yzOH/+fMW84cokagi1Wi0ACLVaXamv89NPQgBCKBRC/PlnKQ1TUoRwcJCNV6yo1D4REVmKO3fuiOPHj4s7d+4IIYS4fVv+GTXFcvu2cX0PDQ0Vffv2Lbb9zTffFC1bthRarVa3LTY2Vjg6OgqNRiMSExMFAHH27Nlix16/fl0AEDt37izxdUeOHCmmTJmi+3nz5s1CoVCIf//9VwghhFarFY0bNxZvv/12iedo27atWLBgge7nxo0bi3nz5pX2do1S9N+1MGO+v3mlpoI9+6y8/SQEMHRoKUnexwd49125zto1RETl4uAA3L5tmsXBoWLeQ3JyMgIDA/WurnTr1g23b9/GxYsX4evri6eeegrt27fHiy++iCVLluC///4DALi6umLo0KEICQnBs88+i08//RSpqam68wgh8PPPP6NPnz66bU8//TQaNmyIFStWAJBXcs6fP4+wsDAAwO3btzFlyhS0bt0aLi4ucHR0RHJycrW4UsNQUwnmzwcaNABOnwbefLOUhpMmAR06sHYNEVE5KRRArVqmWQplkEplbW2NrVu34tdff0WbNm2wYMECtGzZEikpKQDkAN+EhAR07doVa9asQYsWLbB3714AwP79+5GXl4euXbvqzmdlZYWhQ4di5cqV0Gq1WLFiBZ544gk0bdoUADBlyhSsX78es2bNwh9//IGkpCS0b98eOTk5VfOGHwBDTSVwcQGWLpXrn34qZ0cwiLVriIhqvNatWyMhIQGi0PjK3bt3o3bt2mjYsCEAQKFQoFu3bnjvvfdw6NAhKJVKrF+/Xte+U6dOiIyMxJ49e9CuXTt8++23AORTT71794a1tbXea4aFheHChQv44YcfsH79egwfPlzvtYcOHYrnnnsO7du3h4eHB86ePVuJn0DFYaipJD17AiNGyPWwMHmp0qAuXYAxY+T66NGlVO8jIqLqTq1WIykpSW8JDw/HhQsXMGHCBJw4cQIbNmxAVFQUIiIiYGVlhX379mHWrFn466+/cP78efzwww+4evUqWrdujZSUFERGRiIhIQHnzp3Dli1bcOrUKbRu3RoA8NNPP+ndesrXpEkTPPnkkwgPD4dKpcLzzz+v29e8eXP88MMPSEpKwt9//42XX34ZWq22yj6jB1Jho3zMXFUNFNZ/TSEaNZIDysaMKaXhzZtCeHjIhu++W2X9IyKqbkobUGruQkNDBYBiy/Dhw8XOnTtF586dhVKpFB4eHmLatGkiNzdXCCHE8ePHRUhIiKhXr55QqVSiRYsWukG7aWlpol+/fsLT01MolUrRuHFj8c477wiNRiNOnz4tVCqVuF3CiOZvv/1WABBjx47V256SkiKeeOIJYW9vL7y9vcXChQtFUFCQeO2113RtzHWgsEKImvE8cUZGBpydnaFWq+Hk5FRlrxsfDwQHy/WtWwvWi1m3Ts65oFQChw8DLVtWWR+JiKqLu3fvIiUlBU2aNIGdnZ2pu2PWPvnkE2zbtg2bNm0ydVfuq7R/V2O+v3n7qZI99VTBzAjDhwMZGSU0fOEFoFcv1q4hIqIK0bBhQ0RGRpq6G1WKoaYKzJ4NNGkCnD8PvP56CY0UCiA2FrC3B3buBL76qiq7SEREFmbAgAF47LHHTN2NKsVQUwUcHYF75QCwdCmweXMJDX18CmbEfP111q4hIiIyAkNNFQkKAl57Ta6PGAHcvFlCw8K1a6ZOraLeERERVX8MNVVo1iygeXPg0iWZXQwqXLsmLk7eiiIiIj015BmXGqOi/j0ZaqqQg4PMKfm19n76qYSGhWvXjBrF2jVERPfY2toCALIqajZJMgv5/575/77lVa5HumNjYzFnzhykpaXB19cXCxYsgL+/v8G2S5YswVdffYWjR48CAPz8/DBr1iy99kIIREVFYcmSJbh58ya6deuGRYsWoXnz5ro2N27cwIQJE/Dzzz/DysoK/fv3x6effgpHR8cy9dlUj3QbMnUqMHeunNH76FGgbl0DjdRqoFUrIC1NzhEVFVXV3SQiMkupqam4efMm6tevDwcHB705k6h6EUIgKysLV65cgYuLCzw9PYu1Meb72+hQs2bNGgwZMgSLFy9GQEAA5s+fj3Xr1uHkyZOoX79+sfaDBw9Gt27d0LVrV9jZ2WH27NlYv349jh07hgYNGgAAZs+ejejoaKxcuRJNmjTBjBkzcOTIERw/flz3vPozzzyD1NRUfPHFF8jNzUVYWBg6d+6sKwV9P+YUau7eBTp1Ak6cAAYNAkp8C2vXAgMHsnYNEVEhQgikpaXhZomDE6m6cXFxgYeHh8GAWqmhJiAgAJ07d8bChQsBAFqtFt7e3pgwYQKmT59+3+M1Gg3q1KmDhQsXYsiQIRBCwMvLC6+//jqm3JvUUa1Ww93dHXFxcXjppZeQnJyMNm3a4MCBA3jkkUcAAJs3b0avXr1w8eJFeHl53fd1zSnUAMD+/UBgIKDVAt99B/Tvb6CREEDv3sCvvwLduwPbt1fdDGpERGZOo9EgNzfX1N2gB2Rra1tsbqrCjPn+tjHmhXNycpCYmKhXzMfKygrBwcFISEgo0zmysrKQm5sLV1dXAEBKSgrS0tIQXKjUrrOzMwICApCQkICXXnoJCQkJcHFx0QUaAAgODtbNifHcc88Ve53s7GxkFxqLklFi1TvT8PcHpk0DoqPl8JnHHwfq1SvSKL92Tdu2BbVrQkNN0V0iIrNjbW1d6pch1TxGDRS+du0aNBoN3N3d9ba7u7sjLS2tTOeYNm0avLy8dCEm/7jSzpmWllbs1paNjQ1cXV1LfN3o6Gg4OzvrFm9v7zL1rypFRQHt2gFXr8qqwwavmTVpIsfUAKxdQ0REVIoqffopJiYGq1evxvr16yt9zo7IyEio1WrdcuHChUp9vfJQqeRTUDY28hbU2rUlNJw8GWjfnrVriIiISmFUqHFzc4O1tTXS09P1tqenp8PDw6PUY+fOnYuYmBhs2bIFHTp00G3PP660c3p4eODKlSt6+/Py8nDjxo0SX1elUsHJyUlvMUcPPwy89ZZcHztWPuxUDGvXEBER3ZdRoUapVMLPzw/x8fG6bVqtFvHx8QgMDCzxuI8++ggzZ87E5s2b9cbFAECTJk3g4eGhd86MjAzs27dPd87AwEDcvHkTiYmJujbbt2+HVqtFQECAMW/BLL35JtCxI3DjRilzWQYGyp2A/F/WriEiItInjLR69WqhUqlEXFycOH78uAgPDxcuLi4iLS1NCCHEq6++KqZPn65rHxMTI5RKpfjuu+9Eamqqbrl165ZeGxcXF7FhwwZx+PBh0bdvX9GkSRNx584dXZuePXuKTp06iX379ok///xTNG/eXAwaNKjM/Var1QKAUKvVxr7lKvH330LY2goBCPHVVyU0+u8/ITw8ZKP33qvK7hEREZmEMd/fRocaIYRYsGCBaNSokVAqlcLf31/s3btXty8oKEiEhobqfm7cuLEAUGyJiorStdFqtWLGjBnC3d1dqFQq8dRTT4mTJ0/qveb169fFoEGDhKOjo3BychJhYWF6weh+zD3UCCHEBx/IvOLiIsTFiyU0WrNGNlIqhSjyGREREVkaY76/y1VRuDoytzo1huTlybtMf/0F9OoFbNxooCxN4do1TzwBxMezdg0REVksY76/OfeTGbGxkU9DKZXApk3AihUGGuXXrrG3B3bsAL7+usr7SUREZI4YasxMmzbAzJlyffJk4Px5A40K166JiGDtGiIiIjDUmKXXX5cTdWdkACNGlPA0VOHaNW+8UeV9JCIiMjcMNWbI2lqWo7GzA7ZuBb780kCjwrVrVqxg7RoiIqrxGGrMVMuWcl4oQF65SUkx0CgwEBg1Sq6zdg0REdVwDDVmbOJE4LHHgMxMYNgwOaN3MdHRgLs7cPIkMHt2lfeRiIjIXDDUmDErK3lnycFB3l36/HMDjVxcgE8/lesffgj8808V9pCIiMh8MNSYuYceAj76SK5PmwacPm2g0YABQM+eQE4OMGZMCSOLiYiILBtDTTUwZoyss5eVBQwdCmg0RRooFPIyjr09sH07sGqVKbpJRERkUgw11YCVFbB8OeDoCOzeXXC3SU+TJkBUlFyPiJCPehMREdUgDDXVhI8P8PHHcv2tt4ATJww0iogA2rWTxfhYu4aIiGoYhppqZORIoEcP4O5deRsqL69IA1vbgqI2y5cDu3ZVdReJiIhMhqGmGlEogKVLAScnYN++gis3egIDZc0agLVriIioRmGoqWa8vQvG1LzzDnDsmIFG+bVrTpwoeHSKiIjIwjHUVEOhoUDv3vIJ7tBQIDe3SAPWriEiohqIoaYaUijk0Jk6dYDExBIKCefXrsnOZu0aIiKqERhqqikvL2DBArn+/vvA338XaaBQALGxclZM1q4hIqIagKGmGnv5ZaBfP3n7KTRU3o7S07Qpa9cQEVGNwVBTjSkUwOLFQN268krNBx8YaPT666xdQ0RENQJDTTXn7l4w0eWsWXKMjR5bW+CLL+Q6a9cQEZEFY6ixAAMGyEWjkbehipWm6doVGDVKrrN2DRERWSiGGgsRGwvUry/r1rz7roEGrF1DREQWjqHGQri5yfE1gMws+/YVaVCnDjB/vlz/8EPg1Kmq7B4REVGlY6ixIM89BwweDGi18jbUnTtFGgwcCISEsHYNERFZJIYaC/PZZ4CnJ3DyJDBjRpGdCoUcVWxnB8THA998Y5I+EhERVQaGGgvj6lowUfcnnwB//lmkQeHaNZMns3YNERFZDIYaC/S//wFDh8q7S2FhQGZmkQaFa9dMm2aKLhIREVU4hhoLNW8e0LAhcPo08OabRXYWrl2zbBnw++9V3j8iIqKKxlBjoVxcgKVL5fpnnwE7dxZpULh2zahRrF1DRETVHkONBQsJAcLD5XpYGHD7dpEG0dGyuM2JE8CcOVXePyIioopUrlATGxsLHx8f2NnZISAgAPv37y+x7bFjx9C/f3/4+PhAoVBgfn6tlELy9xVdxo0bp2vTvXv3YvtHjx5dnu7XKHPnAo0bA2fPAlOnFtlZuHbNBx+wdg0REVVrRoeaNWvWICIiAlFRUTh48CB8fX0REhKCK1euGGyflZWFpk2bIiYmBh4eHgbbHDhwAKmpqbpl69atAIAXX3xRr93IkSP12n3Eyrj3Vbu2nPIJkMX57n20BV56CejRg7VriIio2jM61HzyyScYOXIkwsLC0KZNGyxevBgODg5Ynv/NWUTnzp0xZ84cvPTSS1CpVAbb1KtXDx4eHrpl48aNeOihhxAUFKTXzsHBQa+dk5OTsd2vkZ58Esi/6DV8OKBWF9rJ2jVERGQhjAo1OTk5SExMRHBwcMEJrKwQHByMhISECulQTk4OVq1ahWHDhkGhUOjt++abb+Dm5oZ27dohMjISWVlZJZ4nOzsbGRkZektNFhMjS9RcuCCf6Nbz0EPAO+/IddauISKiasqoUHPt2jVoNBq4u7vrbXd3d0daWlqFdOjHH3/EzZs3MXToUL3tL7/8MlatWoUdO3YgMjISX3/9NV555ZUSzxMdHQ1nZ2fd4u3tXSH9q64cHYG4OHlhZtky4NdfizR4/XWgbVvWriEiomrL7J5+WrZsGZ555hl4eXnpbQ8PD0dISAjat2+PwYMH46uvvsL69etx5swZg+eJjIyEWq3WLRcuXKiK7pu1xx4DXntNro8YAfz3X6GdSqV+7Zo//qjy/hERET0Io0KNm5sbrK2tkZ6errc9PT29xEHAxjh37hy2bduGESNG3LdtQEAAAOD06dMG96tUKjg5OektJCfobtECuHwZmDSpyM5u3QqeAR81CsjJqeruERERlZtRoUapVMLPzw/x8fG6bVqtFvHx8QgMDHzgzqxYsQL169dH796979s2KSkJAODp6fnAr1uTODjI21BWVsBXXwE//VSkQUyMrF2TnMzaNUREVK0YffspIiICS5YswcqVK5GcnIwxY8YgMzMTYWFhAIAhQ4YgMjJS1z4nJwdJSUlISkpCTk4OLl26hKSkpGJXWLRaLVasWIHQ0FDY2Njo7Ttz5gxmzpyJxMREnD17Fj/99BOGDBmCxx9/HB06dCjP+67RAgMLBguHhxcZF1y4ds3MmXKeBSIioupAlMOCBQtEo0aNhFKpFP7+/mLv3r26fUFBQSI0NFT3c0pKigBQbAkKCtI752+//SYAiJMnTxZ7vfPnz4vHH39cuLq6CpVKJZo1ayamTp0q1Gp1mfusVqsFAKOOsWR37gjRurUQgBAvvVRkp1YrRI8ecmdwsPyZiIjIBIz5/lYIUTOqrWVkZMDZ2RlqtZrja+45cEBetdFogHXrgBdeKLTzzBk5k/fdu8CqVcDgwSbrJxER1VzGfH+b3dNPVHU6dwamT5frY8YAekWhi9auuXGjyvtHRERkDIaaGm7GDKB9e1meZuzYIrMkvP460KYNcPUqa9cQEZHZY6ip4VQqYOVKwMYG+P57YM2aQjsL165ZupS1a4iIyKwx1BA6dQLefluujxsH6BWHfvRRYORIuc7aNUREZMYYaggA8OabMtzcuCEf89a7DcXaNUREVA0w1BAAwNZW3oaytQV+/hn4+utCO11dgXnz5Dpr1xARkZliqCGd9u2Bd9+V6xMnApcuFdo5aBDw9NNAdrZ8VKpmVAIgIqJqhKGG9LzxhnzUW62WQ2l02UWhABYtAuzsgG3bgG+/NWk/iYiIimKoIT02NvI2lEoF/PorsHx5oZ0PPSSfAQeAiAjWriEiIrPCUEPFtG4NfPCBXJ88GTh/vtDOKVNk7ZorVwoq9xEREZkBhhoyaPJkoGtX4NYtYPjwQrehCteuWbIE+PNPk/WRiIioMIYaMsjaGlixArC3l0No8nMMANauISIis8RQQyVq0QKIjpbrU6YA//5baGd+7Zrjx4G5c03SPyIiosIYaqhUEyYAjz8OZGYCw4YBWu29HaxdQ0REZoahhkplZSVvQ9WqBezaBcTGFto5aBAQHAzcvWtgNkwiIqKqxVBD99W0KfDRR3J92jTg1Kl7O/Jr16hUwNatwP/9n8n6SERExFBDZTJ6NPDUU8CdO0BYGKDR3NvRrFlB7ZrJk1m7hoiITIahhsrEygpYtgyoXRvYvRuYP7/QzqlTZXEb1q4hIiITYqihMmvcGPjkE7n+1lvAiRP3drB2DRERmQGGGjLK8OFAz55yXsvQUCAv796Oxx4DRoyQ66xdQ0REJsBQQ0ZRKOTFGGdnYP/+IiVqZs8G6tVj7RoiIjIJhhoyWsOGwKefyvWoKODo0Xs7itauOXPGJP0jIqKaiaGGymXIEOB//5N3mUJDgdzceztefpm1a4iIyCQYaqhcFArgyy+BOnWAgwflrAm6Hfm1a7ZsAVavNmk/iYio5mCooXLz9AQWLpTr778PJCXd21G4ds2kScB//5mgd0REVNMw1NADGTQIeP55+RRUaGihh55Yu4aIiKoYQw09kPy7TW5uwOHDcnwwAP3aNV9+KSv2ERERVSKGGnpg9esDn38u16Ojgb/+urfjscdkYRuAtWuIiKjSMdRQhXjxRWDgQDknVGioLM4HQM6EWa8ecOwY8PHHJu0jERFZNoYaqjALF8qrNsePy/o1AGTtmvy5Fd5/n7VriIio0pQr1MTGxsLHxwd2dnYICAjA/v37S2x77Ngx9O/fHz4+PlAoFJivNxOi9O6770KhUOgtrVq10mtz9+5djBs3DnXr1oWjoyP69++P9PT08nSfKombmxw+AwBz5gB7997bMXiwnOKbtWuIiKgSGR1q1qxZg4iICERFReHgwYPw9fVFSEgIrly5YrB9VlYWmjZtipiYGHh4eJR43rZt2yI1NVW3/FlkUsTJkyfj559/xrp167Br1y5cvnwZzz//vLHdp0rWty/wyiuAVgsMHQrcuQPWriEioqohjOTv7y/GjRun+1mj0QgvLy8RHR1932MbN24s5s2bV2x7VFSU8PX1LfG4mzdvCltbW7Fu3TrdtuTkZAFAJCQklKnfarVaABBqtbpM7an8btwQwtNTCECIiIhCO2bOlBvr15eNiIiI7sOY72+jrtTk5OQgMTERwcHBum1WVlYIDg5GQkLCA4WrU6dOwcvLC02bNsXgwYNx/vx53b7ExETk5ubqvW6rVq3QqFGjEl83OzsbGRkZegtVjTp1gKVL5fq8ecAff9zbwdo1RERUiYwKNdeuXYNGo4G7u7vednd3d6SlpZW7EwEBAYiLi8PmzZuxaNEipKSk4LHHHsOtW7cAAGlpaVAqlXBxcSnz60ZHR8PZ2Vm3eHt7l7t/ZLxevYBhw+TwmbAwIDMT8vbT4sWyAWvXEBFRBTOLp5+eeeYZvPjii+jQoQNCQkKwadMm3Lx5E2vXri33OSMjI6FWq3XLhQsXKrDHVBaffCJn9D5zBoiMvLfx8ccLateMHl1oJkwiIqIHY1SocXNzg7W1dbGnjtLT00sdBGwsFxcXtGjRAqdPnwYAeHh4ICcnBzdv3izz66pUKjg5OektVLWcnYFly+T6ggXAjh33duTXrjl6lLVriIiowhgVapRKJfz8/BAfH6/bptVqER8fj8DAwArr1O3bt3HmzBl4enoCAPz8/GBra6v3uidPnsT58+cr9HWp4vXoIYsJA/J21K1b0K9d8957wL//mqx/RERkOYy+/RQREYElS5Zg5cqVSE5OxpgxY5CZmYmwsDAAwJAhQxCpu9cgBxcnJSUhKSkJOTk5uHTpEpKSknRXYQBgypQp2LVrF86ePYs9e/bgueeeg7W1NQYNGgQAcHZ2xvDhwxEREYEdO3YgMTERYWFhCAwMRJcuXR70M6BKNmcO4OMDnD0rxwoDYO0aIiKqeOV5vGrBggWiUaNGQqlUCn9/f7F3717dvqCgIBEaGqr7OSUlRQAotgQFBenaDBw4UHh6egqlUikaNGggBg4cKE6fPq33mnfu3BFjx44VderUEQ4ODuK5554TqampZe4zH+k2re3b5dPcgBC//XZv4z//CKFSyY3/938m7R8REZknY76/FULUjP9EzsjIgLOzM9RqNcfXmMiECXIqhYYN5XAaZ2cAH3wAzJgBuLsDycnyeXAiIqJ7jPn+Nounn6hmiIkBHnoIuHgRiIi4t3HqVKBVKyA9vdAjUkRERMZjqKEqU6sWEBcnZ01YvhzYtAmyds0XX8gGX3wB7Nljyi4SEVE1xlBDVerRR4HJk+X6yJHAf/9B1q4ZNkxuHDWKtWuIiKhcGGqoyn3wAdCiBXD5MvDaa/c2fvSRnOabtWuIiKicGGqoytnbAytXAlZWwNdfAxs2AKhbl7VriIjogTDUkEl06VJQsyY8HLh2DcArrwBPPsnaNUREVC4MNWQy774LtGkjJ+0ePx5yBPGiRXLw8G+/AWvWmLqLRERUjTDUkMnY2cnbUNbWMr+sWwc52Oatt2SDSZOAIvN9ERERlYShhkzqkUcKytOMHSuv2uCNN1i7hoiIjMZQQyY3YwbQoYMcVzNmDCCUhWrXLF4MJCSYtoNERFQtMNSQySmV8jaUjQ3www/A6tXQr10THs7aNUREdF8MNWQWOnaUV2wAYNw4IDUV+rVr8h/3JiIiKgFDDZmNyEjg4YdlleFRowDhyto1RERUdgw1ZDZsbeVtKKUS+Pln4KuvUFC75s4d1q4hIqJSMdSQWWnXTl6UAeQUChcv3atdo1TK2jVr15q2g0REZLYYasjsTJkC+PsDarWc9FI0L1S75rXXWLuGiIgMYqghs2NjI29DqVTA5s3AsmUApk0DWrZk7RoiIioRQw2ZpVatgA8/lOsREcC5NNauISKi0jHUkNmaNAno1g24dQsYPhzQPhYEhIXJnaxdQ0RERTDUkNmytgZWrADs7YH4+HsXaubMYe0aIiIyiKGGzFrz5kBMjFyfOhX4V10X+PhjueG994CUFNN1joiIzApDDZm98eOBoCAgM1PefdIOfhV44gnWriEiIj0MNWT2rKyA5cuBWrWA338HFsYq5GBhpVI+HrVunam7SEREZoChhqqFpk2BuXPl+vTpwD9g7RoiItLHUEPVxqhRQHCwvOsUFgZoptyrXZOWBrz5pqm7R0REJsZQQ9WGQiEL8dWuDezZA8z7nLVriIioAEMNVSuNGgHz5sn1t98GkusHAUOHysHCo0axdg0RUQ3GUEPVzrBhwDPPANnZQGgokBc9B6hbFzhypCDxEBFRjcNQQ9WOQgEsWQI4OwMHDgBzVrgV1K55913WriEiqqEYaqhaatAA+OwzuR4VBRzpNATo3p21a4iIajCGGqq2Xn0V6NNHDqMJHapA7gLWriEiqsnKFWpiY2Ph4+MDOzs7BAQEYP/+/SW2PXbsGPr37w8fHx8oFArMnz+/WJvo6Gh07twZtWvXRv369dGvXz+cPHlSr0337t2hUCj0ltGjR5en+2QhFAr58JOrK3DoEBD9Q8uCR7tZu4aIqMYxOtSsWbMGERERiIqKwsGDB+Hr64uQkBBcuXLFYPusrCw0bdoUMTEx8PDwMNhm165dGDduHPbu3YutW7ciNzcXPXr0QGZmpl67kSNHIjU1Vbd89NFHxnafLIyHBxAbK9dnzgQO9YwEWrRg7RoiohpIIYRxgw8CAgLQuXNnLFy4EACg1Wrh7e2NCRMmYPr06aUe6+Pjg0mTJmHSpEmltrt69Srq16+PXbt24fHHHwcgr9R07NjR4JWessjIyICzszPUajWcnJzKdQ4yT0IAL74IfP890L498NfHu6Ds0V1eytmzB+jSxdRdJCKicjLm+9uoKzU5OTlITExEcHBwwQmsrBAcHIyECix8plarAQCurq5627/55hu4ubmhXbt2iIyMRFZWVonnyM7ORkZGht5ClkmhAD7/HHBzk091v7+LtWuIiGoio0LNtWvXoNFo4O7urrfd3d0daWlpFdIhrVaLSZMmoVu3bmjXrp1u+8svv4xVq1Zhx44diIyMxNdff41XXnmlxPNER0fD2dlZt3h7e1dI/8g81a8viwoDQEwMcODlebJ2zeHDQDmv7hERUfVidk8/jRs3DkePHsXq1av1toeHhyMkJATt27fH4MGD8dVXX2H9+vU4c+aMwfNERkZCrVbrlgsXLlRF98mE+vcHXnoJ0GiAoZNccDf6XiG+qCjg7FmT9o2IiCqfUaHGzc0N1tbWSE9P19uenp5e4iBgY4wfPx4bN27Ejh070LBhw1LbBgQEAABOnz5tcL9KpYKTk5PeQpZv4ULA3R04fhyIOvVKQe2aceNYu4aIyMIZFWqUSiX8/PwQHx+v26bVahEfH4/AwMByd0IIgfHjx2P9+vXYvn07mjRpct9jkpKSAACenp7lfl2yPHXrAl9+KdfnfqxAwqg4Wbtm0ybgu+9M2jciIqpcRt9+ioiIwJIlS7By5UokJydjzJgxyMzMRFhYGABgyJAhiIyM1LXPyclBUlISkpKSkJOTg0uXLiEpKUnvCsu4ceOwatUqfPvtt6hduzbS0tKQlpaGO3fuAADOnDmDmTNnIjExEWfPnsVPP/2EIUOG4PHHH0eHDh0e9DMgC9OnDzBkCKDVAkOjGiNryjtyx8SJwL1B6EREZIFEOSxYsEA0atRIKJVK4e/vL/bu3avbFxQUJEJDQ3U/p6SkCADFlqCgIF0bQ/sBiBUrVgghhDh//rx4/PHHhaurq1CpVKJZs2Zi6tSpQq1Wl7nParVaADDqGKq+btwQwstLCECIyRNyhWjRQv4wdqypu0ZEREYw5vvb6Do11RXr1NQ8v/4K9OolH/neNe8gHpvkx9o1RETVTKXVqSGqTp55Bhg+XI4PHvrZw8h8eSRr1xARWTCGGrJoH38MeHsD//4LTLf/lLVriIgsGEMNWTRnZ2D5crm+cJk9tg9bJX9g7RoiIovDUEMWLzgYyJ/QfdjaENzq1pO1a4iILBBDDdUIc+YAPj7AuXMKTGnwLWvXEBFZIIYaqhEcHYEVK+T6l2vrYMuApfIH1q4hIrIYDDVUY3TvLjMMAAzf8QpuPuQHpKUBb71l0n4REVHFYKihGmXWLKBZM+DiJQUimv8sN37+ObBvn2k7RkRED4yhhmqUWrWAuDhZg2/FZk9sfPITOVg4PJy1a4iIqjmGGqpxunUDIiLkevixibhR5yFZu+bTT03bMSIieiAMNVQjzZwJtGoFpKZb47VWm+VG1q4hIqrWGGqoRrK3l7ehrKyAVQnN8GObN4GsLNauISKqxhhqqMYKCADeeEOuj0p/D9dsPGTtmu+/N23HiIioXBhqqEZ7912gbVvgynUbjGuxVW5k7RoiomqJoYZqNJUKWLkSsLYG1h5vh7UeE4HUVNauISKqhhhqqMbz8yvIMGOz5iAd9Vm7hoioGmKoIYIMNb6+wPUMJcY0+gWCtWuIiKodhhoiyPktV64EbG2B9ecfwf/VGsnaNURE1QxDDdE9vr7AO+/I9fFYgMvwlLVrzp0zbceIiKhMGGqICpk2TY6x+S9ThVGu30Gwdg0RUbXBUENUiK2tvA2lVAIbb3TFSuthwC+/AD/8YOquERHRfTDUEBXRti3w/vty/TWbWFxEA2DCBNauISIycww1RAZMmQJ06QJkZNthuMP/QaSmAm+/bepuERFRKRhqiAywtpZzQ9nZAVuyHsNSjABiY4H9+03dNSIiKgFDDVEJWrYEPvxQrkfYfIZzwlvWrsnLM23HiIjIIIYaolK89hrw6KPA7Tx7DLP5Gtq/WbuGiMhcMdQQlcLaGlixAnBwALbnPY7FGC2L2bB2DRGR2WGoIbqPZs2A2bPl+lSrj3Emy4O1a4iIzBBDDVEZjB0LPPEEkKW1R5giDtpfNrF2DRGRmWGoISoDKytg+XLA0RH4QzyGzzCRtWuIiMxMuUJNbGwsfHx8YGdnh4CAAOwv5THXY8eOoX///vDx8YFCocD8+fPLdc67d+9i3LhxqFu3LhwdHdG/f3+kp6eXp/tE5eLjA8ydK9cjFTH4J9WRtWuIiMyI0aFmzZo1iIiIQFRUFA4ePAhfX1+EhITgypUrBttnZWWhadOmiImJgYeHR7nPOXnyZPz8889Yt24ddu3ahcuXL+P55583tvtEDyQ8HHj6aeCusMNQxEGzcBFr1xARmQthJH9/fzFu3DjdzxqNRnh5eYno6Oj7Htu4cWMxb948o8958+ZNYWtrK9atW6drk5ycLACIhISEMvVbrVYLAEKtVpepPVFJzp8XwslJCECIjzBFiI4dhcjNNXW3iIgskjHf30ZdqcnJyUFiYiKCg4N126ysrBAcHIyEhIRyhaqynDMxMRG5ubl6bVq1aoVGjRqV+LrZ2dnIyMjQW4gqgrc3MG+eXJ+BmTielA189plpO0VERMbdfrp27Ro0Gg3c3d31tru7uyMtLa1cHSjLOdPS0qBUKuHi4lLm142Ojoazs7Nu8fb2Llf/iAwJCwN69QKyIW9D5b39LmvXEBGZmMU+/RQZGQm1Wq1bLly4YOoukQVRKIAlSwAXF4ED8MdHd8YD48ezdg0RkQkZFWrc3NxgbW1d7Kmj9PT0EgcBV8Q5PTw8kJOTg5s3b5b5dVUqFZycnPQWoork5QUsWKAAALyLd3F44zlg/XoT94qIqOYyKtQolUr4+fkhPj5et02r1SI+Ph6BgYHl6kBZzunn5wdbW1u9NidPnsT58+fL/bpEFWHwYKBvXyAXSoRiJXLHTwY4fouIyCSMvv0UERGBJUuWYOXKlUhOTsaYMWOQmZmJsLAwAMCQIUMQGRmpa5+Tk4OkpCQkJSUhJycHly5dQlJSEk6fPl3mczo7O2P48OGIiIjAjh07kJiYiLCwMAQGBqJLly4P+hkQlZtCAXzxBVC3rkASOmFW6lDWriEiMpXyPF61YMEC0ahRI6FUKoW/v7/Yu3evbl9QUJAIDQ3V/ZySkiIAFFuCgoLKfE4hhLhz544YO3asqFOnjnBwcBDPPfecSE1NLXOf+Ug3VabVq+Uj3jbIEYl4WIj9+03dJSIii2DM97dCiJoxsjEjIwPOzs5Qq9UcX0MVTghgwADgu++A9jiMAx1GQJW4B7CxMXXXiIiqNWO+vy326SeiqqRQAJ9/DtSrq8URdMD7h/uydg0RURVjqCGqIPXqAYu/lP+XisF0HHhzPWvXEBFVIYYaogr0/PPAy4MEtLBGaPYXuDtmMmvXEBFVEYYaogq2YKECHm65SEYbvPNrF9auISKqIgw1RBXM1RX4crktAGAupmDPqJWsXUNEVAUYaogqwbPPAqGv5EHACkOvzUHW9PdN3SUiIovHUENUSeYvsEEDt7s4hRZ4a1FD4MABU3eJiMiiMdQQVRIXF2Dp13YAgE8xEb8P/gLIyzNtp4iILBhDDVEl6tkTGDH4DgSsEHYqErfnLjZ1l4iILBZDDVEl+/hzezRyvYV/8RCmzVAC58+buktERBaJoYaokjk5ActX1wIAfJ4XjviBX7J2DRFRJWCoIaoCTz1thbEv3QAADNs7EhnfbjRxj4iILA9DDVEVmb3EFU1dbuA8GmNKeAZr1xARVTCGGqIq4ugIrFgrb0MtyRqM34Z8Y+IeERFZFoYaoir0+NMqvPacnORy+IZncXP7QRP3iIjIcjDUEFWxWasao3ntVFxCQ0x+8SJr1xARVRCGGqIq5uAAxP2fCgpoEXejDzaO5qBhIqKKwFBDZAJde7vi9R5HAQAjl3fBjcMXTdwjIqLqj6GGyERmrm+HVvZnkSY8MLHXaVN3h4io2mOoITIROwcrrFyuhRU0+OZSd6yfvs/UXSIiqtYYaohMyP+lppje9Q8AwKg5D+Fqym0T94iIqPpiqCEysXd+CUA75Ulc1bphXMgpU3eHiKjaYqghMjGViz1WfnIDNsjFulOdsDb6jKm7RERULTHUEJmBh8cF4q12PwEAxs6oi/RLrF1DRGQshhoiM/HmpkfR0fowrmtcMKrnOU7kTURkJIYaIjOh9HbHyrdOwRY52HD0IXy74Lqpu0REVK0w1BCZkQ5RzyGqURwAYPwUFS5fNm1/iIiqE4YaInNiZYVpP3XDI/gLN3Md8WrPqzh71tSdIiKqHhhqiMyMjW9brBz5J1S4i+1H6qFpU4Fne2vx66+AVmvq3hERmS+GGiIz1ObTUdjmMxI98BuEUGDjJiv06gU0byYwdy5wncNtiIiKKVeoiY2NhY+PD+zs7BAQEID9+/eX2n7dunVo1aoV7Ozs0L59e2zatElvv0KhMLjMmTNH18bHx6fY/piYmPJ0n8j82dvj0aOL8dvHx/BP3UBMxidwwX/4N0WBqVOBhg0FwsKAAwdM3VEiIvNhdKhZs2YNIiIiEBUVhYMHD8LX1xchISG4cuWKwfZ79uzBoEGDMHz4cBw6dAj9+vVDv379cPToUV2b1NRUvWX58uVQKBTo37+/3rnef/99vXYTJkwwtvtE1UetWkBEBJpf2I5PPrXBJY9HsBTD0QkHcfeuAnFxgL8/0LkzEBcH3Llj6g4TEZmWQgjjqmEEBASgc+fOWLhwIQBAq9XC29sbEyZMwPTp04u1HzhwIDIzM7Fx40bdti5duqBjx45YvHixwdfo168fbt26hfj4eN02Hx8fTJo0CZMmTTKmuzoZGRlwdnaGWq2Gk5NTuc5BZFJ37wIrVkDMisb+i56IxTiswUDkQAUAcHUFhg0DRo8GHnrIxH0lIqogxnx/G3WlJicnB4mJiQgODi44gZUVgoODkZCQYPCYhIQEvfYAEBISUmL79PR0/PLLLxg+fHixfTExMahbty46deqEOXPmIC+v5Kqr2dnZyMjI0FuIqjU7O2DMGCjOnEbAlyPwVZN3cRENMRtvwMfqHG7cAObOBZo1A555Bti4EdBoTN1pIqKqY1SouXbtGjQaDdzd3fW2u7u7Iy0tzeAxaWlpRrVfuXIlateujeeff15v+8SJE7F69Wrs2LEDo0aNwqxZs/DGG2+U2Nfo6Gg4OzvrFm9v77K8RSLzp1QCI0cCJ0+i3oo5eKPZepzWNsVG9EYvmy1QKAQ2bwaefVZesYmJAa5eNXWniYgqn9k9/bR8+XIMHjwYdnZ2etsjIiLQvXt3dOjQAaNHj8bHH3+MBQsWIDs72+B5IiMjoVardcuFCxeqovtEVcfWFhg6FEhOhvWqr9C71b/4JS8Ep0QzTFV+Clf7LJw7B0RGAg0bAq++CiQkgNMvEJHFMirUuLm5wdraGunp6Xrb09PT4eHhYfAYDw+PMrf/448/cPLkSYwYMeK+fQkICEBeXh7OllCZTKVSwcnJSW8hskg2NsDgwcDRo8Dq1XiorT0+ypmEi3fqIk41Cv6eF5CTA6xaBXTtCvj5AUuXApmZpu44EVHFMirUKJVK+Pn56Q3g1Wq1iI+PR2BgoMFjAgMD9doDwNatWw22X7ZsGfz8/ODr63vfviQlJcHKygr169c35i0QWS5ra2DgQODwYeD772Hv2xKh2V9iX2ojHFA9irB2+2FnJ3DokLx71aABMHky8M8/pu44EVEFEUZavXq1UKlUIi4uThw/flyEh4cLFxcXkZaWJoQQ4tVXXxXTp0/Xtd+9e7ewsbERc+fOFcnJySIqKkrY2tqKI0eO6J1XrVYLBwcHsWjRomKvuWfPHjFv3jyRlJQkzpw5I1atWiXq1asnhgwZUuZ+q9VqAUCo1Wpj3zJR9aTVCrFhgxB+fkLIu07iuspTzH1svXjIJzd/kwCEePppIdavFyI319SdJiLSZ8z3t9GhRgghFixYIBo1aiSUSqXw9/cXe/fu1e0LCgoSoaGheu3Xrl0rWrRoIZRKpWjbtq345Zdfip3ziy++EPb29uLmzZvF9iUmJoqAgADh7Ows7OzsROvWrcWsWbPE3bt3y9xnhhqqsbRaITZtEiIgQJdiNEo78ev/FopngzOFQlEQbry9hfjgAyHu/TcKEZHJGfP9bXSdmuqKdWqoxhMC2LYNeP994M8/5TZbW5x9YQq+cHkDS9e54No13Wb07w+MGwd06wYoFKbrNhHVbMZ8fzPUENU0QgC7dslws2OH3GZjg+yXw7Cuw/v4/HsPFC4j1aEDMHasHIvs6GiaLhNRzVVpxfeIyAIoFED37sD27cAffwA9egB5eVB9tQSvvNEAe5oNwcEfzmLECMDeXo47Hj0a8PICJkwAkpNN/QaIiAxjqCGqyR59FPjtN1nAplcvQKsFvv4anfo3xZLbg3A5Phnz5wMtWgC3bgELFwJt2gBPPgl89x2Qm2vqN0BEVIChhoiALl2AX34B/voL6NtX3qJavRouXdvgtT9fRPLqv7F1K/Dcc4CVlbxr9eKLgI8P8N57wOXLpn4DREQcU0NEhiQlAR98AHz/fcG2vn2BGTNwob4fvvwSWLIEyK+raWMjA8/YsUBQEAcWE1HF4ZgaInowHTvK+0tHjgAvvSRTyoYNwCOPwHvM/zDzf/tw/jzwf/8HPPYYkJcHrFsHPPEE0K4dEBsLcA5ZIqpqDDVEVLJ27WRyOX5cTh5lZSVvU3XpAuWzIXip4Z/4/Xfg77/lYOJatWTT8eNlxeKxY+XsDUREVYGhhojur1Ur4KuvgJMngbAwOSXDli3yMs2TT6LDjZ1Y9LnA5cvAggVA69bA7dvAokVA+/bA448Da9YAOTmmfiNEZMkYaoio7Jo1A5YvB06dAsLDZZW+HTvkfaegIDjt34bx4wSOHZNPjL/wgsw/f/wh72I1bgy88w5w8aKp3wgRWSIOFCai8jt/HvjoIzlqOP8yTJcuMrn07AkoFLh0Se7+8ksgNVU2sbYG+vSRt6eeeooDi4moZKwobABDDVElunQJmDMH+OIL4O5due2RR4AZM4BnnwUUCuTmyrHGsbHAzp0Fh7ZsCYwZA4SGAi4upug8EZkzhhoDGGqIqkBaGjB3rhxMk5Ult3XsKMNNv35yoDHkYOLPP5fDdG7dks0cHORUDGPHykOIiACGGoMYaoiq0NWrwCefyBLEt2/Lbe3aAW+/XTDQBjLQfPONvHpT+Cmprl1luHnhBUClMkH/ichsMNQYwFBDZALXrwOffiqX/MI1rVoBb70lRw7b2ACQBYz//FNevfnuO1n3BgDq1QNGjABGjZKDjImo5mGoMYChhsiEbt4EPvsMmDdPrgPySaq33pL3nGxtdU3T0oClS+XwnPynpKysgP/9T169efpp3V0sIqoBGGoMYKghMgMZGfJe08cfy6s4ANCkCRAZKUcKK5W6pnl5wM8/y6s327YVnKJZMzmweOhQwNW1artPRFWPocYAhhoiM5JfmW/uXODKFbnN21uGm2HDig2kOXlSNo+LA9Rquc3ODhg0CBg3DvDzq9ruE1HVYagxgKGGyAxlZckCNrNny/tOgJxf4Y03gJEjAXt7veaZmcC338qLPX//XbDd31/emho4UIYdIrIcDDUGMNQQmbE7d4Bly2S4yR9I4+4OTJ1aMKlUIUIAe/fKcLNuXUHdv7p15YWe0aOBpk2r+D0QUaVgqDGAoYaoGsjOlveYoqOBc+fkNjc3YMoUeSmmdu1ih1y5IvPQ4sWywDEgKxQ/84w8pGdP3RPkRFQNMdQYwFBDVI3k5gJffw18+CHw779ym6srEBEhpwB3di52iEYDbNokr9789lvB9iZN5JWbYcNkPiKi6oWhxgCGGqJqKC9PDqL58EPgn3/kNhcX4LXX5FKnjsHDTp+WA4tXrAD++09uU6nkmJuxY+UYHM43RVQ9MNQYwFBDVI1pNMDatcDMmUBystxWuzYwYQIweXKJl2CysoA1a+TVm8TEgu1+fjLcvPSSnJ6BiMwXQ40BDDVEFkCrBb7/XoabI0fktlq15HPdr78O1K9v8DAhgAMHZLhZs0YO3QHkRZ+wMFn3pnnzqnkLRGQchhoDGGqILIhWC/z0E/D++8ChQ3Kbvb0cPDN1KuDpWeKh167J21KLFgEpKQXbe/SQ2ah3bw4sJjInDDUGMNQQWSAhgF9+kVdu9u+X21QqWeNm2jSgYcMSD9Vqgc2bZcXiTZvkqQCgUSM519SIESVe+CGiKsRQYwBDDZEFEwLYskVeudmzR25TKuUjT9On33c2zJQU+Uj4smUFszfY2gIvviiv3gQGcmAxkakY8/3NaeGIqPpTKICQEDnVd3w8EBQkK/ItXiwnixoxAjhzpsTDmzQpqPu3ciUQECCfKv/2W6BbN6BTJ1n4ODOzCt8TERmNoYaILIdCATz5JLBzp1yeeko+Fr5sGdCypZwFM//RcAPs7IAhQ2S14r/+khd67OzklAyjRgFeXvJJ8hMnquoNEZExGGqIyDIFBcnpvXfvlmWFNRp5GaZ1a2DwYOD48VIP9/OTWejSJTmpeLNmcpLxzz6TpwgOBn74QWYmIjIPDDVEZNm6dgV+/RXYtw949lk5Qvjbb4F27WQ1vvxHw0uQX8j45ElZqbhPH8DKSt7l6t8f8PGR45RTU6vm7RBRycoVamJjY+Hj4wM7OzsEBARgf/5TByVYt24dWrVqBTs7O7Rv3x6bNm3S2z906FAoFAq9pWfPnnptbty4gcGDB8PJyQkuLi4YPnw4bt++XZ7uE1FN5O8vHwNPTASee04OLl67FujQAXj++YJHw0tgZSUf+96wQc7cEBkJ1Ksnr+S88458amrgQOD33wuepCKiqmV0qFmzZg0iIiIQFRWFgwcPwtfXFyEhIbhy5YrB9nv27MGgQYMwfPhwHDp0CP369UO/fv1w9OhRvXY9e/ZEamqqbvm///s/vf2DBw/GsWPHsHXrVmzcuBG///47wsPDje0+EdV0Dz8s7xsdPgwMGCDH4axfL7f36SOr9N1H48bArFnAhQvAN9/Ii0F5eTIjBQXJnLRoEXDrVhW8HyIqIIzk7+8vxo0bp/tZo9EILy8vER0dbbD9gAEDRO/evfW2BQQEiFGjRul+Dg0NFX379i3xNY8fPy4AiAMHDui2/frrr0KhUIhLly6Vqd9qtVoAEGq1ukztiaiGOHZMiJdfFsLKSgh5kUWInj2F2L3bqNMcOiREeLgQDg4Fp3F0FGLsWCGOHq2crhPVBMZ8fxt1pSYnJweJiYkIDg7WbbOyskJwcDASEhIMHpOQkKDXHgBCQkKKtd+5cyfq16+Pli1bYsyYMbieXyzi3jlcXFzwyCOP6LYFBwfDysoK+/btM/i62dnZyMjI0FuIiIpp00ZebklOBkJDZTnhzZvls9zBwfJ+Uhl07Ah88YW8HfXpp/Jhq9u3ZXG/du2A7t2Bdevko+JEVDmMCjXXrl2DRqOBu7u73nZ3d3ekpaUZPCYtLe2+7Xv27ImvvvoK8fHxmD17Nnbt2oVnnnkGGo1Gd476RUp72tjYwNXVtcTXjY6OhrOzs27x9vY25q0SUU3TogUQFycf+R4xArCxKah50707sH17mQbLuLgAEyfKjLRtmxyuY20N7Nol73Y1bgxERcnwQ0QVyyyefnrppZfQp08ftG/fHv369cPGjRtx4MAB7Ny5s9znjIyMhFqt1i0XLlyouA4TkeVq2hRYsgQ4fVrOJWVrKxPJU08Bjz4qH4EqQ7hRKOQh338PnD0LzJgBuLvLp6Tef1+GmxdeKHNWIqIyMCrUuLm5wdraGunp6Xrb09PT4eHhYfAYDw8Po9oDQNOmTeHm5obTp0/rzlF0IHJeXh5u3LhR4nlUKhWcnJz0FiKiMmvcWI72PXMGGD9ezim1Z4+seRMQAGzcWOY00rChDDLnzwOrVwOPPy7L5nz/vQw+bdoACxYAanUlvyciC2dUqFEqlfDz80N8fLxum1arRXx8PAIDAw0eExgYqNceALZu3VpiewC4ePEirl+/Ds97M+0GBgbi5s2bSExM1LXZvn07tFotAgICjHkLRETG8faWiSMlBZg8Wc4GfuCArHnj5wf8+KOsfVMGSqV87HvXLlkeZ8wYwNFRViieOBFo0EBeHDp8uHLfEpHFMnYU8urVq4VKpRJxcXHi+PHjIjw8XLi4uIi0tDQhhBCvvvqqmD59uq797t27hY2NjZg7d65ITk4WUVFRwtbWVhw5ckQIIcStW7fElClTREJCgkhJSRHbtm0TDz/8sGjevLm4e/eu7jw9e/YUnTp1Evv27RN//vmnaN68uRg0aFCZ+82nn4ioQqSnC/HGG0LUqlXwmFP79kKsXSuERmP06dRqIRYuFKJNm4LTAUJ06ybEt98KkZ1dCe+BqBox5vvb6FAjhBALFiwQjRo1EkqlUvj7+4u9e/fq9gUFBYnQ0FC99mvXrhUtWrQQSqVStG3bVvzyyy+6fVlZWaJHjx6iXr16wtbWVjRu3FiMHDlSF5LyXb9+XQwaNEg4OjoKJycnERYWJm7dulXmPjPUEFGFunpViDffFKJ27YIk0qaNEN98I0RentGn02qF2LlTiBdfFMLGpuCU9esL8dZbQpw7VwnvgagaMOb7WyFEzRiiZszU5UREZXbjhpwQav78gkExLVoAb70FvPyyfIrKSJcvA0uXykfEL1+W26ys5B2vcePkOBwrs3jMg6jyGfP9zVBDRFQR1Gpg4ULgk09k0AHkk1Rvvgm8+qocUGOk3Fw5s0NsLLBjR8H25s3leJyhQ4E6dSqm+0TmiqHGAIYaIqoSt27Jintz5wLXrsltjRvLyaKGDpVPUZVDcrJ8GGvlSjlbOCDHLL/8MjB2rJzlgcgSMdQYwFBDRFUqM1PeP/roIyC/rEXDhsC0abK4n51duU57+7YsgBwbqz/BeJcuMty8+GK5T01klhhqDGCoISKTuHNHFvObPbtggIynJzB1KjBqFODgUK7TCgHs3i0vCn33XcH0C25uwPDh8tRNmlTQeyAyIYYaAxhqiMik7t4FVqwAoqPl9N4AUL8+MGVKQcGackpPLxhYnH9qhQLo3VtevQkJ4cBiqr4YagxgqCEis5CTIwfGREfLgn4AULcu8Prr8tGmB/j7lJcH/PKLvDW1dWvB9qZNZW4KC5MvRVSdMNQYwFBDRGYlN1cOjvnwQznPFCAfZZo0SZYXdnF5oNP/848cWLxiRcGT5ioV8NJLwMiRQOvW8uUUigd6GaJKx1BjAEMNEZmlvDxgzRrggw/kfAmAvFozcaIMOA94aSUzU843FRsLHDqkv0+lAjw85BAfLy/5v4aWevV4+4pMh6HGAIYaIjJrGo0c8TtzJnDsmNzm6Cgn04yIkMniAQgB7Nsnw82mTQWldMrC2rog/JS0eHnJWcjLUWuQqFQMNQYw1BBRtaDVAuvXy3Dz999ym4ODHBQzZYpMFxXg7l0gLQ1ITdVfLl/W//nq1TJPRg6FQmavkkJP/rqHBx87p7JjqDGAoYaIqhUhgJ9/Bt5/H0hMlNvs7IDwcOCNN+SU3lUgNxe4cqXk0JO/pKXJi01lVadOyaGn8PIAD4WRhWCoMYChhoiqJSGAzZuB996T948AOeXCiBGykF+jRqbt3z1arSygXFLoKbxkZ5f9vI6OpY/3yQ9Ezs4c9GypGGoMYKghompNCGDbNnnl5s8/5TZbWzn1QmRktam0JwTw33+lh578YJSZWfbz2tmVHnry1+vW5aDn6oahxgCGGiKyCEIAu3bJcJM/y6W1NTBkiJw8s1kz0/avAt26VXroyV9u3iz7OW1sig96NnQlqH59Dno2Fww1BjDUEJHF+fNPOaB4yxb5s5WVnOHyrbeAVq1M27cqdOeO/qDnkm6BXb1a9nNaWRUMei7t9peHR7nnKKUyYqgxgKGGiCzW3r0y3GzaJH9WKICBA2W4adfOtH0zI7m5ckqJ+437SU83btBz3br3H/Pj6Vnuab5qPIYaAxhqiMjiJSbKcLNhQ8G2/v2Bt98GOnY0WbeqG41GXtW537iftDQ560VZOTmVPt4nf3Fy4qDnwhhqDGCoIaIaIylJVij+/vuCbX36ADNmAI88YrJuWRohZBHDsoz7ycoq+3nt7e//qHv+oOeaEH4YagxgqCGiGufoUTm31Jo1BRX0evWST0v5+8tHw6nSCVF80HNJt8Dy5+kqC1vb0m975YeievXkWPLqiqHGAIYaIqqxTpwAZs2SE2hqtXKblRXQuDHQvLlcmjUrWG/SRH5jUpXLypK3te437ufatbKf08pKTmFxv1tfHh7m+c/OUGMAQw0R1XinT8tws24dcPt2ye2srfUDT+Hg4+Njnt98NUxOjuFpLopeDbpypSDHloWbW9lufdnbV957K4qhxgCGGiKie4SQ34inTsmgc+pUwXL6dOkDQGxsZLApenWneXMZhFjcxaxoNPrTXJR0CywtTT4dVlbOzoZDT+vWwDPPVOx7YKgxgKGGiKgMhJDfckWDTv7/3rlT8rE2NvLWVdGrO/mBpzoP7LBwWm3xQc8l3QIr7VfgySeB+PiK7Zsx39+M1EREVEChkP8J7uUFBAXp79Nq5TedoSs8Z87Iqb/zfy7K1hZo2tTwFR5vbwYeE7Oykree3NyA9u1LbicEkJFRcvAxdVkkXqkhIqIHp9UCly4ZvsJz5kzps1gqlQWBp+gVHm9vTtZUw/H2kwEMNUREJqLRABcvFr+6c+oU8O+/pVewU6mAhx4yfIWnQQMGnhqAocYAhhoiIjOk0QAXLhS/upMfeEobvWpnVxB4il7h8fJi4LEQDDUGMNQQEVUzeXnA+fOGr/CkpMj9JbG3lwGn6NWd5s3lYzo1oRSvhWCoMYChhojIguTlAefOGb7Ck5JS+oyUDg76YafwuocHA4+ZYagxgKGGiKiGyM0Fzp41XIPn7NnSA4+jo+ErPM2aybK8DDxVrtJDTWxsLObMmYO0tDT4+vpiwYIF8Pf3L7H9unXrMGPGDJw9exbNmzfH7Nmz0atXLwBAbm4u3n77bWzatAn//vsvnJ2dERwcjJiYGHh5eenO4ePjg3PnzumdNzo6GtOnTy9TnxlqiIgIOTky2Bi6wnPuXOnld2vXLvkKT716DDyVpFJDzZo1azBkyBAsXrwYAQEBmD9/PtatW4eTJ0+ifv36xdrv2bMHjz/+OKKjo/G///0P3377LWbPno2DBw+iXbt2UKvVeOGFFzBy5Ej4+vriv//+w2uvvQaNRoO//vpLdx4fHx8MHz4cI0eO1G2rXbs2atWqVaZ+M9QQEVGpsrP1A0/h4HPuXMGkoIY4ORkev9OsmSz+wsBTbpUaagICAtC5c2csXLgQAKDVauHt7Y0JEyYYvGoycOBAZGZmYuPGjbptXbp0QceOHbF48WKDr3HgwAH4+/vj3LlzaNSoEQAZaiZNmoRJkyYZ010dhhoiIiq37Gz5NJahwoMXLpQeeJydDV/dad4cqFu36t5DNVVpFYVzcnKQmJiIyMhI3TYrKysEBwcjISHB4DEJCQmIiIjQ2xYSEoIff/yxxNdRq9VQKBRwcXHR2x4TE4OZM2eiUaNGePnllzF58mTYlDDPSHZ2NrILFXvKyMi4z7sjIiIqgUolJzZq3br4vrt3CwJP0dtaFy4AajXw119yKapOnZKv8Li6Vv77sjBGhZpr165Bo9HA3d1db7u7uztOnDhh8Ji0tDSD7dPS0gy2v3v3LqZNm4ZBgwbpJbKJEyfi4YcfhqurK/bs2YPIyEikpqbik08+MXie6OhovPfee8a8PSIiIuPZ2QFt2silqDt3ZEVlQ1d4Ll0C/vsPOHBALkW5upZ8hafIf/STZFZzP+Xm5mLAgAEQQmDRokV6+wpf7enQoQOUSiVGjRqF6OhoqFSqYueKjIzUOyYjIwPe3t6V13kiIqKi7O3lhEiGJkXKyioIPEWv8Fy+LGeY3LdPLkW5uRl+Qqt5c3m7q4YyKtS4ubnB2toa6enpetvT09Ph4eFh8BgPD48ytc8PNOfOncP27dvve98sICAAeXl5OHv2LFq2bFlsv0qlMhh2iIiIzIKDg5w90tAMkpmZMuAYKjyYlgZcuyaXvXuLH1uvnuErPM2ayQHNFsyoUKNUKuHn54f4+Hj069cPgBwoHB8fj/Hjxxs8JjAwEPHx8XoDfLdu3YrAwEDdz/mB5tSpU9ixYwfqlmHgVFJSEqysrAw+cUVERFSt1aoF+PrKpajbt/XDTuH19HTg6lW57NlT/Nj69Q1f3WnWTD6yXs0ZffspIiICoaGheOSRR+Dv74/58+cjMzMTYWFhAIAhQ4agQYMGiI6OBgC89tprCAoKwscff4zevXtj9erV+Ouvv/Dll18CkIHmhRdewMGDB7Fx40ZoNBrdeBtXV1colUokJCRg3759eOKJJ1C7dm0kJCRg8uTJeOWVV1CnTp2K+iyIiIjMn6Mj0LGjXIrKyDB8hef0aeDKlYJl9+7ix3p4GB6/89BD8jWrgXIV31u4cKGu+F7Hjh3x2WefISAgAADQvXt3+Pj4IC4uTtd+3bp1ePvtt3XF9z766CNd8b2zZ8+iSZMmBl9nx44d6N69Ow4ePIixY8fixIkTyM7ORpMmTfDqq68iIiKizLeY+Eg3ERHVaGp1yVd4rl0r/VhPz5Kv8Dg4VGq3OU2CAQw1REREJbh50/ATWqdPA9evl35sgwYFIScgABgxokK7xlBjAEMNERFROdy4UfIVnv/+02/79NPAli0V+vKVVnyPiIiIahhXV8DfXy5FXb+uH3JKGE5SVRhqiIiIqHzq1pXLvXG1pmZl6g4QERERVQSGGiIiIrIIDDVERERkERhqiIiIyCIw1BAREZFFYKghIiIii8BQQ0RERBaBoYaIiIgsAkMNERERWQSGGiIiIrIIDDVERERkERhqiIiIyCIw1BAREZFFqDGzdAshAAAZGRkm7gkRERGVVf73dv73eGlqTKi5desWAMDb29vEPSEiIiJj3bp1C87OzqW2UYiyRB8LoNVqcfnyZdSuXRsKhaJCz52RkQFvb29cuHABTk5OFXpuS8PPquz4WZUdP6uy42dlHH5eZVdZn5UQArdu3YKXlxesrEofNVNjrtRYWVmhYcOGlfoaTk5O/KUvI35WZcfPquz4WZUdPyvj8PMqu8r4rO53hSYfBwoTERGRRWCoISIiIovAUFMBVCoVoqKioFKpTN0Vs8fPquz4WZUdP6uy42dlHH5eZWcOn1WNGShMRERElo1XaoiIiMgiMNQQERGRRWCoISIiIovAUENEREQWgaGmjGJjY+Hj4wM7OzsEBARg//79pbZft24dWrVqBTs7O7Rv3x6bNm2qop6anjGfVVxcHBQKhd5iZ2dXhb01nd9//x3PPvssvLy8oFAo8OOPP973mJ07d+Lhhx+GSqVCs2bNEBcXV+n9NAfGflY7d+4s9nulUCiQlpZWNR02kejoaHTu3Bm1a9dG/fr10a9fP5w8efK+x9XUv1fl+bxq6t+sRYsWoUOHDrrCeoGBgfj1119LPcYUv1cMNWWwZs0aREREICoqCgcPHoSvry9CQkJw5coVg+337NmDQYMGYfjw4Th06BD69euHfv364ejRo1Xc86pn7GcFyOqTqampuuXcuXNV2GPTyczMhK+vL2JjY8vUPiUlBb1798YTTzyBpKQkTJo0CSNGjMBvv/1WyT01PWM/q3wnT57U+92qX79+JfXQPOzatQvjxo3D3r17sXXrVuTm5qJHjx7IzMws8Zia/PeqPJ8XUDP/ZjVs2BAxMTFITEzEX3/9hSeffBJ9+/bFsWPHDLY32e+VoPvy9/cX48aN0/2s0WiEl5eXiI6ONth+wIABonfv3nrbAgICxKhRoyq1n+bA2M9qxYoVwtnZuYp6Z74AiPXr15fa5o033hBt27bV2zZw4EAREhJSiT0zP2X5rHbs2CEAiP/++69K+mSurly5IgCIXbt2ldimJv+9Kqosnxf/ZhWoU6eOWLp0qcF9pvq94pWa+8jJyUFiYiKCg4N126ysrBAcHIyEhASDxyQkJOi1B4CQkJAS21uK8nxWAHD79m00btwY3t7epSb/mq6m/l49iI4dO8LT0xNPP/00du/eberuVDm1Wg0AcHV1LbENf68KlOXzAvg3S6PRYPXq1cjMzERgYKDBNqb6vWKouY9r165Bo9HA3d1db7u7u3uJ9+fT0tKMam8pyvNZtWzZEsuXL8eGDRuwatUqaLVadO3aFRcvXqyKLlcrJf1eZWRk4M6dOybqlXny9PTE4sWL8f333+P777+Ht7c3unfvjoMHD5q6a1VGq9Vi0qRJ6NatG9q1a1diu5r696qosn5eNflv1pEjR+Do6AiVSoXRo0dj/fr1aNOmjcG2pvq9qjGzdJN5CgwM1Ev6Xbt2RevWrfHFF19g5syZJuwZVWctW7ZEy5YtdT937doVZ86cwbx58/D111+bsGdVZ9y4cTh69Cj+/PNPU3elWijr51WT/2a1bNkSSUlJUKvV+O677xAaGopdu3aVGGxMgVdq7sPNzQ3W1tZIT0/X256eng4PDw+Dx3h4eBjV3lKU57MqytbWFp06dcLp06cro4vVWkm/V05OTrC3tzdRr6oPf3//GvN7NX78eGzcuBE7duxAw4YNS21bU/9eFWbM51VUTfqbpVQq0axZM/j5+SE6Ohq+vr749NNPDbY11e8VQ819KJVK+Pn5IT4+XrdNq9UiPj6+xHuJgYGBeu0BYOvWrSW2txTl+ayK0mg0OHLkCDw9PSurm9VWTf29qihJSUkW/3slhMD48eOxfv16bN++HU2aNLnvMTX596o8n1dRNflvllarRXZ2tsF9Jvu9qtRhyBZi9erVQqVSibi4OHH8+HERHh4uXFxcRFpamhBCiFdffVVMnz5d13737t3CxsZGzJ07VyQnJ4uoqChha2srjhw5Yqq3UGWM/azee+898dtvv4kzZ86IxMRE8dJLLwk7Oztx7NgxU72FKnPr1i1x6NAhcejQIQFAfPLJJ+LQoUPi3LlzQgghpk+fLl599VVd+3///Vc4ODiIqVOniuTkZBEbGyusra3F5s2bTfUWqoyxn9W8efPEjz/+KE6dOiWOHDkiXnvtNWFlZSW2bdtmqrdQJcaMGSOcnZ3Fzp07RWpqqm7JysrSteHfqwLl+bxq6t+s6dOni127domUlBRx+PBhMX36dKFQKMSWLVuEEObze8VQU0YLFiwQjRo1EkqlUvj7+4u9e/fq9gUFBYnQ0FC99mvXrhUtWrQQSqVStG3bVvzyyy9V3GPTMeazmjRpkq6tu7u76NWrlzh48KAJel318h87Lrrkfz6hoaEiKCio2DEdO3YUSqVSNG3aVKxYsaLK+20Kxn5Ws2fPFg899JCws7MTrq6uonv37mL79u2m6XwVMvQZAdD7PeHfqwLl+bxq6t+sYcOGicaNGwulUinq1asnnnrqKV2gEcJ8fq8UQghRudeCiIiIiCofx9QQERGRRWCoISIiIovAUENEREQWgaGGiIiILAJDDREREVkEhhoiIiKyCAw1REREZBEYaoiIiMgiMNQQERGRRWCoISIiIovAUENEREQWgaGGiIiILML/A7tZvY0da4XYAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "plt.plot(np.arange(len(train_loss)), train_loss, c=\"r\", label=\"Loss/Train\")\n", "plt.plot(np.arange(len(val_loss)), val_loss, c=\"b\", label=\"Loss/Val\")\n", "plt.legend()" ] }, { "cell_type": "markdown", "id": "4701538e-a016-4c78-8bd4-1cd063989210", "metadata": {}, "source": "### Evaluation\n\nWe're going to measure the accuracy of the models. However, the outputs are padded, so we have to ignore the padded indexes, regardless of the predictions. " }, { "cell_type": "code", "execution_count": 18, "id": "4670c616-a9ac-4753-96d0-41d745da1205", "metadata": {}, "outputs": [], "source": [ "def categorical_accuracy(preds, actual, pad_idx=1):\n", " non_padding_indices = jnp.nonzero((actual != pad_idx))\n", "\n", " matches = jnp.equal(preds[non_padding_indices], actual[non_padding_indices])\n", " acc = jnp.sum(matches) / actual[non_padding_indices].shape[0]\n", "\n", " return acc" ] }, { "cell_type": "code", "execution_count": 19, "id": "4efd3148-d628-45e1-9f94-45150c8196d5", "metadata": {}, "outputs": [], "source": [ "import jax.nn as jnn\n", "\n", "@jit\n", "def infer(params, words, labels):\n", " logits = model.apply(params, words, rngs={\"dropout\": dropout_key})\n", " proba = jnn.log_softmax(logits, axis=-1)\n", " preds = jnp.argmax(proba, axis=-1)\n", "\n", " return preds\n", "\n", "@jit\n", "def batch_infer(params, words, labels):\n", " preds = vmap(infer, in_axes=(None, 0, 0))(params, words, labels)\n", " return preds" ] }, { "cell_type": "code", "execution_count": 20, "id": "92c5c5c5-565d-44d5-8ae2-6914ae8fc5a6", "metadata": {}, "outputs": [], "source": [ "def evaluate(params, test_loader):\n", " acc_per_batch = list()\n", " for batch in tqdm(test_loader):\n", " words, labels = batch\n", " preds = batch_infer(params, words, labels)\n", "\n", " acc = categorical_accuracy(preds, labels)\n", " acc_per_batch.append(acc)\n", "\n", " mean_acc = jnp.mean(jnp.array(acc_per_batch), axis=-1)\n", "\n", " return mean_acc" ] }, { "cell_type": "code", "execution_count": 21, "id": "87d3323a-9bca-44ab-83f3-225e35d87c67", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "afd052e612284e1daa3bcdfcf9d5419b", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/12 [00:00