{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "060dcdeb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: graphviz in /home/gerald/miniforge3/lib/python3.12/site-packages (0.20.3)\n"
]
}
],
"source": [
"!pip install graphviz"
]
},
{
"cell_type": "markdown",
"id": "5f7a3b51",
"metadata": {},
"source": [
"# Backpropagation and computation graph\n",
"\n",
"The objective of the current exercise is to introduce how works a deep-learning library. To this end we will devellop in this exercise a python library that build a computational using only scalar operations (consequently it is not yet necessary to use numpy library).\n",
"\n",
"The main object we will manipulate will be the ComputationalGraphNode, this object (or class in python) will have different methods and attributes (discuted during the lecture session). \n",
"We will consider two type of nodes, a classical nodes and parameters node (that will sum the gradient at each backward step).\n",
"\n",
"\n",
"**Below a code for printing the graphs ! Do not modify the code** \n"
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "46ec199a",
"metadata": {},
"outputs": [],
"source": [
"import graphviz\n",
"name_operation_list = {'Addition': '+', 'Multiplication': '*', 'Substraction': '-'}\n",
"def createbackwardgraph(root, parent_index=None, graph=None, nodelist={}, edge_list=set()):\n",
" if graph is None:\n",
" graph = graphviz.Digraph(filename = \"digraph.gv\")\n",
" \n",
" if root not in nodelist:\n",
" nodelist[root] = str(len(nodelist))\n",
"\n",
" node_name = root.name if 'name' in root.__dict__ else \"unknow\"\n",
" if root.gradient_function is not None:\n",
" node_name = root.gradient_function.__qualname__.split(\".\")[0] + \" : \" + node_name\n",
" node_name += f'\\n v = {root.value} \\n g = {root.cumulative_gradient}' \n",
"\n",
" parent_index = nodelist[root]\n",
" graph.node(parent_index, node_name)\n",
"\n",
"\n",
" if root.previous_node_list is None:\n",
" return\n",
"\n",
" for node in root.previous_node_list:\n",
"\n",
" if node not in nodelist:\n",
" nodelist[node] = str(len(nodelist))\n",
"\n",
" node_name = node.name if 'name' in node.__dict__ else \"unknow\"\n",
" if node.gradient_function is not None:\n",
" node_name = node.gradient_function.__qualname__.split(\".\")[0] + \" : \" + node_name\n",
" node_name += f'\\n v = {node.value} \\n g = {node.cumulative_gradient}' \n",
"\n",
" node_index = nodelist[node]\n",
" graph.node(node_index, node_name)\n",
" if not (nodelist[node], nodelist[root]) in edge_list:\n",
" edge_list.add((nodelist[node], nodelist[root]))\n",
" graph.edge(nodelist[node], nodelist[root])\n",
" elif root.previous_node_list[0] == root.previous_node_list[-1]: \n",
" graph.edge(nodelist[node], nodelist[root])\n",
" createbackwardgraph(node, None, graph=graph, nodelist=nodelist, edge_list=edge_list)\n",
" return graph\n"
]
},
{
"cell_type": "markdown",
"id": "fda64f7e-9d88-4498-9dc6-4540ecf679d0",
"metadata": {},
"source": [
"## Implement the computational graph node\n",
"\n",
"A node of the computation graph needs to store different information :\n",
"* A backward operation (computing gradient of the model according to its input)\n",
"* The value of the node (a float)\n",
"* The gradient of the node (updated/accumulated when backward is called)\n",
"* The list of previous nodes (input that produced the current node)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "90480840",
"metadata": {},
"outputs": [],
"source": [
"from typing import List, Tuple\n",
"\n",
"class ComputationGraphNode(object):\n",
" def __init__(self, value : float, gradient_function = None , previous_node_list: List = None, name=None):\n",
" if name is not None:\n",
" self.name = name\n",
" self.value = value\n",
" self.cumulative_gradient = .0\n",
" self.gradient_function = gradient_function\n",
" self.previous_node_list = previous_node_list\n",
"\n",
" def __str__(self) -> str:\n",
" return f'ComputationGraphNode with value = {self.value}'\n",
" def __repr__(self) -> str :\n",
" return f'ComputationGraphNode with value = {self.value}'\n",
"\n",
" def backward(self, gradient : float = 1.) -> float:\n",
" self.cumulative_gradient += gradient\n",
"\n",
" # call backward with on the previous node (with correct gradient)\n",
" \n",
"\n",
"class Parameter(ComputationGraphNode):\n",
" def backward(self, gradient=1) -> float:\n",
" self.cumulative_gradient += gradient"
]
},
{
"cell_type": "markdown",
"id": "bfac8fb7",
"metadata": {},
"source": [
"## implement add, mul, sub and relu operations\n",
"\n",
"For each operations we should define a forward function and a gradient_function function. To ease the implementation we will consider an operation as an object that:\n",
"* In the method forward takes previous nodes and compute the net node\n",
"* In backward methods takes previous input nodes and compute the gradient for each input node\n",
"\n",
"\n",
"Implement the methods of the following operations :"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb74f457",
"metadata": {},
"outputs": [],
"source": [
"class Operation(object):\n",
" @staticmethod\n",
" def forward(*args, **kwargs) -> Tuple[float]:\n",
" raise NotImplementedError(\"This is an abstract class, you should inherit from the class\")\n",
" @staticmethod\n",
" def gradient_function(*args, **kwargs) -> ComputationGraphNode:\n",
" raise NotImplementedError(\"This is an abstract class, you should inherit from the class\")\n",
"\n",
"class Addition(object):\n",
" @staticmethod\n",
" def gradient_function(node_a: ComputationGraphNode, node_b: ComputationGraphNode, output_grad: float):\n",
" return output_grad, output_grad\n",
" \n",
" @staticmethod\n",
" def forward(node_a: ComputationGraphNode, node_b: ComputationGraphNode):\n",
" return ComputationGraphNode(node_a.value + node_b.value, Addition.gradient_function, [node_a, node_b])\n",
"\n",
"class Substraction(object):\n",
" @staticmethod\n",
" def gradient_function(node_a: ComputationGraphNode, node_b: ComputationGraphNode, output_grad: float):\n",
" raise NotImplementedError(\"Implement the function\")\n",
" @staticmethod\n",
" def forward(node_a: ComputationGraphNode, node_b: ComputationGraphNode):\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
"\n",
"class Multiplication(object):\n",
" @staticmethod\n",
" def gradient_function(node_a: ComputationGraphNode, node_b: ComputationGraphNode, output_grad: float):\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
" @staticmethod\n",
" def forward(node_a: ComputationGraphNode, node_b: ComputationGraphNode):\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
"class ReLU(object):\n",
" @staticmethod\n",
" def gradient_function(node_a: ComputationGraphNode, output_grad: float):\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
" @staticmethod\n",
" def forward(node_a: ComputationGraphNode):\n",
" raise NotImplementedError(\"Implement the function\")\n"
]
},
{
"cell_type": "markdown",
"id": "5887584b",
"metadata": {},
"source": [
"### Overloading the operators (not necessary, but easier to use)"
]
},
{
"cell_type": "code",
"execution_count": 120,
"id": "30bdb1d1",
"metadata": {},
"outputs": [],
"source": [
"ComputationGraphNode.__add__ = Addition.forward\n",
"ComputationGraphNode.__sub__ = Substraction.forward\n",
"ComputationGraphNode.__mul__ = Multiplication.forward"
]
},
{
"cell_type": "markdown",
"id": "f67b7bb3",
"metadata": {},
"source": [
"**Code/propose different examples and verify that when calling backward on the last node you obtain the correct gradient !!**\n",
"\n",
"\n",
"NB : You can use createbackwardgraph(loss) to show the computation graph and values"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d0da9c2",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x1 = ComputationGraphNode(.1, name='x1')\n",
"x2 = .....\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "ea01de50",
"metadata": {},
"source": [
"## Create a module that apply linear transformation\n",
"\n",
"We consider now a new class namely Module, all the module you will implement will inherit from this class. Notice also that module may have parameters (node of type `Parameter`) that will be returned calling `my_module.parameter`\n"
]
},
{
"cell_type": "code",
"execution_count": 124,
"id": "8927639c",
"metadata": {},
"outputs": [],
"source": [
"class Module:\n",
" def __init__(self):\n",
" raise NotImplemented(\"\")\n",
" \n",
" def parameters(self):\n",
" ret = []\n",
" for name in dir(self):\n",
" o = self.__getattribute__(name)\n",
"\n",
" if type(o) is Parameter:\n",
" ret.append(o)\n",
" if isinstance(o, Module) or isinstance(o, ModuleList):\n",
" ret.extend(o.parameters())\n",
" return ret\n",
"\n",
"class ModuleList(list):\n",
" def parameters(self):\n",
" ret = []\n",
" for m in self:\n",
" if type(m) is Parameter:\n",
" ret.append(m)\n",
" elif isinstance(m, Module) or isinstance(m, ModuleList):\n",
" ret.extend(m.parameters())\n",
" return ret"
]
},
{
"cell_type": "markdown",
"id": "20680304",
"metadata": {},
"source": [
"Implement the two modules :\n",
"\n",
"* Linear which apply on a list of node a linear transformation with bias $Wx^t + b$\n",
"* The relu function that compute from a list the relu for each element"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbe2479c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[ComputationGraphNode with value = 0.6931947456281193,\n",
" ComputationGraphNode with value = -0.05401745103124978,\n",
" ComputationGraphNode with value = -0.18556659917864893]"
]
},
"execution_count": 125,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import random\n",
"class Linear(Module):\n",
" def __init__(self, input_size, output_size):\n",
" self.weights = ModuleList([ModuleList([Parameter(random.random() - 0.5) for i in range(input_size)]) for j in range(output_size)])\n",
" self.bias = ... # initialise all bias at 0.01\n",
"\n",
" def __call__(self, node_list : List[ComputationGraphNode]) -> List[ComputationGraphNode]:\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
"def relu(x_list : List[ComputationGraphNode]) -> List[ComputationGraphNode]:\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
"l = Linear(2, 3)\n",
"l([Parameter(0), Parameter(1)])"
]
},
{
"cell_type": "code",
"execution_count": 126,
"id": "f96d96b6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[ComputationGraphNode with value = 0.01,\n",
" ComputationGraphNode with value = 0.01,\n",
" ComputationGraphNode with value = 0.01,\n",
" ComputationGraphNode with value = 0.3709708747268021,\n",
" ComputationGraphNode with value = 0.3122238709013172,\n",
" ComputationGraphNode with value = -0.32452137256185565,\n",
" ComputationGraphNode with value = 0.26050392153060586,\n",
" ComputationGraphNode with value = -0.04208783788285286,\n",
" ComputationGraphNode with value = -0.15347876129579607]"
]
},
"execution_count": 126,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"l.parameters()"
]
},
{
"cell_type": "markdown",
"id": "8064b262",
"metadata": {},
"source": [
"## Create your network similarly\n",
"Create a two layer networks with as activation a sigmoid function :\n",
"* input_size being the size of the input X\n",
"* intermediate_size the hidden layer size\n",
"* output_size the size of the outpu (for binary classification it is 1)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "502968eb",
"metadata": {},
"outputs": [],
"source": [
"class MyTwoLayerNetwork(Module):\n",
" def __init__(self, input_size, intermediate_size, output_size = 1):\n",
" ### Initialise submodule necessary for computation\n",
" raise NotImplementedError(\"Implement the function\")\n",
"\n",
" \n",
" def __call__(self, node_list : List[ComputationGraphNode]) -> List[ComputationGraphNode]:\n",
" ### Apply the neural network on the input\n",
" raise NotImplementedError(\"Implement the function\")\n"
]
},
{
"cell_type": "markdown",
"id": "079e801f",
"metadata": {},
"source": [
"### Implement the following functions :\n",
"* hing_loss computing the hinge loss from node\n",
"* update_weight that will update all the weights given in parameters $\\theta^t = \\theta^{t-1} - \\lambda \\nabla_{\\theta^{t-1}} L$\n",
"* zero_grad that set all accumulated gradient of the nodes to 0"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acf58c9c",
"metadata": {},
"outputs": [],
"source": [
"## Create your loss function\n",
"\n",
"def hinge_loss(x : ComputationGraphNode, y: ComputationGraphNode):\n",
" return ReLU.forward(ComputationGraphNode(1) - x * y)\n",
"\n",
"def update_weight(parameter_list : List[ComputationGraphNode], learning_rate: float):\n",
" \n",
"def zero_grad(parameter_list):\n"
]
},
{
"cell_type": "markdown",
"id": "41ef4db0",
"metadata": {},
"source": [
"## Build a network that can solve the following problem\n",
"\n",
"Train your network on the following dataset and provide accuracy of the model on the training data (only for this example else consider separated trianing and testing set)"
]
},
{
"cell_type": "code",
"execution_count": 129,
"id": "1fde92c1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 129,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGdCAYAAADwjmIIAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAX7xJREFUeJzt3X18E3W2P/BPWmhLgbRAoWmxy/OiXSpFsKWoqwvVVlDkiruA7vKwiD9RVMAHYO8Ci3gFhOuyKorXBcGriMj6gKhdEUVFC7hAFQS50q2A2BRppYEWCiTz+yNMaNpMMjOZSWYmn/fr1VdtOklmGktOz/ec87UJgiCAiIiIyGLion0CRERERHpgkENERESWxCCHiIiILIlBDhEREVkSgxwiIiKyJAY5REREZEkMcoiIiMiSGOQQERGRJbWI9glEg8fjwY8//oi2bdvCZrNF+3SIiIhIBkEQcPLkSWRmZiIuLnSeJiaDnB9//BFZWVnRPg0iIiJS4ciRI7jkkktCHheTQU7btm0BeH9Idrs9ymdDREREcrhcLmRlZfnex0OJySBHXKKy2+0McoiIiExGbqkJC4+JiIjIkhjkEBERkSUxyCEiIiJLYpBDRERElsQgh4iIiCyJQQ4RERFZEoMcIiIisiQGOURERGRJMTkMkEzK4wYOfQGcqgLapANdBgFx8dE+KyIiMigGOWQO+zYAJTMA148Xb7NnAsWLgOzh0TsvIiIyLC5XkfHt2wCsG+sf4ACAq9J7+74N0TkvIiIyNAY5ZGwetzeDAyHANy/cVjLTexwREVEjDHLI2A590TyD40cAXEe9xxERETXCIIeM7VSVtscREVHMYJBDxtYmXdvjiIgoZjDIIWPrMsjbRQWbxAE2wN7ZexwREVEjDHLI2OLivW3iAJoHOhe+Ll7IeTlERNSMrkHOp59+iptvvhmZmZmw2Wx46623Qt5ny5YtuOKKK5CYmIiePXti1apVzY5ZtmwZunbtiqSkJOTn52PHjh3anzwZR/Zw4HcvAfYM/9vtmd7b5c7J8biBis+APeu9n9mRRURkaboOA6yrq0Pfvn3xxz/+EbfeemvI4ysqKjBs2DDcfffdeOWVV7B582bceeedyMjIQFFREQDgtddew/Tp07F8+XLk5+dj6dKlKCoqwoEDB9CpUyc9L4eiKXs4cOkw9ROPOUyQiCjm2ARBCDSARPsnstnw5ptvYsSIEZLHzJgxA++++y727t3ru2306NE4ceIESkpKAAD5+fm48sor8cwzzwAAPB4PsrKycN9992HmzJmyzsXlciElJQW1tbWw2+3qL4rMQRwm2GzWzoXlLiXZICIiihql79+GqskpLS1FYWGh321FRUUoLS0FAJw9exY7d+70OyYuLg6FhYW+YwJpaGiAy+Xy+6AYwWGCREQxy1BBjtPpRHq6fytweno6XC4XTp8+jePHj8Ptdgc8xul0Sj7uggULkJKS4vvIysrS5fzJgDhMkIgoZhkqyNHLrFmzUFtb6/s4cuRItE+JIoXDBImIYpahdiF3OByoqvJ/s6mqqoLdbkerVq0QHx+P+Pj4gMc4HA7Jx01MTERiYqIu50wGx2GCREQxy1CZnIKCAmzevNnvtk2bNqGgoAAAkJCQgP79+/sd4/F4sHnzZt8xRH44TJCIKGbpGuScOnUKZWVlKCsrA+BtES8rK8Phw4cBeJeRxo4d6zv+7rvvxr///W888sgj+Pbbb/Hss89i3bp1mDZtmu+Y6dOn44UXXsDq1auxf/9+TJ48GXV1dZgwYYKel0JmxWGCREQxS9flqn/961/4zW9+4/t6+vTpAIBx48Zh1apVqKys9AU8ANCtWze8++67mDZtGv72t7/hkksuwd///nffjBwAGDVqFH766SfMmTMHTqcTubm5KCkpaVaMTOQjDhMMOCdnIdvHiYgsKmJzcoyEc3JilMetfpigkZ6DiChGKX3/NlThMZGu4uKBbtfo9/icqkxEZCiGKjwmMi1xqnLTmTyuSu/t+zZE57yIiGIYgxyicHGqMhGRITHIIQoXpyoTERkSgxyicHGqMhGRITHIIQoXpyoTERkSgxyicHGqMhGRITHIIQoXpyoTERkSgxwiLYhTle0Z/rfbM723c04OEVHEcRggkVayhwOXDpM/8ZjTkYmIdMUgh0hLcqcqczoyEZHuuFxFFGmcjkwm4/YIKC2vxttlR1FaXg23J+a2PCSTYiaHKJJCTke2eacjXzqMS1dkCCV7KzHvnX2orD3juy0jJQlzb85GcZ+MIPckij5mcogiidORyURK9lZi8su7/AIcAHDWnsHkl3ehZG9llM6MSB4GOUSRxOnIZBJuj4B57+wLtiMb5r2zj0tXZGgMcogiidORySR2VNQ0y+A0JgCorD2DHRU1kTspIoUY5BBFEqcjk0kcOykd4Kg5jigaGOQQRRKnI5NJdGqbpOlxRNHAIIco0jgdmUwgr1t7ZKQkBcs5IiMlCXnd2kfytIgUYQs5UTQonY5MFGHxcTbMvTkbk1/eBRv8hx6Igc/cm7MRHycVBhFFHzM5RNEiTkfOuc37mQEOGUxxnww89/sr4EjxX5JypCThud9fwTk5ZHjM5JAy3G+JKKYU98nA9dkO7KiowbGTZ9CprXeJSq8MjtsjROy5yPoY5JB83G+JKCbFx9lQ0KOD7s+j9XRltQETAy3rsAmCEHOTnFwuF1JSUlBbWwu73R7t0zEHcb+lZqPBLvzis2CWiMIgTleW+BdG8fKY2oCJ21gYm9L3b9bkUGgh91uCd78ljzuSZyWfxw1UfAbsWe/9bNTzJIpRWk9XVrsdBbexsB4GORSamfdb2rcBWNoHWH0T8I+J3s9L+3CnbyID0XK6stqAidtYWBODHArNrPstiUtsTQM0V6X3dgY6RIag5XRltQETt7GwJgY5FJoZ91sy+xIbUQzRcrqy2oCJ21hYE4McCs2M+y2ZeYmNKMZoOV1ZbcCk5H5uj4DS8mq8XXYUpeXVXMIyMLaQU2jifkvrxgJSs0+Ntt+SWZfYiGKQltOVxYDJWXsmYB7XBu8ww6YBk9z7/Vx3Flcv+ojdVybBTA7JY7b9ltQssbELiyhqtJquLAZMgOQWuAEDJjn3G943A/euYfeVmXBODufkKGOWiccet7eLylWJwHU5Nm+ANnWP9/w56JDIELQaxKf1nJzZwy7D/Hf3SxYni5merTMGc3CgjpS+f0ckyFm2bBkWL14Mp9OJvn374umnn0ZeXl7AY6+77jp88sknzW4fOnQo3n33XQDA+PHjsXr1ar/vFxUVoaSkRNb5MMiJEb4BhkDABLiYgeKgQyJL0nLi8Y6KGox5YVvI+746aWBEpkPHKqXv37rX5Lz22muYPn06li9fjvz8fCxduhRFRUU4cOAAOnXq1Oz4N954A2fPnvV9XV1djb59++K3v/2t33HFxcV48cUXfV8nJibqdxFkTuISW8AMzULv90N2Ydm8XViXDjNmxoqIJKndjiLQ/dh9ZU66BzlPPvkkJk2ahAkTJgAAli9fjnfffRcrV67EzJkzmx3fvr1/MdjatWuRnJzcLMhJTEyEw+HQ78TJGrKHewMUqSU2JV1Y3a6JyCkTkfFo2eZOkaNr4fHZs2exc+dOFBYWXnzCuDgUFhaitLRU1mOsWLECo0ePRuvWrf1u37JlCzp16oTevXtj8uTJqK6u1vTcyULi4r0BSs5t3s+NMzLswiIiGbRsc6fI0TXIOX78ONxuN9LT/Ttd0tPT4XQ6Q95/x44d2Lt3L+68806/24uLi/HSSy9h8+bNWLRoET755BPceOONcLsDd8M0NDTA5XL5fRABMOegQyKKOLVdWxRdhm4hX7FiBXJycpoVKY8ePRrDhw9HTk4ORowYgY0bN+LLL7/Eli1bAj7OggULkJKS4vvIysqKwNmTKZhx0CERRYVWbe4UObrW5KSlpSE+Ph5VVf6p/qqqqpD1NHV1dVi7di0effTRkM/TvXt3pKWl4eDBgxgyZEiz78+aNQvTp0/3fe1yuRjokJcZBx0SGYhWLd9mUdwnA9dnO2Lqms1M1yAnISEB/fv3x+bNmzFixAgAgMfjwebNmzFlypSg93399dfR0NCA3//+9yGf54cffkB1dTUyMgJH0YmJiey+ImlyurCIqBm1s2jMTm3XFkWe7nNyXnvtNYwbNw7PP/888vLysHTpUqxbtw7ffvst0tPTMXbsWHTu3BkLFizwu98111yDzp07Y+3atX63nzp1CvPmzcPIkSPhcDhQXl6ORx55BCdPnsSePXtkBTOck0MBmWXQIZEBlOytxOSXd0lNl1K8fBNrGSFSx3BzckaNGoWffvoJc+bMgdPpRG5uLkpKSnzFyIcPH0ZcnH9p0IEDB7B161Z88MEHzR4vPj4eX3/9NVavXo0TJ04gMzMTN9xwA+bPn89sDYVH7MIioqDcHgHz3tkXbLoU5r2zD9dnO2QFKnIyQkqDIAZNBHBbB2ZyiIgUKi2v1mz6r5yMEABFy2KxuowWC5S+fxu6u4qIiIxHq+m/oTJCADDzjT2Y/LL8TTHFoMkIm2i6PQJKy6vxdtlRlJZXw+2JuZxC1Om+XEVERNai1fTfHRU1khteAt5A50T9OcnvNV0W03oZLRzMJhkDMzlkHB43UPEZsGe997Mn8HBHIoourab/hrvPkwCgsvYMdlTUAJAXNDU+Xi9GyibFOmZyyBj2bZBo4V4U+RZudlkZDotIjUWc/jv55V1S06VkTf/Vap8nMVgywiaaRsomEYMcMoJ9Gy4M42vyz4Kr0nv7716KXKBjpGCLADDtb1Ti9N+mr41DwWsjZoSctWcCBgVyicGSETbRVJJN4qwd/THIodD0zGx43N6gItjfPSUzvTuJ651NMVKwRQCkO2/EtD9H6UdXuNN/Q2WEBACpyS1RW38u4L8QNniDKnFZLFTQ1PR4PUQjm8RMpzQGORSc3pmNQ1/4P3YzAuA66j1Ozxk2HjfwzgMwRLAll8WX1Zj2N4dwp/+GyggBkL0sptUyWjginU1ipjM4BjkkLRKZjVNVoY9pepweb+6fLgFOBytGjFCwJVcMLKsx7R87QmWElCyLabGMFo5IZpOY6QyNQQ4FFqllpDbpyo7T483d4wa2PyvvWLlBmZ5iZFnNCEWkFDnBMkJKl8WiuYlmpLJJzHTKwxbyWBasZVvJMlI4ugzyBinBmlHtnb3HiW/uTc9LfHPft0HdORz6Ajh9Qt6xcoMyvYQMPuENPi3Qfm+EIlIyDjEIuiW3Mwp6dAj5xq30eC2J2SRHiv//m46UJM2yK0Zplzc6ZnJiVaiMiJplJDXi4r3PuW4sIPV3T/FC72e9Mktyr6FVe2+wFU1GqWGKACMUkRKppXc2iZlOeZjJiUVyMiJKl5HCkT3cu8Rib/LXjT3z4tKLnpkludeQf3f0C3sjFXwagJj2B5rn+SJVRBrLuCVB+PTMJjHTKQ8zObFGbq3N/WXeIMNVKXGszft9rTIb2cO9WRipgmI939zFJTPJa4U3i/Prh5Q/ttYiGXwaQLSLSGMVO3aMj5lOeRjkxBq5GZEj2+UtI2mZ2YiLl15i0fPNPeiS2QU3/y36WRxARkCmcfBpANEsIo1FSjp2OJ8leozQLm8GXK6KNUoyInKWkSJFSYGyGpLX2hn43f8ap1tJDMgASC7iaB18GkA0i0hjiZxdwee9sw9nz3vwtw//D/3nb8KYF7bhgbVlGPPCNly96CPuyxRBkShwNjubIAgxt9DqcrmQkpKC2tpa2O32aJ9OZFV8Bqy+KfRxRY97syJt0oGsfG9mJ9pD53yt00DAv1u0CLzMMmAvYOF4Z2+AY5SAjEyntLwaY17YFvK4NoktcKrhfLPbxdCTb7CRFUsZNaXv3wxyYi3I8biBpX2C15/Y4gDBc/FrIw2Z45v7RUYJyIxyHhS2t8uO4oG1ZWE9hlgLsnXGYMu+0VL0KH3/Zk1OrJFTf9I4wAG8AcW6Pxhj2SZUgXIsCVbDFCkxMHk5lmjRicNJ1GQkrMmJRVL1J5L1Lhe884AxhsyJb+45t3kDnENfBB5oaGbBBjUahV7DGSlqxI4dLfIvsT6fhYyBmZxY1TQjctIJfPCfwe9zugb4fivQ/drInGMoWmQRjLjUYobsiJF2j49hWtdiBOvYUSrW57OQMTDIiWWNlzs2PybvPhWfGSPIkdy/6cLS2sB7gN5DgwctUsHEFeOBDj2iE/SYZV+qGJq8bFR6zbKRmk3UvnVL1NSdk/UYGRLzWWKpQJaMgUEOecn9d8YI/x4FzSJcsO1Z74dUBiRYkLTl8YtfRzKDYqbsSAxNXjaiULNslt1+Bdq1TlAdTASaTdS/Sztcu/hjyeFzjQWaz2L1AYMM4IyJQQ55dbkawGKZx0VZyCxCI4EyIHKCpGD3b0qrJS8zZUdibPKykciZZTPl1V1ovAuDmmAi0K7goZayUpNbYuGtOc2eR8mAQTOyegBnZiw8Jq9u1wCt2gU/plX76L+5AsCB9xQcHGBnbiVBUqidvfdt8Lbkr74J+MdE7+elfdQV3ZopO6L3cEaSFGr3aQBous2UGEyEO6hPavhcanJLTCv8JXb++fpmb+pyBwxqtTdWpPfcEgO4pq+JVj9zCg8zOeQVFw/c/JS3nkWK0q0N9Cjq9biBr9cpvFOTDIjiIEEig6J1/YyZsiNyd4+P9rKaBanpWrqw2Il57+zD9dmOsJZRlG6zESoo07LlPNIZlVABnFY/c1KPmRy6KHu4dxZO2yb/GLTNVD4jR8sMR2OHvgDqj6u7rxjcqA0SGgdHIetnIJ39kWK27IiRtv2wIKmMhNqupcbBRLiUbLMhNygLt+U8GhkVJQEcRQczOWYSiXZnLYbt6dkhFM5SjRjcyNl1PNj9AX3qZ8yYHRH/f/l+q7fzzgZv3ZYRljVNLFhG4vpsR9Ddp0OJ9PwauUFZOC3nemdUpIqKIxXAkXoMcswikrNT5EzSlQq49O4QUpWFabIzt5ypz8HuD+hXPyNmRwK+1gbduuLbd5uc72LjzfUxETlFuuHMson0/BpxwKBUUCZuAxGo5VwuPZfEggWckQjgKDxcrjIDo02WDbYUpSTDoUbIJZ2mJDIgklOfZd5fz/qZ7OHA1L3AuI3AyBXez1P3GDNgMNr/myYnt0j3+mxHwALgYEkKG6Tn1yg9RyWFveKAQfEcmp4TELjlXAm9MiqhlsB+rmsIOiFaq585qcdMjtEZbXZKqKWogZPlPY7aZSelWZhgGZCmS3PV5cDOF4GTlaHvH3LJK0D2Rwkj7EsVitH+37QAJRmJQAXAP9c14N41u33HirQKJtQW9koNGHRoVBSsR0ZFzhLY/Hf3Y/awbNy7pnlWTaufOYWHQY7Ryc2MRGK7BTlvanI7n8LpEJJc0ukM3PA40LqD/HqipsHErx+SV49kxvoZrZlpro9JKM1IBJpl81ycTZdgItxZN0q7spTQY0lMbsDZrnWCrgEchYdBjtHJzXi8Pha4+Wl9lzTkvKnVHweSOwD1NdAlwyHSazdyJRkUM9bPaMlMc31MQouMhB7BhFaFvYGCMi0E23NLbUZFScB5S25n3QI4Cg+DHKOTm/E4fUL/vY3kvlldPgrY9hx0z3AYYUlHr2DLDMw018cktMpIaB1MRHLWjVpaL4kpDTj1CuAoPBEpPF62bBm6du2KpKQk5OfnY8eOHZLHrlq1Cjabze8jKcn/fzZBEDBnzhxkZGSgVatWKCwsxHfffaf3ZWjD4/a22u5Z7/0cao6K0kJbpbNZlJD7ZtV7aGzNTxGDrZzbvJ9jIcABzDfXxwQiUaSrhllapYv7ZGDrjMF4ddJA/G10Ll6dNBBbZwxWtWQkBpwsKjY33YOc1157DdOnT8fcuXOxa9cu9O3bF0VFRTh27Jjkfex2OyorK30fhw4d8vv+E088gaeeegrLly/H9u3b0bp1axQVFeHMGYPPIlAzIE+s/ZAlzM6lUJS8qZmpQ4jU8ft/U+It2ep1STqQ2jrBkZIUtT2ezNQqrWRQYajHMWLAScrYBEHQdWOP/Px8XHnllXjmmWcAAB6PB1lZWbjvvvswc+bMZsevWrUKU6dOxYkTJwI+niAIyMzMxIMPPoiHHnoIAFBbW4v09HSsWrUKo0ePDnlOLpcLKSkpqK2thd1uV39xSkh1JYm/LqEyHPs2AO/cD5z+OfRzjVzhzSrowXcdQMClKLNnapQMXIzEcEYzCDjDqXNs1CXpyEi7Wrs9Aq5e9FHIZbStMwZb7k2fm28ai9L3b11rcs6ePYudO3di1qxZvtvi4uJQWFiI0tJSyfudOnUKXbp0gcfjwRVXXIHHH38cv/rVrwAAFRUVcDqdKCws9B2fkpKC/Px8lJaWBgxyGhoa0NDQ4Pva5XJpcXnyadFqmz0cSLQD/3tL6OfTswbCysW2SgYuRnI4o9HFcl2SjoxU46FHYa9Z6NkVRvrTNcg5fvw43G430tP933TT09Px7bffBrxP7969sXLlSlx++eWora3FkiVLMGjQIHzzzTe45JJL4HQ6fY/R9DHF7zW1YMECzJs3T4MrUkmrVttu1+g7m0UuK76pKdmKQs9tK8zKCEXgpCu9Z91EmpJMmZECTlLGcN1VBQUFKCgo8H09aNAgXHbZZXj++ecxf/58VY85a9YsTJ8+3fe1y+VCVlZW2Ocqm1attkaazWKlNzUlmTaAA/AoZjXNaqS1TgRswPFTDSgtrzZNhoNLULFD1yAnLS0N8fHxqKryf/OuqqqCw+GQ9RgtW7ZEv379cPDgQQDw3a+qqgoZGRf/Z6yqqkJubm7Ax0hMTERiYqKKK9CIlq22Vl4uihalW1FwAB7FMDGrUbK3Eg+t/8p0gUK4Qw3JXHTtrkpISED//v2xefNm320ejwebN2/2y9YE43a7sWfPHl9A061bNzgcDr/HdLlc2L59u+zHjDitW23ZuaQtJZk2DsAjCrmnU8neyoD3U7LvldI9suSQuzeYFs+lFz1+Llam+3LV9OnTMW7cOAwYMAB5eXlYunQp6urqMGHCBADA2LFj0blzZyxYsAAA8Oijj2LgwIHo2bMnTpw4gcWLF+PQoUO48847AQA2mw1Tp07FY489hl69eqFbt26YPXs2MjMzMWLECL0vRx09lpmstFwUbXoMteMAPLIotdOPlSwR6bWcZIahhsFwmU053YOcUaNG4aeffsKcOXPgdDqRm5uLkpISX+Hw4cOHERd3MaH0888/Y9KkSXA6nWjXrh369++PL774AtnZ2b5jHnnkEdTV1eGuu+7CiRMncPXVV6OkpKTZ0EBD4TKTcSndbNMIxd9EEdS4SPf4yQbFgUKwJaK7X96FaYW90DWt9YVNRs/i3jXSx/7xqq64Ptuhqv7HLEMNA+Eymzq6z8kxoqjMyRFZebaKma9Nyfwfq88Kopgg1V3U9Paf6xow/939QQObQP42Ohe35Hb2zdiRe/84GyBnBUZNBqO0vBpjXtgW8rhXJw00VCYn1M/QynOKmjLUnBwKwKrLTGafG6Mk08asHJmc1LLH8L4Z2PBVpeKAJhBx+nGoJaKm5JaYqMlg6LFbeSSYfZktmhjkUPisMjdGyfwfK84KopggtexRWXsGz39aEfbjNw0U9Fr6CVb/I5WlMutQQzMvs0UbgxwKjxbTnI1ESabNqlk5sqxgRcNaCBQo6LmflVT9T7DiXDMONTTT3mFGwyDHaiJdF6PVNGci0p3SpSOlAgUKoZaItCBmMOQW55ptqwazLrMZAYMcK4lGXQznxhCZhh7LGf859DLUnj4HQEBB9zQMbFITEmyJSCud2iYpbm0301YNZl1mMwJdhwFSBIl1MU2zKmJdzL4N+jyvHjNm9OZxAxWfAXvWez973NE+I6KI0HI5wwagXXJLrNj6bzzz8UE883E57lixHVcv+qjZMEBxiciRIv/55bxf2+Bdisrr1l5Rca4ZSf0MHSlJbB8PgpkcK4hmXYzSGTPRZvYuMKIwaLV0JGYTfq4/1+x7Ul1PTZeIvj9ej6Uf/h+AwJmJZ8b0Q7vWifhwnxMrPv8+4DkAFzMYsVCcW9wnA4MvTcf/ln6PQzX16NI+GX8o6IqEFsxXSGGQYwXRrIsx0qahoVilC4xIJXHZ4+6Xd4X1OI6UJJw+58aJAEFOsK6npktEvR1tQhYAF/TogCu7tQ95XCwU5wYqqv771grDFkwbAYMcK9C7LiZUMbMZ5sZYrQuMDEmqddlIrs92IDW5ZcAAJZCMlCTMHpaNdq0TfNfl8Qi4Y8V2yfvIndsitwBYznFWL87lxGN1GORYgZ51MXKXd4w+N4ZdYKQzs+wrtKOiRlaAM+U3PXFVz7SAQcfbZUdlPZecpSG5BcChjrNyca7a/cKIhcfWoPUu5yKlxczi3Jic27yfjRLgAOwCI12p3ZU7GuTWpPRKb4OCHh0CvmkadWlIqjg3pVVLTC3sheuzHRE9H61YvahaTwxyrECsiwHQPNBRWRcTcnkH3uUds3QmmbELjEwh1F/ZgPevbLfc/Qp0pkWAIi4NBfmzytf1FGnFfTKwdcZgTCv8JVJbtQQAnDh9Dn/98LuAnV9mEAtF1XphkGMVYl2MvUla3J6prqBWyfKOGeiV7aKYZ7a/srUIUMSlIfH4pvcHors0tGmfE0s//D+cOO2/LGfEzJocRs2cmQGDHCvJHg5M3QuM2wiMXOH9PHWPusJfqy3vKM12cZYOyRTOX9luj4DS8mq8XXYUpeXVEcn2aBWgGHVui9kya3IYOXNmdCw8thqt9lMy6vJOONtWyO0C4ywdUkDtX9nRLFTWav8mI26PYMUdu61cVK03BjkUmBGH/GkRfITqAuMsHVJITeuyEdqBtQpQjLY9glXrV8y4sagRMMihwIIO+YP36+xbvMGCnq3iYubmwHvAtmebf19N8CGV7TLbLJ1Ib8ZKASn9K9tI7cBGC1C0YOX6FSNmzoyOQQ5Jk1rescUBgscbdGx7Vr+lnECZm2Y0DD7MNEuHS2qGouSvbCsupxiJ1YcCWjEw1RODHAqu8fKOmE0RPP7H6LGUI7VsFJBGwYeWxdZ6Zlm4pGZIcv/KtupyilGwfoUaY5BDocXFe9+k37xL4gCNl3KCLhsFEW6nl1bF1npmWcy2pBZj5PyVbeXlFKNg/QqJGOSQPJFcygn5XBLC7fTSothabpZFbabHTEtqFJDVl1OMgvUrBDDIIbkiOTdH8WNo1OkV7o7qcrMsHg/wwSx1mR6rzS+KQVxOiRzWrxCHAZI8kZybo+gxVG5bISWcydFysyzrx8nfD6wpo84vIkWMOkiPyGqYySF5Ijk3J+RzNdJ0kJ8W1O6oHlb2RGY9jRHnF5EqXE4h0h+DHJIn3KUczZ7rgoH3AL2H6jcbRs3k6LCzJzLqaSL5OpDuuJxCpC8uV5F8Wm8Cquq5OgO/+1+geIE3EDDSm3nITUBlCpURiuTrQBQF0djTi6zJJghCzP3f43K5kJKSgtraWtjt9mifjvlEctKu2ab6+rqrgOZZFpm/auM2yssime1nQyRDNPf0IuNT+v7NIIdBDmkt4JyczkDR48A/Z4Wup5m6h8EKxSSpPb3E3CiLsknp+zdrcih2RCrzEaxw2RbHehqiACK+pxczoTGBQQ5ZU9N/wOqrL2RRIrTXk1ThstR+YHp0iRGZSET39OLebzGDQQ5Zj6yNPRG9vZ7UtqgTSXB7BNO3okdsTy8N9n6zws87VjDIIWtRurGnOJvml8XAke2RCzrE/cDEQOfQFwx0SBWrFOpGZE8vDfZ+s8rPO1ZEpIV82bJl6Nq1K5KSkpCfn48dO3ZIHvvCCy/gmmuuQbt27dCuXTsUFhY2O378+PGw2Wx+H8XFxXpfBhmdqo09L8ymefJSYPVNwD8mej8v7RN6+nA49m3wPkckn5MsRyzUbbrM46w9g8kv70LJ3soonZly4p5eUvkQG7zBRFh7einZ+y0AK/28Y4XuQc5rr72G6dOnY+7cudi1axf69u2LoqIiHDt2LODxW7ZswZgxY/Dxxx+jtLQUWVlZuOGGG3D06FG/44qLi1FZWen7ePXVV/W+FDI6tRt7At6ancbkbrOghphtUru1AxFCF+oC3kJds8yYEff0AppPmtJsT68w9n6z2s87Vuge5Dz55JOYNGkSJkyYgOzsbCxfvhzJyclYuXJlwONfeeUV3HPPPcjNzcWll16Kv//97/B4PNi8ebPfcYmJiXA4HL6Pdu3a6X0p1uFxAxWfAXvWez973NE+I21ouinlhX+oSmZq+/MJmS7X4TnJkpQU6pqF7nt6hbH3mxV/3rFA15qcs2fPYufOnZg1a5bvtri4OBQWFqK0tFTWY9TX1+PcuXNo394/RbllyxZ06tQJ7dq1w+DBg/HYY4+hQ4fAFfcNDQ1oaGjwfe1yuVRcjUVYuatA800pZWyzoJSSdLlWz0mWFLFC3QjTdU+vMPZ+s+rP2+p0zeQcP34cbrcb6en+bz7p6elwOp2yHmPGjBnIzMxEYWGh77bi4mK89NJL2Lx5MxYtWoRPPvkEN954I9zuwH/9LliwACkpKb6PrKws9RdlZlZfJtFqW4WmtMwQhZEuJ2osIoW6USLu6XVLbmcU9OigXeeSuPcbAMlFMYlZVVb+eVuZofeuWrhwIdauXYs333wTSUkX/8cZPXo0hg8fjpycHIwYMQIbN27El19+iS1btgR8nFmzZqG2ttb3ceTIkQhdgYHEwjJJ0H/AAkhOk/e4WmaIwkiXh82qy5QxSmmhrpn3g9L03FXu/RaRwmjSnK7LVWlpaYiPj0dVlf9fpVVVVXA4HEHvu2TJEixcuBAffvghLr/88qDHdu/eHWlpaTh48CCGDBnS7PuJiYlITExUfgFWEivLJJLD9joDNzwOtO5wsU08Kx94qq+q1LVqYaTLw2LlZcoYJRbqTn55l9T8bF+hrpnbnnU5dxWzqpT8vMk4dM3kJCQkoH///n5Fw2IRcUFBgeT9nnjiCcyfPx8lJSUYMGBAyOf54YcfUF1djYwMY/+yRlUsLZNkDwem7vVudDlyhffz1D1AnxHeAC7nNu/nFgmqU9eqhZEuV83qy5QxTE6hbiTbnrXOFul67uJUcvHfAxm/c7oXRpPmdN+g87XXXsO4cePw/PPPIy8vD0uXLsW6devw7bffIj09HWPHjkXnzp2xYMECAMCiRYswZ84crFmzBldddZXvcdq0aYM2bdrg1KlTmDdvHkaOHAmHw4Hy8nI88sgjOHnyJPbs2SMrYxOTG3RWfOadxRKK3B2wrURqQ009t1mI1HN63N75O5JZPG4KagVSE3jdHgFXL/pIsivIBu8b9NYZg8POQGidcYnkuas5N048jg7DbdA5atQo/PTTT5gzZw6cTidyc3NRUlLiK0Y+fPgw4uIuJpSee+45nD17Frfddpvf48ydOxd/+ctfEB8fj6+//hqrV6/GiRMnkJmZiRtuuAHz58/nklQwei2TWGGTu2hssxCp54yVZcoYJxbqNhWp/aCkdg8XMy5qshwR3ctKIamfNxlPRLZ1mDJlCqZMmRLwe02Lhb///vugj9WqVSv885//1OjMYoi4TKLlDthWqvOQ2lDT7M8ZS8uU1Ewk2p712j2cLdukBUN3V5HGVHYVBMQ6D3OIZjcXRV0k2p71GpLHlm3SAjfojDVaLJNosMkdRUi0urnIEMS2Z2ftGalXH44w2571yrhE4tzJ+pjJiUUqugr8hLnJHUVQNLq5yDAisR+UXhmXiOxlRZbHIIeUY52HuWi5TEmmo3fbs55D8tiyTeHichUpxzoP84lGBxkZhp77Qek9JE/XvazI8nSfk2NEMTknR0u+2Ssh6jw4e4UoZph5qjKZh+Hm5JAF6dGOTkSmZrSMCwf2EcAgh9SS3CMqU99JwURkWEYZksesEom4XMXlqvBYYeIxEVmG1PRlMYfDgmVz43IVRVY0JgUTEQWg1/RlMi+2kBMRkSXoNX2ZzIuZHCIiMjWxyPj9vZWyjud+V7GDQQ4REZlWoCLjULjfVexgkENERKZUsrcSd7+8S/bx3O8q9jDIISIiw2s696Z/l3aY+cYe2ffnflexiUEOEREZWqAlqTaJLXCq4bzsx3BwTk5MYpBDRESGJTX3Rm6AM7agC27sk8GJxzGKQQ4RERlSsLk3ct3YJ8MQU5gpOjgnh4iIDCnU3JtQUlu1ZJFxjGOQQ0REhhTuPJsJV3XlElWMY5BDRESGFM48m3bJLTFlcC8Nz4bMiEEOEREZUl639shISYLSXIwNwIJbc5jFIQY5RERkTPFxNsy9ORsAmgU64tepyS39bs9ISeJO4+TD7ioiIjKs4j4ZeO73VzSbkyPOvbk+2+E3JJCt4tSYTRCEcLrzTMnlciElJQW1tbWw2+3RPh0iIgqh6cRjBjOxSen7NzM5RERkePFxNs67IcVYk0NERESWxCCHiIiILIlBDhEREVkSgxwiIiKyJAY5REREZEkMcoiIiMiSGOQQERGRJUUkyFm2bBm6du2KpKQk5OfnY8eOHUGPf/3113HppZciKSkJOTk5eO+99/y+LwgC5syZg4yMDLRq1QqFhYX47rvv9LwEIiIiMhndg5zXXnsN06dPx9y5c7Fr1y707dsXRUVFOHbsWMDjv/jiC4wZMwYTJ07E7t27MWLECIwYMQJ79+71HfPEE0/gqaeewvLly7F9+3a0bt0aRUVFOHPmTMDHJCIiotij+7YO+fn5uPLKK/HMM88AADweD7KysnDfffdh5syZzY4fNWoU6urqsHHjRt9tAwcORG5uLpYvXw5BEJCZmYkHH3wQDz30EACgtrYW6enpWLVqFUaPHh3ynLitAxERkfkoff/WNZNz9uxZ7Ny5E4WFhRefMC4OhYWFKC0tDXif0tJSv+MBoKioyHd8RUUFnE6n3zEpKSnIz8+XfMyGhga4XC6/DyIiIrI2XYOc48ePw+12Iz093e/29PR0OJ3OgPdxOp1Bjxc/K3nMBQsWICUlxfeRlZWl6nqIiIjIPGKiu2rWrFmora31fRw5ciTap0REREQ60zXISUtLQ3x8PKqqqvxur6qqgsPhCHgfh8MR9Hjxs5LHTExMhN1u9/sgIiIia9M1yElISED//v2xefNm320ejwebN29GQUFBwPsUFBT4HQ8AmzZt8h3frVs3OBwOv2NcLhe2b98u+ZhEREQUe1ro/QTTp0/HuHHjMGDAAOTl5WHp0qWoq6vDhAkTAABjx45F586dsWDBAgDAAw88gGuvvRb//d//jWHDhmHt2rX417/+hf/5n/8BANhsNkydOhWPPfYYevXqhW7dumH27NnIzMzEiBEj9L4cIiIiMgndg5xRo0bhp59+wpw5c+B0OpGbm4uSkhJf4fDhw4cRF3cxoTRo0CCsWbMGf/7zn/GnP/0JvXr1wltvvYU+ffr4jnnkkUdQV1eHu+66CydOnMDVV1+NkpISJCUl6X05REREZBK6z8kxIs7JISIiMh9DzckhIiIiihYGOURERGRJutfkkPW5PQJ2VNTg2Mkz6NQ2CXnd2iM+zhbt0yIiohjHIIfCUrK3EvPe2YfK2oubo2akJGHuzdko7pMRxTMjIqJYx+UqUq1kbyUmv7zLL8ABAGftGUx+eRdK9lZG6cyIiIgY5JBKbo+Aee/sQ6DWPPG2ee/sg9sTc817RERkEAxySJUdFTXNMjiNCQAqa89gR0VN5E6KiIioEQY5pMqxk9IBjprjiIiItMYgh1Tp1FbedGm5xxEREWmN3VWkSl639shISYKz9kzAuhwbAEeKt53czNgeT0RkXgxySJX4OBvm3pyNyS/vgg3wC3TEEGDuzdmmDgjYHk9EZG5crooBbo+A0vJqvF12FKXl1Zp1PBX3ycBzv78CjhT/JSlHShKe+/0Vpg4E2B5PRGR+zORYnN7ZiOI+Gbg+22GpJZ1Q7fE2eNvjr892mPo6iYisjpkcC4tUNiI+zoaCHh1wS25nFPToYPo3frbHExFZA4Mci+KwPvXYHk9EZA0MciyK2Qj12B5PRGQNDHIsitkI9cT2eKlFNxu8dU1mb48nIrI6BjkWxWyEemJ7PIBmgY5V2uOJiGIBgxyLYjYiPFZujyciihVsIbeoWBjWpzcrtscTEcUSmyAIMdde43K5kJKSgtraWtjt9mifjq44tZeIiKxC6fs3MzkWx2yE/ri/FRGRMTHIiQHisD7SHjNlRETGxcJjIgmh9vzi/lZERMbGTA5RAKEyNNzfiojI+JjJIWpCToaGE6WJiIyPQQ5RI3L3/HK6OFGaiMjoGOQQNSI3Q1NzqkHW43GiNBFR9LAmx6DYlhwdcjMv7VsnICMlCc7aMwGzPjZ4pyNzojQRUfQwyDEgtiVHj9zMiyOlFSdKExEZHJerDIZtydGlZM8v7m9FRGRszOQYCNuSo0/pnl+cKE1EZFy6ZXJqampwxx13wG63IzU1FRMnTsSpU6eCHn/fffehd+/eaNWqFX7xi1/g/vvvR21trd9xNput2cfatWv1uoyIYltyeEIN75NLaYZGnCh9S25nFPTowACHiMggdMvk3HHHHaisrMSmTZtw7tw5TJgwAXfddRfWrFkT8Pgff/wRP/74I5YsWYLs7GwcOnQId999N3788UesX7/e79gXX3wRxcXFvq9TU1P1uoyIklv0yrbk5rSuY2KGhojI/HTZhXz//v3Izs7Gl19+iQEDBgAASkpKMHToUPzwww/IzMyU9Tivv/46fv/736Ourg4tWnjjMZvNhjfffBMjRoxQfX5G3YW8tLwaY17YFvK4VycN5F5UjYh1TE3/RxbDEdbHEBFZg9L3b12Wq0pLS5GamuoLcACgsLAQcXFx2L59u+zHES9CDHBE9957L9LS0pCXl4eVK1ciVJzW0NAAl8vl92FESopeyUvu8D61S1dEFAM8bqDiM2DPeu9njzvaZ0Qa0WW5yul0olOnTv5P1KIF2rdvD6fTKesxjh8/jvnz5+Ouu+7yu/3RRx/F4MGDkZycjA8++AD33HMPTp06hfvvv1/ysRYsWIB58+Ypv5AIU1r0GsvEOUKfH/xJdh0Ts19E1My+DUDJDMD148Xb7JlA8SIge3j0zos0oSjImTlzJhYtWhT0mP3794d1QoA3HTVs2DBkZ2fjL3/5i9/3Zs+e7fvvfv36oa6uDosXLw4a5MyaNQvTp0/3e/ysrKywz1MPYtFr0/oSB+fk+ASqvwmFdUxEOvC4gUNfAKeqgDbpQJdBQFx8tM9Kvn0bgHVjgaa5YFel9/bfvcRAx+QUBTkPPvggxo8fH/SY7t27w+Fw4NixY363nz9/HjU1NXA4HEHvf/LkSRQXF6Nt27Z488030bJly6DH5+fnY/78+WhoaEBiYmLAYxITEyW/Z0QsepUmVX8TCrdXINKY2TMgHrf3/IMN7SiZCVw6TJvAzewBoUkpCnI6duyIjh07hjyuoKAAJ06cwM6dO9G/f38AwEcffQSPx4P8/HzJ+7lcLhQVFSExMREbNmxAUlLoN6aysjK0a9fOVEGMHGJbMl0UrP5GCrdXINKBFTIgh77wD9CaEQDXUe9x3a4J77nMHhCKTBio6VKTc9lll6G4uBiTJk3C8uXLce7cOUyZMgWjR4/2dVYdPXoUQ4YMwUsvvYS8vDy4XC7ccMMNqK+vx8svv+xXINyxY0fEx8fjnXfeQVVVFQYOHIikpCRs2rQJjz/+OB566CE9LoMMJtQcoaZYx0Skg0hnQPRyqkrb46RYISAETBuo6TYn55VXXsGUKVMwZMgQxMXFYeTIkXjqqad83z937hwOHDiA+vp6AMCuXbt8nVc9e/b0e6yKigp07doVLVu2xLJlyzBt2jQIgoCePXviySefxKRJk/S6DDIQpXU1rGMKjJu/UlgimQHRU5t0bY8LxCoBoYkDNd2CnPbt20sO/gOArl27+rV+X3fddSFbwYuLi/2GAFJskVtXM+U3PXFVzzS+eQfAzV8pbJHKgOityyBvJsJVicBBiM37/S6D1D+HFQJCkwdq3KCTTEPuHKFp1/+S2ysEwM1fTc4os1wikQGJhLh471ILADT7V+XC18ULw3vjtkJAqCRQMyAGOWQa4hwhQPKfJNbfSODQRJPbtwFY2gdYfRPwj4nez0v7eG+PNDEDEuzPDXvn8DIgkZI93LvUYm+SxbRnarMEY4WA0OSBGnchJ1PhHCF1lGz+yq4+g4lWPUTTTpqsfODIdu/XV4wHtjwOSI0tDTcDEgni9bnPAiOWA4IA1B/XtmsoEktiejN5oMYgh0zHKnOEIlkAzM1fTSpa9RCBOmlscYDgufh1q/beczj988Xb7JneAMegRag+wTqFtKyNEZfE1o2FaQNCkwdqDHLIlMw+R0iLAmAlQZLcom0OTTSYaBSuSmWOGgc4wMXg5ro/AR16mGZuSsQzY+KSWMCgygQBockDNQY5RBEmNbVZLACWs2u60iBJLNp21p6R+luMQxONKNL1EEEzR01dyCTtWg1M3RP4Tc5ow+POnwU2TkXEM2PZw72PaaSfhRImDtQY5BBFUKgCYBu8BcDXZzskszJqgiRu/mpSka6HCJk5aipIJslow+P2bQA2TgPqq4McpGNLd1y8cdvE5TBpoMbuKqIIUlIAHEg4XVJi0bYjxX9JypGSJCt7RFEQ6U4mtRmhpvcTl4SaBkziklCku8LE86k/Lu94g3YKRZ0YqOXc5v1s8AAHYCaHKKLCLQAOt0vKKkXbMSPS9RBqM0KN72e04XGKluAuMGinECnHIIcogsItANaiS8rsRdsxJ5L1EHXVzbuoggrQWaNnsbSaGh9FS3BhdgoZrQaJGOQQ6UGq86l/l3Zo3zoBNXVnA94vVAEwu6RiVCTqIfZtANaPh/yMh0QmSa9iabU1PkqfR05mLFAw8+27xqpB0ovJAjkGORRz9J5PI9X5NLxvBjZ8VRk0wAGCFwCzSyqG6Vm4qmZJRyqTpKRYWu4bZjht33LPJzkNuOmvoQOSQMFWq/bA6QB1dCbYwFIRoxWTy8Agh2KK3htUSnU+VdaewfOfVgS9r5ypzeySIl3IXdIpetwbNAQLSOQOj6uv9m5NEeoNM5waH4/b+9Gqnf/QwqaS04Dp+4EWCdLHANLBVqAAR875mYlJdyJndxXFDL03qAzW+RRK+9Yt8cnDv5EVaLFLijQnd0mnTXrozho5G1/2GQm8Pl5e95XaDSLF/b7+95YgAY7N+3HTX0MHOGqyXcHOz0xCBprwBnLR2jQ2CGZyyPLcHgHb/l2Nmf/YE9Z8mlBCdT4FU1N3DjsP/Sy7IJhdUqQprefxBCuWvuFx4INZkJ2ZUVPjI5V1aCq5A3D577yZHo87eKZF8QyhIOdnNtGYvK0RBjmkWCT3XApXoOWpQLTYoDLcfZ+U3p9dUqQZPfYnkiqWVvqGqTQAk5NxadkaaJnknZuz7VnvR6jaknCDFDO3pZt4J3IGOaSI3jUtWpKqjwkmnEAl3I4mdkSR5uQW9uo1jydQsbTSN0ylAZicjMu5Ou9HY6FqS1QHKcbewFIWE+9Ezpockk3vmhYtqa2PCSfQEDuflOa0bPAGiuyIIk2JNSmrbwL+MdH7eWkf6WnD4hKTvckfK/ZMbYtKlb5hyqnxaRyAqc4mhKgtCTl9OhDjb2ApS6Qnb2uIQQ7JEs52ApHi9ggoLa/G22VHserzCkX1MVoEGmLnk/h4cp8XYEcUaUzttgrZw4Gpe4FxG4GRK7yfp+7RtmtGzRumkgAsrGxCkCJhOcFWq3ahzy9aPG6g4jNgz3rvZyVFwkoDTQPhchXJEu52AnqTW3sTiJaBhtj5FGxOTuPb5bSNEykS7rYKem8kqXZpTO5AxJDLWzJIZYNCTZ826gaWWsy3MelO5DZBEKL3p3eUuFwupKSkoLa2Fna7PdqnYwpvlx3FA2vLQh73t9G5uCW3s/4n1Iia2pvG9KgpkirONlPRNplUxWfepalQxm2MbidMwDfeztq8Yfq6qwBVgU6on42Zpv5Kdppd+HdHaaYpyteu9P2bmRySxajbCYQzmyY1uSWWjbkCA3t00DzQkOp8YkcU6c4snTB6blUhlXVomwmcP3Nhbk4YXWRNs13iUpDRgh49NkvVO9OnMQY5JItRtxNQM5tGDGcW3pqDq3qlaX9SRNFkpk4YPd8wpYKob9/VtovMyFsdmHi+jVZYeEyyBCuqFf+puLGPdzhdJIuP1bR8czIwWZqJO2E0JwZRjac0a9lFprbAO1LMktXTETM5JJtUUa3NBggCsPLz77Hy8+8V17iEU6cid3ls9rDLkNY2kXUwZH3hzLwxU61JOLRYKtNjKUhrZsrq6YSFxyw8VkwMSj7c58SKz79v9n0xfJCTLQl3uKDbI+DqRR+FXEbbOmOwoQMbFiQbjBXe7JUW9hp52cWIzFDg7XFf2AQ1xCDFqXtM8/83C49Jd/FxNuR1a4/p68oCfl/8VfrTm3sw+NJ0JLQIvCoq1RUlDheUEyRZYVduM02RjglWebNXkq0w6Q7TUWWGpSC9JlmbCGtySBU5Bb81decwcMHmgJOQtRwuaOZduc00RTomGL3GQqlANSlNabHDdDiD5szKLEtBkZpkbVDM5JAqcgt+a+rOBszKaD1cMNCu3P27tMPOQz/j7bKjhlwCChXoabEzOilghhoLPYTbgWOVzJdSemxqqhc92/UNjkEOqaJ0Hk7TN2u5QZKS7qnGM2hK9lbi2sUfG3oJyOhTpGNOrLbbhrPs8s1bwOvjmt8eC8tcZlsKMtl8G61wuYpUUbIZZeM3a5GewwXNsgSkR6BHYTBDjYUe1C677H0LWD9B4mCZy1xmF+NLQWbATA6p0rjgV67Gb9Z6DRcMZwko0h1OegZ67NZSwSw1FlpTs+yybwOwPkAGx49FM19NqVkKskL3nknolsmpqanBHXfcAbvdjtTUVEycOBGnTp0Kep/rrrsONpvN7+Puu+/2O+bw4cMYNmwYkpOT0alTJzz88MM4f/68XpdBQYgFv+1bt5R1fOM361DDBQF1XVFKloAaK9lbiasXfYQxL2zDA2vLMOaFbbh60Ue6Zn1CZcPU7owejWuxhFgdoqd0h2lf7ZJMVst8BSKnwFu0b4O3rXv1TcA/Jno/L+1jvqJ2k9AtyLnjjjvwzTffYNOmTdi4cSM+/fRT3HXXXSHvN2nSJFRWVvo+nnjiCd/33G43hg0bhrNnz+KLL77A6tWrsWrVKsyZM0evy6AQivtkYNusQrRvnSB5jNSbtR5dUWqWgKK1vKVHoGeWpTpDUvpmbyVKll1C1i41YbXMVzis1r1nAroMA9y/fz+ys7Px5ZdfYsCAAQCAkpISDB06FD/88AMyMzMD3u+6665Dbm4uli5dGvD777//Pm666Sb8+OOPSE/3/uIsX74cM2bMwE8//YSEBOk32sY4DFB74psrEHhWTbCgRculldLyaox5YVvI416dNBAFPTr4hglKZX8iMUxQqzk5RrgWS9Bzd+xokrNEIueYPeu9GQg57J1NNWhOV77BfFIBovkG80WDIYYBlpaWIjU11RfgAEBhYSHi4uKwfft2/Md//IfkfV955RW8/PLLcDgcuPnmmzF79mwkJyf7HjcnJ8cX4ABAUVERJk+ejG+++Qb9+vXT43JIBqktHxwy3qy13Jlbaa2PETqcArW/qwn0jHAtlmDFdlu5bd5yOnCUZGasmvlSI1a796JMlyDH6XSiU6dO/k/UogXat28Pp9Mpeb/bb78dXbp0QWZmJr7++mvMmDEDBw4cwBtvvOF73MYBDgDf18Eet6GhAQ0NDb6vXS6X4mui0LR6sw6H0gnIcpe3Pj94XNdr0SLQY7eWhqzUbhtqmvFvVwHJHeQHdCELlQHY4oHbVpo786W1WO3eizJFQc7MmTOxaNGioMfs379f9ck0rtnJyclBRkYGhgwZgvLycvTo0UP14y5YsADz5s1TfX+ST8usjFpKskpyO5ee+fgg/rHrB13m7Egt1yldxtOzW4tMSs404/UTAMFz8eZQg/yCzoe5YORK4Fcjwjp1y4nV7r0oUxTkPPjggxg/fnzQY7p37w6Hw4Fjx4753X7+/HnU1NTA4XDIfr78/HwAwMGDB9GjRw84HA7s2LHD75iqKm/UG+xxZ82ahenTp/u+drlcyMrKkn0eZD5ys0qhlrcaU7KnllxS9TjD+2Zgw1eViup09GrLJxOTUyTcOMAB5A3yEwuVrVi7pBczTUi2EEVBTseOHdGxY8eQxxUUFODEiRPYuXMn+vfvDwD46KOP4PF4fIGLHGVlZQCAjIwM3+P+13/9F44dO+ZbDtu0aRPsdjuys7MlHycxMRGJiYmyn5fCZ4Q5LXKySsGWt5oKNmdHzfVKbVBaWXsGz39a0ez4UEGWFTYrJY2pWvqQuYWFFWuX9GS2CckWoUt3FQDceOONqKqqwvLly3Hu3DlMmDABAwYMwJo1awAAR48exZAhQ/DSSy8hLy8P5eXlWLNmDYYOHYoOHTrg66+/xrRp03DJJZfgk08+AeBtIc/NzUVmZiaeeOIJOJ1O/OEPf8Cdd96Jxx9/XPa5sbtKX2bcVTvQOQcjdmdJ3TfU9YbqhJIip0PKjD9/0knFZ945LGqN22id2iSjsGr3XoQYorsK8HZJTZkyBUOGDEFcXBxGjhyJp556yvf9c+fO4cCBA6ivrwcAJCQk4MMPP8TSpUtRV1eHrKwsjBw5En/+859994mPj8fGjRsxefJkFBQUoHXr1hg3bhweffRRvS6DFJLKTuix1KMlcXnrr5sO4JmPy0MeLxbvqr1eObu4ByKnQ8oIBeBkEHKKhINhEaz2mAGLKN2CnPbt2/uyNoF07doVjZNIWVlZvoxNMF26dMF7772nyTmStoy2q7bSJaT4OBuu6tlRVpDTqW2S7OsdfGk6dh762e88wu1wCnV/IxSAkwHIKRIOpjr07wKpYKXuPYPj3lUUNjGY+PzgT4aZ06J2yUZJ8a7cuTQDF2xGTd1Zv/MYfWV4he/skIpxSvY+kioStsU1Lzpuaucq4NcPMctApsUgh8KitJYF0H9OSzhLZkqKd+VeR+MARzyPv374HVKTW6K2/pyiv63ZIUWyB/s1FmiJ5PutwCcLgz/XyR85nI5MTbe9q8j6pPZJCkXPLESoJSTAu4Tk9kiHFnL31FJ7HeJSlkjuwh07pCisvY+abiKZ1kvecxqhLsfj9hZR71nv/exxR/uMyCSYySFVggUTUiKRhdBqawM5xbtKZuwEOo8T9ecwrfCXWPvlYVlzcuRskUEWFnKwn4y278bMMpxOTeaK6AIGOaSK0u6gSGUhtNzaIFTxrpIZO1K6piVj64zBAYOpR4ovY4cUXaT13kdmGE4XakuKYAMLicAgh1RSWlcTqSxEpLc2kNpCon3rlqipOyfrPKSCKTkdUkYYukgqKCkcFmm995HRh9NpnbmimMQgh1SRGyRM+U1PXNUzLWJvvuFubaAmaAi0tJWblYpfzS1BkNIfxNmA/l3ayb+4Jjj0z6TULr/osbwkuT1Dpn7D6eQGeNy1mzTAIEdDsfRXtdxgYtr1vzT0LuSNhRM0NM26lJZXBw1wAMAjADsP/ayqnd6sQxdjXjjLL3otL0VyOJ2SAI+7dpMG2F2lkZK9lbh60UcY88I2PLC2DGNe2IarF32Ekr2V0T41XYjBRCDR7gKS2x3VmFSnmBg0iK+j2yOgtLwab5cdRWl5tWSXlpa1QU1p0UFGUSBnR/CSmdKdQ+LyEoDmPXlhLi817bzSK8BR0hlmlsJoMjRmcjQQy39V21u1QO3p8363pSa3xIJbc6J6zUq2NpA7udjjETD/3f2yMj3h1AaFygjK7SD766YDuKpnR0tnFE1Fi+WXaCwvaUFNfY0ZCqPJ8BjkhMloWxlESsneStz98q6A3/u5PnTBbSTI3dpAbtBwz5rdzb4nFciqrQ2Ss2QmN/vzzMfleObjctbpGIVWyy9m3PtITYAnpzC66HH/n0NWPnBku3l+LqQ7Bjlh0moui5m4PQJmvrEn6DEz39hjmsAunAnMUoGsmtoguRlBpZ1hsZBRNAUtl1+MsveR3CJitQFe9nDgt6uAdx8E6o9fvN2eCfQZCfxzVvCtKjhPJ+axJidMetZeGNW28mqcCJGtOVF/DtvKqyN0RvIFqqkJt528cSDbmJLaICV1NmKWSG74yDodgxCXXyRfORtg72ye5Zd9G4ClfYDVNwH/mOj9vLRP4KnLagO8fRu8gUzjACe5A/Cr/wC+eLp5dqjpXlxyJkGTpTGTE6ZIz2UxgtJ/Hw990IXjruqVpvPZyCe1FDR72GWqJxc3FiiQlVsbpDQjqHQIoRUziqZj9Lk0SijtElNTXyP1HPXVQOkzMk/0wn3fuR9ISgG6Xm3cn6+a2UkUEjM5YQr1V7UN3jdSa22oqHS3pegL1j1175rdGN7Xm1WR6FmRRSqQFWuDbsntjIIeHQIu4SnNCEplieTen6JELBy2N1k2tGeaZ3qvmi4xpZ1hQZ9DhdM/Ay8Nl840RZuSrBgpwiAnTI1bqaXeIK22oaLcTIBRMgZyloI2fFWJZbcHXlp69vYrdA9k1WQEi/tkYOuMwXh10kBM+U1PTZ8nEuS241tO9nBg6l5g3EZg5Arv56l7zBHgAMqKiBtTEuCFfA6VjLh8Fc6mqxQSl6s0IDXa36obKg7s3gGpyS2D1uW0S26Jgd2NEeTIXQpq1zpBch+puDioGjAol9puLDFLlNetPf6x6wfVk54jLeanNRulcFiNcLrE5HaG6Tbgz2DbQXDrCt0xyNGIkrksZhcfZ8PCW3MkW8gBYMGtOYa5diVLQY3bzpvOq1l2e79mc3K0CmTDmdSsxf0jSaqLrLL2DO5+eReevb0fhl6eGZVzIxnC7RKTE+DpOuDPQNtBcOsK3THI0ZDcuSxWUNwnA8t/fwX+smEfnC5j/zUud4nm++P1vv+WLlLORrvWCboEsuFmBM2QUQy2dCia8upuPAMbhl4e/fOlACIxpC/kczR6LrV1O0bYDoJbV+jOJghCjCyEX+RyuZCSkoLa2lrY7fZon46pmWG/LrdHwFULN8Ppagh6XEZKErbOGIxN+5wBMw3iVek9bybcn6mRX5PS8mqMeWGbrGOXc66Pcfk6n4CAeUMtiqhDPceg+4C964PPyQlm3MboZ0cqPvMWGYdihHM1CKXv3wxyGORYVuM3+8/+7zjW7/oh5H1euTMfD73+lWQNj1jbsnXGYMMEDmbydtlRPLC2TNaxGfw5G1vAzTY7a7u9RKjnaNp2nZXv/Xr9eG9HVUAXMk1T90S/zsXj9nZRhcqKGeFcDULp+zeXq8iSAi03yVFaXh1zE6wjSUl3V8z+nM0yLyUS20uEeo5A9T09rgNufip4Fsgo84isNDvJoBjkkOVIFbbKI+9enDejjthFJjf4jLmfc8DMhYG3JohEl5ia5zDTRqZmOlcTYpBDliKnsDUQcRmqoHsanvm4POTxRpo3YyZiF1iwzrzGYurnrHSKMAVnpo1MzXSuJsMghywl1EycQBq3WA/s0UHVvJqYocFSSnGfDDx7ez9MeXU3pOb/xdzPmfNS9GGmeURmOlcT4cRjshQ1yxuNN8yMxQnWsmk4en7o5Zl4ZswVAb8Xkz9ntVOEiSgoZnLIUtJaJ8o67j+HXoZO9sSALdZmmDcTcTospQy9PAPL4/hzBsB5KaGYpRibDIdBDlmLzD/8szPtuKqn9A7psTTBOiQdl1L4c74g3CnCVma2YmwyFAY5ZCnHTwUf+KfkuFiaYB2UzqPn+XNGZKYImxGLsSlMrMkhS1GzmzeFwKUU/YnzUgBIVoPF2ryUkBlEeDOIHnckz4pMhkEOWYo4h0VqscMG7yTdmOna0QKXUiJDnJdib1KLZM+MzYwFi7FJA1yuoqCMvA9SIGbajds0uJQSOZyXchEziKQBBjkkSWonbqN3vrA7SmMcPR9ZVp2XorRDihlEZdiBFpBuG3TW1NTgvvvuwzvvvIO4uDiMHDkSf/vb39CmTZuAx3///ffo1q1bwO+tW7cOv/3tb70nbGv+F/irr76K0aNHyz43btAZmtTWCJHaiVsLZstCGV4kNmQka1LTIcXNK+WLoQ40w+xCfuONN6KyshLPP/88zp07hwkTJuDKK6/EmjVrAh7vdrvx008/+d32P//zP1i8eDEqKyt9wZHNZsOLL76I4uJi33GpqalISpJfSMogJzi3R8DViz7iTtzUHP9aJKWkOqTEP5mC1Rv57gsEzCDGYq1SU+H8fE3IELuQ79+/HyUlJfjyyy8xYMAAAMDTTz+NoUOHYsmSJcjMzGx2n/j4eDgcDr/b3nzzTfzud79rlv1JTU1tdixpJ9TWCNyJO4ZZdSmF9BHujCVuXhkctwMJSZfuqtLSUqSmpvoCHAAoLCxEXFwctm/fLusxdu7cibKyMkycOLHZ9+69916kpaUhLy8PK1euRKhkVENDA1wul98HSZO7NULM7RBNRMpo0SGVPRyYuhcYtxEYucL7eeoeBjgAO9Bk0CWT43Q60alTJ/8natEC7du3h9PplPUYK1aswGWXXYZBg/w7Nh599FEMHjwYycnJ+OCDD3DPPffg1KlTuP/++yUfa8GCBZg3b57yC4lRnDVDRJrQqkOKGcTA2IEWkqJMzsyZM2Gz2YJ+fPvtt2Gf1OnTp7FmzZqAWZzZs2fjqquuQr9+/TBjxgw88sgjWLx4cdDHmzVrFmpra30fR44cCfscjcjtEVBaXo23y46itLwabqktnkPgrBki0gQ7pPTFn29IijI5Dz74IMaPHx/0mO7du8PhcODYsWN+t58/fx41NTWyamnWr1+P+vp6jB07NuSx+fn5mD9/PhoaGpCYGHhzxsTERMnvWYWW7d6cNUNEmuCMJW01LfzPyg/x8wVgiwPqqiN6mkaiKMjp2LEjOnbsGPK4goICnDhxAjt37kT//v0BAB999BE8Hg/y8/ND3n/FihUYPny4rOcqKytDu3btLB/EBCPV7u2sPYPJL+9S1e7NWTNEFDbOWNKOVJt4n9uAL56Wvp/gAdaPB+Ks1WUll64t5FVVVVi+fLmvhXzAgAG+FvKjR49iyJAheOmll5CXl+e738GDB/HLX/4S7733nl+bOAC88847qKqqwsCBA5GUlIRNmzbhoYcewkMPPaSo5sZKLeR6t3tz1gwRhY0zlsITqk28YAqwbZk3oAnIOjOFDNFCDgCvvPIKpkyZgiFDhviGAT711FO+7587dw4HDhxAfX293/1WrlyJSy65BDfccEOzx2zZsiWWLVuGadOmQRAE9OzZE08++SQmTZqk12UYnt7t3twhmojCxu0q1JPTJv7Vq0ECnAvHiV1WMVbArVsmx8islMl5u+woHlhbFvK4v43OxS25nfU/ISIi0k7FZ8Dqm7R5rJErgJzbtHmsKFH6/s1dyE2O7d5ERBamZft3DHZZMcgxObZ7ExHpxOP2ZlL2rPd+9rgjfw5yA5PkDkCwdwJ755jsYmOQY3JiuzfQ/H9vtnsTEam0b4N3g9DVNwH/mOj9vLSP9/ZIEtvwQwUwQ5+8+HXT7wMx28XGIMcCxHZvR4r/kpQjJckUu4UTERmK2M3UdMsEV6X39kgGOmIbPoCgAUyfEd59vuxN/r23Z1puk04lWHhs8sLjxtjuTUQUJo/bm7GR3BMqSu3Yctvwmw4MtFgXm2FayCny2O5NRBQmJZteRrIdW24bPvf58sMgh4iISGTkTS8ZwCjGmhwiIiIRN720FAY5REREIrndTDHYjm1GDHKIiIhEcruZLFTMa2UMcoiIiBrLHs52bLWMMECxERYeExERNcVNRZUL2Oae6c2MRSkwZJBDREQUCLuZmpOawyMOUGy6W7o4QDFKGTAGOURERBSaVKamaAHwz1loFuAAF26zASUzvZmxCGfCWJNDREREwQXb6uL1cfIHKEYYgxwiIiKS5nF7MziSmRqZojBAkUEOERERSQu51YVMURigyJocIiIikhZ2BubCpqZRGKDITA4RERFJU5SBMdYARQY5REREJE3uVhe3rTbcAEUuVxEREZE0cauLdWPhDXQaFxs3ytRkDweybzbUAEUGOURERBScuNVFwInGCy9magw2QJFBDhEREYVmwq0uGOQQERGRPAbL1ITCwmMiIiKyJAY5REREZEkMcoiIiMiSGOQQERGRJTHIISIiIktikENERESWxCCHiIiILIlBDhEREVkSgxwiIiKypJiceCwI3s3FXC5XlM+EiIiI5BLft8X38VBiMsg5efIkACArKyvKZ0JERERKnTx5EikpKSGPswlywyEL8Xg8+PHHH9G2bVvYbN5t4l0uF7KysnDkyBHY7fYon6G+YulaAV6vlcXStQK8XquLpetVe62CIODkyZPIzMxEXFzoipuYzOTExcXhkksuCfg9u91u+f+5RLF0rQCv18pi6VoBXq/VxdL1qrlWORkcEQuPiYiIyJIY5BAREZElMci5IDExEXPnzkViYmK0T0V3sXStAK/XymLpWgFer9XF0vVG6lpjsvCYiIiIrI+ZHCIiIrIkBjlERERkSQxyiIiIyJIY5BAREZElxUyQ81//9V8YNGgQkpOTkZqaKus+giBgzpw5yMjIQKtWrVBYWIjvvvvO75iamhrccccdsNvtSE1NxcSJE3Hq1CkdrkAZpef1/fffw2azBfx4/fXXfccF+v7atWsjcUmS1LwG1113XbPruPvuu/2OOXz4MIYNG4bk5GR06tQJDz/8MM6fP6/npcii9Hprampw3333oXfv3mjVqhV+8Ytf4P7770dtba3fcUZ5bZctW4auXbsiKSkJ+fn52LFjR9DjX3/9dVx66aVISkpCTk4O3nvvPb/vy/k9jiYl1/vCCy/gmmuuQbt27dCuXTsUFhY2O378+PHNXsfi4mK9L0MWJde6atWqZteRlJTkd4yVXttA/ybZbDYMGzbMd4xRX9tPP/0UN998MzIzM2Gz2fDWW2+FvM+WLVtwxRVXIDExET179sSqVauaHaP034KAhBgxZ84c4cknnxSmT58upKSkyLrPwoULhZSUFOGtt94SvvrqK2H48OFCt27dhNOnT/uOKS4uFvr27Sts27ZN+Oyzz4SePXsKY8aM0ekq5FN6XufPnxcqKyv9PubNmye0adNGOHnypO84AMKLL77od1zjn0c0qHkNrr32WmHSpEl+11FbW+v7/vnz54U+ffoIhYWFwu7du4X33ntPSEtLE2bNmqX35YSk9Hr37Nkj3HrrrcKGDRuEgwcPCps3bxZ69eoljBw50u84I7y2a9euFRISEoSVK1cK33zzjTBp0iQhNTVVqKqqCnj8559/LsTHxwtPPPGEsG/fPuHPf/6z0LJlS2HPnj2+Y+T8HkeL0uu9/fbbhWXLlgm7d+8W9u/fL4wfP15ISUkRfvjhB98x48aNE4qLi/1ex5qamkhdkiSl1/riiy8Kdrvd7zqcTqffMVZ6baurq/2ude/evUJ8fLzw4osv+o4x6mv73nvvCf/5n/8pvPHGGwIA4c033wx6/L///W8hOTlZmD59urBv3z7h6aefFuLj44WSkhLfMUp/flJiJsgRvfjii7KCHI/HIzgcDmHx4sW+206cOCEkJiYKr776qiAIgrBv3z4BgPDll1/6jnn//fcFm80mHD16VPNzl0ur88rNzRX++Mc/+t0m53/gSFJ7rddee63wwAMPSH7/vffeE+Li4vz+UX3uuecEu90uNDQ0aHLuamj12q5bt05ISEgQzp0757vNCK9tXl6ecO+99/q+drvdQmZmprBgwYKAx//ud78Thg0b5ndbfn6+8P/+3/8TBEHe73E0Kb3eps6fPy+0bdtWWL16te+2cePGCbfccovWpxo2pdca6t9qq7+2f/3rX4W2bdsKp06d8t1m1Ne2MTn/jjzyyCPCr371K7/bRo0aJRQVFfm+DvfnJ4qZ5SqlKioq4HQ6UVhY6LstJSUF+fn5KC0tBQCUlpYiNTUVAwYM8B1TWFiIuLg4bN++PeLnLNLivHbu3ImysjJMnDix2ffuvfdepKWlIS8vDytXrpS95b0ewrnWV155BWlpaejTpw9mzZqF+vp6v8fNyclBenq677aioiK4XC5888032l+ITFr9P1dbWwu73Y4WLfy3r4vma3v27Fns3LnT73cuLi4OhYWFvt+5pkpLS/2OB7yvk3i8nN/jaFFzvU3V19fj3LlzaN++vd/tW7ZsQadOndC7d29MnjwZ1dXVmp67Umqv9dSpU+jSpQuysrJwyy23+P3uWf21XbFiBUaPHo3WrVv73W6011aNUL+3Wvz8RDG5QaccTqcTAPze5MSvxe85nU506tTJ7/stWrRA+/btfcdEgxbntWLFClx22WUYNGiQ3+2PPvooBg8ejOTkZHzwwQe45557cOrUKdx///2anb8Saq/19ttvR5cuXZCZmYmvv/4aM2bMwIEDB/DGG2/4HjfQay9+L1q0eG2PHz+O+fPn46677vK7Pdqv7fHjx+F2uwP+3L/99tuA95F6nRr/joq3SR0TLWqut6kZM2YgMzPT782guLgYt956K7p164by8nL86U9/wo033ojS0lLEx8dreg1yqbnW3r17Y+XKlbj88stRW1uLJUuWYNCgQfjmm29wySWXWPq13bFjB/bu3YsVK1b43W7E11YNqd9bl8uF06dP4+effw77d0Nk6iBn5syZWLRoUdBj9u/fj0svvTRCZ6QvudcbrtOnT2PNmjWYPXt2s+81vq1fv36oq6vD4sWLNX8j1PtaG7/B5+TkICMjA0OGDEF5eTl69Oih+nHVitRr63K5MGzYMGRnZ+Mvf/mL3/ci9dqSNhYuXIi1a9diy5YtfgW5o0eP9v13Tk4OLr/8cvTo0QNbtmzBkCFDonGqqhQUFKCgoMD39aBBg3DZZZfh+eefx/z586N4ZvpbsWIFcnJykJeX53e7VV7bSDJ1kPPggw9i/PjxQY/p3r27qsd2OBwAgKqqKmRkZPhur6qqQm5uru+YY8eO+d3v/PnzqKmp8d1fS3KvN9zzWr9+Perr6zF27NiQx+bn52P+/PloaGjQdA+SSF2rKD8/HwBw8OBB9OjRAw6Ho1klf1VVFQCY9rU9efIkiouL0bZtW7z55pto2bJl0OP1em2lpKWlIT4+3vdzFlVVVUlem8PhCHq8nN/jaFFzvaIlS5Zg4cKF+PDDD3H55ZcHPbZ79+5IS0vDwYMHo/ZGGM61ilq2bIl+/frh4MGDAKz72tbV1WHt2rV49NFHQz6PEV5bNaR+b+12O1q1aoX4+Piw/3/xUVTBYwFKC4+XLFniu622tjZg4fG//vUv3zH//Oc/DVN4rPa8rr322madN1Iee+wxoV27dqrPNVxavQZbt24VAAhfffWVIAgXC48bV/I///zzgt1uF86cOaPdBSik9npra2uFgQMHCtdee61QV1cn67mi8drm5eUJU6ZM8X3tdruFzp07By08vummm/xuKygoaFZ4HOz3OJqUXq8gCMKiRYsEu90ulJaWynqOI0eOCDabTXj77bfDPt9wqLnWxs6fPy/07t1bmDZtmiAI1nxtBcH7HpWYmCgcP3485HMY5bVtDDILj/v06eN325gxY5oVHofz/4vvfBQdbWKHDh0Sdu/e7WuL3r17t7B7926/9ujevXsLb7zxhu/rhQsXCqmpqcLbb78tfP3118Itt9wSsIW8X79+wvbt24WtW7cKvXr1MkwLebDz+uGHH4TevXsL27dv97vfd999J9hsNuH9999v9pgbNmwQXnjhBWHPnj3Cd999Jzz77LNCcnKyMGfOHN2vJxil13rw4EHh0UcfFf71r38JFRUVwttvvy10795d+PWvf+27j9hCfsMNNwhlZWVCSUmJ0LFjR8O0kCu53traWiE/P1/IyckRDh486Nd+ev78eUEQjPParl27VkhMTBRWrVol7Nu3T7jrrruE1NRUX5fbH/7wB2HmzJm+4z///HOhRYsWwpIlS4T9+/cLc+fODdhCHur3OFqUXu/ChQuFhIQEYf369X6vo/jv2MmTJ4WHHnpIKC0tFSoqKoQPP/xQuOKKK4RevXpFNTgXBOXXOm/ePOGf//ynUF5eLuzcuVMYPXq0kJSUJHzzzTe+Y6z02oquvvpqYdSoUc1uN/Jre/LkSd97KgDhySefFHbv3i0cOnRIEARBmDlzpvCHP/zBd7zYQv7www8L+/fvF5YtWxawhTzYz0+umAlyxo0bJwBo9vHxxx/7jsGFOSEij8cjzJ49W0hPTxcSExOFIUOGCAcOHPB73OrqamHMmDFCmzZtBLvdLkyYMMEvcIqWUOdVUVHR7PoFQRBmzZolZGVlCW63u9ljvv/++0Jubq7Qpk0boXXr1kLfvn2F5cuXBzw2kpRe6+HDh4Vf//rXQvv27YXExEShZ8+ewsMPP+w3J0cQBOH7778XbrzxRqFVq1ZCWlqa8OCDD/q1XEeL0uv9+OOPA/6/D0CoqKgQBMFYr+3TTz8t/OIXvxASEhKEvLw8Ydu2bb7vXXvttcK4ceP8jl+3bp3wy1/+UkhISBB+9atfCe+++67f9+X8HkeTkuvt0qVLwNdx7ty5giAIQn19vXDDDTcIHTt2FFq2bCl06dJFmDRpkuI3Br0oudapU6f6jk1PTxeGDh0q7Nq1y+/xrPTaCoIgfPvttwIA4YMPPmj2WEZ+baX+jRGvb9y4ccK1117b7D65ublCQkKC0L17d7/3XlGwn59cNkGIYv8vERERkU44J4eIiIgsiUEOERERWRKDHCIiIrIkBjlERERkSQxyiIiIyJIY5BAREZElMcghIiIiS2KQQ0RERJbEIIeIiIgsiUEOERERWRKDHCIiIrIkBjlERERkSf8flinbAqfPujsAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"np.random.seed(42)\n",
"dataset = list(zip(np.random.randn(50,2)*0.2 + [0.5, 0.5], np.ones(50))) \\\n",
" + list(zip(np.random.randn(50,2)*0.2 + [-0.5, 0.5], np.zeros(50)))\\\n",
" + list(zip(np.random.randn(50,2)*0.2 + [0.5, -0.5], np.zeros(50))) \\\n",
" + list(zip(np.random.randn(50,2)*0.2 + [-0.5, -0.5], np.ones(50)))\n",
"X, Y = np.array([x for (x, _) in dataset]), np.array([y for (_, y) in dataset]) \n",
"\n",
"plt.scatter(X[Y==1][:, 0],X[Y==1][:, 1])\n",
"plt.scatter(X[Y==0][:, 0],X[Y==0][:, 1])"
]
},
{
"cell_type": "code",
"execution_count": 139,
"id": "77212760",
"metadata": {},
"outputs": [],
"source": [
"model = MyTwoLayerNetwork(2, 4, 1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5e383715",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The loss for the epoch 0 is 0.9617411191954615\n",
"The loss for the epoch 1 is 0.9292992151788343\n",
"The loss for the epoch 2 is 0.8846996706346144\n",
"The loss for the epoch 3 is 0.82486640029748\n",
"The loss for the epoch 4 is 0.774978622862398\n",
"The loss for the epoch 5 is 0.7349209583470298\n",
"The loss for the epoch 6 is 0.6969667990769776\n",
"The loss for the epoch 7 is 0.6579152527209665\n",
"The loss for the epoch 8 is 0.6194708822398083\n",
"The loss for the epoch 9 is 0.587391692679086\n",
"The loss for the epoch 10 is 0.558362708152681\n",
"The loss for the epoch 11 is 0.5309317122115291\n",
"The loss for the epoch 12 is 0.5025536046225325\n",
"The loss for the epoch 13 is 0.47526993856452415\n",
"The loss for the epoch 14 is 0.44837597523622863\n",
"The loss for the epoch 15 is 0.4213699380091326\n",
"The loss for the epoch 16 is 0.3946085786657498\n",
"The loss for the epoch 17 is 0.3674278447636486\n",
"The loss for the epoch 18 is 0.340176283992231\n",
"The loss for the epoch 19 is 0.3126937007008613\n",
"The loss for the epoch 20 is 0.2841036569230373\n",
"The loss for the epoch 21 is 0.25703186346791435\n",
"The loss for the epoch 22 is 0.2280447275582623\n",
"The loss for the epoch 23 is 0.2002222576269299\n",
"The loss for the epoch 24 is 0.18587532016917524\n",
"The loss for the epoch 25 is 0.18228814316310257\n",
"The loss for the epoch 26 is 0.17776939893175647\n",
"The loss for the epoch 27 is 0.17377607127251568\n",
"The loss for the epoch 28 is 0.17116504536816776\n",
"The loss for the epoch 29 is 0.16822492694298327\n"
]
}
],
"source": [
"n_epoch = 30\n",
"w = [0.1, 0.1]\n",
"b = 0.1\n",
"learning_rate = 1e-3\n",
"parameter_list = model.parameters()\n",
"for epoch in range(n_epoch):\n",
" loss_values = []\n",
" shuffled_index = np.arange(len(X))\n",
" np.random.shuffle(shuffled_index)\n",
" \n",
" for i in shuffled_index:\n",
" x, y = X[i], Y[i]\n",
" pred = \n",
" loss =\n",
" loss.backward()\n",
" loss_values.append(loss.value)\n",
"\n",
" update_weight(TODO)\n",
" zero_grad(TODO)\n",
"\n",
" #print(pred)\n",
" print(f'The loss for the epoch {epoch} is {np.mean(loss_values)}')\n",
" loss_values = []"
]
},
{
"cell_type": "markdown",
"id": "816165e9",
"metadata": {},
"source": [
"### Evaluate accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "96202711",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.96"
]
},
"execution_count": 142,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# test \n",
"score = 0\n",
"for i in shuffled_index:\n",
" x, y = X[i], Y[i]\n",
" pred = model.forward([ComputationGraphNode(x_val) for x_val in x])\n",
" score += 1 if (pred[0].value * (2 * y - 1) > 0) else 0\n",
"score/len( X)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}