<jupyter_start><jupyter_text>Project 4: Question Answering on Squad CS4740/5740 Fall 2020 Project Submission Due: Monday, December 21, 11:59pmPlease submit a **pdf file** of this notebook on **Gradescope**, and **ipynb** on **CMS**. For instructions on generating pdf and ipynb files, please refer to the pinned Piazza post @695. Name: <NAME>Netid: etl43 This assignment is to be done **on your own**, i.e., without any teammate.  Notes---1. Please read through the entire notebook before you start coding. That might inform your code structure.2. Google colab does **not** provide good synchronization; we do not recommend multiple people to work on the same notebook at the same time.  Overview---In this project you will develop NLP models for question-answering tasks. Specifically, you will train models on the SQuAD 2.0 dataset which contains questions on text segments from Wikipedia. The dataset aims to measure the reading comprehension capabilities of machines. Some examples from the main paper are given below. For Project 4, the task is divided into two parts. First, you will decide on a model by yourself and train it on this task. Secondly, you will fine-tune a pretrained model.  ![Screen Shot 2020-11-16 at 12.38.55 PM.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABHwAAAU0CAYAAACq9ACaAAAKJGlDQ1BJQ0MgUHJvZmlsZQAASImFVgdUU9kWve+lN0oSQodQQ5HeAkjvRUE6iEoIEGoMgQAiKiLiCIwoIiJgQ0dFFBwLIGNBRLENCirYJ8ggoHwHCzbQ/BeYGR3/X/+frJOz38495+5331srGwDSZo5AkA7LAZDBzxaG+Lgzo6JjmLjfARHIAgVgBCw53CyBW3BwIEDir/rPeDcAIGm9bSKd9Z+//8+gJiRmcQGAY<KEY>//<KEY>EEEEAAAQRqUoCAT01eNgqNAAIIIIAAAggggAACCCCAAAIIxAsQ8Im3YQ8CCCCAAAIIIIAAAggggAACCCBQkwIEfGryslFoBBBAAAEEEEAAAQQQQAABBBBAIF6AgE+8DXsQQAABBBBAAAEEEEAAAQQQQACBmhQg4FOTl41CI4AAAggggAACCCCAAAIIIIAAAvECBHzibdiDAAIIIIAAAggggAACCCCAAAII1KQAAZ+avGwUGgEEEEAAAQQQQAABBBBAAAEEEIgXIOATb8MeBBBAAAEEEEAAAQQQQAABBBBAoCYFCPjU5GWj0AgggAACCCCAAAIIIIAAAggggEC8AAGfeBv2IIAAAggggAACCCCAAAIIIIAAAjUpQMCnJi8bhUYAAQQQQAABBOqngOd58uOPP0qbNm2kRYsWNY+wdOlSmTp1ap2pT81fECqAAAII1CEBAj516GIur6r84x//kEceeSQ4fe/eveWiiy4K1pMuPPHEE/LMM88EyTfccEM55ZRTgvWkC++8847ccsstQfKDDjpIdtttt2D9pptuknfffTdYv+eee6RBgwbBOgsIIIBAEoEFCxbIJ598EiTVz5H11ltPmjVrFmyr5MLyPn8l61Yo7/pa70Im9WHfzJkz5fLLL5e3337b/7ubNWuWX+1WrVrJWmutJQ899JD06dMnU4oLLrhAxo0bF+R56623SvPmzYN1dyFN2gkTJvjlffXVV2Xs2LGi6wsXLvSz69ixo3Tv3l222WYbGTRokPTv3989DcsIIIAAAgikEiDgk4qLxPkErrzySjnzzDODXU2aNPF/eVthhRWCbUkWtt56a3nrrbeCpPqlZ8qUKdKoUaNgW5KFY489Vm677bYg6WuvvSbbbbddsL7GGmvI119/HawvWbJEGjZsGKyzgAACCCQRGDNmjPTt2zeU9I477pCBAweGtlVqZXmfv1L1KpZvfa13MZe6vP/ll1+Wo48+WiZOnBhbzQ8//FA22mij2P2l7Nh4441l2LBhwaEadNJWRfmmJGknTZok559/vugPTdqqp9ikQeQDDjjATx8XaCqWB/sRQAABBOq3AAGf+n39M6m9Bk80iOJO2uLnwAMPdDcVXP7pp5+kc+fOosEXd9IA0JZbbuluKriszbz1l7HvvvvOT9e+fXv54YcfpHHjxsFxBHwCChYQQKAMgXyBh9tvv12OOuqoMnJNfujyPn/ykmabsr7WO1vF2slNW+RuscUWRQMktRDw0ZbLH3/8cWr87bffXrQ1NUGf1HQcgAACCNR7AQI+9f4WyAZAmxyPHDkyyOywww6Te++9N1gvtvDggw/KIYcckpPs7LPP9ptw5+yI2fDRRx+FfuHT17k0b3ci4ONqsIwAAqUKLO/Aw/I+f6lu5R5XX+tdrlstHq+v722wwQYyfPjwoPj6/eLQQw+VHj16+P3evP7666KvcmurmZVWWilIl8VCklY79jxJ0p5zzjlyxRVX2EPkV7/6lWy77bay9tpry+qrr+7/OPXZZ5/535/0dS93Gjp0qJxwwgnuJpYRQAABBBAoLmBaRDAhULbA6aef7pm7LfivU6dOnmmtkzhfE5gJjnXzWXPNNRPnoQlN30GhfEywJ+d486UqlCZNOXMyYwMCCNTpce3vAAA5BElEQVRbgfHjx4c+S/Szy7TwWWYey/v8y6yikRPV13pHGOrFqukzJ/Q3dskllyzTeptXxELnN690xZ4/SdrPP//cz8+0RPZMn0OxeZnXvTzTUjB0btM/UWx6diCAAAIIIBAnQAuf4jExUiQQePPNN/0OBt2k2rGiNsMuNi1evNh/nevnn3/2k7Zr106mT58eHPbNN98k7ohx0003lQ8++MA/Vl/j0lE8ND93qrUWPv/+97+Dvo06dOgg2kcREwIIVIeAvnbqfl5pp809e/ZcZoVLe/5q+zwptTxp673MLshyPlGpnsu52LGn/8Mf/iDaUbJO2teevv6dtn/A2MwT7EjSasdmkzTtAw88IPvss4+0bNnSHpp3rq+md+vWLbRvxowZ0rZt29A2VhBAAAEEECgkQMCnkA77Egto0EabUuuXMTtp0+XLLrvMrsbO3WCRvp+ur3G5o3xdd911ctJJJ8Ueb3doB89du3YVE930N2lHzdphc3SqtYCP1uNf//qXX41+/frJiBEjolViHQEEEEgkUG2fJ9VWnkSIVZyornm6P+KYFr/y5ZdfLlP9pEEcLVSatEkrod9pJk+eHCQ3LYT817+CDSwggAACCCBQRICATxEgdicX0Hfq9ZcrO+kwqV988YVdjZ2fddZZ8te//tXfv+OOO8qQIUNk8803D9LvtNNOoiN0FJvuuuuu0Og411xzTd5h3Qn4FJNkPwII1FWBagsIVFt5av261zVP/SFJB17QSfvy0X76luWUJoiTJm2SOugPaeb1eNFWPXb69ttvc1r92H3MEUAAAQQQyCdAwCefCttKEnj00UdlwIABoWPHjh0rq6yySmhbdEV/tbMdMv7lL3+RU089VXRIdh3+VKemTZv6HTPGDYVq89tvv/3kySeftKsyevTonCGTdScBn4CIBQQQqGcC1RYQqLby1PrtUNc8dfROfTVbp/oW8LnhhhvkxBNPDG7JFi1ayJw5c0SHamdCAAEEEEAgqQABn6RSpCsqoL9CrbjiirJo0aIg7fXXXx/6whLs+O+C9s+z6qqrBps/+eQT0T4w9tprL3n22WeD7Y8//rhoQCduWrhwoR8kmj17tp9EgzpfffVV3uQEfPKysBEBBOqBQLUFBKqtPLV+C9Q1z/oa8HnsscdERxk1g0oEt2S+UUeDnSwggAACCCAQI0DAJwaGzaUJ6OtX//znP4ODd955Z3nppZeC9eiCBoROPvlkf7O+q66dFOqkw4+6/fYcccQRcvfdd/v78v3vlVdeET2XnQoN5x4X8NG+f7TDy//85z+izab11zT9RXHDDTf0g1JJf1XTXyM1WKUtjDSgpf9pEEyHkNW8jjzyyLwtj2zZNXA2bdo0uyoHH3ywvPfee/567969Q742kXbmHO2c2u4rd64uen4NumlLrEmTJokG2Fq3bu3326R10jL279+/4KnKdXEznzhxYiiwqJ3kaifdcdOECRNEm8frpP1ErbzyynFJ5euvv/bvWe03SYd/nj9/vt9JpgYz9bhddtlFdtttN7/+sZmYHdoJuX5p1/tfzz9r1izfS5v9H3LIIf69UOh4uy+r8tj8ks71b1HrrlOrVq1Cwx1rnxLaKbu+sjl16lQ/jf797b777tKoUSN/Pd///vGPfwT9Uej9Gg3ivvvuu/Lxxx/7fzunnXZa3lcXsnK15Sv3/ta/Bx06WifthLVLly4269C8Ep56gmLnz/rzpNy/46zKU6zeIXxnRT8HtG+3p556SkaOHOnfj9oJsPaPpq8h6781GmQoNr3//vui/anopJ+FtnWr3k9vvPGGv08/+/XzppR/R4qd3+7PytPmZ+dZOdn8ks7N6FQybty4IPkmm2wS9A1oRrbybYOdZkG3aSvg6FTufWrzS/OaVpq0Nv988+eff1723nvv0L9x+n3kww8/FG0RzYQAAggggEAqAfPlhAmBzARMB8uhYUTNFzHPPOjG5m8CREF6EwgJ0pmOiYPt5ob2zMN2wWHeTXAolN48jAZ5RRfyDctuvkh5q622WigPPa/9b6uttvJMECaaVWjdtE7yfv/733vmC35wnD3enZvAkWceZj0TNAkdb1euuuqqgse7edllE8Syh2c6Nw/2nvnFuGh5tE7mC6pngmU558/Kxc04eg3Ng5W7O2dZh7O1Vjo3D985acwvqd7gwYM9E7AIpXWPs8t6jffcc0/PNK/PyUc3mD6nPNP3QsF8DjvsMM8EL/IerxuzLE/sSQrs2HrrrYPya321rC+88IJnAjuxRmZEGe/OO++MzdXNU6+hnXQoc72Hra/O77vvPrs7mGfhGmRmFrK4v92hmPXvwLQydE8RLLt1z8pTMy92/qw+T7L6O86qPMXqHcA7CybA45mH8tB95t5zuqzXZtCgQUU/782Dd5CPCeb7ZzEBXs8EvoPt0byT/DviFDfRYlae7smydHLzTbJsBn6I9Yt66rrpAzCUbVb3qc3Uvc/0fOUOy27zjZubYKFngjshA/0eZQKUcYewHQEEEEAAgYICOqIREwKZCZgWEaEvKvoF6Yknnsibv35x0i8y9kvcww8/HErXt2/fYJ+meeedd0L73RX3gb5YcCgaLDAth4oGafT85hfggg/o5lfhUHltveLmpjWEZ37NdKvhL5fyBd60jsrJp9wNr7/+umdaK6Sqk/mlO+e0Wbm4GUevYdqAj2kh5GbnB1dMh+Gp6qoPhuZX8Jx8zAhznhk+OFFepnNyb968eaE8dEWDPVmUJyfjFBvcAIXew1tuuWWiOmndTWuwvGdy89RraFoHefp3kO9vxA34qEcWrm6hsrq/kz4QunXPylPrU+z8WX2eZPV3nFV5itXbvda6rIFI/XzKd6/l26af96NGjYpmE6y7n0Ea8DF9zyXKu9i/I8EJEi5k5WlPl7WTzTfpPG3AxwzbHso6q/vUZprmPkuT1ubvzvXfAtNaNXQfNWvWzDMtI91kLCOAAAIIIJBKgIBPKi4SJxFYe+21Q19YBg4cmPcwfSi0X7T1ITHagsZ0Vhjs13R/+tOf8uZj+uoJpTNN8vOmsxvdL+r2/Hbeq1cvTx/MzCtSoTztfvMKms0mZ+62UDCvwHjmtR//IeDmm2/2zIhh3rbbbpuT5+WXX56Tj3mdzdPj7X/aasCeX+d2uzvX4FiWk3klLW8rDg2mmf6VvDPOOMM79thjPdPcPpTODKGbU4ysXNyMo9ew3IDP/fffHzLWVj7HHHOM38pEf7W/5ZZbvFNOOcUz/UsF6bSlQHS6+OKLg/32mukv/novmw7JPfMql9ekSZNQGm1NEJ2yKk803zTr0QCFrY/OzetYnul7KzawpcGwDz74IOd0bp76YKP3i5uvu+wGgLNytQXK8v5O+pDn1t2tZzmeWp9i58/q8ySrv+OsylOs3vZa61w/t6PmZmAAz7yK6l1yySXe6aef7m2zzTY5afTzLhoctvlGP4Pc/Mv5d8Tmn3SelaeerxJOSeth02lLQvffNtdVl919uqyfre6U1X1q80xzn6VJa/N353/+859D96B+jr744otuEpYRQAABBBBILUDAJzUZBxQTOPfcc0NfWrSVSL6WLPr6k/0yt9lmm+Vkq6+P2P06X3/99XPS6AZ9kHbTxbUosgfn+6Ju+nDwTP89Nok/1y9a0V+E48qgB+gDnbZY0i+gU6ZMCeWlK9pK4YQTTgiVVR8oTD8pOWndDWaUs+AY/XW40pP+ymj6CgrOqbZmaFwv7jU5ff3uwAMP9DQwpUGg6FQJl+g1LDfgo4EY9x565plnotUI1vW1NdP3jKevEbrT+PHjPdOHS5CPepx33nk5r+5pOrf8es9EHyqzKI9btlKW8wUo1llnHc/0SRRkZ/oP8YNibdu2DeptHffZZ58gnV3Il6dNrw8355xzjvfqq6/6LX/sMVm6ap5Z399JH/Ly1b1cT61P0vNrWp1K/TypxN9xOeVJWm/Tf1bO5/ihhx7qmT5efA/3f/qZH21hscMOO+T998v9G7b3cBb/jrjlKWW51OtbKadS6uAeo/9GWt/27du7u/IuZ32fJr3PtDBp0kYLr6++R79v3HjjjdFkrCOAAAIIIJBagIBPajIOKCagD8T2C5qdDxs2LHSYBoDc14W075TopIEQ/QXP5qFz01FnNJmn/SLYNNr8uVCfQXpwvi/qpoPenHx1wyOPPBLkrefQ/OMmfcfeDEMft9vfrn2+aPDEllfnzz33XMFjSv0CXzDTAjuHDBkSKp++LlcsoKLZ6TXWvpCiUyVcotewWPncV/7UPBpg0YCjvSbaAkeDc2knDQLZPHQe1yJN89Xzmw6gg/QXXHBB6HRZlCeUYQkr0QCF/p1Nnz49b04a9NMWP279TSfanun8PJQ+mqdNr315mU7OQ2ntSpaummfW93fSh7xo3bPw1PokPb+m1anUz5NK/B2XU56k9d5jjz1C9+VRRx3lO8T9T/v30sCNvTd1fu+99+Ykj34Gabos/h3JOVHKDaVe30o5pSx+TvK0AZ+s79Ok95kWPE3aaEW1Jal7z2nfefl+KIsexzoCCCCAAALFBAj4FBNif2oBfViOvkevzebdyYzEE/pyky9QoOm1I2D3S9Btt93mZuPp+/5uJ7tmBKXQ/nwr0S/qW2yxRb5k/jZtDRDtQDHuoTc2k8gOfcXNrdPVV18dSRFeLfULfDiXZGvanN5tpaLlfPrpp5MdXGaqNC7Ra1huwEd/xXeviRklJVVt9Pzu8doSLNq/TzRD9zWl6D1Ybnmi5yplPRqgyNcht5uv9onjGuiy1tGdonlqGjPKW04rKHtM1q6VuL+TPuRF656FpzolPb81rfTnSZq/Yy1TqeVJUm8zYmTontS+zvQeKDbpa7juvRz9+9Tjo59B+dLY81Ti3xGbd3ReimclnaLlS7ueNuCTNP+k92mS+8yeM01ae4ydR1t1xn0nsumZI4AAAgggkFSAgE9SKdKlEoh+mdJfs93psssuC75Q65fwuF+y7rjjjiCdfgGPviai/Xy4X8yTNIGOflG/4YYb3KLlLGuHnO45oq1Dcg4osuHSSy8N5af9RxSaSvkCXyi/QvvMcMWhsukrZKW0dil0jrh9aVyi17DcgI++euVeY33NSluW6K+uGlQsNulrhO7xJ598sqd9SxX6TzvitMfo+dy/gXLLU6y8SfanDVBonvr3aeukczV0p2ie2oJPWwfFTVm7VuL+TvqQF617sYCPmhTz1DRJz69pdar050mav+NyypOk3tF+4KI/PPggef6nf4vRHy2iLdCin0HL+t+RPMX2N5VyfSvpFFfOpNsrFfBJep8muc9sXdKktcfYuf5baz879VVPJgQQQAABBLISIOCTlST5hAS0WbX98qJzfU3GfdXK7SDz6KOPDh3rruiwyW6nxW3atPEWLFgQJNEOmt3zaD8ExaboF3X9dbPQVOx1oELH5tunDwZumXV0l0JTKV/gC+VXaN91110XKluh15IK5VPKvjQu0WtYbsBHh/LVV5Dc6+Iu6z2g/X48+uijocCMreeFF14Ye6ybT6Hl77//3mbnlVueIKMyFkoJUEQDsNGOraN5Fvrb16Jn7VqJ+zvpQ1607kkCPsU81Sjp+TWtTpX+PEnzd1xOeZLUW1vauX9z2ll30klfqXGPffLJJ0OHRj+DlvW/I6HCOCulXN9KOjlFK2mxUgGfpPdpkvvMVixNWnuMzrU1qAb97f2m/SAyIYAAAgggkJUAAZ+sJMknJDB79uycoc5tXzU6HLs7UlGxV4Z+9atfBV+E9AuR/WKtv8K6/QAV6lDZLVz0i/qXX37p7s5ZLjXgo0NOawuFa6+91jvzzDP9EZr0IaJ79+6h+lRTwEcfwO2XTp3ffvvtOR7lbsjCJXoNyw34aJ3013+37nHL+qU+2oF1tCVG3LGFtkfvw3LKU+410uNLCVB8/vnnIcNOnTqFihLNU0c/KzRl7VqJ+zvpQ1607kkCPsU81S7p+a1zKQEBe6w7z+LvWPMrtTxJ6h3tL63Y54Rbv+OPPz50L0dfvY1+BkX/ft28dLnUf0ei+RRbL8Wzkk7Fyltsf7kBn3Lv0yT3ma1DmrT2GJ3rD1vuvw233nqru5tlBBBAAAEEyhIg4FMWHwcXEvjNb34T+hJjRzVyX2XRkXk0OFRocvs60S9FOiS4TtF+gM4///xC2QT7ol/Uiz0EpPmirq976VDrW265ZahvIffLXHS5mgI+2geSW77HH388cCtnIWuXSl1DDXBFO2x1PeyyBizdUd3cjsM1jfZLk+Y/fVDQL/3RqdTyRPMpZb2UAIX2j2KNdK79a7lTNM98Q7e76bN2rcT9nfQhL1r3JAGfYp5qlfT81rWUgIA9Nuu/Y8231PIkqbfbakLvR+2QOemkr9q697K+huxOlfoMcs9RynIpnpV0KqUO7jGlBHyyvE+T3Ge2vGnS2mN0Pm7cuNC9lqYlmpsPywgggAACCOQTIOCTT4VtmQjor/fuF+Y11ljDz1cDNnb7brvtVvRc+nqLTa9z25LnqquuCm0v9vBoT1SpL+oPPvhg0WCB+3qarVM1BXyir8hdeeWVlq3keSVcKnUNtZI61Lj+wnrAAQfkDNFsr5nOdZStKVOm+C4HHXRQ6F5M0u9PUtBSypM070LpSglQvPfeeyEHHbnLnaJ55gtyuemzdq3E/Z30IS9a9yQBn2KeapX0/Na1lICAHluJv2PNt9TyJKl3t27dQvdj3ChaWo7o9Nvf/jZ0rAby3amSn0HuedIul+JZSae05Y+mTxvwyfo+TXKf2TKnSWuP0bn+6OT+2/L3v//d3c0yAggggAACZQkQ8CmLj4MLCeivbO6XGF3Wfkq05YPdnqSTZT2H23GyBk1+/PFHz21BpA/fbqe3hcpViS/qp512WlAnWzdtvbTnnnv6wQN90NBf8RYtWuQl7TvA1qGUL/D22LRzHR7cll/nxxxzTNosQukr5VKJaxgquLPyww8/eNp5+M477xyyUZ9rrrnGT6l9Lrhu9rVDJ5vMFpOUJ4uTlRKg0ECZ66D3vztF8ywW8MnaNev7W+uW9CEvWvckAZ9inmnOb69DKZ8nlfo71jKVUp6k9daRs9z7sdgrhNZI5xqsdI+977773N05o3Rl2VI0dKKUK6V4VtIpZfFzkqcJ+FTiPk36960FT5PWrejChQv9/sq0s3/9r9jrge6xLCOAAAIIIFBMgIBPMSH2lyXgBnf0y7MOq96wYcPgi/T48eMT5X/ccccFx2g++ite69atg23HHntsonw0UdbBgq+//jrUsbQ2j9fOYd1Oqt3CXX/99UG5tS5pWvj07dvXzSrzZe1nyX3I6dixo1fqMPSVdElzDbVDTB0Nyq2XBiNLmYYOHRrK58ADD/Sz0X6o3Pzt9lLOkeaYuPKkySMubdoAhY7mtummm4YczjnnnFD20TyLBXyyds3y/rYVS/qQF617sYBPEk8tQ9Lz2/K6AYEknyeV/DvWMqUtj61Hknrra8Tu3+Wvf/3rRD8M6LWJtsaMDgiQ5jNIy5zm1WBbx1LmpXhW0qmUOrjHJA34VOo+TXKf2fKmSWuPYY4AAggggEClBQj4VFq4nuc/ePDg0BfutdZaK1hPM/To888/HxynX+A333zz0LrtEDoJd9Zf1O+///5QWfRXxkLTWWedFUqfJuDToUOHQlmXvU+DI127dg2VL82IIXq8nSrpEr2G0SGTbRl0Hn0lUO+fUgM++kus+wC52Wab+afSlltuB+Ka5uWXX3aLUZHluPJkcbJogOKVV14pmO3NN98cslEDHdbenaJ5Fgv4ZO2a5f1t65X0IS9a9yw8tQxJz2/L6wYEknyeVPLvWMuUtjy2Hknq/emnn+bck1qfQpMG2qJ5a6AoOkU/g6qxhU+S66v1qqRT1C3tetKAT6Xu0+i9oINOxE1p0sblwXYEEEAAAQSyFiDgk7Uo+YUEhg0blvOF2z4wpwkkzJs3L6eVhs2nZcuW3ty5c0PnLbSS9Rd17WDRlkXn2k9I3KRfFvVXdTd9sYDPuuuuG0qf78EiyybgQ4YMCZ2vWbNm3l133RVXJX+7PrhroKtdu3ZBukq6aLDQNdT7LG465JBDQmn1uGjAR1uiaJ8dxfqBivapsvfeewenjQY3dTS2999/P9ifb0HvB+0cdocddgjtzqI8oQxLWIkGKNZee21vzpw5eXPSPifatm0bctZrpA/P7hTNs1jAR4/N0lXzy+r+1rx0SvqQF617Fp5pzu8X1vwv7edJJf+OtUxpy2PrkdR92223Dd2XnTt39t544w2bTWi+YMEC78QTTwyl1w7aNWgcnbL+dySaf6nrpXpWyqnUetjjkgZ8KnWfJr3PtLxp0tr66Vw/J7VDcfufdtbOhAACCCCAQFYCBHyykiSfvALar050GHL7oO6OcpT34MhGfbi2x7rzaD8hkcNyVrP+oj5t2rSc5v86DLv2M2QnDVg9/PDDOa+TaT2KBXy23377UL33228//8FbO/PVvmLs/jQj0Nhy5Ztrq5HoF1ct58CBA/0HJX3o1wejr776ynvyySc9fR2gRYsWQRntl9VKumhn3+49oEGT6KTD8UaHVrbHRAM+GnCx+zTvBx54wPvoo4+CQKKa6OhybmsETX/RRRcFp1WTqJu+3nfFFVd47usg8+fP90eY05Ze+vCp+ay22mpBPrqQRXlCGZawEg1QaDn1Fc133nnHsy25tGWV2kdff9G0L730Us5Zo3kmCfhk6aoFyur+tpWLXvO4FgDRumfhqWVIen5bXvt5oefX/4p9nlTy71jLlLY8th5J6z127NicYKS+Vnz22Wf7f+N6P2gdX3jhBX9AAOti5xpwzDdl/e9IvnOUsq1Uz0o5lVIH95ikAZ9K3adJ7zMtc5q0bh11cAR7v+n88MMPd3ezjAACCCCAQFkCBHzK4uPgJAJ//OMfQ19m9AtNp06dcn79L5aXDlHtfimyy9ovUJqpEl/UdeQwWx4711+G9UF+zTXXjG2dpGmLBXwuvPDCnLz1ATv6kH3vvfemYSiYdsSIETmvKNl66Xl1uG27Hp2PHDkyyLtSLvkCOZtssomnfTnpPm2toy2/omWz69GAjw282P12bo3dgJbd16ZNm1AgRys9atSoIIhj09m5vvLVs2fPUB9Wdp8OBe9OWZXHzTPtcr4AhS2v1l37d7Lr0fk+++yT93TRPJMEfDSjrFxtobK6vzW/pA950bq7ZqV6pjm/rXspnyeV+jvWMpVSnrT11tYf2om+a55kecstt/QDhNbOnVfi3xE3/1KXS/XU81XCqdR62OOSBnw0fSXu06R/33r+NGk1vZ0uu+yy0L1JwMfKMEcAAQQQyEKAgE8WiuRRUCDa/45+0S7lC42O8GUfwO2XdV3X7WmmSnxRf+2113LKZstYbF4s4PPZZ595jRs3Dn0hzJenBjqynDQossEGGxQ9r1uWrbbaynMf4ivl8sUXX+QNnLhlscvaKsV9aNDtbsBHW6ust956qeqpry9pS5d8k7bm2XjjjRPnp6/MHX300UFWWZcnyDjlQjRAoQ8z0b8/a+zOBw0aFPuQHM3TvVeKFa9c12j+WdzfmmfSh7xo3bPwTHN+W/9SPk8q9XesZSqlPKXUWztijvaz5d637rIG6y+++GJ/VEXrFp1X4t+R6DlKWS/V054rayebb6lz97O7ffv2BbOpxH2a9O9bC5YmrVsRAj6uBssIIIAAAlkLEPDJWpT8cgTy9b/z6KOP5qRLsiE6CpCup53cL+raUmXKlCkFs3BHV9FXAeLSP/XUU34fNu6Dg13Wh3od1ls7sX3rrbdCwYAkfRlde+21eR+29Vfr/fff39PRjPTVl6wnvXY6lHH//v1DZbb10rmOgKWvhbz77rt5T18plxtvvDG2FY+2yNlll108vc+049999903KH/cNdRXuE444QRPOzp16+cu6ytamlehPoMUQd20fIXctJXb+eefHwqQuYBZlsfNN+lyNEChD4Kvv/66py2p8rXw6t27t6et8ApN2vLKeup10Nfu0kxZuLrny+L+dl+h0cBsXD9HlfDUuiQ9v1vvUj5PKvV3rOUqpTyl1Hv27Nne1Vdf7XXr1i24D+39qHNtabXTTjvl7bPH9dPlSv07Ej1PKeuleLrnydLJzbeUZTfgowG7YlPW92ma+yxNWrce11xzTeh+LOUHMTc/lhFAAAEEEHAFGuiK+aLDhAACGQiY4IKYB3Uxnf+KLpvXXsR00iympYyYV4zKOoNpySQvvviimC/jYvpF8v8zAQUxrU3KyjfJwfoxYYa9FdNni5i+HsQEsMS8rib9+vUT8/AkpuVHwWwq5aImaj1mzBgxgSdZddVV/XIlKVNcgU0LGzGtSWT8+PH+fyYoISYIJOZhQ8yoXP5y3LHR7epm+jry3caNGycmQCcmgOjfE+b1LjEBgughOetZlicn8wIbttlmG3nzzTeDFCbg49dfN5h+asS0cBLTWbiYUd3EBDP8+zFIXOGFLFzdIpZ7f7t5xS1Xm2cpnyeV+jtWs1LKE2ddbLv+TelnmXm1z/88M8FXMa3y/M8zE4gsdnhN7M/Cs1adKnmf1sTFp5AIIIAAAgg4AgR8HAwWEUAAAQR+ESgUoMAovQCe6c04AgEEEEAAAQQQQKA8AQI+5flxNAIIIFAnBQhQZHtZ8czWk9wQQAABBBBAAAEEigsQ8CluRAoEEECg3gkQoMj2kuOZrSe5IYAAAggggAACCBQXIOBT3IgUCCCAQL0TIECR7SXHM1tPckMAAQQQQAABBBAoLkDAp7gRKRBAAIF6J0CAIttLjme2nuSGAAIIIIAAAgggUFyAgE9xI1IggAAC9U6AAEW2lxzPbD3JDQEEEEAAAQQQQKC4AAGf4kakQAABBOqdwJ577inPPfdcUG93WPZgIwuJBfBMTEVCBBBAAAEEEEAAgYwECPhkBEk2CCCAQF0SmDVrlkyePDmoUo8ePaR58+bBOgvpBPBM50VqBBBAAAEEEEAAgfIFCPiUb0gOCCCAAAIIIIAAAggggAACCCCAQFUJEPCpqstBYRBAAAEEEEAAAQQQQAABBBBAAIHyBQj4lG9IDggggAACCCCAAAIIIIAAAggggEBVCRDwqarLQWEQQAABBBBAAAEEEEAAAQQQQACB8gUI+JRvSA4IIIAAAggggAACCCCAAAIIIIBAVQkQ8Kmqy0FhEEAAAQQQQAABBBBAAAEEEEAAgfIFCPiUb0gOCCCAAAIIIIAAAggggAACCCCAQFUJEPCpqstBYRBAAAEEEEAAAQQQQAABBBBAAIHyBQj4lG9IDggggAACCCCAAAIIIIAAAggggEBVCRDwqarLQWEQQAABBBBAAAEEEEAAAQQQQACB8gUI+JRvSA4IIIAAAggggAACCCCAAAIIIIBAVQkQ8Kmqy0FhEEAAAQQQQAABBBBAAAEEEEAAgfIFCPiUb0gOCGQq8NMrb8uMf74ti3/86b/5NpAmnTvKSn84WFr07p7pucgMAQQQQAABBBBAAAEEEECgbgoQ8Kmb15Va1bDAV7scIt6CBTk1aL3tVtJr8Mk529mAAAIIIIAAAggggAACCCCAQFSAgE9UhHUElqPA0oWLZPjOB+UtQfN11pK+Qy/Ku4+NCCCAAAIIIIAAAggggAACCLgCBHxcDZYRWM4CBQM+a60pfW8cspxLyOkRQAABBBBAAAEEEEAAAQRqQYCATy1cJcpYbwSWLlosw3f6Xd76Nl9vHel73YU5+xbPmC2zPx0uS/U1sEaNpEWfntJilW456diAAAIIIIAAAggggAACCCBQfwQI+NSfa01Na0Rg1O/PlIVjx+aUtu1vdpYeZx0b2j7r4y9lwmmDRTzvf9sbNJDet18lLfv2/N82lhBAAAEEEEAAAQQQQAABBOqVAAGfenW5qWwtCMwZMVZmvP4fWTJnnl/cBiaA03iljtJp752lcasWoSpMvv9pmXb7/aFturLiiUdL5/12zdnOBgQQQAABBBBAAAEEEEAAgfohQMCnflxnallHBSbf9bhMu+fhnNp1HHiIdDl8n5ztbEAAAQQQQAABBBBAAAEEEKgfAgR86sd1ppZ1VICATx29sFQLAQQQQAABBBBAAAEEEChTgIBPmYAcXlhg4bTpsmDCpOD1pMYrtJHG7VeQpp3aS8PmzXIOXjx9pswbN0mWzl8gbTZYUxo2axpKo/uXLl1qtjfLeb1JEy76eYbM+2aCf4we38B0Ypxv0vxnffyVLJoyVRbPmCVLZs4x/eCYfFs2l0amjK3WXV1arraKNGjYMN/hVbONgE/VXAoKggACCCCAAAIIIIAAAghUlQABn6q6HHWjMPO/myLTHn1B5gz7VBZNnJi/UiaQ0rxfP2m3907SZqN1ZNozr8iM51+VJT/9FKRf6czjpdPu2/nrP736jvx40z2yeNq0YH+z1VaTVW+7zF9fMPlH+e6aO2TuBx+JiQj523oN/bO0Xqd/kH7hT9Nl+mvvyux3PpR5n39pokOLgn35Fhq0aClttttSuhwzQJqYIJU7LTXHTrhoqCwcZ+q3dIk0bNFCWqy7pnQddLA0bNLETZqz/N1tD8vcdz+WpfNMHz0NG0lTM6pWr8EnB8GluSPHyY8PPSsLRo35pS4maNWsTy/pfvYfpZEJSC2eOVu+vfI2WfjNOFk6Y6YsmT0r5xyN2rSVhm1aB9sbNG8hrTZZV1Y+7hB/24Q/3yTzR4wWWbxYxJS3ac/u0uW4g6X5yisFx+RbmPzAMzLLXAtvvim7lqvvKtLz/BNiA2v58mAbAggggAACCCCAAAIIIIBA5QUI+FTeuF6dYerfX5cp198qsrBwMCUJijsq1ZgzLpV5H5pgTmTqe99QEwCZJd+ee4VppTMjtLfHXy+QtibIodP8id/LNwNPKxrkCWXw35VG7TvIKkOHSPPuXYLdc74eI+P+eFawbhdW/MMR0vmg39rVnPlPr/1Hvh9yVc72PnddKy16d/e3jzvvKpnz9n9y0nT787nSbouN5MenXpYfrjPGJUx97r7OD16N2OvInKObdO8uq951pQlYNc7Zpxviztv7zmukZZ8eeY9hIwIIIIAAAggggAACCCCAwPIRIOCzfNzr5Fmn/3uYTDr/iqCFTepKNjYtYxb/L1DUfN21pe/1g/1svjnhQpn/hWmVE5lWvvAMmXzlLbJ0zuzIHhE34DPzg89k4plDctIk3dCkWzdZ7f5rRUfM0klfHRu53zE5dW3cubP0f/Sm2GxH/+FPsmDEyPB+01Km/9N3SuM2rfztcXXteMxh0uWQvWTq06/IlGv/Fs4j4ZoGyJr36CojDjhOFv/4Y85RK536R+m0144523XD6GPONa2ORoX3admfukMat/1fa6JwAtYQQAABBBBAAAEEEEAAAQSWhwABn+WhXkfPOfKwU/O+wtV4pZWkzfZbSiMT0Jg/erzMfu2NvAJd/3SytF5/TZln0sz9arS0Mn3wtN1obT9tXBBE8148ZUre/HpeO0TamPx0mjtqnIw95oxQutbbbS1tfr2BtFxjVWlmWu94i5fI97c8JNOfeCaUzq50veB06bD9r+2qfHPc+TJ/+PBg3S6s8re/SKv+fexqMF/4408y6oBjg3W70GL9daXPtRfYVYmra8ejD5Uuh+4ti81w7d/feJ8s+Ga8LJ70Xd5XuqRxY/OaVq8gT+2bSF/p6nLEfv62Hx75u/x4893BfrvQuGtX6ffg9UFgy26f/+1k+ebQE+xqMG+x0QbS56r/C9ZZQAABBBBAAAEEEEAAAQQQqA4BAj7VcR1qvhSzTYBm/KBzcurRqO0Ksup914p21mynyfc+JdPufMCuBvPW228jvS44MVh3F+KCIG4aXW7YsqU0XbWPNO+/qqxkhiW3rWa0s+cRew/0kzdq3Ua6nDVI2m29SfRw8TxPRh44KG/rF/cVMz0wLmjSfsA+QV857gl+ePQfph+iu9xN/nLnk4+VFffZOdgeV1cb8AkSmgXti2fWK6+5m/zlYsOyL5k7X0bu/wdZOtd0Vh2Z7Ktj7ubv73hMfrrvEXeTv9zl7BOl427b5GxnAwIIIIAAAggggAACCCCAwPIVIOCzfP3rzNmnvfSmTL7s+pz6tNlhW79TX3dHXEuXpn16y2p3/tVNGizHBUFsgoatW0vHQ/eXTvvtEttp8qiBZ0njTh2k29l/kKYd29tDc+bjhwzN2wqpxfrrmZY45wfpF5oRvkYN+GOwbhcad+ki/R++wa4G87wtgkzn1f2euC3UKXRcXbMM+Gihvrv5Afn5kaeC8tmF5uutI32vu9Cu+vORh5rWW99GOuA2nT2v/sydpiPpFqG0rCCAAAIIIIAAAggggAACCCx/AQI+y/8a1IkSTL7nCZl210M5dWl34D7SbdAhoe3ekiXy1Q4DQtv8laZNZK2Xc/PQfXFBEN2nwZ7et/1FmnftrKtlT5OG3mte63o2J598AanRR51tXq36Jidt79uvkpar/u+VKh2eftT+ps8f04LInZqvsYb0vflid1NsXbMO+Cyc9rMJWB33y0hdoRKI9L7jamnZt6e/dd64STLmyJMjKURabbaprHJ5bsfVOQnZgAACCCCAAAIIIIAAAgggsMwFCPgsc/K6eUIdrnvabfflVC5fwGep6Stn+I55Aj6m0+a1/pk+4NPjygul7cbr5Jw7yYalCxbK/ElTZOH3P8jin2eaV5zmyqy3P5D5n32Rc3jT3qYF0l3hFkhxr6e1P3h/WfnY3wV5/PD4i/LjDbcH63ZhxeOOlM4D9rCr/jwuuJV1wEdPNuGSG2XWP18PnV9X2u68g/T4kwkGmWnyPU+aYN6D/rL7v67nnyYddtjc3cQyAggggAACCCCAAAIIIIBAlQgQ8KmSC1HrxZj+9ocy6bzLc6rRerutpNeF4dYhcR0Aa4fB/R8ampOHbogLgminzf0fuTHvMXEbZ385Sqa/9JbM+3y4LJwwQcS0OEoy5Qv4+MO9H5bb75CO6tXvgeuCbOPKv9ojt0jTlToF6XQhLm0lAj5zx0yUsQNPDZ1fVxo0a+aPvtXIdPY8+lgzstjI8Mhiun/1Z++Shs2a5hzLBgQQQAABBBBAAAEEEEAAgeUvQMBn+V+DOlECbSXzzSHH59RFX7fqe+dV0rRzx2DfpKH35R0Jq+WmG0vvv+R2/KwHxgVBWm9rAkqDwwGl4ESRhVmffCXfX327LNIgT9ykw65HXruySfMFfHTfyMNPy5tnn7uulRa9u4vfYfS+R+cM4d6sb19Z9Y4rbPbBPK6ulQj46EnHnP5nmTfs4+D8dmGl046TtltuJKN0+PmISetttpReF51ikzJHAAEEEEAAAQQQQAABBBCoMgECPlV2QWq5OKMGni0Lx+T2Z9NohXamv5eNRRo1lIWTJsv8Tz/LW81ul/5J2m2+Yd59sUGQ3x9shhrfN+8x7sa4V680TaO2bWWFPXf1h4FvvdZqMvmux/N2ZhwX8Pn+9kfkp/sfc0/nL3c4bIB0PeoA+fGpl+WH627N2R83klZsXf87LLubUamjdLl5zBz2hUw8fbC7yV9u1q+ftNtzR5ly5U05+7pdfI6028pcUyYEEEAAAQQQQAABBBBAAIGqFCDgU5WXpTYLNeuzr2XCKRfktGRJUpsWG20gfa76v9ikaYIg0Uz8183ONy1pIq1UNF3bXXeUroMOlcZtWweHxY1eFRfwmfvNBBl71GnB8XahSc9e0u/eq+Sbk4fkDXL1vW+oNO/R1SYP5mnqmkXAR0+sI5gtHDMmKINdaLHB+jLv40/sqj9v2LKV9H/mDjMaWuPQdlYQQAABBBBAAAEEEEAAAQSqR4CAT/Vci5oviY6+NWL/42TJzz+lqkurzTaRnoNPkYbNm8UelyYIEs1kzBmXyrwPP4pulnxDxmuitAEfPWbEgBNk8ZTJuhiaet14mYw/6bycfoKa9OxpgkFXh9LalTR1zSrg89Mrb8v3f77WFqHgvM1O20vP/xtUMA07EUAAAQQQQAABBBBAAAEElq8AAZ/l61+nzj7j/U/l27N+GWK81Zaby5Kffpb5w7/O27KmcefO0nLDdaXtdr+WFX61XlGHNEGQaGbD9zhSls6eHd0sPa+/RNqsu3rO9nHnXCFz3v0gZ3tcCx9NOOnG+2X6Y0/nHKMdUS/+/vuc7R0OPUC6Hp1npDKTMk1d4wI+HQ490OR/YM554zZosG7kgONl8dSpcUmC7d3/cr6ssGnxaxYcwAICCCCAAAIIIIAAAggggMAyFyDgs8zJ6+4Jv7v1Yfn5wcf9CtpWIEsXLpKFU6bKkjnz/Fe9GpjXgBp3aCdNO7ZLBZEmCBLN+KudDxZv4cLoZul146XSeq1+oe0a+Bi+x+/Fmzc3tF1XmvYxw7LfGR6W3Saa89VoGTcof4fTNo07733H1dKyb093U7Ccpq4T/3KrzHz+5eBYu7DCnr+R7qcNtKuJ5j88/Hf58Za7C6Zt1HYFM3rXrdKgUaOC6diJAAIIIIAAAggggAACCCCwfAUI+Cxf/zp19olX3SEzn3shqFPz1VeXFuuv+cvQ3Q0b/rLddNzcsLHp+6VxI/8VrmbdVpJmPbsVDQClCYIEBfjvwjcnXijzP/8yulmiI3x5po+f7256IG9LHT24UAsf3T9ivz/I4mnTdLHg1LhLF+n/8A2xadLU9fvbHzUdRj+ak1fTPn1NcOqXEcA0iPXDI/8w6Z6QTsccIivus3NOet2wZO58Gbn/sbJ0bm6wyx7Qdo9dpccZZsQxJgQQQAABBBBAAAEEEEAAgaoWIOBT1Zentgo3+Z4nZdpdD5ZU6AYtWkjLTTaUTgf8Rlqv0z8njzRBkOjBcUERTddq6y2k3a7byJIZs2T6C6/L/M++iB4erBcL+Hx79Z0y49nng/RxC+0O2Fu6HX9o3O5Ur3RNffoVmXLt3/Lm1WrLX4s0aCjzvxge9KvUfK01pe+NQ/Km142TNOD16FOx+3ted7G0WW+N2P3sQAABBBBAAAEEEEAAAQQQqA4BAj7VcR3qRClmfTpcJpx8ftl16Xj4AOky8IBQPuUEfPSVsjHHnC1LZs4I5Zl2pVjAZ9YnX/0ySlmRjFe56XJpteaqsanS1HXumIkyduCpsXlFdzRo2lTWfDk+KLdw2s8yasBxIosXRw+VRh06SP8n/iYNGjTI2ccGBBBAAAEEEEAAAQQQQACB6hIg4FNd16OmS7PY9NMzct+jxVuwoOx6dDn3ZOm4y1ZBPhMvvVlmvvxqsG4XVh58hrTfdjO7Gjuf/eVIGX/yhSaQsSg2je5o0q2bdDx8P5l82fU56VptvpmscukZOdvtBm/pUhl5yMl5O2m2aexQ7XY93zxtXeOGfc+Xd5JXsiZccqPM+ufrOYe32/e30u2kI3K2swEBBBBAAAEEEEAAAQQQQKD6BAj4VN81qckSzRv7rYw/fYgZmSvdkOxxldVRvPo9cmPQmmTmR1/KdxddY169mv7LIaZPoCY9ekifG4ZI4zat4rIJbddRxKZcf5cs+vbb0HZdadiylbQfsJd0/t0e4i1ZKmNPGiwLxo4zHdv80tKlUdu20uWsQdJuy41zjnU3zDadN39/5d9k4YSJ4VYyTZpI0x49ZeWz/yit+vd2D8lZTlvXhT9Mkwl/+qssGD06Jy/d0HillaTlphtIhz12KHpuTT/n6zEy7o9n6WJoWuXmK6TVGn1D21hBAAEEEEAAAQQQQAABBBCoTgECPtV5XWquVCMPP10WTRgfKneDps2kSffu0qht69B2b+kS8UwHwUtmzpTFP/wQ2ueurPb4rdK0Uwd3U9nL2gpnzvDRMt8EqBb/PEMatW4lLVZbxQ9k1PLIU0sXLf6lXqPHy5LZc00Aq7npDHtlabV6X2kc8S+G+PO/3pPvBodHI9OgUX8TgGNCAAEEEEAAAQQQQAABBBCoDQECPrVxnaq6lAun/iSjzOhO7qQtZvred13R0bcWz5gtU598UabdZ0aaMsEYd+p57RBpY0b5Ylq2AmPPvEzmfjAsdNL2v9tXVv7jwaFtrCCAAAIIIIAAAggggAACCFSvAAGf6r02NVMyfQVp4mmmfxxnatKrl/S75ypnS+HFUQPPkoVjxoQSrTz4TNM/z69C21iprEDc61y977xGWvbpUdmTkzsCCCCAAAIIIIAAAggggEBmAgR8MqOsvxnlHSnK9LHT48oLpe2GaxWFmTtqnIw7/v/EWxju7Ln37VdJy1V7FT2eBOkFpj7zT1nw7WRZ8cDfSNMVf3ltbvH0mTLujEtz+gIqNpR7+rNzBAIIIIAAAggggAACCCCAQKUFCPhUWrge5O+PTnXAIFk8bWpObXVkq3a7byfNe3eXZit1Eu0nx1uyRBaYjoYXmH50pr/0psx+652c17kaNGsmq//9HmnYpHFOnmwoX2D0UWfLgm++MT06N5ZmfftIw1YtZcHI0bJ09uxw5uZ6rXLDpXTWHFZhDQEEEEAAAQQQQAABBBCoegECPlV/iWqjgNNefEMmXz60eGEbNBDxvKLpVvzjkf6IWUUTkqAkgW9Ovkjmf/p50WM7HHqgdD36wKLpSIAAAggggAACCCCAAAIIIFBdAgR8qut61HRpJt/9hEy795Gc1jppK9Vu39/KyiccJg3Ma2FMlREYf9H1Mvv1Nwtmrh01dz32d1yHgkrsRAABBBBAAAEEEEAAAQSqU4CAT3Vel5ot1ZwRY+THB54xrUe+lCUzpievR6PG0nLD9aTjQXsm6vcnecakzCfw3S0Pys8PP5lvlzRs3Vq6nHEcHWbn1WEjAggggAACCCCAAAIIIFAbAgR8auM61WQptTPn2cM+l0U//CRL582XpXPnmY6ZF4mY17oaNmsqjdq0Nv3H9JAWq/WWFn26m/56mtRkPWux0Itnzpbpr78rCyZ8L0tnmX57zDVpvGJ7aW46yV5hi424FrV4USkzAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oIAAZ+6cBWpAwIIIIAAAggggAACCCCAAAIIIOAIEPBxMFhEAAEEEEAAAQQQQAABBBBAAAEE6oLA/wMAAP//ep/KmwAAQABJREFU7J0H3BTF+fiH3qSDCiIqRcSG2AsI9vg3scSosSUGS2LvivmINYm9d0WN2DV2sUWxKxYEKyAooIL0Jr3tf54zs7/dvd27vd3Z97279zt8XmZ3dnbmme8zMzv73OxMPUc7hYMABCAAAQhAAAIQgAAEIAABCEAAAhCoGgL1MPhUjS4pCAQgAAEIQAACEIAABCAAAQhAAAIQyBHA4ENFgAAEIAABCEAAAhCAAAQgAAEIQAACVUYAg0+VKZTiQAACEIAABCAAAQhAAAIQgAAEIAABDD7UAQhAAAIQgAAEIAABCEAAAhCAAAQgUGUEMPhUmUIpDgQgAAEIQAACEIAABCAAAQhAAAIQwOBDHYAABCAAAQhAAAIQgAAEIAABCEAAAlVGAINPlSmU4kAAAhCAAAQgAAEIQAACEIAABCAAAQw+1AEIQAACEIAABCAAAQhAAAIQgAAEIFBlBDD4VJlCKQ4EIAABCEAAAhCAAAQgAAEIQAACEMDgQx2AAAQgAAEIQAACEIAABCAAAQhAAAJVRgCDT5UplOJAAAIQgAAEIAABCEAAAhCAAAQgAAEMPtQBCEAAAhCAAAQgAAEIQAACEIAABCBQZQQw+FSZQikOBCAAAQhAAAIQgAAEIAABCEAAAhDA4EMdgAAEIAABCEAAAhCAAAQgAAEIQAACVUYAg0+VKZTiQAACEIAABCAAAQhAAAIQgAAEIAABDD7UAQhAAAIQgAAEIAABCEAAAhCAAAQgUGUEMPhUmUIpDgQgAAEIQAACEIAABCAAAQhAAAIQwOBDHYAABCAAAQhAAAIQgAAEIAABCEAAAlVGAINPlSmU4kAAAhCAAAQgAAEIQAACEIAABCAAAQw+1AEIQAACEIAABCAAAQhAAAIQgAAEIFBlBDD4VJlCKQ4EIAABCEAAAhCAAAQgAAEIQAACEMDgQx2AAAQgAAEIQAACEIAABCAAAQhAAAJVRgCDT5UplOJAAAIQgAAEIAABCEAAAhCAAAQgAAEMPtQBCEAAAhCAAAQgAAEIQAACEIAABCBQZQQw+FSZQikOBCAAAQhAAAIQgAAEIAABCEAAAhDA4EMdgAAEIAABCEAAAhCAAAQgAAEIQAACVUYAg0+VKZTiQAACEIAABCAAAQhAAAIQgAAEIAABDD7UAQhAAAIQgAAEIAABCEAAAhCAAAQgUGUEMPhUmUIpDgQgAAEIQAACEIAABCAAAQhAAAIQwOBDHYAABCAAAQhAAAIQgAAEIAABCEAAAlVGAINPlSmU4kAAAhCAAAQgAAEIQAACEIAABCAAAQw+1AEIQAACEIAABCAAAQhAAAIQgAAEIFBlBDD4VJlCKQ4EIAABCEAAAhCAAAQgAAEIQAACEMDgQx2AAAQgAAEIQAACEIAABCAAAQhAAAJVRgCDT5UplOJAAAIQgAAEIAABCEAAAhCAAAQgAAEMPtQBCEAAAhCAAAQgAAEIQAACEIAABCBQZQQw+FSZQikOBCAAAQhAAAIQgAAEIAABCEAAAhDA4EMdgAAEIAABCECgqgk4jqNmzZqlWrZsqZo1a1bVZa2LhUO/dVHrlBkCEIAABOIQwOAThxJxChIYPny4evzxx904G220kbr00kvd87gHTz31lHruuefc6FtvvbU644wz3PO4Bx988IG688473eiHH3642nfffd3z22+/XY0cOdI9f+CBB1S9evXccw5+JbB8+XI1ZswYF4cw6tOnj2rSpIkbxkH1EKBdVI8uSy1Jtbb1hQsXqiuvvFK99957ub7sl19+yaFp0aKF2myzzdSjjz6qunXrViou4pcJAfRbJorQYpRDH1IOMtSGRupqueOw/uc//6nGjx/vRr3llltU69at3XMOIFBnCOhfRXAQSEXgmmuucXSDcf8aNWrkzJ8/v+Q0+/fv76Yh6bVv395ZtWpVyekcf/zxvnRGjBjhS2OTTTbxXV+9erXvOie/Evjuu+98nEQn9957L3iqlADtokoVG6NY1djWX331VWf99dfP68O8z6pPP/00Bh2ilCMB9FteWimHPqQcZKgNrdTVcsdhveuuu/qeAdOnT49zG3EgUHUEmOGjR3+4dATGjRunevfu7UtEZvwceuihvrBCJ3PnzlVrr7220sYXX7R3331X9evXzxdW6ES3UNWlSxc1bdq0XLS2bduqmTNnqoYNG7q3iawis3GSZ/369c0p/v8IfP/996p79+4+HkOHDlXHHnusL4yT6iBAu6gOPSYpRbW1dZnBucsuu6g1a9YUxKENPmqbbbYpGIeL5UcA/ZafTsqhDykHGWpDM3W13HFYDxgwQL3zzjtuVG3wUeuss457zgEE6goBDD51RdMZl7NXr17q22+/dXM5+uij1bBhw9zzYgePPPKIOvLII/OinX/++bkp+XkXIgI+++wz3wBePueStL2OF1svjehjBhHRbKrxCu2iGrUar0zV1Nbl84a+ffuqsWPHuoWX59FRRx2l9IwfNXv2bPXmm28q+fRXPudl8O9iqogD9FueaiqHPqQcZKgN7dTVcsdhjcEnDiXi1AkCVTdniQLVCoGzzz7bN22yQ4cOTimfSmnDjO9+3fhy55tuumlJ5dFrB/nS0caevPv5dCUPSWjAlClTfCxFJ3qGT2hcAiufAO2i8nWYtATV1NbvvvtuX7/1j3/8IykW7itDAui3DJWiRSqHPqQcZKgN7dTVcsdhzSddcSgRpy4QYIZPnTDrZV9ImTIplnSvk4UyZVp9MafX6cl9zjVv3rxc1DZt2ii9BpB7m/4+OfbCmttvv7365JNPcvfKZ1yyK4uk53WVNpPh/fffV/Jpm7h27dqpE044wVucTI8lX68uZNHmrl27ZponiacjkLS+VFq7SEeJu4MEqqWt//Wvf1XaKJArnnyqK58Ls0hnUNuVe45+y1d35dCHlCJD0mdlVhpII08p5c5K/nJMlxk+5agVZKoNAhh8aoN6FeYpRhuZGi+Da+MGDx6srrjiCnMa6XuNRU2bNlXyGZd3l6+bbrpJnXbaaZH3mwszZsxQnTp1UtpSmwvabbfdlF6w2Vx2/Up7sZVyvPXWWzn5N954Y9+OA26hOIDA/wgkrS+V1i5QOATCCHiN/nqGqPr666/DohFWoQTQb4UqrgzFTvqszKoo5SZPVuWsyXQx+NQkbfIqZwIYfMpZOxUmm6yR8PDDD7tSy7a3X331lXsedXDeeecpvdNX7vKee+6pLrvsMrXzzju70ffaay/12muvuedRB/fff78aNGiQe/mGG24I3da90l5sGQS4KuUgBoGk9aXS2kUMFESpgwTkhwdZqF+crOUj67rhqocA+q0eXdZ2SZI+K7OSu9zkyaqcNZkuBp+apE1e5UwAg085a6fCZHviiSfUYYcd5pN60qRJasMNN/SFBU/kV1izwObVV1+tzjzzTKW3ZFcLFy7MRW3cuHFuoc2WLVsGb/WdH3zwwerpp592wyZOnJi3y5RcrLQXWwYBrko5iEEgaX2ptHYRAwVR6iAB2e1RPuUVh8Gn+ioA+q0+ndZWiZI+K7OSt9zkyaqcNZkuBp+apE1e5UwAg085a6fCZFuwYIHq2LGjWrlypSv5zTffrE499VT3PHgg6/P06NHDDR4zZoySdWIOOOAA9fzzz7vh//nPf5QYdKLcihUrckaiRYsW5aLIy+s333wTGr3SXmwZBISqkcAIAknrS6W1i4jiE1zHCWAQqO4KgH6rW781Wbqkz8qsZCw3ebIqZ02mi8GnJmmTVzkTwOBTztqpQNnk86vXX3/dlXzvvfdWr776qnsePBCD0Omnn54LlvV3pk2blju+5ZZbfOv2/PnPf1b//ve/g7e75//973+V5GVcoe3co15sZe0fWTTvww8/VD/99JNq1qxZ7hfirbfeOmeUqlevnkm+oC+/LouxSmYYiUFL/sQIJlsCS1rHHHNM6Mwjk6gYzubMmWNO1RFHHKE++uij3PlGG23k42siyWLOwcWpzbU0/tSpU5VsgyuuefPmat11141Mbty4cTldy3pDsk3osmXLVKtWrXJGwM6dO6t99tlH7bvvvmqttdaKTMPGhbT8vTJ8/PHH6ssvv8wFidxmBpvUlbfffjt3TfQra0/JbIJS64okLOtfyVpTzzzzjPr222/V9OnTc4vMynpN8lmk1H15yYlytupL1u1C5JeF2Z988slcPfnhhx/UL7/8klv7a9ttt1VHHnlkjl9UOU348OHDc4zkXOp80BA8cuRINXr06Fz7O+uss9R6661nbk3tx60PjRo1UnrXs5z+dtxxx7x816xZk6s/sr6M9BPevqZ79+6qpvoar2Bx23pcBmnahFeuYsfCcvLkyW607bbbzl1LrkuXLjnO7kV9IGEyazTMpW2Lkmaa+hmXbU3VrzBGScLScLWp3ySyyz0yLpHnmbgWLVrk+qzcif5P+mvZoEI+X589e3YuWMYi++23n2rQoIGJluenqSfexGz0qSY9ea7JWEN+YJNZ19InyI9p8uyTT+nk+SbjkV69eplbfH7cPiQLnkaQQjLYelaavMRPM96wKU+hcnvl9R6naZfedOQ4bt9VU88FI18hg4+MQaTtynNY2lGTJk2U3tVLHXjggbGfwSYf8W22RW+6HEPACgHdweMgYI2AXmDZtyWuHlg7+qUuMn1tIHLja0OIG2/8+PFuuK7ojp45VHCbd72osy++7sTdtIIHYdtPf/rpp07Pnj19aUi+5q9///6ONsIEk/Kd69lJzl/+8hdHP9Dc+8z9Xl+/zDn6RdTRAynf/ebkuuuuK3i/Ny1zrF/Wze1W/W222caVReTWM6jy0l+9erVzySWXOHpw68Y1cgV9YbP//vs7ixcvzksnbYAt/l459OeGbpm0wS53SRssHD3gdcODZYxTV0we2sDjaGNHZFqStjA76aSTIuufrfqSVbswZdXrcDkdOnQoWNajjz7a0YMmc0uo791mVWQ2bujQoY60A68+HnzwQXPZip+kPgwcONARPRv32GOPFaw/+lfeogyyqOtx2rqUIQmDUtqE4RTX1xsF+HTu1X/YsV4zLjRpG21REk5TP5OwzaJ+hQJKGJiWqy39JhQ/d5tXp9IfSx/18ssvO9qwE/nc04Zm57777ovM1ptm0n7MVp8qQmrDlSN9T1ib8YbJOEC/EDv6h7G8ssXtQ7xlt8XTCFNIBlvPSsnLRh9sU55C5TZsvH7adulNS46T9F1ZPheMfN66JvVY6vlTTz2V66elLnvrtjmW9qh/gDNJxPJttsVYGRIJAiUSkB2NcBCwRkDP7MjrQKVzDXN6jR5HDEKmk5UXIa/Tv3S71yTOBx984L3sO+7WrZsbt5hxKPhiq2cOFTXSSP56xkXBFzE9C8OVwZSpkK9/AXT0r5e+cshJkkGAnh2Vl46NAO8gQsoiOvM6MfbohbZLKrcM8PQvS95krBzb4u8VxltXxOCj15eKVdZidUXykJcB/ctprPRM/ZswYYJXvNyxrfriLavkZ6tdSB3Ru+45eovsWGXVC7Y7S5cuzSunCfAO4ERm/au6I20prK3ZNvh4GZVSH/TMPEdeXPVsxlA5g7KL4SrY1kz5xc+irhdr6yb/pAzitAmTRyl+qQYBva13XvK22qIknKZ+JmVru37lAUoYYIOrDf0mFN+9zatTaav9+vWL1Y6lz9OzZdx0vAfeNEvtx2z3qW+++aajZ+/GKpPpq+TZFXRx+xBv2W3xNLIUksHWs1LystEH25SnULkNG+PbaJcmLeMn7buyei4YuYJ1Tc/EjlXP5QfMV155xSQT6dtui5EZcQECKQlg8EkJkNvzCWy++ea+DlXvnJUfSYfIQMgMHmRgFJxBo9f+ca9LvL///e+h6ei1enzx9CcwofFMoPfBZPI3/gYbbJAbsOtPpHxpmuv6EzSTTJ7vnV2gp307+vOlnIHgjjvucPSOYY78EmvSMf6VV16Zl47+nM2R+81f8FcIE+71xTiWhSs2iHjooYd8ZZKH5PHHH+/Ii7bMhLnzzjudM844w9HrMrnxZEZLFs4Wf69sWdUVqUemDhhfL1Tu6Onyzj/+8Q/n7LPPdvRU5Lw4Ysz88ccfvSI6tupLVmW9/PLL88ohM6SkfetF2h39KZejP1HxxZEZTVHOO4Dr2rWro7dp9t1reIofNCJHpRk3PA2joHFPL2bvyC+c+rO0UPml7US5LOp6sbZuZEnDoFD/adIv1ZfZFt6+0Kt/OfZek2Opd15nsy1KumnqZxq2NuuXl0/SY1tc0+o3qfze+7w6DdYvab96HcJIg7b8wPHJJ594k8sde9MstR+z2afqz89DZynJs0avpeicc845zgknnODoTyV98aTfDbq4fYi37LZ4GlkKyWDrWSl52eiDbcpTqNyGjfi22qU3TTlO03dl8Vww8kXVNRlXb7XVVrlxusge9oOUXpLAkfeLQs5mWyyUD9cgkJYABp+0BLk/j8AFF1zge4GRX47CZrLI50/mYa/XuchLR6ZMm+viS+cc5uSl0RsvakaRuTfswdS6dWtHr99jouR8se4HB9FRMsgN8mCRGUvyQjFjxgxfWnIivwSccsopPlllUKXXBsiL6w2QF0NTPvk1pCZdsUGEvKwb2cR/7rnnIsWTKeB6vRVHPr/LwmXBP4u6or8bz6tXRx11lKPXAsjDInVQXga8jPfYY4/Q9mRuTlpfsijrlClTHL32kyu/DLIuvPDCvM8ZJZ43f2lHQcOWKV/UAE4YycvV4MGDnTfeeCM388fcY8v3ymh0ErfvMPHFmKzXyfCJ9OKLL/o4SdwddtjBF8d7kkVdL9bWTf5pGBTqP036aX3pUw3rtm3bFkwui7aYpn6mYWvKbKN+FYQW42IWXE22pejX3JPWD9PpFlts4ej1Cd2k9XosuR865CXR6ML4Bx10kBvPHISlaeIX6sds9qkyk1Jmh5l8xddr9ThRn8TLp/aHHnqoI/24GIGCLm4fElb2tDyNLHFlkPhJn5Vyr5TB9ngvjTxxyp1lu0zTd2X5XAira/Ljq16jStToOr0GlyOzi71tQY6lvkc5m20xKg/CIWCLAAYfWyRJxyUgL/bBTnPUqFHudTkQA5B3CrGsARN0YgiRX2S9aemF6YLRcr+Smzh60bWCawbJzWEPJr3QcF66EvD444/78pf0o5x886u3oY+6nAuXtWtkQGXkFf+FF14oeE+aQUDBhGNcLDaIEEOdKYvM0hCjVm25LPhnUVd++9vfusyE3bHHHlsQmV7g0hGjguEs/rBhwyLvSVpfsiirGPi8ckfN0pPCiIFHL+7txr/oootCyxg2gJM8ZD0wvQBy6D22AtMwEhnlRSloWDayyaw4L6uWLVuaS3l+FnW9WFs3QqRhUKj/NOmn9UsxCGTRFtPUzzRsbdavtDrIgquRqRT9mnvS+kGdysy8+fPnhyYrRhGZ8eNtyw0bNnT0RhC++ME0Tfxi/ZjNPvWyyy7zySmfxutNCHxyhp3IeE7WPQy6uH1IsOw2eBpZ4sog8ZM+K+XeLPrgNPLEKXeW7TJN35XlcyFY12QNyagfWeW9RGa1mbYovoxrw37AlTpgsy1KejgIZEkAg0+WdOto2vLSH/y+WT5T8Tq9i46vUw0bPEh8WRzQ2/nec8893mRy62J4FwvWO0H5roedBB9Mu+yyS1i0XJj8AqZ30PHJEDXQi0wkcEE+cfOW6frrrw/E8J+mGQT4Uyr9rNggQmabeMvy0ksvlZ5JDd9RCn/bdUXvYOfjJWsvFVukWPDIZ4FezoXqbNL6Yrus8uLglVl+xSu2dpN3enRUGYMDOMlD7x6TN2soi2qVlpEsihrlZEH04KdtS5YsiYoeK7yUul6srZsM0zJI238aOaL8uAaBrNpimvqZlm1N168wHWTF1eQVV78mvg0/qNOwBYu9+ciaON6+T46lb/O6YJpx+jGbfao8d7yzLyX/Z5991itiycdx+5Bg2W3wNMLGlUHiJ31Wmrzi+KX0wWnkKVburNtl2r4rq+dCqXVNflQOztK76qqr8lRtsy3mJU4ABDIggMEnA6gk6TjBh5z8guN1V1xxhTsgkpfesE++JP69997rxpMBSXBqtKzRIeHm77bbbvNmE3ocfDDdeuutofFMoCzOatIXP+pTExO/mP+vf/3Ll56s11LIpRkEFEo3zrVigwj5PMfLRqY4y68esn6PLLZZjq4U/rbrSnBdqqAhNIqXtI+gETVqNkvS+mK7rPJppbduyILF8j18oT+9VbF7j9SlsH4hOICTWYDyq3pNuLSM7r777oJiBmf/Rf2yWDARz8VS6nqxtm6STcsgbf9p5Ijy4xoEsmqLaepnWrY1Xb/CdJAVV5NXXP2a+Db8oE6LGSgkTxmrePs/eS56XTDNOP2YzT51xIgRPvnkc/G0M3Tj9iHBstvgadjGlUHiJ31Wmrzi+KX0wWnkKVburNtl2r4rq+dCkroWHNcG267o3WZbjFOPiAOBtAQw+KQlyP2hBGS6q3ewI79ce7dn9y5Ie9xxx4WmIYGyhaJ8BmHSks8cli9f7saXBZrNNfHlG+ViLvhgkl8+CjnvDmCSR9oHkxiYvDLLzk+FXJpBQKF041wrNoiQrUllurq3PN5jYSfr0zzxxBOhL+9xZLAdpxT+tuuKzHLx8pEFM+O64Ja5Tz/9dOitSeuL7bJefPHFvrJ6yx33+Oeff84rY3AAV6j/yLs5ZUBaRu+8805BCbp06eJjNnPmzILxi10spa4Xa+smr7QM0vafRo4oP65BIKu2mKZ+pmVb0/UrTAdZcTV5xdWviW/DD+o0joEi+GNUcLOCYJpx+jGbfepNN93k62sKfW4bl2HcPiRYdhs8jYxxZZD4SZ+VJq84fil9cBp5ipU763aZtu/K6rmQpK55N5SRsUqw7YrebbVF+VFH3lvi/Mn7EA4CSQlg8ElKjvsKEpDPE2ThQe+LnVmrRrYb9n66UGwasSxe6k3HGGjk13/vOkDyQIvjgg+mr7/+uuBtSQ0+sl20/Apw4403Oueee25uNyJ5aQ++1FWywUfAySwVr36ijmVAErUYZEEFJLxog7/tuhKcwRFnvQRT/JNPPtnHOepTwKSDRttlDf7CHVUvCoWHtc3gAK7QblaGnS0/LaOoWVlGvmDfENfgY6OuF3thMDKmZZDVwN7IF9cgkFVbTFM/07LNqn4ZtnH8rLiavOPq18S34Qd1GsdA8eWXX/r66w4dOvhECaYZpx+z2aeKgcnb9w4dOtQnX5KTuH1IsOw2eBp548og8ZM+K01eXt9GH5xGnmLlzrpdpu27snouJKlrwZ1/g21X9G6rLQbfb7xtMngsy1cU2zXMWyc5hoCXAAYfLw2OrRL4f//v//kGFGZ3Ju9nG2IUEuNQIedd10M6QNkmVFxwHaAhQ4YUSsa9FnwwFXvpLsXgIw8t2Wq9X79+vi1Mgx2397zSDT4CVgaLwYWFvWU0x2Loi1q01lVQigPb/G3XFflMybAQXxZkjuvk0z/vvfJZZJhLOmi0XVb5jNMrr6yzU8qfDGDDftEKDuDCtjwO42IjzDajoEylGHxs1/ViLwxG1qwZmHyS+nENAlm1xTT1M2u2pdSvpPyz4mrkiatfE9+GH9RpHAOFrJHj7f/kZc3rgmnG6cds9qmy3qFXPpnVkNbF7UOCZbfB08geVwaJn/RZafKy3QenkadYubNul1n3XYZ5qX5Wdc1WWyzF4CPtNU5bKZUR8esGAQw+dUPPtVJK+cXKO6Do3bt3Tg4x2Jjwfffdt6hs8tmQiS++mclz3XXX+cLjDJgks6weTI888khRo4f38zRTpmow+AhX2ZZW1pA45JBD8rYSN2UVX3ZiSrs2ieQXdFnwt11X1ltvPV+djdodLlg2Of/d737nu1cMi2Eu6aDRdlkPP/xwn7y21nQKDuDCjEJhXGyE2WYUlCnuC3kWdb3YC4ORNWsGJp+kflyDQFZtMU39zJpt3PqVlL3clxVXI1Nc/Zr4NvygTuO8dH300Ue+/k927vK6YJpx+jGbfWrwc/hrr73WK16i47h9SLDsNngagePKIPGTPivl3iz64DTyFCt31u0y675LmCdxSerayJEjfW23e/fueVnbaovy47D8EBr3T/oVHASSEMDgk4Qa98QiIL9+eF/05VjW5JBf+U14nEWWJTPvwsliNJk1a5bjnUEkRoSwBV7DBM3iwXTWWWe5ZTJlk9lLsgWkGEHkxX7y5MnOypUrnVK+6Rb50wwCwspfSlixQUShtORzFFl0e++9985jc8MNNxS6teRrWfG3XVdk5ylTP8SPM43fwAhu9SvbeIe5pPXFdlkvuOACX1nNp5hhMpcSFhzAxXlRKiX9QnFtMwrmFeeFPKu6HretZ80gyKTU87gGgazaYpr6mTXbOPWrVN7B+FlxNfnE1a+Jb8MP6jSOgUKe+96+XsYCXhdMM04/ZrNPveiii3zyHX/88V7xEh3H7UOCZbfB0wgcVwaJn/RZmVUfnFQeKUuxcmfdLrPuu6SMSVySuia7AXvbrvzYFnQ222Iwbc4hkAUBDD5ZUCVNl4DXuCMdqHSk9evXdzvTKVOmuHELHZx44onuPZKO/Lqy1lpruWEnnHBCodt912w/mMaNG+dbWFqmzsqCiN5Fqr0C3Hzzza7cUpZSZviE/dLgTdv2cbFBRNz8brnlFl+ZDz300Li3Fo2XJX/bdUU+a/QOJHbaaadYhkoZEAdnh0UtUO4dNJZSX2yXVdbm8pbVls6DA7g4L0pFK1HMCLYZBbMt9kKeZV2P29azZhBkUup5XINAVm0xTf3Mmm2x+lUq67D4WXE1ecXVr4lvww/qtJiBQna72n777X393+DBg32iBNOM04/Z7FNlTUVv/9y+fXsn7dbYcfuQYNlt8DRw48og8ZM8K7Psg5PIE7fcWbfLrPsuU85S/VLrmqQfnP123nnn5WVrsy3mJU4ABDIggMEnA6gk+X8ELrnkEt+gYrPNNnPPt9hii/+LWOTopZdecu+TQcrOO+/sOzcLQhdJJnfZ9oPpoYce8skiv/4UcvLw8A60SjH4tGvXrlDS1q+VMngqlPmKFSt8Zd5xxx0LRS/pWpb8bdeVzz//3MdB6oHIX8jJy0NQD2IoinLeQWMp9cV2WWU2m3dRdSnra6+9FiV27PDgAC7Oi1LsxItEtM0omF2xF/Is63qwjsni+mEuawZheZYSFtcgkFVbTFM/s2ZbrH6VwjkqblZcTX5x9Wvi2/CDOv3vf/9bMNk77rgjr59/8sknffcE04zTj9nsU1etWuV06tTJJ6fMWojr5P6gi9uHBMtug6eRJa4MEj/JszLLPjiJPHHLnXW7zLrvMuUs1Q/WtTfeeKNgEiNGjMj7cS3YdiUBm22xoEBchIAlAhh8LIEkmXACo0aN8g0ovIaOUgYXS5cudVq0aBGaVvPmzZ0lS5aECxASavvBJFtre8slvw5EOXmJklkX3vjFDD5bbrmlL37YItNhuxlFyVBKeLHBk/xqKdNdi62fFFzP4MADDyxFjIJxs+Rvu65IQQYOHOjT59prr+28/fbboWVcvny5c+qpp/riy7fesq5VlEtaX7Ioa9DgKy+cH3/8cZTouXBpI7JA9R577BEaLziAi/OiFJpQgsAsGHnFKPZCnmVdL9bWjZxZMzD5JPVLMQhk0RbT1M+s2RarX0mZB+/LgqvJoxT9mnvS+kGdbr755s7ixYtDk33xxRedVq1a+fps+XFLDPdeF0wzbj9ms0+97LLLfHI2adLEuf/++71i5h2LnPKjVps2bfKuxe1DgmW3wdMIE1cGiZ/kWZllH5xEnlLKnWW7zLrvMuUs1Q/WNdliPartyiYaXbt29bUJ0Umw7RoZbLZFkyY+BLIigMEnK7KkmyMg6+oEB5nG2FHqbk1iJDD3ev3gt/HF0Nt+MM2ZMyfvFwHZhl3WGTJODFaPPfZY3oLRUo5iBp/dd9/dV+6DDz4498CSRZJlXRRzvZQdn4xcxfxigyd5KTe6kAW4H374Yeezzz5zDXAys0d2ZfP+ciXxL7300mJZx76eJX/bdUUKNWnSpLwXAvnM8fzzz8+xE2ZSppdffjm3QLnha3wZZBRypj6Y+HHrSxZlFYNVsA7JJ49XXXWV4/0kbdmyZbld92T2mxjARPaePXuGFjM4gIv7ohSaWImBWTDyihDsK4PbsmdZ14N6qvYZPsI9i7aYpn7Wdv3y1sU0x1lwNfKUg8FH+if5XP2DDz5wzEyXiRMn5gzVwU9vJe6rr75qxHf9pPXEZp8qz5pguxd5Bw0alPsRQl6MJT/ZCvrpp5925LOgZs2auc982YnM64JpRfUhwbLb4GnkiCuDxE/yrMyyD04iTynlzrJdZt13mXKW6kfVNe+PlLJT8DXXXJM3LpN6+corr0RmabMtRmbCBQhYIoDBxxJIkokm8Le//c0dIEgHKn8dOnSItJpHpSRbf5v7vb6sC1SKy+LBJDuHeWWSY5mJIS+tm266aeTsJIlXzOBz8cUX56Utg8rgwHLYsGGlYIgVt9jgybycB8tuZPMODk2cli1b+l72YwlSJFJW/LOoK1IU+ZVQFvU2TOL6sqODDNILuaT1JauyTpgwwTXiBMspn3zJL2redb1MnNatW4cWMziAq0sGHwGSVV0v1taNMrKqJyb9tH6pBgHbbTFN/cyabTGDYlr23vttczVpl6pfc18aP6hT00eJL88zWf/GG+Y9Puigg0KzDqZZSj9ms08dP3583qe3Rn55jst28uY86H/77be+ssXtQ4Jl96ablKcRJK4MEj/pszKrPjipPFKWuOXOql1m3XdJGZO4QnVN2u0GG2yQN5Y29THOTHSbbTFJ+bgHAnEJYPCJS4p4iQkE19+RzvRPf/pTyenJDl/GkGA6ZDmX8FJcFg+msO9+jYzF/GIGny+++MJp2LBh5KDLpH/yySeXgiFW3EKDCPlls0+fPkXlMvKJL1Pd5VdR2y4r/lnUFVN2WagyuMaNl5X3WIyHl19+ee67cXN/lJ+0vmRZVpnNI1OpvWUqdCyfFhx33HGhRQwO4Ep5UQpNsITALBmJGHFeyLOq64XauhdR1gy8eSU5TmIQsNkW09TPrNnGqV9JmEfdY5OrySOJfs29Sf2gTqWtBMciYf3ZSSedFGmgD6ZZaj9ms0+VHVX79u0bu3+Wsvbv398Jyhy3DwmW3QZPo9u4Mkj8pM/KrPrgpPJIWUopdxbtMuu+S8qYxAXrWrdu3WLVcxlPyzo9cZzNthgnP+JAIAkBDD5JqHFPSQTC1t954oknSkrDRA7ufCHnpTrvg0l+vZoxY0bBJLwPCJmJEBX/mWeeyX3XHjbwkxdY2Z5cFn979913fQ+cOGsZ3XjjjaEDTJkl8oc//<KEY>//<KEY>//<KEY>///<KEY>ARIgARIgARIgARIgARIggX5NgAKfnOLNE/hgVyxbeLH00kvnhBT2VgiBjxi7TiXqjTfeiMTocJIn2Kp55plnUn7KCnzw8N///vdUuDYznENo9Mknn6TisX9UeT50XrA72xprrJGwQbq//e1vRw8++GD06aefRrKsL5LlfMn9zTbbLL5u5wPnnawzblr4mwRIgARIgARIgARIgARIgARIoH8ToMAnp3zzBD4QUrjbcMsuXjmhpW9hW3Kxf+MVDKR9tv6qIvBxl3SNHTs2FbDszJUILKaffvrIXr6mHqsIfPTZcePGRRAiuQIf/P7tb3+r3jKPZZ5vIi/Q4DnwwANb0j/TTDNFYtcnvi47rkXHHntsBO0in+tknfGlh9dIgARIgARIgARIgARIgARIgAT6L4F+JfBZZ511ckvqr3/9a2rCXmdJFyKCloctwHCFKFmJEeO9scbIzDPPHEGQUNZVEfjI1uiptF5wwQVJtLfffnvqXhbHIgIfaLvMO++8kRh5TsLXE+R17733TsUFfmuvvbZ6ibVl6jwfMi9Jov53cvnll8dp/+pXvxphORq0pKD5M2rUqOjMM8+Mpk6d6j7S8rtTdaYlIbxAAiRAAiRAAiRAAiRAAiRAAiTQrwn0K4HPKquskltYoQU+EIzYAh9o/CCOdk52c4qfO/TQQ9t59d6vIvDB0ilNKwQWb731VhL2vvvum9yDHxgf9rkHHngg5c9nw+ff//537Oe4447zBRFfO+uss1LhLLjggonfus+HzEuSKDmBPSHZUS3C0r1XXnnFvlXqvFN1plQi6ZkESIAESIAESIAESIAESIAESKDnCfS0wMfV5lhooYVyC8Q1mjtkyJBc/wsssEBKMAHDx7a76qqrUvchLJlHdmd66qmnbG+p81/+8pfxMxC6vPPOO6l7RX+4Ah/sppXnnn322VhYoQIfd0exjTbaKJWP1VZbzRvc2WefnfK35pprtvhTgc2KK67Ycs++YGu6fO9730tu1X0+ZF40UZdeemnCb/LkyXq50rGJOjNp0qRYSLfttttGV1xxRaV08SESIAESIAESIAESIAESIAESIIH+RaCnBT5PPvlkSgAxzTTTpHZPuvLKK6M99tgjKbHx48en/EPokue+9rWvpfzffffdKe8w2LvIIouk/ECogqVa+++/f4T0weDvv/71rwgT/e9+97uJ3yJ2a1KRWT9cgQ/ivPfeey0f/3+KNG6//fZJvBBy6a5b6mufffZJ7qtQCFveq/vwww+jc845J9m9S/3YS+L++c9/xt5VYAM/f/jDHzSIluP666+fxIn41dV9PmRekCaU36BBg5K0jhgxIhaq3HfffXH5TpkyJcLW7Vl2ezRfemyiziy++OJJ+sD9xBNP1Oh4JAESIAESIAESIAESIAESIAESGKAEelrg895770Vf/vKXU5PdJZdcMjr33HNjQQ+W4NjaKtBsUWGFHp9++mlv0cMAs/rRo28iDWPIiEf9FDkefvjh3jiLXvQJfGafffboxhtvTAUB4cIuu+ySpA2aQK7QCg/ccsstiR9N/+c+97lo0003jbbaaqsI25Djum+L9dVXXz3CUrrFFlssjtsW2ECLBwIR1yH9SC/CxBbwtkZU3edD5gXpxi5hyqTdcbrppotZbbjhhhGWtKkQzM1/yDrz8MMPt6QPgkU6EiABEiABEiABEiABEiABEiCBgU2gpwU+KLqddtqpZcKrE3Msr4JwAXZ1TjjhhGiWWWZp8Tt06NAIS3ael23W4XDEbywP03D0qLswuUKi6667rkX7RZ+xj9jN6aCDDorjqfPPJ/DReJZaaqlozz33jLnA+LFeRz6zdhF7//33vZpK+iyOK6ywQqypZGu72Pd33XXXOEu2wAb3IWQ67bTToscffzzCzmQQSi266KJxuqCRddFFF6VQ1H0+ZF6QMBjYtpef2Xludw4NMghkfC5UnYGmkZuO5ZZbzhclr5EACZAACZAACZAACZAACZAACQwgAj0v8IEB3WHDhrVMeocPHx4Le1CW0PpxJ8Xu7+222y4uduy+5N5zf6+77rotVQQ7WMEgr6txhGch2Nhkk02iiRMntjxX5YIr8IG2zIUXXhhhi3A3rUjPj3/843j3q7y4sCxp5ZVXbnl+rrnmis4444xY8IHnoeVkx4G8YRkVBCNwKrCB4Ancdcty+xmcYxnS/fffHz9j/6v7PMIKlRdN12uvvRYba3bzUOT3V77yleiNN97QoFLHEHUGS8mWX375pEygZYSljHQkQAIkQAIkQAIkQAIkQAIkQAIDm8A0yL5MXHvaiU0aIwaVjWiRGBF6mCWWWMKIfZmO5AlpEa0LI/Z7jGgEGdEUitMiS5eCpUc0ZYwss0rCE4GPEaGEeffdd80TTzxhZBlSfO+b3/ymkeU9cToSzzknqAoilDIPPfSQ+eSTT8z8889vRLPHyJK11FMPPvigkW3Xjdg4MmJw2cw222zJfVlGZmSJVpxnPCdbscdpEsPR5vXXXzdf//rXjQh7zHzzzZc8Y5/UfV7DCpEXDUuMg5vNN9/cIG3nn39+zEO2YDevvvqqEQGVkaWFBuWOI/5eeOEF87e//c2IECwOYty4cUaWx2lwLce6dUaEPkaWssXxbrbZZqnyaImMF0iABEiABEiABEiABEiABEiABAYEgX4h8BkQJWVlMkvgY3nhaSACEPCMHDkyFoDdcMMNZr311isU8ttvv23EfpKR5YFGDIcbsR9V6Dl6IgESIAESIAESIAESIAESIAESIIEQBCjwCUGxj8OgwKdvgENrDBpO0MCBoAcCnzIOml6yPb3ZfffdzSmnnFLmUfolARIgARIgARIgARIgARIgARIggVoEKPCpha8zD1Pg0zfcl156afPnP/85jkzsFJmjjjqqVMQHHnigOeSQQ4xscW9kG/pSz9IzCZAACZAACZAACZAACZAACZAACdQhQIFPHXodetYV+MCGDuzj0IUj8J///MeIweXYbg9ChT2je+65J7ZbVCQWMW5t9t57byM7pcU2jWSb+yKP0Q8JkAAJkAAJkAAJkAAJkAAJkAAJBCFAgU8QjH0byHPPPWdkq/AkUtkJK152JDtmJdd4Uo8ADC7LlvLmnXfeSQIaPHiw+cUvfmE23nhjg3PXwVjz1VdfbS6//HIjO5DFwiEIiSAsoiMBEiABEiABEiABEiABEiABEiCBviRAgU9f0g4UF5YIbbjhhqnQZIvvePer1EX+qEVg7Nix5uijj24JA4I17JI2aNAgM/3008c7pEHrCrukqZMt6c0ll1wS79Km13gkARIgARIgARIgARIgARIgARIggb4iQIFPX5EOEA+2nZ8wYUJsF2bKlCmpEIcOHWogoMA28HPPPXdq2/aUR/4oTABb02Np1mGHHRZvv17kwbnmmsuMGTPGjB492kDzio4ESIAESIAESIAESIAESIAESIAEOkGAAp9OUK8Q58cff2zmmGMOg+2+27kRI0YYbCdOF4YA7CMde+yxsbBt4sSJZurUqUnA0047rRkyZIhZZZVVzPDhw2PjzBT0JHh4QgIkQAIkQAIkQAIkQAIkQAIk0CECFPh0CHyVaN98883UsqGsMGaYYQYDQ850zRCAXZ8XX3zRzDLLLGb22Wc3NMjcDGeGSgIkQAIkQAIkQAIkQAIkQAIkUJ0ABT7V2fFJEiABEiCB/2vHjmkAAAAQhvl3jQeuHRVAQsoHAQIECBAgQIAAAQIEkgIOn+QsShEgQIAAAQIECBAgQIAAAQIEfgGHz28nSYAAAQIECBAgQIAAAQIECBBICjh8krMoRYAAAQIECBAgQIAAAQIECBD4BRw+v50kAQIECBAgQIAAAQIECBAgQCAp4PBJzqIUAQIECBAgQIAAAQIECBAgQOAXcPj8dpIECBAgQIAAAQIECBAgQIAAgaSAwyc5i1IECBAgQIAAAQIECBAgQIAAgV/A4fPbSRIgQIAAAQIECBAgQIAAAQIEkgIOn+QsShEgQIAAAQIECBAgQIAAAQIEfgGHz28nSYAAAQIECBAgQIAAAQIECBBICjh8krMoRYAAAQIECBAgQIAAAQIECBD4BRw+v50kAQIECBAgQIAAAQIECBAgQCAp4PBJzqIUAQIECBAgQIAAAQIECBAgQOAXcPj8dpIECBAgQIAAAQIECBAgQIAAgaSAwyc5i1IECBAgQIAAAQIECBAgQIAAgV/A4fPbSRIgQIAAAQIECBAgQIAAAQIEkgIOn+QsShEgQIAAAQIECBAgQIAAAQIEfgGHz28nSYAAAQIECBAgQIAAAQIECBBICjh8krMoRYAAAQIECBAgQIAAAQIECBD4BRw+v50kAQIECBAgQIAAAQIECBAgQCAp4PBJzqIUAQIECBAgQIAAAQIECBAgQOAXcPj8dpIECBAgQIAAAQIECBAgQIAAgaSAwyc5i1IECBAgQIAAAQIECBAgQIAAgV/A4fPbSRIgQIAAAQIECBAgQIAAAQIEkgIOn+QsShEgQIAAAQIECBAgQIAAAQIEfgGHz28nSYAAAQIECBAgQIAAAQIECBBICjh8krMoRYAAAQIECBAgQIAAAQIECBD4BRw+v50kAQIECBAgQIAAAQIECBAgQCAp4PBJzqIUAQIECBAgQIAAAQIECBAgQOAXcPj8dpIECBAgQIAAAQIECBAgQIAAgaSAwyc5i1IECBAgQIAAAQIECBAgQIAAgV/A4fPbSRIgQIAAAQIECBAgQIAAAQIEkgIDIxrcJZf7pJwAAAAASUVORK5CYII=)  Task and dataset---Obtain the data from [here](https://p4-squad-data.s3.us-east-2.amazonaws.com/train-v2.0.json). You also may consider doing so in a code block below.```bash!wget https://p4-squad-data.s3.us-east-2.amazonaws.com/train-v2.0.json```This gives you the dataset containing about 130,319 examples. You should split this into a **train** and **validation** dataset as in Project 2.<jupyter_code>gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
  print('Select the Runtime > "Change runtime type" menu to enable a GPU accelerator, ')
  print('and then re-execute this cell.')
else:
  print(gpu_info)
!wget https://p4-squad-data.s3.us-east-2.amazonaws.com/train-v2.0.json
!pip install transformers
!python -m spacy download en_core_web_md
from google.colab import drive

drive.mount('/content/drive')
import collections
import json
import string
import re
import time
import math

import numpy as np
import transformers

import torch
import torch.nn as nn
from torch import bmm
from torch.nn import Linear, GRU, Softmax, LogSoftmax
from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence
from torch.utils.data import Dataset, DataLoader, SubsetRandomSampler
from torch.optim import Adam

import spacy
from tqdm import tqdm

nlp = spacy.load('en_core_web_md')
# Use GPU where possible
get_device = lambda : "cuda:0" if torch.cuda.is_available() else "cpu"
with open("/content/drive/MyDrive/CS 5740/P4/Dataset/train-v2.0.json", "r") as fp:
    dataset = json.load(fp)
data = dataset['data']
num_subjects = len(data)
num_paragraphs = 0
num_questions = 0
num_impossible = 0
num_plural_answers = 0
for subject in data:
  num_paragraphs += len(subject['paragraphs'])
  for paragraph in subject['paragraphs']:
    num_questions += len(paragraph['qas'])
    for question in paragraph['qas']:
      if question['is_impossible']:
        num_impossible += 1
      if len(question['answers']) > 1:
        num_plural_answers += 1
print("The number of subjects: ", num_subjects)
print("The number of paragraphs: ", num_paragraphs)
print("The number of questions: ", num_questions)
print("The number of unanswerable questions: ", num_impossible)
print("The number of plural answers: ", num_plural_answers)
def get_token_for_char(context, char_idx):
    for i, token in enumerate(context):
        if char_idx == token.idx:
            return i
        if char_idx < token.idx:
            return i - 1
    return len(context) - 1

test_ex = nlp("The blue bird flies.")
test_ex[get_token_for_char(test_ex, len("The blue bird flies."))].text == "."
class Preprocessor:
    def __init__(self, nlp=None):
        self.nlp = nlp if nlp is not None else spacy.load('en_core_web_md')

    def preprocess(self, data, is_val=False):
        """
            Takes a squad data and parses to DrQA format vectors for context,
            question, and answer.
        """
        nlp = self.nlp
        dataset = []
        for topic in tqdm(data, position=0, leave=True, desc="Generating dataset"):
            for paragraph in topic['paragraphs']:
                context = nlp(paragraph['context'])
                self.context = context
                context_dict = [{"emb": word.vector, "pos": word.pos_, "ner": word.ent_type_, "exact": 0.0, "lower": 0.0, "lemma": 0.0} for word in context]
                context_words = [word.text for word in context]
                context_lemmas = [word.lemma_ for word in context]
                for question in paragraph['qas']:
                    question_words = collections.defaultdict(int)
                    question_nlp = nlp(question['question'])
                    for word in question_nlp:
                        question_words[word.text] = 1
                    for j, feature in enumerate(context_dict):
                        if context_words[j] in question_words:
                            feature['exact'] = 1.0
                        if context_words[j].lower() in question_words:
                            feature['lower'] = 1.0
                        if context_lemmas[j] in question_words:
                            feature['lemma'] = 1.0
                    question_words = {}
                    question_emb = [word.vector for word in question_nlp]
                    start = -1
                    end = -1
                    unans = 0
                    if question['is_impossible']:
                        unans = 1
                    else:
                        for answer in question['answers']:
                            ans_start = answer['answer_start']
                            ans_end = ans_start + len(answer['text'])
                            start = get_token_for_char(context, ans_start)
                            end = get_token_for_char(context, ans_end)
                    if is_val:
                        dataset.append({
                            'text': context_words,
                            'answer_text': answer['text'],
                            'context': context_dict,
                            'question': question_emb,
                            'answer': {
                                'unans': unans,
                                'start': start,
                                'end': end}})
                    else:
                        dataset.append({
                            'context': context_dict,
                            'question': question_emb,
                            'answer': {
                                'unans': unans,
                                'start': start,
                                'end': end}})
        return dataset

preprocessor = Preprocessor(nlp)
dataset = preprocessor.preprocess(data[math.floor(0.8*len(data)):])
val_dataset = preprocessor.preprocess(data[math.ceil(0.8*len(data)):], is_val=True)
def get_feature_dicts(dataset):
    ner = {}
    pos = {}
    for features in dataset:
        for feature in features['context']:
            f_ner = feature['ner']
            f_pos = feature['pos']
            if f_ner not in ner:
                ner[f_ner] = len(ner)
            if f_pos not in pos:
                pos[f_pos] = len(pos)
    return ner, pos

ner, pos = get_feature_dicts(dataset)
embedding = nn.Embedding(len(ner), len(ner))
embedding.weight.data = torch.eye(len(ner))
pos_embedding = nn.Embedding(len(pos), len(pos))
pos_embedding.weight.data = torch.eye(len(pos))<jupyter_output><empty_output><jupyter_text>Q1: Initial data observationsWhat are your initial observations after you explore the dataset?**Answer:**Named entities may not be written consistently, as is the case with "Beyoncé" which may or may not be written with a diacritic. Some answer spans are ambiguous, eg:- Excludes definite article: - Question: 'When did Beyoncé rise to fame?' - Dataset answer: 'late 1990s' - Possible answer: 'in the late 1990s'- Some answers include incomplete spans: - eg. 'split with Luckett and Rober' (missing -son at the end)- Sometimes possible answer sequences are repeated many times - eg. "Beyoncé" in the first paragraph as an answer to which artist.- Which leads to cases where Beyoncé is qualifying other words and may lead to a deceivingly high F1-score.Some basic statistics:- The number of subjects:  442- The number of paragraphs:  19035- The number of questions:  130319- The number of unanswerable questions:  43498 The two metrics we will be using are **exact match** and **F1**. The exact match metric measures the percentage of predictions that match any one of the ground truth answers **exactly**. The F1 score is a looser metric, which measures the average overlap between the prediction and ground truth answer.The F1 metric on answers is defined as:$$F_1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$$where $Precision$ is:$$Precision = \frac{TP}{TP + FP} = \frac{\text{ of matching tokens}}{\text{ of predicted tokens}}$$and $Recall$ is:$$Recall = \frac{TP}{TP + FN} = \frac{\text{ of matching tokens}}{\text{ of gold tokens}}$$and $TP$ refers to the number of true positives; $FP$ is the number of false positives; and $FN$ is the number of false negatives. (See Jurafsky & Martin Ch. 4.7).These are then also averaged with the "has answer" prediction where we replace the individual question-answer pair score with a 1 if it correctly predicts that the question is not answerable, and keep the original score otherwise.  Q1.1: The metricsHow much time does it take to compute these metrics? When will these metrics correlate well/poorly with human judgement of answer quality? Please implement the metric yourself.Each round of the validation set with roughly 18k sample questions and answers (%20 of the train dataset) took approximately 353 seconds to compute. A complexity analysis is noted below.F1-score based on overlapping terms works fairly well for ambiguous answers. For example, "the airline United" vs "United," both are reasonable answers, so the overlapping permits some measure of success if one or the other is selected. However, it is also a bad measure for the same reason: it cannot account for a plurality of possible solutions. A good measure would encapsulate all reasonable responses and perhaps weight based on relevance, as is the case with confusing "Washington" terms, then precision to "Washington D. C." would be preferred.On the other hand, F1-measure does have the issue of permissiveness. "Batman not in Gotham" is weighted equally with "not Batman in Gotham" (because of the imprecision of the F1-measure defined). As well, it permits odd cases to be considered somewhat precise, such as if we "asked how is the weather?" and the response were "she said the weather is nice," which includes two unrelated terms. Therefore, the exact measure is a better measure in this case, since it excludes misleading matches.\* **Complexity analysis**: Let $n = |ground truth|$ be the length of the ground truth answer, $m = |predicted|$ be the length of the ground truth answer. The exact metric takes $O(\min(m, n))$time, since as soon as one token fails to match, we return unmatched. The F1 score can be computed by generating a dictionary of counts and comparing the dictionaries, which also takes $O(\min(m,n)) + O(\min(m,n))$ complexity. Therefore, over the entire dataset, presuming the same $m, n$ represent the maximal length ground truth and predicted and $l$ number of samples to compare, then it takes $O(\min(m,n) \times l)$ complexity to compute the metrics.<jupyter_code>from collections import Counter
# Some potentially useful helper functions for the metrics
def remove_articles(text):
    regex = re.compile(r'\b(a|an|the)\b', re.UNICODE)
    return re.sub(regex, ' ', text)

def white_space_fix(text):
    return ' '.join(text.split())

def remove_punc(text):
    exclude = set(string.punctuation)
    return ''.join(ch for ch in text if ch not in exclude)

def lower(text):
    return text.lower()

def normalize_answer(s):
    """Lower text and remove punctuation, articles and extra whitespace."""
    return white_space_fix(remove_articles(remove_punc(lower(s))))

def get_tokens(s):
    if not s:
        return []
    return normalize_answer(s).split()
# Another batch of helper functions, feel free to modify as needed
def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans):
    """apply_no_ans_threshold applies the no-answer threshold (or mapping) to
    the no-ans probabilities/predictions.

    :param scores: A mapping of qid to the f1 or em score
    :type scores: dict[qid: str, score: float]
    :param na_probs: A mapping of qid to the no answer prediction/probability
    :type na_probs: dict[qid: str, score: float]
    :qid_to_has_ans: A mapping of qid to answerability
    :type qid_to_has_ans: dict[qid: str, answerability: bool]
    :returns: A mapping of qid to new scores accounting for answerability
    :rtype: dict[qid: str, score: float]
    """
    new_scores = {}
    return new_scores

def make_eval_dict(exact_scores, f1_scores, qid_list=None):
    """make_eval_dict makes a dictionary from the individual scores of qa pairs.
    
    :param exact_scores: Mapping of qid to the em score
    :type exact_scores: dict[qid: str, score: float]
    :param f1_scores: Mapping of qid to f1 score
    :type f1_scores: dict[qid: str, score: float]
    :param qid_list: List of qid's to aggregate, used to separate HasAns and
        NoAns scores for better analysis of models
    :type qid_list: List[qid: str]
    :returns: mapping of relevant evaluation metrics to their scores
    :rtype: OrderedDict[value: str, score: float]
    """
    return collections.OrderedDict([
        ("exact", 0.),
        ("f1", 0.),
        ("total", 0.),
    ])

def merge_eval(main_eval, new_eval, prefix):
    """merge_eval is a helper to merge two outputs of make_eval_dict.

    This is useful for cases when we want to have the aggregate scores, but also
    want to look at the scores for HasAns and for NoAns. Using make_eval_dict,
    we can take only those that have an answer, compute and merge with the
    original eval_dict.

    :param main_eval: The main evaluation dictionary that you report from
    :type main_eval: dict[value: str, score: float]
    :param new_eval: The new evaluation dictionary that you want to merge in
    :type new_eval: dict[value: str, score: float]
    :param prefix: The prefix you want attached to the new_eval scores.
        For example, HasAns as a prefix can be used to get HasAns_f1.
    :type prefix: str
    :returns: combination of main_eval and new_eval with prefix on new_eval keys
    :rtype: OrderedDict[value: str, score: float]
    """
    for k in new_eval:
        main_eval['%s_%s' % (prefix, k)] = new_eval[k]
# metric implementation here
def compute_exact(a_gold, a_pred):
    return normalize_answer(a_gold) == normalize_answer(a_pred)

print("Should not be 0: ", compute_exact("john went to to the dock", "to the dock"))
print("Should be 1: ", compute_exact("to the dock", "to the dock"))
print("Should be 0: ", compute_exact("dock the to", "to the dock"))

def compute_f1(a_gold, a_pred):
    assert(len(a_gold) is not None)
    assert(len(a_pred) is not None)
    a_gold_tokens = get_tokens(a_gold)
    a_pred_tokens = get_tokens(a_pred)
    a_gold_count = Counter(a_gold_tokens)
    a_pred_count = Counter(a_pred_tokens)
    TP = sum((a_gold_count & a_pred_count).values())
    TPplusFP = len(a_gold_tokens)
    TPplusFN = len(a_pred_tokens)
    if TPplusFP > 0 and TPplusFN > 0:
      precision = TP/TPplusFP
      recall = TP/TPplusFN
      if precision > 0 and recall > 0:
        return 2*(precision*recall)/(precision+recall)
    return 0

print("Should not be 1 or 0: ", compute_f1("john went to to the dock", "to the dock"))
print("Should be 1: ", compute_f1("to the dock", "to the dock"))
print("Should be 1: ", compute_f1("dock the to", "to the dock"))
print("Should be 0: ", compute_f1("wicker basket", "to the dock"))<jupyter_output>Should not be 0:  False
Should be 1:  True
Should be 0:  False
Should not be 1 or 0:  0.5714285714285715
Should be 1:  1.0
Should be 1:  1.0
Should be 0:  0<jupyter_text>Model 1: Pick your own model---In this part of the project you will implement a model to perform well on the SQuAD dataset. The features that you use are entirely up to you. As always, a good strategy is to work through lots of examples from the training data to determine what types of features would be most useful. You are **NOT** allowed to use (i) deep contextualized pre-trained language models (e.g. ELMo, GPT, BERT) since they will be covered in Part B, or (ii) any existing code written for the SQUAD task. Otherwise, feel free to re-use any code you have written for previous projects in the class as well as **general** ML/NLP tools (e.g., PyTorch, spaCy). Please use the code block below to code, prototype, and test your language model:Code of Academic Integrity:  We encourage collaboration regarding ideas, etc. However, please **do not copy code from online or share code with other students**. We will be running programs to detect plagiarism.<jupyter_code>max_length = 0
for example in dataset:
  max_length = max(max_length, len(example['context']))
max_ans = 0
for example in dataset:
  max_ans = max(max_ans, example['answer']['start'])
# Checking this to see how realistic a non-RNN model is.
max_length, max_ans
# Convenience embedding. One hot encodings for NER and POS.
# Uses torch.eye to construct an index-to-index_vector matrix.
ner_embedding = nn.Embedding(len(ner), len(ner))
ner_embedding.weight.data = torch.eye(len(ner))
pos_embedding = nn.Embedding(len(pos), len(pos))
pos_embedding.weight.data = torch.eye(len(pos))
import pickle
pickle.dump( ner, open( "ner.p", "wb" ) )
pickle.dump( pos, open( "pos.p", "wb" ) )
def get_example_tensors(example):
    question = torch.Tensor(example['question'])
    context = torch.Tensor([feature['emb'] for feature in example['context']])

    # This is the most painful tensor to construct. We need to concat a vector
    # of POS and NER, given by the Embedding above, then add exact / lower /
    # lemma matches (binary scalars). The hard part is making it all the right
    # dimensions, hence the weird transposing. This is probably less efficient
    # than constructing the other way, but it works -- don't touch what already
    # works!
    features = torch.cat([embedding(torch.LongTensor([ner[feature['ner']] for feature in example['context']])),
        pos_embedding(torch.LongTensor([pos[feature['pos']] for feature in example['context']])),
        torch.LongTensor([[feature['exact'] for feature in example['context']]]).transpose(-1, 0),
        torch.LongTensor([[feature['lower'] for feature in example['context']]]).transpose(-1, 0),
        torch.LongTensor([[feature['lemma'] for feature in example['context']]]).transpose(-1, 0)], dim=1)
    return context, features, question
# Because we're separating the model into "answerable" and "answer", we need a separate dataset of answerable questions
ans_train_dataset = []
ans_val_dataset = []
for example in dataset:
    if not example['answer']['unans']:
        ans_train_dataset.append(example)
for example in val_dataset:
    if not example['answer']['unans']:
        ans_val_dataset.append(example)
class QADataset(Dataset):
    """QADataset is a torch dataset to interact with the QA data.

    :param data: The vectorized dataset with input and expected output values
    :type data: List[Tuple[List[torch.Tensor], int]]
    """
    def __init__(self, data):
        """
            :param data: An array of examples with preprocesed context,
                         question, and answer
            :type data: dict[]
        """
        self.data = data
        self.len = len(data)
    
    def __len__(self):
        """__len__ returns the number of samples in the dataset.

        :returns: number of samples in dataset
        :rtype: int
        """
        return self.len
    
    def __getitem__(self, index):
        """__getitem__ returns the tensor, output pair for a given index

        :param index: index within dataset to return
        :type index: int
        :returns: A tuple (x, y) where x is model input and y is our label
        :rtype: Tuple[torch.Tensor, int]
        """
        example = self.data[index]
        # This is factored out for validation, because we need to have access
        # to strings to perform F1 / exact metrics.
        context, features, question = get_example_tensors(example)
        ans_start = torch.LongTensor([0])
        ans_end = torch.LongTensor([0])
        unans = torch.LongTensor([0])
        # It isn't clear the best way to handle unanswerable questions. I opted
        # to stay inside the DrQA design with a final output layer for
        # unanswerable questions akin to the start / end output layers. However,
        # the consequence is that for batching, the loss includes these invalid
        # start / end values. If I had more time, I might make the initial
        # position a 'no answer' position and increment starts and ends by 1.
        if example['answer']['unans']:
            unans = torch.LongTensor([1])
        else:
            ans_start = torch.LongTensor([example['answer']['start']])
            ans_end = torch.LongTensor([example['answer']['end']])
        return context, features, question, unans, ans_start, ans_end

qa_dataset = QADataset(dataset)
qa_val_dataset = QADataset(val_dataset)
ans_dataset = QADataset(ans_train_dataset)
# We have variable length sequences. In order to use data loader, we have to
# custom handle collating the tensors. :'(

def padding_fn(batch):
    (cs, fs, qs, us, ss, es) = zip(*batch)
    cs_pad = pad_sequence(cs)
    fs_pad = pad_sequence(fs)
    qs_pad = pad_sequence(qs)
    us_pad = torch.cat(us)
    ss_pad = torch.cat(ss)
    es_pad = torch.cat(es)
    return (cs_pad, fs_pad, qs_pad, us_pad, ss_pad, es_pad)
train_sampler = SubsetRandomSampler([i for i in range(len(qa_dataset))])
ans_train_sampler = SubsetRandomSampler([i for i in range(len(ans_dataset))])
val_sampler = SubsetRandomSampler([i for i in range(len(qa_val_dataset))]) 
dataloader = DataLoader(qa_dataset, batch_size=32, sampler=train_sampler, collate_fn=padding_fn)
ans_dataloader = DataLoader(ans_dataset, batch_size=32, sampler=ans_train_sampler, collate_fn=padding_fn)
val_dataloader = DataLoader(qa_val_dataset, batch_size=32, sampler=val_sampler, collate_fn=padding_fn)
class Importance(nn.Module):
    def __init__(self, hidden_size):
        super(Importance, self).__init__()
        self.linear = Linear(hidden_size, 1)
        self.softmax = nn.Softmax()

    def forward(self, hidden):
        """
            We computer the linear transform of each embedding to b_j
            :param input: seq length x batch x hidden_size representing the
                          contextual embeddings of the sequence of tokens.
            :type input: nn.Tensor
        """
        # Temporarily pretend this is a linear model to fit to get weights
        view = hidden.view(-1, hidden.size(-1))
        # Revert back to seq len x batch
        b_j = self.linear(view).view(hidden.size(0), hidden.size(1))
        return b_j


class QuestionEncoder(nn.Module):
    def __init__(self, embedding_size, num_layers=3, hidden_size=128):
        super(QuestionEncoder, self).__init__()
        self.hidden_size = hidden_size
        self.gru = GRU(embedding_size, hidden_size, num_layers, bidirectional=True)
        self.importance = Importance(2*hidden_size) # Bilinear

    def forward(self, input):
        """
            Takes a word embedding encoded sequence and generates an attentive
            contextual output.
        """
        hidden, _ = self.gru(input, None)
        W = self.importance(hidden)
        # Generate weighted average. BMM would give sequence averages if we
        # don't transpose :/
        return W.transpose(0, 1).unsqueeze(1).bmm(hidden.transpose(0,1))


class AlignedEmbedding(nn.Module):
    """
        Nearly identical to Importance, except that forward takes two input
        sequences and generates the weighted attention of the first aligned to
        the second.
    """
    def __init__(self, input_size):
        super(AlignedEmbedding, self).__init__()
        self.linear = Linear(input_size, input_size)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, context, question):
        """
            :param context: batch x input_size x hidden_size representing the
                          contextual embeddings of the sequence of tokens.
            :type context: nn.Tensor
            :param question: batch x input_size x hidden_size representing the
                          contextual embeddings of the sequence of tokens.
            :type question: nn.Tensor
        """
        context_view = context.view(-1, context.size(-1))
        question_view = question.view(-1, question.size(-1))
        Wc = self.relu(self.linear(context_view)).view(context.size()).transpose(0, 1)
        Wq = self.relu(self.linear(question_view)).view(question.size()).transpose(0, 1).transpose(1, 2)
        WcWq = Wc.bmm(Wq).view(-1, question.size(0))
        a = self.softmax(WcWq).view(-1, context.size(0), question.size(0))
        return a.bmm(question.transpose(0, 1)).transpose(0, 1)


class ParagraphEncoder(nn.Module):
    def __init__(self, embedding_size, features_size, num_layers=3, hidden_size=128):
        super(ParagraphEncoder, self).__init__()
        self.hidden_size = hidden_size
        self.aligned_embedding = AlignedEmbedding(embedding_size)
        self.gru = GRU(2*embedding_size + features_size, hidden_size, num_layers, bidirectional=True)

    def forward(self, context, features, question):
        """
            Takes:
            - Paragraph embeddings
            - Question embeddings
            - Paragraph features (POS, exact matches, etc.)

            Constructs:
            - Aligned embedding

            Outputs:
            - Paragraph hidden representations
        """
        W = self.aligned_embedding(context, question)
        output, _ = self.gru(torch.cat([context, features, W], dim=2), None)
        return output


class BilinearAttention(nn.Module):
    def __init__(self, context_emb_size, question_emb_size):
        super(BilinearAttention, self).__init__()
        self.linear = Linear(question_emb_size, context_emb_size)
        self.softmax = nn.Softmax(dim=-1)
        self.logsoftmax = nn.LogSoftmax(dim=-1)

    def forward(self, context_embedding, question_embedding):
        # Batch first for bmm:
        context_view = context_embedding.transpose(0, 1)
        # Get the linear transformation:
        Wq = self.linear(question_embedding).transpose(1, 2)
        pWq = context_view.bmm(Wq).squeeze(2)
        if self.training:
            return self.logsoftmax(pWq)
        else:
            return self.softmax(pWq)


class UnanswerableAttention(nn.Module):
    def __init__(self, context_emb_size, question_emb_size):
        super(UnanswerableAttention, self).__init__()
        self.linear = Linear(question_emb_size, context_emb_size)
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, context_embedding, question_embedding):
        # Batch first for bmm:
        context_view = context_embedding.transpose(0, 1)
        # Get the linear transformation:
        Wq = self.linear(question_embedding).transpose(1, 2)
        pWq = context_view.bmm(Wq).squeeze(2)
        return self.softmax(pWq)


class Reader(nn.Module):
    def __init__(self, embedding_size, features_size, hidden_size=128):
        super(Reader, self).__init__()
        self.paragraph_encoder = ParagraphEncoder(embedding_size, features_size)
        self.question_encoder = QuestionEncoder(embedding_size)
        self.start_bilinear_attention = BilinearAttention(2*hidden_size, 2*hidden_size)
        self.end_bilinear_attention = BilinearAttention(2*hidden_size, 2*hidden_size)
        self.loss = nn.NLLLoss()


    def compute_Loss(self, predicted_vector, gold_label):
      return self.loss(predicted_vector, gold_label)	


    def forward(self, context, features, question):
        question_emb = self.question_encoder(question)
        context_emb = self.paragraph_encoder(context, features, question)
        P_start = self.start_bilinear_attention(context_emb, question_emb)
        P_end = self.end_bilinear_attention(context_emb, question_emb)
        return P_start, P_end


class Unanswerable(nn.Module):
    def __init__(self, embedding_size, features_size, hidden_size=128):
        super(Unanswerable, self).__init__()
        self.paragraph_encoder = ParagraphEncoder(embedding_size, features_size)
        self.question_encoder = QuestionEncoder(embedding_size)
        self.bilinear_attention = UnanswerableAttention(2*hidden_size, 2*hidden_size)
        self.linear = Linear(2*hidden_size, 2)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=-1)
        self.logsoftmax = nn.LogSoftmax(dim=-1)
        self.loss = nn.NLLLoss()


    def compute_Loss(self, predicted_vector, gold_label):
      return self.loss(predicted_vector, gold_label)	


    def forward(self, context, features, question):
        question_emb = self.question_encoder(question)
        context_emb = self.paragraph_encoder(context, features, question)
        W = self.bilinear_attention(context_emb, question_emb)
        Wcontext = W.unsqueeze(1).bmm(context_emb.transpose(0,1))
        output = self.relu(self.linear(Wcontext)).squeeze()
        if self.training:
            return self.logsoftmax(output)
        else:
            return self.softmax(output)

# Testing the question encoder.
QuestionEncoder(300)(q).size()
# Testing the Paragraph encoder.
ParagraphEncoder(300, 40)(c, f, q).size()
# Test the reader
res = Reader(300, 40)(c, f, q)
print(res[0].size())
print(res[1].size())
# Test the unanswerable
Unanswerable(300, 40)(c, f, q).size()<jupyter_output><empty_output><jupyter_text>Q2 : Explain your  implementationQ2.1: Explain here your implementation (e.g. **which algorithms/data structures** you used). Make clear which parts were implemented from scratch vs. obtained via an existing package. **Answer:***QA Model*The model is inspired largely from the "Reader" model from the DrQA paper [1]. There are three primary differences: 1) I did not implement the term frequency, since I believed that the RNN / LSTM / GRU should encapsulate that information in the hidden representation; 2) I did not use drop layers, which would have been a good addition and certainly a future direction; and 3) I opted for GRU over the suggested LSTM for faster training. The model has three pieces: 1) Question encoding, 2) Document / paragraph / context encoding, and 3) Start / end labeling.1. Question encoding - A GloVe embedding is encoded by SpaCy - A simple GRU layer followed by - An importance encoding using a linear projection and vector matrix multiplication (Wq). - The importance encoding uses a weighted-average to generate attention for the labeling.2. Document embedding - A SpaCy word embedding + one-hot encoded NER + one-hot encoded POS + exact match flag + lemma match flag + lower match flag - A "Question alignment" uses a cross multiplication of the question and document embeddings (WcWq), which is normalized (Softmax) and appended to the document embedding. - Followed by GRU to generate a hidden representation of the aligned contextual embedding.3. Start / end labeling - A bilinear attention layer is used by multiplying the averaged question attention with each sequential document hidden represention - A softmax is applied over the sequence to generate a probability distribution over the start / end terms.*Unanswerable Model*The model is inspired by the DRQA model [1], except that the Start / End layer is substitued with an attention layer. The attention layer is based on the question attention layer from DRQA, generating a weighted average of the hidden representations. The attention is then fed through a linear layer to generate a binary classification output assigning unanswerable or answerable.1. Question encoding - A GloVe embedding is encoded by SpaCy - A simple GRU layer followed by - An importance encoding using a linear projection and vector matrix multiplication (Wq). - The importance encoding uses a weighted-average to generate attention for the labeling.2. Document embedding - A SpaCy word embedding + one-hot encoded NER + one-hot encoded POS + exact match flag + lemma match flag + lower match flag - A "Question alignment" uses a cross multiplication of the question and document embeddings (WcWq), which is normalized (Softmax) and appended to the document embedding. - Followed by GRU to generate a hidden representation of the aligned contextual embedding.3. Attentive Output - An importance encoding using a linear projection and vector matrix multiplication (Wq). - The importance encoding uses a weighted-average to generate attention for the labeling. - A softmax is applied over the binary output to generate a probability distribution over answerable / unanswerable.*Packages used** SpaCy was used for GloVe embeddings, NER, and POS.* PyTorch for the NN modeling* NumPy, TQDM, Collections, etc. The usualsQ2.2: Explain and motivate any design choices providing the intuition behind them (e.g. which methods you used for your implementation, why?).[1] Chen, Danqi, et al. "Reading wikipedia to answer open-domain questions." arXiv preprint arXiv:1704.00051 (2017).**Answer:**DRQA was chosen to explore the content of the course and because it utilizes alignment and attention. Aside from this, it is naturally extendible to include unanswerable labeling in a similar manner to the start / end prediction.Initially, I designed an unanswerable model that included predicting start, end, and unanswerability, but the performance was poor. There were other options for unanswerability:- Two separate models for prediction- A hyperparameter for a threshold for answer satisfiability- Checking if the best sequences were outperformed by nonsensical cases or didn't select a span as "unanswerable"The latter two options seemed more manually tuning than learning, so I opted for the first. In order to modify DRQA to perform binary labeling, I considered both using the hidden representation as in P3 and performing a linear layer over the last output, but given that the attention contextualized important information about terms, it seemed as effective while maximized parallelizability for performance through matrix multiplications.Q2.3: What features did you use and why? Please motivate your choices.**Answer:**The original reader model included question and document embeddings, POS, NER, and flags for matches that are exact, lemmatized, or lower between the question and document, as well as term frequency. All except the term frequency were adopted in this model. Term frequency is already represented by the word embeddings, attention, and hidden representations and seemed an unnecessary overhead. However, the original paper pointed out that matches are extremely important, so those remained.  Validation Step<jupyter_code>model = Reader(300, 40)
model.to(get_device())
def train(model, dataloader, optimizer):
  model.train()
  total = 0
  loss = 0
  correct = 0

  accumulated_loss = 0
  for c, f, q, u, s, e in tqdm(dataloader, position=0, leave=True, desc="Training Batches"):
    start, end = model(c.to(get_device()), f.to(get_device()), q.to(get_device()))
    start_loss = model.compute_Loss(start, s.to(get_device()))
    end_loss = model.compute_Loss(end, e.to(get_device()))

    accumulated_loss += start_loss.detach().to("cpu")

    # We need to zero out the gradient buffer, otherwise we accumulate all the
    # gradients over the loop. This default behavior is _only_ useful if we
    # intend to run the gradient updates ourselves, which we defer to PyTorch.
    total_loss = start_loss + end_loss
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()
  accumulated_loss /= len(dataloader)
  return accumulated_loss
optimizer = Adam(model.parameters(), lr=0.0001)
train(model, ans_dataloader, optimizer)
torch.save({
            'epoch': 2,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'f1': 0.3905598336673982
            }, "drqa2")
def get_start_end(pred_start, pred_end):
    """
        Given a predicted start and end arrays, generates the most likely pair
        of start and end positions within 15 of each other.
    """
    start = 0
    end = 0
    best = 0
    # A more efficient implementation would do partial dot products and return
    # the highest value between partials.
    for i in range(len(pred_start)):
        for j in range(i, min(i+15, len(pred_start))):
            curr = pred_start[i] * pred_end[j]
            if curr > best:
                best = curr
                start = i
                end = j
    return start, end
def predict_answer(model, context, features, question):
    '''
        Given a question and context, generates the start and end positions.
        
        :param question: The question to answer
        :type question: str
        :param features: Features in the question.
        :type features: str
        :param context: The context in which to search for the answer
        :type context: str
        :return: Start and end positions
        :rtype: Tuple[int, int]
    '''
    # Need to batchify
    q = question.unsqueeze(1).to(get_device())
    f = features.unsqueeze(1).to(get_device())
    c = context.unsqueeze(1).to(get_device())
    start, end = model(c, f, q)
    s_cpu = start.transpose(0, 1).squeeze(1).detach().cpu().numpy()
    e_cpu = end.transpose(0, 1).squeeze(1).detach().cpu().numpy()
    return get_start_end(s_cpu, e_cpu)
def evaluate_qa(model, dataset):
    """
        Takes a QA model and a non-Dataloader dataset and evaluates the model.

        :param model: A trained model to evaluate
        :type model: Reader
        :param dataset: An array containing preprocessed, but not tensorfied,
                        validation dataset.
        :type dataset: Tupe[]
        :return: A pair of (answer, hasAns) that represents the model's prediction
            on the answer and whether this question is answerable
        :rtype: Tuple[str, bool]
    """
    start = time.time()
    model.eval()
    f1s = []
    exacts = []
    for example in tqdm(dataset, position=0, leave=True, desc="QA model validation"):
        text = example['text']
        answer = example['answer_text']
        c, f, q = get_example_tensors(example)
        p_s, p_e = predict_answer(model, c, f, q)
        pred_ans = " ".join(text[p_s:p_e])
        f1s.append(compute_f1(answer, pred_ans))
        exacts.append(compute_exact(answer, pred_ans))

    print("")
    avg_f1 = sum(f1s)/len(f1s)
    avg_exact = sum(exacts)/len(exacts)
    print("Total time:", time.time() - start)
    return avg_f1, avg_exact
# f1_res, exact_res = evaluate_qa(model, ans_val_dataset)
# Model loading code
checkpoint = torch.load("/content/drive/MyDrive/CS 5740/P4/directory/drqa2")
optimizer = Adam(model.parameters(), lr=0.0001)
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
model.load_state_dict(checkpoint['model_state_dict'])
model.to(get_device())
for i in range(1):
    train(model, ans_dataloader, optimizer)
    f1_res, exact_res = evaluate_qa(model, ans_val_dataset)
    print("F1: ", f1_res)
    print("Exact: ", exact_res)
    qa_f1.append(f1_res)
    qa_exact.append(exact_res)
epochs = [i for i in range(len(qa_f1))]
plt.plot(epochs, qa_f1, '-')
plt.xlabel("Epochs")
plt.ylabel("F1 Score")
plt.title("DRQA model QA F1 Scores of epochs")
epochs = [i for i in range(len(qa_f1))]
plt.plot(epochs, qa_exact, '-')
plt.xlabel("Epochs")
plt.ylabel("Exact Match %")
plt.title("DRQA model QA Exact Match over epochs")
model_unans = Unanswerable(300, 40)
model_unans.to(get_device())
def train_unans(model, dataloader, optimizer):
  model.train()
  total = 0
  loss = 0
  correct = 0

  accumulated_loss = 0
  for c, f, q, u, _, _ in tqdm(dataloader, position=0, leave=True, desc="Training Batches"):
    unans = model(c.to(get_device()), f.to(get_device()), q.to(get_device()))
    # print(unans)
    batch_loss = model.compute_Loss(unans, u.to(get_device()))

    accumulated_loss += batch_loss.detach().to("cpu")

    # We need to zero out the gradient buffer, otherwise we accumulate all the
    # gradients over the loop. This default behavior is _only_ useful if we
    # intend to run the gradient updates ourselves, which we defer to PyTorch.
    optimizer.zero_grad()
    batch_loss.backward()
    optimizer.step()
  accumulated_loss /= len(dataloader)
  return accumulated_loss
def evaluate_unans(model, dataloader):
    model.eval()
    correct = 0
    total = 0
    for c, f, q, u, _, _ in tqdm(dataloader, position=0, leave=True, desc="Training Batches"):
        unans = model(c.to(get_device()), f.to(get_device()), q.to(get_device()))
        total += unans.size()[0]
        result = torch.topk(unans, 1, dim=1)[1].squeeze(1)
        correct += (u.to(get_device()) == result).cpu().numpy().sum()
        if total > 1000:
            break
    print()
    return correct / total
optimizer_unans = Adam(model.parameters(), lr=0.0001)
losses_unans = []
accs_unans = []
for epoch in range(3):
    loss = train_unans(model_unans, dataloader, optimizer_unans)
    accuracy = evaluate_unans(model_unans, val_dataloader)
    losses_unans.append(loss)
    accs_unans.append(accuracy)
    print("Epoch", epoch, "loss:", loss)
    print("Epoch", epoch, "accuracy:", accuracy)
evaluate_unans(model_unans, val_dataloader)
torch.save({
            'epoch': 2,
            'model_state_dict': model_unans.state_dict(),
            'optimizer_state_dict': optimizer_unans.state_dict(),
            'accuracy': 0.6063285943505354
            }, "drqa_unans")
import matplotlib.pyplot as plt

plt.rcParams['figure.dpi'] = 150
epochs = [1, 2, 3]
plt.plot(epochs, accs_unans, '-')
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.title("Unanswerable model accuracy")
epochs = [1, 2, 3]
plt.plot(epochs, losses_unans, '-')
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.title("Unanswerable model losses")<jupyter_output><empty_output><jupyter_text>Q3: Results AnalysisQ3.1: Explain here how you evaluated the models. Summarize the performance of your system and any variations that you experimented with on the validation datasets. Put the results into clearly labeled tables or diagrams and include your observations and analysis. **Answer:**The QA+Unanswerable was split into two models for QA and unanswerable questions. To evaluate each model, I used separate metrics. For the DRQA-inspired model for QA, I used F1-score and exact match solely on a validation set of questions with answers. In this case, the performance of the model achieved ~0.42 F1-score using overlapping sequences. For exact match, the highest performance was ~0.23 accuracy. See figures below:![DRQA F1 Scores](https://drive.google.com/uc?export=view&id=1eSUXfb8FiSWSDIvPlPpbURNUL_mIIfzf)![DRQA Exact Scores](https://drive.google.com/uc?export=view&id=1PUhFgmQPlm7sTpGKAa3HEbfz4eP8J7C6)For the Unanswerable model, I measured performance by accuracy in detecting a question that was answerable or unanswerable. The model achieved an accuracy of ~0.71. See figures below:![Unanswerable accuracy](https://drive.google.com/uc?export=view&id=1sm73BT1BjnpdaDfrkEusmsEiiJ5JRi1k)![Unanswerable loss](https://drive.google.com/uc?export=view&id=1bZCFQsvSd7L-F2fKw6goobqDYzlnyJJX)Despite being similar models, the unanswerable question achieved higher accuracy more quickly. This is likely due to the fact that it is easier to detect key words associated with answer spans that are relevant than to achieve an exact answer span.Q3.2: When did the system work well? When did it fail?  Any ideas as to why? How might you improve the system?The system performed poorly in exact matches. The primary challenge is that the start / end predictions are based on a heuristic algorithm, which does not lend to relationships between words in the span. In the future, I would opt for a span labelling algorithm, such as BIO tagging to determine possible answer sequences. However, the algorithm worked particularly well for questions that required numerical / date named-entity recognition, such as "1930" as the year when an event occurred. This is likely due to the encoding included NER as well as the alignment matching the NER.**Answer:**Q3.3: What is the effect of the method used for handling unknown words? Note: Even if you choose to skip over unknown words, that is still a _method_ of handling out-of-vocabulary terms.**Answer:**SpaCy handles unknown words with an UNK embedding. However, since the features included have matching, NER, and POS, there is still sufficient information to make matches between unseen words, such as names and dates.  Part B: Fine-tune a Pre-trained Language Model---In this part, the goal is to build a strong system by fine-tuning a pre-trained language model. You can use **any** public code, **any** pre-trained language models, and **any** hardware resources -- anything other than code specifically written for SQUAD. A reasonable baseline F1 score to aim for is around 0.7.<jupyter_code># Load the base pretrained model from Huggable transformers
from transformers import BertModel, BertTokenizerFast
from math import floor, ceil
bert_tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
# The following code is borrowed and adapted from: https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

def read_squad(data):
    contexts = []
    questions = []
    answers = []
    for group in data:
        for passage in group['paragraphs']:
            context = passage['context']
            for qa in passage['qas']:
                question = qa['question']
                for answer in qa['answers']:
                    contexts.append(context)
                    questions.append(question)
                    answers.append(answer)

    return contexts, questions, answers
contexts, questions, answers = read_squad(data)
# The following code is borrowed and adapted from: https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

def add_end_idx(answers, contexts):
    for answer, context in zip(answers, contexts):
        gold_text = answer['text']
        start_idx = answer['answer_start']
        end_idx = start_idx + len(gold_text)

        # sometimes squad answers are off by a character or two – fix this
        if context[start_idx:end_idx] == gold_text:
            answer['answer_end'] = end_idx
        elif context[start_idx-1:end_idx-1] == gold_text:
            answer['answer_start'] = start_idx - 1
            answer['answer_end'] = end_idx - 1
        elif context[start_idx-2:end_idx-2] == gold_text:
            answer['answer_start'] = start_idx - 2
            answer['answer_end'] = end_idx - 2

add_end_idx(answers, contexts)
train_idx = floor(len(contexts)*0.8)
val_idx = ceil(len(contexts)*0.8)
train_encodings = bert_tokenizer(contexts[:train_idx], questions[:train_idx], truncation=True, padding=True, max_length=512)
val_encodings = bert_tokenizer(contexts[val_idx:], questions[val_idx:], truncation=True, padding=True, max_length=512)
# The following code is adapted from: https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0
# The original code has off-by-one errors, so I changed the code substantially.

def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_pos = encodings.char_to_token(i, answers[i]['answer_start'])
        # if None, the answer passage has been truncated
        if start_pos is None:
            start_pos = bert_tokenizer.model_max_length
        # else:
        #     start_pos += 1
        start_positions.append(start_pos)
        end_pos = encodings.char_to_token(i, answers[i]['answer_end']-1)
        if end_pos is None:
            end_pos = bert_tokenizer.model_max_length
        else:
            end_pos += 1
        end_positions.append(end_pos)
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, answers[:train_idx])
add_token_positions(val_encodings, answers[val_idx:])
# The following code is borrowed and adapted from: https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

class SquadDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

    def __len__(self):
        return len(self.encodings.input_ids)

train_dataset = SquadDataset(train_encodings)
val_dataset = SquadDataset(val_encodings)
train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=1, shuffle=False)
# A basic layer over the BERT model for fine-tuning to QA.
class BertQA(nn.Module):
    def __init__(self, hidden_size=768, num_layers=1):
        super(BertQA, self).__init__()
        self.bert_model = BertModel.from_pretrained('bert-base-uncased')
        self.start_linear = Linear(hidden_size, 1)
        self.end_linear = Linear(hidden_size, 1)
        self.softmax = Softmax(dim=-1)
        self.logsoftmax = LogSoftmax(dim=-1)

    def compute_Loss(self, predicted_vector, gold_label):
      ignore_index = predicted_vector.size(1)
      gold_label.clamp_(0, ignore_index)
      loss = nn.NLLLoss(ignore_index=ignore_index)
      return loss(predicted_vector, gold_label)	

    def forward(self, input_ids, attention_mask):
        embeddings = self.bert_model(input_ids, attention_mask=attention_mask)[0]
        hidden_view = embeddings.view(-1, embeddings.size(-1))
        start = self.start_linear(hidden_view).view(-1, embeddings.size(1))
        start = start.masked_fill_(~attention_mask.bool(), float('-inf'))
        end = self.end_linear(hidden_view).view(-1, embeddings.size(1))
        end = end.masked_fill_(~attention_mask.bool(), float('-inf'))
        if self.training:
            return self.logsoftmax(start), self.logsoftmax(end)
        else:
            return self.softmax(start), self.softmax(end)
model = BertQA()
model.to(get_device())
from transformers import AdamW
def predict(input_ids, s, e):
    s_idx = 0
    e_idx = 0
    best = 0
    # A more efficient implementation would do partial dot products and return
    # the highest value between partials.
    for i in range(len(s)):
        for j in range(i, min(i+15, len(e))):
            curr = s[i] * e[j]
            if curr > best:
                best = curr
                s_idx = i
                e_idx = j
    sep_idx = (input_ids == bert_tokenizer.sep_token_id).nonzero()[0]
    if e_idx < s_idx or (e_idx <= 0 and s_idx <= 0) or e_idx > sep_idx:
        return "", False
    else:
        return bert_tokenizer.decode(input_ids[s_idx:e_idx+1]), True
# Model loading code
# checkpoint = torch.load("/content/drive/MyDrive/CS 5740/P4/Best/bert2_0")
# opt = AdamW(model.parameters(), lr=3e-5)
# opt.load_state_dict(checkpoint['optimizer_state_dict'])
# model.load_state_dict(checkpoint['model_state_dict'])
# model.to(get_device())
def evaluate(model, val_loader):
    torch.set_grad_enabled(False)
    model.eval()
    start_time = time.time()
    f1s = []
    exacts = []
    i = 0
    for batch in tqdm(val_loader, position=0, leave=True, desc="Validation"):
        input_ids = batch['input_ids'].to(get_device())
        attention_mask = batch['attention_mask'].to(get_device())
        start_positions = batch['start_positions'].to(get_device())
        end_positions = batch['end_positions'].to(get_device())
        start, end = model(input_ids, attention_mask=attention_mask)
        s_cpu = start.detach().cpu()[0]
        e_cpu = end.detach().cpu()[0]
        in_cpu = input_ids.cpu()[0]
        pred_ans, pred_unans = predict(in_cpu, s_cpu, e_cpu)
        gold_ans = bert_tokenizer.decode(in_cpu[start_positions.cpu()[0]:end_positions.cpu()[0]+1])
        f1s.append(compute_f1(gold_ans, pred_ans))
        exacts.append(compute_exact(gold_ans, pred_ans))
        i += 1
    torch.set_grad_enabled(True)

    print("")
    avg_f1 = sum(f1s)/len(f1s)
    avg_exact = sum(exacts)/len(exacts)
    print("Total time:", time.time() - start_time)
    return avg_f1, avg_exact
evaluate(model, val_loader)
# opt = AdamW(model.parameters(), lr=3e-5)

f1s = [0.8346922246029286]
exacts = [0.6487025948103793]

for epoch in range(1, 3):
    accumulated_loss = 0
    model.train()
    for batch in tqdm(train_loader, position=0, leave=True, desc="Training Batches"):
        input_ids = batch['input_ids'].to(get_device())
        attention_mask = batch['attention_mask'].to(get_device())
        start, end = model(input_ids, attention_mask=attention_mask)
        start_positions = batch['start_positions'].to(get_device())
        end_positions = batch['end_positions'].to(get_device())
        start_loss = model.compute_Loss(start, start_positions)
        end_loss = model.compute_Loss(end, end_positions)
        total_loss = start_loss + end_loss
        accumulated_loss += total_loss.detach().cpu()
        opt.zero_grad()
        total_loss.backward()
        opt.step()
    f1, exact = evaluate(model, val_loader)
    f1s.append(f1)
    exacts.append(exact)
    print("F1: ", f1)
    print("Exact: ", exact)
    torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': opt.state_dict(),
            }, f"bert2_{epoch}")
    print("Accumulated loss:", accumulated_loss/len(train_loader))
import matplotlib.pyplot as plt

plt.rcParams['figure.dpi'] = 150
epochs = [1, 2, 3]
plt.plot(epochs, f1s, '-')
plt.xlabel("Epochs")
plt.ylabel("F1 score")
plt.title("Bert model F1-scores")
epochs = [1, 2, 3]
plt.plot(epochs, exacts, '-')
plt.xlabel("Epochs")
plt.ylabel("Exact Accuracy")
plt.title("Bert model exact accuracy")
torch.save({
            'epoch': 1,
            'model_state_dict': model.state_dict(),
            }, "/content/drive/MyDrive/CS 5740/P4/content/drive/MyDrive/CS 5740/P4/bert3")

torch.save({
            'epoch': 2,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': opt.state_dict(),
            }, "/content/drive/MyDrive/CS 5740/P4/content/drive/MyDrive/CS 5740/P4/bert4")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07325980262810702, 'fraction_numerical': 0.07255264130033247, 'mean_word_length': 6.109326930291888, 'pattern_counts': {'":': 6, '<': 31, '<?xml version=': 0, '>': 38, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '390643', 'license': 'Apache-2.0', 'n_tokens_mistral': 48326, 'n_tokens_neox': 44159, 'n_words': 5411, 'path': 'nlp/QuestionAnswering.ipynb', 'repo_name': 'elouie/CodeSamples', 'size': '329566'}