<jupyter_start><jupyter_text>Project Workflow ![WhatsApp Image 2021-12-11 at 3.49.47 AM.jpeg](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAGoBQADAREAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD9U6ACgBAQwDKQQeQR3oAWgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA8W/aY0H4y654b8r4b3wbSvIkXVNPtFKX1wOvyPn50KggxptY5I/eBtqgGx8MdH+KFv8ADnwxBF4k8PWUcekWirbXXhy4aaICJcI7fbFyw6E7V57DpQB6jQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAU<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAU<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB4r8MP+Tm/jf/ANenhX/<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFAB<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>QAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQ<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQ<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>AUAFABQAUAFABQAUAFABQAU<KEY>AUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>aNABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAeU+Avh34d1my1rX7u58QRXuo+Jdba6ez8R6jaJIYtRuIIyY4Z1TKxRRoMDogHavfx+Y1qMqdGKi4xhTteEHa8Iyeri3q236s+SynJ8NiadXETc1KVWrflq1Ip2qTitIzS0jFLbZI63Svhr4V0fW7bxHAur3Oo2cUsFvNqGuX195KS7fMCLcTOq7ti5wP4R6V59XMsRWpOi+VRdm0oQje21+WK2uz2MPkuEw9eOJjzOcU0nKpUnZO17Kcmleyv6HU1wHqhQAUAFABQAUAFABQByHgj/AJGb4g/9jHB/6aNOr0cd/Aw3+B/+nKh42V/71jf+vq/9M0Tr6849kKACgAoAKACgAoAKACgAoAKACgDxT9jD/k2HwD/14zf+lMtduYf7zP8AroeflX+50/T9T2uuI9AKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAPN/j6obwXo5/u+NvCB/8r9gP6104X43/AIZf+ks5Mb/DX+KH/pSPSK5jrCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA87+F//I7/ABe/7HO2/wDUe0euiv8ABT/w/wDt0jlw/wDEq/4l/wCkRPRK5zqCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA5D4Wf8ize/9jH4h/8ATvd16Oafx4/4KX/puB42Rf7rP/r7X/8AT1Q6+vOPZOc+IviqTwP4E17xdDa/aZdJsJrmKIqSHkVTsDY527sZPYZNduXYVY7F08O3ZSaXy6/PsebnGPeV5fWxkVdwi2l5paX8r7+RwuqeMPFHgXxPHpv/AAkd34ylOnwm/tJIbe0gtb26vrS2swkkUOYkkM1yQkjytshzkkEt6tLB0MdQc+RUld2d5NuMYSlO6ctWrR1Sirv7vAr5ji8rxSpe1dd8q5k1GKjOdSnCnZxjeKlzTdpOT5Y3u7XejqHxkuBf6jpfh/wjNqdzaS6mLXM7xpeR2KQLMyssT7WN1P5A3DZiKRy4wqNjTyaPJGpWqcqfJfROzm5NbtX9yPNprqkk9WumtxJL2k6WHoucoupbVpSVNRUrNRlZ+0lyK/u+7KTktIuv4n+LfiTSl1LR4/CCafrbW0J0iG8uHk+0vPNbW0ch8uPyti3N1tdPOEgVFbaA4xeFymhV5Kjqc0LvmaVrWUpNau93GN0+Xlu7X01zx3EGKw/tKKo8tWy5FJt3cpQgnouWynOzXPzWSdkpaS/8LRvF8SavYWlvDMLWS7R2v9SS2021gsba3e7nEwtzJkT3iQsG3qPJdspgoZ/suLoQnJ2vbaLcm5ykoq3Nb4YOStZ6pa3uq/t2axVWnBJ2cvikowiqcYOcubk5vjqKLTuvdbvGzTj0z4ya7qbacyeAhAl3faTpk0U2oOtzHdXdpFdzIsQgO4W8Eu9i7JnY/C45qrk1KlzXrXspyVkrWjJxTvzfakrKydrrcmhxHiK7g1h7KUqcGnJ8ylOEakklya+zjK7bcb2e1j1OvAPrAoA8htvGnirw5448e2Oh/CLxP4qgfXLeVrzTLzS4oo3OlWAMRF3dwvuAUNkKVwwwScgeviqcJ4fDOU0vce9/+flTsmfP4CrOnjMao03L94tuX/nzS7tGn/wtP4if9G1ePv8AwaeH/wD5ZVw+wp/8/V90v/kT1frFX/nzL74//JB/wtP4if8ARtXj7/waeH//AJZUewp/8/V90v8A5EPrFX/nzL74/wDyQf8AC0/iJ/0bV4+/8Gnh/wD+WVHsKf8Az9X3S/8AkQ+sVf8AnzL74/8AyQf8LT+In/RtXj7/AMGnh/8A+WVHsKf/AD9X3S/+RD6xV/58y++P/wAkH/C0/iJ/0bV4+/8ABp4f/wDllR7Cn/z9X3S/+RD6xV/58y++P/yQf8LT+In/AEbV4+/8Gnh//wCWVHsKf/P1fdL/AORD6xV/58y++P8A8kH/AAtP4if9G1ePv/Bp4f8A/llR7Cn/AM/V90v/AJEPrFX/AJ8y++P/AMkH/C0/iJ/0bV4+/wDBp4f/APllR7Cn/wA/V90v/kQ+sVf+fMvvj/8AJB/wtP4if9G1ePv/AAaeH/8A5ZUewp/8/V90v/kQ+sVf+fMvvj/8kH/C0/iJ/wBG1ePv/Bp4f/8AllR7Cn/z9X3S/wDkQ+sVf+fMvvj/APJB/wALT+In/RtXj7/waeH/AP5ZUewp/wDP1fdL/wCRD6xV/wCfMvvj/wDJB/wtP4if9G1ePv8AwaeH/wD5ZUewp/8AP1fdL/5EPrFX/nzL74//ACR598ANQ+Lfwo+D3hn4ea/+zr4yutQ0W2eGeaz1bQWhctK7jaX1BWxhh1AroxSo160qkais/KX+Ry4KVfDUI0pUndecf/kj0H/hafxE/wCjavH3/g08P/8Ayyrn9hT/AOfq+6X/AMidX1ir/wA+ZffH/wCSD/hafxE/6Nq8ff8Ag08P/wDyyo9hT/5+r7pf/Ih9Yq/8+ZffH/5IP+Fp/ET/AKNq8ff+DTw//wDLKj2FP/n6vul/8iH1ir/z5l98f/kg/wCFp/ET/o2rx9/4NPD/AP8ALKj2FP8A5+r7pf8AyIfWKv8Az5l98f8A5IP+Fp/ET/o2rx9/4NPD/wD8sqPYU/8An6vul/8AIh9Yq/8APmX3x/8Akg/4Wn8RP+javH3/AINPD/8A8sqPYU/+fq+6X/yIfWKv/PmX3x/+SD/hafxE/wCjavH3/g08P/8Ayyo9hT/5+r7pf/Ih9Yq/8+ZffH/5IP8AhafxE/6Nq8ff+DTw/wD/ACyo9hT/AOfq+6X/AMiH1ir/AM+ZffH/AOSD/hafxE/6Nq8ff+DTw/8A/LKj2FP/AJ+r7pf/ACIfWKv/AD5l98f/AJIP+Fp/ET/o2rx9/wCDTw//APLKj2FP/n6vul/8iH1ir/z5l98f/kg/4Wn8RP8Ao2rx9/4NPD//AMsqPYU/+fq+6X/yIfWKv/PmX3x/+SD/AIWn8RP+javH3/g08P8A/wAsqPYU/wDn6vul/wDIh9Yq/wDPmX3x/wDkg/4Wn8RP+javH3/g08P/APyyo9hT/wCfq+6X/wAiH1ir/wA+ZffH/wCSD/hafxE/6Nq8ff8Ag08P/wDyyo9hT/5+r7pf/Ih9Yq/8+ZffH/5IP+Fp/ET/AKNq8ff+DTw//wDLKj2FP/n6vul/8iH1ir/z5l98f/kg/wCFp/ET/o2rx9/4NPD/AP8ALKj2FP8A5+r7pf8AyIfWKv8Az5l98f8A5IP+Fp/ET/o2rx9/4NPD/wD8sqPYU/8An6vul/8AIh9Yq/8APmX3x/8AkjnfHniX4ieNtDttG/4Z68fWX2bWtH1fzPt3h+Td9h1G3vPLx/aYxv8As+zdn5d+7DY2nSlCnTlze0WzX2uqa/l8zGtUq1oqPspLVPePRp/zeR0X/C0/iJ/0bV4+/wDBp4f/APllWfsKf/P1fdL/AORNvrFX/nzL74//ACQf8LT+In/RtXj7/wAGnh//AOWVHsKf/P1fdL/5EPrFX/nzL74//JB/wtP4if8ARtXj7/waeH//AJZUewp/8/V90v8A5EPrFX/nzL74/wDyQf8AC0/iJ/0bV4+/8Gnh/wD+WVHsKf8Az9X3S/8AkQ+sVf8AnzL74/8AyQf8LT+In/RtXj7/AMGnh/8A+WVHsKf/AD9X3S/+RD6xV/58y++P/wAkH/C0/iJ/0bV4+/8ABp4f/wDllR7Cn/z9X3S/+RD6xV/58y++P/yQf8LT+In/AEbV4+/8Gnh//wCWVHsKf/P1fdL/AORD6xV/58y++P8A8kH/AAtP4if9G1ePv/Bp4f8A/llR7Cn/AM/V90v/AJEPrFX/AJ8y++P/AMkH/C0/iJ/0bV4+/wDBp4f/APllR7Cn/wA/V90v/kQ+sVf+fMvvj/8AJB/wtP4if9G1ePv/AAaeH/8A5ZUewp/8/V90v/kQ+sVf+fMvvj/8kH/C0/iJ/wBG1ePv/Bp4f/8AllR7Cn/z9X3S/wDkQ+sVf+fMvvj/APJB/wALT+In/RtXj7/waeH/AP5ZUewp/wDP1fdL/wCRD6xV/wCfMvvj/wDJB/wtP4if9G1ePv8AwaeH/wD5ZUewp/8AP1fdL/5EPrFX/nzL74//ACQf8LT+In/RtXj7/wAGnh//AOWVHsKf/P1fdL/5EPrFX/nzL74//JB/wtP4if8ARtXj7/waeH//AJZUewp/8/V90v8A5EPrFX/nzL74/wDyQf8AC0/iJ/0bV4+/8Gnh/wD+WVHsKf8Az9X3S/8AkQ+sVf8AnzL74/8AyQf8LT+In/RtXj7/AMGnh/8A+WVHsKf/AD9X3S/+RD6xV/58y++P/wAkH/C0/iJ/0bV4+/8ABp4f/wDllR7Cn/z9X3S/+RD6xV/58y++P/yQf8LT+In/AEbV4+/8Gnh//wCWVHsKf/P1fdL/AORD6xV/58y++P8A8kH/AAtP4if9G1ePv/Bp4f8A/llR7Cn/AM/V90v/AJEPrFX/AJ8y++P/AMkH/C0/iJ/0bV4+/wDBp4f/APllR7Cn/wA/V90v/kQ+sVf+fMvvj/8AJB/wtP4if9G1ePv/AAaeH/8A5ZUewp/8/V90v/kQ+sVf+fMvvj/8kH/C0/iJ/wBG1ePv/Bp4f/8AllR7Cn/z9X3S/wDkQ+sVf+fMvvj/APJEc/xL+LN0q2+ifs4eIYbuR1VZNb1/SLWzQEjc0kltc3MwAGT8sLk4xTVGktZVF8k7/ikvxE8RXekaTv5uKX4Nv8DY+EXgbV/A3hm6HinUrXUPEuv6nda7rt1aLItu15Ow/dwiRmcRQxLDBHkglIVJAJIEV6qqy91WSVl6L/Pc0wtGVGD53eTbb9X+iWi9Dt6wOgKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDkPhZ/yLN7/2MfiH/wBO93Xo5p/Hj/gpf+m4HjZF/us/+vtf/wBPVDr6849kQgMCrAEHgg96A3OfsPh74H0zR7/w7Z+FtNXSNTuDc3OnNAHtXkIQHELZRR8inaoC7gWxkknsqZhiqtSNaVR88VZSvrbXru993rbTY82llGAoUZ4aFKPs5u7ja8b6fZei2WiSV9bXbZYfwX4Oe1srJ/CejNb6bO11ZQmwiKW0zMWMka7cI5YliwwckmoWNxKlKSqSvJWbu9VtZ91boavLME4QpujC0HeK5VaLbvdaaO+t1rcR/BPgySDUrWTwjorQ6zIJtSjNhEVvZA24PMNuJG3c5bJzzTWNxKcJKpK8NI6vReXb5CeWYJxqQdGFqjvJcqtJ73lpq7663J5/DHhq5S3jufD2mSraXjahbh7SNhFdMzO06ZHyyFnZi4+Ylic5JqI4qvG7jNq65Xq9VtZ+VklbY0lgcLNRUqcXyy5lotJNtuS00k22773bZLHoWiQzC5h0axSUXTXwkW3QN9paMxtNnGfMMbMhfqVJGcHFS69WS5XJ2tbd7Xvb0vrba+pccLQjLmUFe/Nsvias5erTtfe2mxerI3CgDkPBH/IzfEH/ALGOD/00adXo47+Bhv8AA/8A05UPGyv/AHrG/wDX1f8ApmidfXnHshQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAF<KEY>QAUAFABQAUAFABQAUAFABQByHws/5Fm9/wCxj8Q/+ne7r0c0/jx/wUv/AE3A8bIv91n/ANfa/wD6eqHX15x7IUAFABQAUAFABQAUAFAHIeCP+Rm+IP8A2McH/po06vRx38DDf4H/AOnKh42V/wC9Y3/r6v8A0zROvrzj2QoAKACgAoAKACgAoAKACgAoAKAC<KEY>AFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB//2Q==)<jupyter_code>!pip install mlxtend --upgrade --no-deps
import re
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
import pandas as pd
from sklearn.utils import shuffle
from sklearn.preprocessing import LabelEncoder
from yellowbrick.cluster import SilhouetteVisualizer
import numpy as np 
import matplotlib.pyplot as plt 
from sklearn.metrics import silhouette_score ,silhouette_samples
from sklearn.cluster import KMeans
from mlxtend.evaluate import bias_variance_decomp
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.manifold import TSNE
from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, plot_confusion_matrix
import nltk
nltk.download('wordnet')
nltk.download("stopwords")
print(stopwords.words('english'))
nltk.download('punkt')
def build_model(model, model_name, X_train, X_test, y_train, y_test, cv ):

  global general_model
  general_model = model.fit(X_train, y_train) # train the model   
  # make prediction of X_test and print Classification report
  global y_pred
  y_pred = model.predict(X_test)
  # 2- Applying K_Fold Cross Vaildation

  cv_accuracies = cross_val_score(estimator = model , X= X_train , y = y_train ,cv = cv)
  accuracy_avg = cv_accuracies.mean()# measure the accuracy of the model (bais)
  test_accuracy = accuracy_score(y_test, y_pred)
  # Evaluate the model performance using metrics.accuracy_score to measure the score
  print("cross_validation accuracy :  ", cv_accuracies)
  print("\n average cross validation accuracy :  " , accuracy_avg)
  print("\n testing accuracy :  "  ,test_accuracy,"\n")
  print("----------------------------------------------------------------------------------------------")

 

  print(classification_report(y_test, y_pred))

  # print Confusion Matrix
  print('\nConfusion Matrix:\n')
  print(confusion_matrix(y_test, y_pred))
  print("----------------------------------------------------------------------------------------------")
  plot_confusion_matrix(general_model, X_test, y_test, xticks_rotation='vertical')
 
  plot_learning_curve(general_model,"Learning Curve of "+ model_name,X_train, y_train, ylim=(0, 1.1), cv=cv, n_jobs=8)

  print("----------------------------------------------------------------------------------------------")
  return general_model, y_pred


def which_one_misclassify(original_data, X_train, y_train, X_test, y_test ,y_pred , document = True):
  errors  = [] 
  correct = []
  predict = []
  gg=np.array(y_test)

  for index ,text in enumerate(X_test) :
    if y_pred[index] != gg[index]:
      if document == True:
        error = original_data.doc_content[index]
      else :
        error  = original_data.paragraph[index]
      errors.append(error)
      corr = gg[index]
      correct.append(corr)
      pred = y_pred[index]
      predict.append(pred)
  document_df = pd.DataFrame()
  document_df['doc_error'] = errors
  document_df['correct']   = correct
  document_df['Predicted'] = predict
  #document_df['book_name'] = 
  print(" The Documents that the Model Misclassify are  : " , len(errors))
  print("----------------------------------------------------------------------------------------------\n")

  #label_encoder object knows how to understand word labels. 
  label_encoder = LabelEncoder()
  X_train_copy = np.copy(X_train)
  X_test_copy = np.copy(X_test)
  y_train_copy = np.copy(y_train)
  y_test_copy = np.copy(y_test)

  avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
         general_model,
         np.array(X_train_copy),
         label_encoder.fit_transform(y_train_copy),
         np.array(X_test_copy),
         label_encoder.fit_transform(y_test_copy),num_rounds=2,
         random_seed=123)
  
  print('Average bias: %.3f' % avg_bias)
  print('Average variance: %.3f' % avg_var)
  print("----------------------------------------------------------------------------------------------")

  return document_df 

def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.1, 1.0,5)):
  
    plt.figure()
    plt.title(title)
    if ylim is not None:
        plt.ylim(*ylim)
    plt.xlabel("Training examples")
    plt.ylabel("Score")
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, '-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, '-', color="g",
             label="Cross-validation score")

    plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left');
    return plt

  
from pathlib import Path
from google.colab import drive
drive.mount('/content/drive')
my_dir_path = "drive/MyDrive/history_data"
doc_list = []
kings_name = []
for file in Path(my_dir_path).iterdir():
    kings_name.append(file.stem)
    ls = [ ]
    with open(file, "r") as file1:
        i = 0;
        for line in file1:
            ls.append(line)
            i = i + 1;
        doc_list.append(ls)
        
document_names = [name for name in  kings_name]        
paragraphs_content = []   #book_content
from pathlib import Path
from google.colab import drive
drive.mount('/content/drive')
my_dir_path = "drive/MyDrive/history_data"
doc_text = []
kings_name = []
for file in Path(my_dir_path).iterdir():
    kings_name.append(file.stem)
    with open(file, "r") as file1:
       text = file1.read()
       doc_text.append(text)

      
doc_data = pd.DataFrame()
doc_data['document']= document_names
doc_data['doc_content'] = doc_text
  
doc_lemma = []
for i in range(len(doc_data['doc_content'])):
  text= re.findall(r"[a-zA-Z]{3,}", doc_data['doc_content'][i])
  lemmatizer = WordNetLemmatizer()
  lst = []
  for j in text:
    W = j.lower()
    w = lemmatizer.lemmatize(W)
    if w not in set(stopwords.words('english')):
      lst.append(str(w))
      l = " ".join(lst)
  doc_lemma.append(l)

doc_data['lemma_doc'] = doc_lemma
doc_data.head()
data = pd.DataFrame()
for i in range(len(doc_list)):
  one_docu_df = pd.DataFrame()
  for j in range(len(doc_list[i])):
    dic = {}
    dic['paragraph'] = doc_list[i][j]
    dic['doc_name'] = document_names[i]
    df = pd.DataFrame(dic , index = [i])
    one_docu_df = one_docu_df.append(df)
  data = data.append(one_docu_df)

data= shuffle(data)
data.reset_index(inplace = True)
dh = data[data['paragraph'].map(len) == 1]
data.drop(dh.index, axis=0, inplace=True)
data.reset_index(inplace=True)
list_lemma = []
for i in range(len(data['paragraph'])):
  text= re.findall(r"[a-zA-Z]{3,}", data['paragraph'][i])
  lemmatizer = WordNetLemmatizer()
  lst = []
  for j in text:
    W = j.lower()
    w = lemmatizer.lemmatize(W)
    if w not in set(stopwords.words('english')):
      lst.append(str(w))
      l = " ".join(lst)
  list_lemma.append(l)

data['lemma'] = list_lemma
data<jupyter_output><empty_output><jupyter_text>doc_data will be used in clustering and then for clssifier 1 data will be used in classifier 2 question as tfidf based documents (tfidf of doc_data) will be passed to classifier1 question as tfidf based documents (tfidf of data ) will be passed to classifier2 question string will be passed to the bm25 and bert<jupyter_code>question = "when was Khufu born?"
question2 = " when was Tutankhamun born?"

text= re.findall(r"[a-zA-Z]{3,}", question)
lemmatizer = WordNetLemmatizer()
lst_word = []
for word in text:
    W = word.lower()
    w = lemmatizer.lemmatize(W)
    if w not in set(stopwords.words('english')):
      lst_word.append(str(w))
question_words =[" ".join(lst_word)]
question_words = pd.Series(question_words)
type(question_words)

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer_doc = TfidfVectorizer()
tfidf_doc = vectorizer_doc.fit_transform(doc_data['lemma_doc'])
question_tfidf_1 = vectorizer_doc.transform(question_words)
question_4_cls_1 = question_tfidf_1.toarray()
print(question_4_cls_1.shape)


from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
tfidf_doc = vectorizer.fit_transform(doc_data['lemma_doc'])
X_tfidf_doc = tfidf_doc.toarray()
tfidf_df_doc = pd.DataFrame(tfidf_doc.toarray(), columns=vectorizer.get_feature_names())
tfidf_df_doc
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
tfidf = vectorizer.fit_transform(data['lemma'])
X_tfidf = tfidf.toarray()
tfidf_df = pd.DataFrame(tfidf.toarray(), columns=vectorizer.get_feature_names())
tfidf_df<jupyter_output><empty_output><jupyter_text>**K-Means**<jupyter_code>def build_kMeans(clusters, X_data):
  kmeans_model= KMeans(n_clusters= clusters, init='k-means++', random_state=0)
  y_pred = kmeans_model.fit_predict(X_data)
  return kmeans_model, y_pred

# method to reduce dimension of data to visualize it
def data_with_tsna(desired_data):
  tsna = TSNE(n_components= 2, random_state= 42, init='random')
  data_tsna = tsna.fit_transform(desired_data) 
  return data_tsna


def visualize_silhouette_kmeans(X_data, model, title):
  # Instantiate the clustering model and visualizer
  visualizer = SilhouetteVisualizer(model, colors='yellowbrick')
  # Fit the data to the visualizer
  visualizer.fit(X_data)
  plt.title("Silhouette Score of K-means With " + title)

tfidf_data_doc = data_with_tsna(X_tfidf_doc)
# tfidf_data_pca = data_with_pca(X_tfidf)
tfidf_kmeans, tfidf_kmeans_pred = build_kMeans(3,tfidf_data_doc)
print("Silhouette Score of K-means With TFIDF    :  {:.4f}".format(silhouette_score(tfidf_data_doc, tfidf_kmeans_pred)))
visualize_silhouette_kmeans(tfidf_data_doc, tfidf_kmeans, "TFIDF");
doc_data['Cluster_label'] = tfidf_kmeans_pred
doc_data
# plot word cloud for the most frequent 50 words in each book  
import wordcloud

for label in doc_data['Cluster_label'].unique():
  books = doc_data[doc_data["Cluster_label"]==label]["lemma_doc"]

  #wordcloud 
  print(f"\n The most frequent 50 words of Cluster: {label}\n")
  wc = wordcloud.WordCloud(background_color='white', max_words=50, 
                          max_font_size=35)
  wc = wc.generate(str(books))
  plt.axis('off')
  plt.imshow(wc, cmap=None)
  plt.show()<jupyter_output>The most frequent 50 words of Cluster: 0<jupyter_text>First Classification with Cluster Labels  KNN<jupyter_code>from sklearn.model_selection import train_test_split
X_data= tfidf_doc.toarray()
y_data = doc_data['Cluster_label']

# Split data to train and test
X_train1, X_test1, y_train1, y_test1 = train_test_split(X_data , y_data, test_size= 0.3, random_state= 42)
knn_cluster =  KNeighborsClassifier(n_neighbors = 5, algorithm= 'kd_tree', p=2)

KNN_model1, knn_pred1 = build_model(knn_cluster, "KNN with TFIDF", X_train1, X_test1, y_train1, y_test1 ,4)
which_one_misclassify(doc_data ,X_train1, y_train1, X_test1, y_test1 ,knn_pred1 , document = True)<jupyter_output>cross_validation accuracy :   [0.42857143 0.33333333 0.66666667 0.5       ]

 average cross validation accuracy :   0.4821428571428571

 testing accuracy :   0.75 

----------------------------------------------------------------------------------------------
              precision    recall  f1-score   support

           0       0.71      1.00      0.83         5
           1       0.80      1.00      0.89         4
           2       0.00      0.00      0.00         3

    accuracy                           0.75        12
   macro avg       0.50      0.67      0.57        12
weighted avg       0.56      0.75      0.64        12


Confusion Matrix:

[[5 0 0]
 [0 4 0]
 [2 1 0]]
----------------------------------------------------------------------------------------------<jupyter_text>Decision Tree<jupyter_code># Build DT model
Decision_tree_tfidf = DecisionTreeClassifier(random_state=0, max_depth= 28)
DT_model, DT_pred1  = build_model(Decision_tree_tfidf, "Decision Tree with TFIDF", X_train1, X_test1, y_train1, y_test1,4)
which_one_misclassify(doc_data , X_train1, y_train1, X_test1, y_test1 ,DT_pred1 ,document=True)<jupyter_output>cross_validation accuracy :   [0.42857143 0.33333333 0.33333333 0.5       ]

 average cross validation accuracy :   0.3988095238095238

 testing accuracy :   0.5 

----------------------------------------------------------------------------------------------
              precision    recall  f1-score   support

           0       0.75      0.60      0.67         5
           1       0.67      0.50      0.57         4
           2       0.20      0.33      0.25         3

    accuracy                           0.50        12
   macro avg       0.54      0.48      0.50        12
weighted avg       0.58      0.50      0.53        12


Confusion Matrix:

[[3 0 2]
 [0 2 2]
 [1 1 1]]
----------------------------------------------------------------------------------------------<jupyter_text>SVM<jupyter_code># Build SVM model
svm_tfidf = svm.SVC(kernel='linear')
svm_model , svm_pred= build_model(svm_tfidf, "SVM with TFIDF", X_train1, X_test1, y_train1, y_test1 ,4 )
which_one_misclassify(doc_data , X_train1, y_train1, X_test1, y_test1 ,svm_pred , document=True)


predicted_cluster = KNN_model1.predict(question_4_cls_1)
classifier1_output= predicted_cluster[0]

import pickle
with open(r'first_classifier.pkl','wb') as cl1_model_pkl:
  pickle.dump(KNN_model1, cl1_model_pkl, protocol=2)
classifier1_output<jupyter_output><empty_output><jupyter_text>Second Classification with document labels Ranking Documents with BM25<jupyter_code>!pip install rank_bm25
from rank_bm25 import BM25Okapi
docu_4_cls2= doc_data[doc_data["Cluster_label"] == classifier1_output]
candidate_docs = list(docu_4_cls2['doc_content'])
tokenized_corpus = [doc.split(" ") for doc in candidate_docs]
bm25 = BM25Okapi(tokenized_corpus)

tokenized_query = question.split(" ")
doc_scores = bm25.get_scores(tokenized_query)
top_rank_para =  bm25.get_top_n(tokenized_query, candidate_docs, n=3)[1]
candidate_document = docu_4_cls2.document[docu_4_cls2['doc_content']==top_rank_para ]
candidate_document = list(candidate_document)[0]
candidate_document<jupyter_output><empty_output><jupyter_text>Ranking Paragrahs with BM25<jupyter_code>selected_document = data[data['doc_name'] == candidate_document]
selected_document = selected_document['paragraph']
corpus = list(selected_document)
tokenized_paras = [doc.split(" ") for doc in corpus]
bm25 = BM25Okapi(tokenized_paras)
tokenized_query = question.split(" ")

doc_scores = bm25.get_scores(tokenized_query)
top_rank_para =  bm25.get_top_n(tokenized_query, corpus, n=3)[2]
top_rank_para<jupyter_output><empty_output><jupyter_text>langauage model (BERT)<jupyter_code>!pip install transformers
from transformers import BertForQuestionAnswering
import torch
from transformers import BertTokenizer
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
# Apply the tokenizer to the input text, treating them as a text-pair.
input_ids = tokenizer.encode(question, top_rank_para)

#print('The input has a total of {:} tokens.'.format(len(input_ids)))
# # BERT only needs the token IDs, but for the purpose of inspecting the 
# # tokenizer's behavior, let's also get the token strings and display them.
# tokens = tokenizer.convert_ids_to_tokens(input_ids)

# # For each token and its id...
# for token, id in zip(tokens, input_ids):
    
#     # If this is the [SEP] token, add some space around it to make it stand out.
#     if id == tokenizer.sep_token_id:
#         print('')
    
#     # Print the token string and its ID in two columns.
#     print('{:<12} {:>6,}'.format(token, id))

#     if id == tokenizer.sep_token_id:
#         print('')
    
# Search the input_ids for the first instance of the `[SEP]` token.
sep_index = input_ids.index(tokenizer.sep_token_id)

# The number of segment A tokens includes the [SEP] token istelf.
num_seg_a = sep_index + 1

# The remainder are segment B.
num_seg_b = len(input_ids) - num_seg_a

# Construct the list of 0s and 1s.
segment_ids = [0]*num_seg_a + [1]*num_seg_b

# There should be a segment_id for every input token.
assert len(segment_ids) == len(input_ids)
# Run our example through the model.
outputs = model(torch.tensor([input_ids]), # The tokens representing our input text.
                             token_type_ids=torch.tensor([segment_ids]), # The segment IDs to differentiate question from answer_text
                             return_dict=True) 

start_scores = outputs.start_logits
end_scores = outputs.end_logits

# Find the tokens with the highest `start` and `end` scores.
answer_start = torch.argmax(start_scores)
answer_end = torch.argmax(end_scores)

# Combine the tokens in the answer and print it out.
answer = ' '.join(tokens[answer_start:answer_end+1])
# Start with the first token.
answer = tokens[answer_start]

# Select the remaining answer tokens and join them with whitespace.
for i in range(answer_start + 1, answer_end + 1):
    
    # If it's a subword token, then recombine it with the previous token.
    if tokens[i][0:2] == '##':
        answer += tokens[i][2:]
    
    # Otherwise, add a space then the token.
    else:
        answer += ' ' + tokens[i]

print('Answer: "' + answer + '"')
# import pickle
# with open(r'linearmodel.pkl','wb') as model_pkl:
#     pickle.dump(model, model_pkl, protocol=2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12052923904199243, 'fraction_numerical': 0.0518625959599533, 'mean_word_length': 4.957428872497366, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '410088', 'license': 'MIT', 'n_tokens_mistral': 14036, 'n_tokens_neox': 13308, 'n_words': 1384, 'path': 'question_answering_(final_NLP_prpject_).ipynb', 'repo_name': 'Aboalarbe/Question-Answering-NMEC-Guide', 'size': '109402'}