\n",
+ "\n",
+ " \n",
+ " ![](\"https://raw.githubusercontent.com/ultralytics/assets/main/yolov5/v70/splash.png\")
\n",
+ "\n",
+ "\n",
+ "
\n", + "![\"Run](\"https://assets.paperspace.io/img/gradient-badge.svg\")
\n",
+ " ![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
\n",
+ " ![\"Open](\"https://kaggle.com/static/images/open-in-kaggle.svg\")
\n",
+ "
\n", + "\n", + "This YOLOv5 🚀 notebook by Ultralytics presents simple train, validate and predict examples to help start your AI adventure.
See GitHub for community support or contact us for professional support.\n", + "\n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7mGmQbAO5pQb"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "Clone GitHub [repository](https://github.com/ultralytics/yolov5), install [dependencies](https://github.com/ultralytics/yolov5/blob/master/requirements.txt) and check PyTorch and GPU."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wbvMlHd_QwMG",
+ "outputId": "0806e375-610d-4ec0-c867-763dbb518279"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "YOLOv5 🚀 v7.0-3-g61ebf5e Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Setup complete ✅ (2 CPUs, 12.7 GB RAM, 22.6/78.2 GB disk)\n"
+ ]
+ }
+ ],
+ "source": [
+ "!git clone https://github.com/ultralytics/yolov5 # clone\n",
+ "%cd yolov5\n",
+ "%pip install -qr requirements.txt # install\n",
+ "\n",
+ "import torch\n",
+ "import utils\n",
+ "display = utils.notebook_init() # checks"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4JnkELT0cIJg"
+ },
+ "source": [
+ "# 1. Predict\n",
+ "\n",
+ "`classify/predict.py` runs YOLOv5 Classification inference on a variety of sources, downloading models automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases), and saving results to `runs/predict-cls`. Example inference sources are:\n",
+ "\n",
+ "```shell\n",
+ "python classify/predict.py --source 0 # webcam\n",
+ " img.jpg # image \n",
+ " vid.mp4 # video\n",
+ " screen # screenshot\n",
+ " path/ # directory\n",
+ " 'path/*.jpg' # glob\n",
+ " 'https://youtu.be/Zgi9g1ksQHc' # YouTube\n",
+ " 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "zR9ZbuQCH7FX",
+ "outputId": "50504ef7-aa3e-4281-a4e3-d0c7df3c0ffe"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[34m\u001b[1mclassify/predict: \u001b[0mweights=['yolov5s-cls.pt'], source=data/images, data=data/coco128.yaml, imgsz=[224, 224], device=, view_img=False, save_txt=False, nosave=False, augment=False, visualize=False, update=False, project=runs/predict-cls, name=exp, exist_ok=False, half=False, dnn=False, vid_stride=1\n",
+ "YOLOv5 🚀 v7.0-3-g61ebf5e Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n",
+ "\n",
+ "Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s-cls.pt to yolov5s-cls.pt...\n",
+ "100% 10.5M/10.5M [00:00<00:00, 12.3MB/s]\n",
+ "\n",
+ "Fusing layers... \n",
+ "Model summary: 117 layers, 5447688 parameters, 0 gradients, 11.4 GFLOPs\n",
+ "image 1/2 /content/yolov5/data/images/bus.jpg: 224x224 minibus 0.39, police van 0.24, amphibious vehicle 0.05, recreational vehicle 0.04, trolleybus 0.03, 3.9ms\n",
+ "image 2/2 /content/yolov5/data/images/zidane.jpg: 224x224 suit 0.38, bow tie 0.19, bridegroom 0.18, rugby ball 0.04, stage 0.02, 4.6ms\n",
+ "Speed: 0.3ms pre-process, 4.3ms inference, 1.5ms NMS per image at shape (1, 3, 224, 224)\n",
+ "Results saved to \u001b[1mruns/predict-cls/exp\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python classify/predict.py --weights yolov5s-cls.pt --img 224 --source data/images\n",
+ "# display.Image(filename='runs/predict-cls/exp/zidane.jpg', width=600)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hkAzDWJ7cWTr"
+ },
+ "source": [
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n", + "
\n",
+ " \n",
+ " \n",
+ "\n", + "\n", + "This YOLOv5 🚀 notebook by Ultralytics presents simple train, validate and predict examples to help start your AI adventure.
See GitHub for community support or contact us for professional support.\n", + "\n", + "
![](\"https://user-images.githubusercontent.com/26833433/202808393-50deb439-ae1b-4246-a685-7560c9b37211.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0eq1SMWl6Sfn"
+ },
+ "source": [
+ "# 2. Validate\n",
+ "Validate a model's accuracy on the [Imagenet](https://image-net.org/) dataset's `val` or `test` splits. Models are downloaded automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases). To show results by class use the `--verbose` flag."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "WQPtK1QYVaD_",
+ "outputId": "20fc0630-141e-4a90-ea06-342cbd7ce496"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "--2022-11-22 19:53:40-- https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar\n",
+ "Resolving image-net.org (image-net.org)... 171.64.68.16\n",
+ "Connecting to image-net.org (image-net.org)|171.64.68.16|:443... connected.\n",
+ "HTTP request sent, awaiting response... 200 OK\n",
+ "Length: 6744924160 (6.3G) [application/x-tar]\n",
+ "Saving to: ‘ILSVRC2012_img_val.tar’\n",
+ "\n",
+ "ILSVRC2012_img_val. 100%[===================>] 6.28G 16.1MB/s in 10m 52s \n",
+ "\n",
+ "2022-11-22 20:04:32 (9.87 MB/s) - ‘ILSVRC2012_img_val.tar’ saved [6744924160/6744924160]\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Download Imagenet val (6.3G, 50000 images)\n",
+ "!bash data/scripts/get_imagenet.sh --val"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "X58w8JLpMnjH",
+ "outputId": "41843132-98e2-4c25-d474-4cd7b246fb8e"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[34m\u001b[1mclassify/val: \u001b[0mdata=../datasets/imagenet, weights=['yolov5s-cls.pt'], batch_size=128, imgsz=224, device=, workers=8, verbose=True, project=runs/val-cls, name=exp, exist_ok=False, half=True, dnn=False\n",
+ "YOLOv5 🚀 v7.0-3-g61ebf5e Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n",
+ "\n",
+ "Fusing layers... \n",
+ "Model summary: 117 layers, 5447688 parameters, 0 gradients, 11.4 GFLOPs\n",
+ "validating: 100% 391/391 [04:57<00:00, 1.31it/s]\n",
+ " Class Images top1_acc top5_acc\n",
+ " all 50000 0.715 0.902\n",
+ " tench 50 0.94 0.98\n",
+ " goldfish 50 0.88 0.92\n",
+ " great white shark 50 0.78 0.96\n",
+ " tiger shark 50 0.68 0.96\n",
+ " hammerhead shark 50 0.82 0.92\n",
+ " electric ray 50 0.76 0.9\n",
+ " stingray 50 0.7 0.9\n",
+ " cock 50 0.78 0.92\n",
+ " hen 50 0.84 0.96\n",
+ " ostrich 50 0.98 1\n",
+ " brambling 50 0.9 0.96\n",
+ " goldfinch 50 0.92 0.98\n",
+ " house finch 50 0.88 0.96\n",
+ " junco 50 0.94 0.98\n",
+ " indigo bunting 50 0.86 0.88\n",
+ " American robin 50 0.9 0.96\n",
+ " bulbul 50 0.84 0.96\n",
+ " jay 50 0.9 0.96\n",
+ " magpie 50 0.84 0.96\n",
+ " chickadee 50 0.9 1\n",
+ " American dipper 50 0.82 0.92\n",
+ " kite 50 0.76 0.94\n",
+ " bald eagle 50 0.92 1\n",
+ " vulture 50 0.96 1\n",
+ " great grey owl 50 0.94 0.98\n",
+ " fire salamander 50 0.96 0.98\n",
+ " smooth newt 50 0.58 0.94\n",
+ " newt 50 0.74 0.9\n",
+ " spotted salamander 50 0.86 0.94\n",
+ " axolotl 50 0.86 0.96\n",
+ " American bullfrog 50 0.78 0.92\n",
+ " tree frog 50 0.84 0.96\n",
+ " tailed frog 50 0.48 0.8\n",
+ " loggerhead sea turtle 50 0.68 0.94\n",
+ " leatherback sea turtle 50 0.5 0.8\n",
+ " mud turtle 50 0.64 0.84\n",
+ " terrapin 50 0.52 0.98\n",
+ " box turtle 50 0.84 0.98\n",
+ " banded gecko 50 0.7 0.88\n",
+ " green iguana 50 0.76 0.94\n",
+ " Carolina anole 50 0.58 0.96\n",
+ "desert grassland whiptail lizard 50 0.82 0.94\n",
+ " agama 50 0.74 0.92\n",
+ " frilled-necked lizard 50 0.84 0.86\n",
+ " alligator lizard 50 0.58 0.78\n",
+ " Gila monster 50 0.72 0.8\n",
+ " European green lizard 50 0.42 0.9\n",
+ " chameleon 50 0.76 0.84\n",
+ " Komodo dragon 50 0.86 0.96\n",
+ " Nile crocodile 50 0.7 0.84\n",
+ " American alligator 50 0.76 0.96\n",
+ " triceratops 50 0.9 0.94\n",
+ " worm snake 50 0.76 0.88\n",
+ " ring-necked snake 50 0.8 0.92\n",
+ " eastern hog-nosed snake 50 0.58 0.88\n",
+ " smooth green snake 50 0.6 0.94\n",
+ " kingsnake 50 0.82 0.9\n",
+ " garter snake 50 0.88 0.94\n",
+ " water snake 50 0.7 0.94\n",
+ " vine snake 50 0.66 0.76\n",
+ " night snake 50 0.34 0.82\n",
+ " boa constrictor 50 0.8 0.96\n",
+ " African rock python 50 0.48 0.76\n",
+ " Indian cobra 50 0.82 0.94\n",
+ " green mamba 50 0.54 0.86\n",
+ " sea snake 50 0.62 0.9\n",
+ " Saharan horned viper 50 0.56 0.86\n",
+ "eastern diamondback rattlesnake 50 0.6 0.86\n",
+ " sidewinder 50 0.28 0.86\n",
+ " trilobite 50 0.98 0.98\n",
+ " harvestman 50 0.86 0.94\n",
+ " scorpion 50 0.86 0.94\n",
+ " yellow garden spider 50 0.92 0.96\n",
+ " barn spider 50 0.38 0.98\n",
+ " European garden spider 50 0.62 0.98\n",
+ " southern black widow 50 0.88 0.94\n",
+ " tarantula 50 0.94 1\n",
+ " wolf spider 50 0.82 0.92\n",
+ " tick 50 0.74 0.84\n",
+ " centipede 50 0.68 0.82\n",
+ " black grouse 50 0.88 0.98\n",
+ " ptarmigan 50 0.78 0.94\n",
+ " ruffed grouse 50 0.88 1\n",
+ " prairie grouse 50 0.92 1\n",
+ " peacock 50 0.88 0.9\n",
+ " quail 50 0.9 0.94\n",
+ " partridge 50 0.74 0.96\n",
+ " grey parrot 50 0.9 0.96\n",
+ " macaw 50 0.88 0.98\n",
+ "sulphur-crested cockatoo 50 0.86 0.92\n",
+ " lorikeet 50 0.96 1\n",
+ " coucal 50 0.82 0.88\n",
+ " bee eater 50 0.96 0.98\n",
+ " hornbill 50 0.9 0.96\n",
+ " hummingbird 50 0.88 0.96\n",
+ " jacamar 50 0.92 0.94\n",
+ " toucan 50 0.84 0.94\n",
+ " duck 50 0.76 0.94\n",
+ " red-breasted merganser 50 0.86 0.96\n",
+ " goose 50 0.74 0.96\n",
+ " black swan 50 0.94 0.98\n",
+ " tusker 50 0.54 0.92\n",
+ " echidna 50 0.98 1\n",
+ " platypus 50 0.72 0.84\n",
+ " wallaby 50 0.78 0.88\n",
+ " koala 50 0.84 0.92\n",
+ " wombat 50 0.78 0.84\n",
+ " jellyfish 50 0.88 0.96\n",
+ " sea anemone 50 0.72 0.9\n",
+ " brain coral 50 0.88 0.96\n",
+ " flatworm 50 0.8 0.98\n",
+ " nematode 50 0.86 0.9\n",
+ " conch 50 0.74 0.88\n",
+ " snail 50 0.78 0.88\n",
+ " slug 50 0.74 0.82\n",
+ " sea slug 50 0.88 0.98\n",
+ " chiton 50 0.88 0.98\n",
+ " chambered nautilus 50 0.88 0.92\n",
+ " Dungeness crab 50 0.78 0.94\n",
+ " rock crab 50 0.68 0.86\n",
+ " fiddler crab 50 0.64 0.86\n",
+ " red king crab 50 0.76 0.96\n",
+ " American lobster 50 0.78 0.96\n",
+ " spiny lobster 50 0.74 0.88\n",
+ " crayfish 50 0.56 0.86\n",
+ " hermit crab 50 0.78 0.96\n",
+ " isopod 50 0.66 0.78\n",
+ " white stork 50 0.88 0.96\n",
+ " black stork 50 0.84 0.98\n",
+ " spoonbill 50 0.96 1\n",
+ " flamingo 50 0.94 1\n",
+ " little blue heron 50 0.92 0.98\n",
+ " great egret 50 0.9 0.96\n",
+ " bittern 50 0.86 0.94\n",
+ " crane (bird) 50 0.62 0.9\n",
+ " limpkin 50 0.98 1\n",
+ " common gallinule 50 0.92 0.96\n",
+ " American coot 50 0.9 0.98\n",
+ " bustard 50 0.92 0.96\n",
+ " ruddy turnstone 50 0.94 1\n",
+ " dunlin 50 0.86 0.94\n",
+ " common redshank 50 0.9 0.96\n",
+ " dowitcher 50 0.84 0.96\n",
+ " oystercatcher 50 0.86 0.94\n",
+ " pelican 50 0.92 0.96\n",
+ " king penguin 50 0.88 0.96\n",
+ " albatross 50 0.9 1\n",
+ " grey whale 50 0.84 0.92\n",
+ " killer whale 50 0.92 1\n",
+ " dugong 50 0.84 0.96\n",
+ " sea lion 50 0.82 0.92\n",
+ " Chihuahua 50 0.66 0.84\n",
+ " Japanese Chin 50 0.72 0.98\n",
+ " Maltese 50 0.76 0.94\n",
+ " Pekingese 50 0.84 0.94\n",
+ " Shih Tzu 50 0.74 0.96\n",
+ " King Charles Spaniel 50 0.88 0.98\n",
+ " Papillon 50 0.86 0.94\n",
+ " toy terrier 50 0.48 0.94\n",
+ " Rhodesian Ridgeback 50 0.76 0.98\n",
+ " Afghan Hound 50 0.84 1\n",
+ " Basset Hound 50 0.8 0.92\n",
+ " Beagle 50 0.82 0.96\n",
+ " Bloodhound 50 0.48 0.72\n",
+ " Bluetick Coonhound 50 0.86 0.94\n",
+ " Black and Tan Coonhound 50 0.54 0.8\n",
+ "Treeing Walker Coonhound 50 0.66 0.98\n",
+ " English foxhound 50 0.32 0.84\n",
+ " Redbone Coonhound 50 0.62 0.94\n",
+ " borzoi 50 0.92 1\n",
+ " Irish Wolfhound 50 0.48 0.88\n",
+ " Italian Greyhound 50 0.76 0.98\n",
+ " Whippet 50 0.74 0.92\n",
+ " Ibizan Hound 50 0.6 0.86\n",
+ " Norwegian Elkhound 50 0.88 0.98\n",
+ " Otterhound 50 0.62 0.9\n",
+ " Saluki 50 0.72 0.92\n",
+ " Scottish Deerhound 50 0.86 0.98\n",
+ " Weimaraner 50 0.88 0.94\n",
+ "Staffordshire Bull Terrier 50 0.66 0.98\n",
+ "American Staffordshire Terrier 50 0.64 0.92\n",
+ " Bedlington Terrier 50 0.9 0.92\n",
+ " Border Terrier 50 0.86 0.92\n",
+ " Kerry Blue Terrier 50 0.78 0.98\n",
+ " Irish Terrier 50 0.7 0.96\n",
+ " Norfolk Terrier 50 0.68 0.9\n",
+ " Norwich Terrier 50 0.72 1\n",
+ " Yorkshire Terrier 50 0.66 0.9\n",
+ " Wire Fox Terrier 50 0.64 0.98\n",
+ " Lakeland Terrier 50 0.74 0.92\n",
+ " Sealyham Terrier 50 0.76 0.9\n",
+ " Airedale Terrier 50 0.82 0.92\n",
+ " Cairn Terrier 50 0.76 0.9\n",
+ " Australian Terrier 50 0.48 0.84\n",
+ " Dandie Dinmont Terrier 50 0.82 0.92\n",
+ " Boston Terrier 50 0.92 1\n",
+ " Miniature Schnauzer 50 0.68 0.9\n",
+ " Giant Schnauzer 50 0.72 0.98\n",
+ " Standard Schnauzer 50 0.74 1\n",
+ " Scottish Terrier 50 0.76 0.96\n",
+ " Tibetan Terrier 50 0.48 1\n",
+ "Australian Silky Terrier 50 0.66 0.96\n",
+ "Soft-coated Wheaten Terrier 50 0.74 0.96\n",
+ "West Highland White Terrier 50 0.88 0.96\n",
+ " Lhasa Apso 50 0.68 0.96\n",
+ " Flat-Coated Retriever 50 0.72 0.94\n",
+ " Curly-coated Retriever 50 0.82 0.94\n",
+ " Golden Retriever 50 0.86 0.94\n",
+ " Labrador Retriever 50 0.82 0.94\n",
+ "Chesapeake Bay Retriever 50 0.76 0.96\n",
+ "German Shorthaired Pointer 50 0.8 0.96\n",
+ " Vizsla 50 0.68 0.96\n",
+ " English Setter 50 0.7 1\n",
+ " Irish Setter 50 0.8 0.9\n",
+ " Gordon Setter 50 0.84 0.92\n",
+ " Brittany 50 0.84 0.96\n",
+ " Clumber Spaniel 50 0.92 0.96\n",
+ "English Springer Spaniel 50 0.88 1\n",
+ " Welsh Springer Spaniel 50 0.92 1\n",
+ " Cocker Spaniels 50 0.7 0.94\n",
+ " Sussex Spaniel 50 0.72 0.92\n",
+ " Irish Water Spaniel 50 0.88 0.98\n",
+ " Kuvasz 50 0.66 0.9\n",
+ " Schipperke 50 0.9 0.98\n",
+ " Groenendael 50 0.8 0.94\n",
+ " Malinois 50 0.86 0.98\n",
+ " Briard 50 0.52 0.8\n",
+ " Australian Kelpie 50 0.6 0.88\n",
+ " Komondor 50 0.88 0.94\n",
+ " Old English Sheepdog 50 0.94 0.98\n",
+ " Shetland Sheepdog 50 0.74 0.9\n",
+ " collie 50 0.6 0.96\n",
+ " Border Collie 50 0.74 0.96\n",
+ " Bouvier des Flandres 50 0.78 0.94\n",
+ " Rottweiler 50 0.88 0.96\n",
+ " German Shepherd Dog 50 0.8 0.98\n",
+ " Dobermann 50 0.68 0.96\n",
+ " Miniature Pinscher 50 0.76 0.88\n",
+ "Greater Swiss Mountain Dog 50 0.68 0.94\n",
+ " Bernese Mountain Dog 50 0.96 1\n",
+ " Appenzeller Sennenhund 50 0.22 1\n",
+ " Entlebucher Sennenhund 50 0.64 0.98\n",
+ " Boxer 50 0.7 0.92\n",
+ " Bullmastiff 50 0.78 0.98\n",
+ " Tibetan Mastiff 50 0.88 0.96\n",
+ " French Bulldog 50 0.84 0.94\n",
+ " Great Dane 50 0.54 0.9\n",
+ " St. Bernard 50 0.92 1\n",
+ " husky 50 0.46 0.98\n",
+ " Alaskan Malamute 50 0.76 0.96\n",
+ " Siberian Husky 50 0.46 0.98\n",
+ " Dalmatian 50 0.94 0.98\n",
+ " Affenpinscher 50 0.78 0.9\n",
+ " Basenji 50 0.92 0.94\n",
+ " pug 50 0.94 0.98\n",
+ " Leonberger 50 1 1\n",
+ " Newfoundland 50 0.78 0.96\n",
+ " Pyrenean Mountain Dog 50 0.78 0.96\n",
+ " Samoyed 50 0.96 1\n",
+ " Pomeranian 50 0.98 1\n",
+ " Chow Chow 50 0.9 0.96\n",
+ " Keeshond 50 0.88 0.94\n",
+ " Griffon Bruxellois 50 0.84 0.98\n",
+ " Pembroke Welsh Corgi 50 0.82 0.94\n",
+ " Cardigan Welsh Corgi 50 0.66 0.98\n",
+ " Toy Poodle 50 0.52 0.88\n",
+ " Miniature Poodle 50 0.52 0.92\n",
+ " Standard Poodle 50 0.8 1\n",
+ " Mexican hairless dog 50 0.88 0.98\n",
+ " grey wolf 50 0.82 0.92\n",
+ " Alaskan tundra wolf 50 0.78 0.98\n",
+ " red wolf 50 0.48 0.9\n",
+ " coyote 50 0.64 0.86\n",
+ " dingo 50 0.76 0.88\n",
+ " dhole 50 0.9 0.98\n",
+ " African wild dog 50 0.98 1\n",
+ " hyena 50 0.88 0.96\n",
+ " red fox 50 0.54 0.92\n",
+ " kit fox 50 0.72 0.98\n",
+ " Arctic fox 50 0.94 1\n",
+ " grey fox 50 0.7 0.94\n",
+ " tabby cat 50 0.54 0.92\n",
+ " tiger cat 50 0.22 0.94\n",
+ " Persian cat 50 0.9 0.98\n",
+ " Siamese cat 50 0.96 1\n",
+ " Egyptian Mau 50 0.54 0.8\n",
+ " cougar 50 0.9 1\n",
+ " lynx 50 0.72 0.88\n",
+ " leopard 50 0.78 0.98\n",
+ " snow leopard 50 0.9 0.98\n",
+ " jaguar 50 0.7 0.94\n",
+ " lion 50 0.9 0.98\n",
+ " tiger 50 0.92 0.98\n",
+ " cheetah 50 0.94 0.98\n",
+ " brown bear 50 0.94 0.98\n",
+ " American black bear 50 0.8 1\n",
+ " polar bear 50 0.84 0.96\n",
+ " sloth bear 50 0.72 0.92\n",
+ " mongoose 50 0.7 0.92\n",
+ " meerkat 50 0.82 0.92\n",
+ " tiger beetle 50 0.92 0.94\n",
+ " ladybug 50 0.86 0.94\n",
+ " ground beetle 50 0.64 0.94\n",
+ " longhorn beetle 50 0.62 0.88\n",
+ " leaf beetle 50 0.64 0.98\n",
+ " dung beetle 50 0.86 0.98\n",
+ " rhinoceros beetle 50 0.86 0.94\n",
+ " weevil 50 0.9 1\n",
+ " fly 50 0.78 0.94\n",
+ " bee 50 0.68 0.94\n",
+ " ant 50 0.68 0.78\n",
+ " grasshopper 50 0.5 0.92\n",
+ " cricket 50 0.64 0.92\n",
+ " stick insect 50 0.64 0.92\n",
+ " cockroach 50 0.72 0.8\n",
+ " mantis 50 0.64 0.86\n",
+ " cicada 50 0.9 0.96\n",
+ " leafhopper 50 0.88 0.94\n",
+ " lacewing 50 0.78 0.92\n",
+ " dragonfly 50 0.82 0.98\n",
+ " damselfly 50 0.82 1\n",
+ " red admiral 50 0.94 0.96\n",
+ " ringlet 50 0.86 0.98\n",
+ " monarch butterfly 50 0.9 0.92\n",
+ " small white 50 0.9 1\n",
+ " sulphur butterfly 50 0.92 1\n",
+ "gossamer-winged butterfly 50 0.88 1\n",
+ " starfish 50 0.88 0.92\n",
+ " sea urchin 50 0.84 0.94\n",
+ " sea cucumber 50 0.66 0.84\n",
+ " cottontail rabbit 50 0.72 0.94\n",
+ " hare 50 0.84 0.96\n",
+ " Angora rabbit 50 0.94 0.98\n",
+ " hamster 50 0.96 1\n",
+ " porcupine 50 0.88 0.98\n",
+ " fox squirrel 50 0.76 0.94\n",
+ " marmot 50 0.92 0.96\n",
+ " beaver 50 0.78 0.94\n",
+ " guinea pig 50 0.78 0.94\n",
+ " common sorrel 50 0.96 0.98\n",
+ " zebra 50 0.94 0.96\n",
+ " pig 50 0.5 0.76\n",
+ " wild boar 50 0.84 0.96\n",
+ " warthog 50 0.84 0.96\n",
+ " hippopotamus 50 0.88 0.96\n",
+ " ox 50 0.48 0.94\n",
+ " water buffalo 50 0.78 0.94\n",
+ " bison 50 0.88 0.96\n",
+ " ram 50 0.58 0.92\n",
+ " bighorn sheep 50 0.66 1\n",
+ " Alpine ibex 50 0.92 0.98\n",
+ " hartebeest 50 0.94 1\n",
+ " impala 50 0.82 0.96\n",
+ " gazelle 50 0.7 0.96\n",
+ " dromedary 50 0.9 1\n",
+ " llama 50 0.82 0.94\n",
+ " weasel 50 0.44 0.92\n",
+ " mink 50 0.78 0.96\n",
+ " European polecat 50 0.46 0.9\n",
+ " black-footed ferret 50 0.68 0.96\n",
+ " otter 50 0.66 0.88\n",
+ " skunk 50 0.96 0.96\n",
+ " badger 50 0.86 0.92\n",
+ " armadillo 50 0.88 0.9\n",
+ " three-toed sloth 50 0.96 1\n",
+ " orangutan 50 0.78 0.92\n",
+ " gorilla 50 0.82 0.94\n",
+ " chimpanzee 50 0.84 0.94\n",
+ " gibbon 50 0.76 0.86\n",
+ " siamang 50 0.68 0.94\n",
+ " guenon 50 0.8 0.94\n",
+ " patas monkey 50 0.62 0.82\n",
+ " baboon 50 0.9 0.98\n",
+ " macaque 50 0.8 0.86\n",
+ " langur 50 0.6 0.82\n",
+ " black-and-white colobus 50 0.86 0.9\n",
+ " proboscis monkey 50 1 1\n",
+ " marmoset 50 0.74 0.98\n",
+ " white-headed capuchin 50 0.72 0.9\n",
+ " howler monkey 50 0.86 0.94\n",
+ " titi 50 0.5 0.9\n",
+ "Geoffroy's spider monkey 50 0.42 0.8\n",
+ " common squirrel monkey 50 0.76 0.92\n",
+ " ring-tailed lemur 50 0.72 0.94\n",
+ " indri 50 0.9 0.96\n",
+ " Asian elephant 50 0.58 0.92\n",
+ " African bush elephant 50 0.7 0.98\n",
+ " red panda 50 0.94 0.94\n",
+ " giant panda 50 0.94 0.98\n",
+ " snoek 50 0.74 0.9\n",
+ " eel 50 0.6 0.84\n",
+ " coho salmon 50 0.84 0.96\n",
+ " rock beauty 50 0.88 0.98\n",
+ " clownfish 50 0.78 0.98\n",
+ " sturgeon 50 0.68 0.94\n",
+ " garfish 50 0.62 0.8\n",
+ " lionfish 50 0.96 0.96\n",
+ " pufferfish 50 0.88 0.96\n",
+ " abacus 50 0.74 0.88\n",
+ " abaya 50 0.84 0.92\n",
+ " academic gown 50 0.42 0.86\n",
+ " accordion 50 0.8 0.9\n",
+ " acoustic guitar 50 0.5 0.76\n",
+ " aircraft carrier 50 0.8 0.96\n",
+ " airliner 50 0.92 1\n",
+ " airship 50 0.76 0.82\n",
+ " altar 50 0.64 0.98\n",
+ " ambulance 50 0.88 0.98\n",
+ " amphibious vehicle 50 0.64 0.94\n",
+ " analog clock 50 0.52 0.92\n",
+ " apiary 50 0.82 0.96\n",
+ " apron 50 0.7 0.84\n",
+ " waste container 50 0.4 0.8\n",
+ " assault rifle 50 0.42 0.84\n",
+ " backpack 50 0.34 0.64\n",
+ " bakery 50 0.4 0.68\n",
+ " balance beam 50 0.8 0.98\n",
+ " balloon 50 0.86 0.96\n",
+ " ballpoint pen 50 0.52 0.96\n",
+ " Band-Aid 50 0.7 0.9\n",
+ " banjo 50 0.84 1\n",
+ " baluster 50 0.68 0.94\n",
+ " barbell 50 0.56 0.9\n",
+ " barber chair 50 0.7 0.92\n",
+ " barbershop 50 0.54 0.86\n",
+ " barn 50 0.96 0.96\n",
+ " barometer 50 0.84 0.98\n",
+ " barrel 50 0.56 0.88\n",
+ " wheelbarrow 50 0.66 0.88\n",
+ " baseball 50 0.74 0.98\n",
+ " basketball 50 0.88 0.98\n",
+ " bassinet 50 0.66 0.92\n",
+ " bassoon 50 0.74 0.98\n",
+ " swimming cap 50 0.62 0.88\n",
+ " bath towel 50 0.54 0.78\n",
+ " bathtub 50 0.4 0.88\n",
+ " station wagon 50 0.66 0.84\n",
+ " lighthouse 50 0.78 0.94\n",
+ " beaker 50 0.52 0.68\n",
+ " military cap 50 0.84 0.96\n",
+ " beer bottle 50 0.66 0.88\n",
+ " beer glass 50 0.6 0.84\n",
+ " bell-cot 50 0.56 0.96\n",
+ " bib 50 0.58 0.82\n",
+ " tandem bicycle 50 0.86 0.96\n",
+ " bikini 50 0.56 0.88\n",
+ " ring binder 50 0.64 0.84\n",
+ " binoculars 50 0.54 0.78\n",
+ " birdhouse 50 0.86 0.94\n",
+ " boathouse 50 0.74 0.92\n",
+ " bobsleigh 50 0.92 0.96\n",
+ " bolo tie 50 0.8 0.94\n",
+ " poke bonnet 50 0.64 0.86\n",
+ " bookcase 50 0.66 0.92\n",
+ " bookstore 50 0.62 0.88\n",
+ " bottle cap 50 0.58 0.7\n",
+ " bow 50 0.72 0.86\n",
+ " bow tie 50 0.7 0.9\n",
+ " brass 50 0.92 0.96\n",
+ " bra 50 0.5 0.7\n",
+ " breakwater 50 0.62 0.86\n",
+ " breastplate 50 0.4 0.9\n",
+ " broom 50 0.6 0.86\n",
+ " bucket 50 0.66 0.8\n",
+ " buckle 50 0.5 0.68\n",
+ " bulletproof vest 50 0.5 0.78\n",
+ " high-speed train 50 0.94 0.96\n",
+ " butcher shop 50 0.74 0.94\n",
+ " taxicab 50 0.64 0.86\n",
+ " cauldron 50 0.44 0.66\n",
+ " candle 50 0.48 0.74\n",
+ " cannon 50 0.88 0.94\n",
+ " canoe 50 0.94 1\n",
+ " can opener 50 0.66 0.86\n",
+ " cardigan 50 0.68 0.8\n",
+ " car mirror 50 0.94 0.96\n",
+ " carousel 50 0.94 0.98\n",
+ " tool kit 50 0.56 0.78\n",
+ " carton 50 0.42 0.7\n",
+ " car wheel 50 0.38 0.74\n",
+ "automated teller machine 50 0.76 0.94\n",
+ " cassette 50 0.52 0.8\n",
+ " cassette player 50 0.28 0.9\n",
+ " castle 50 0.78 0.88\n",
+ " catamaran 50 0.78 1\n",
+ " CD player 50 0.52 0.82\n",
+ " cello 50 0.82 1\n",
+ " mobile phone 50 0.68 0.86\n",
+ " chain 50 0.38 0.66\n",
+ " chain-link fence 50 0.7 0.84\n",
+ " chain mail 50 0.64 0.9\n",
+ " chainsaw 50 0.84 0.92\n",
+ " chest 50 0.68 0.92\n",
+ " chiffonier 50 0.26 0.64\n",
+ " chime 50 0.62 0.84\n",
+ " china cabinet 50 0.82 0.96\n",
+ " Christmas stocking 50 0.92 0.94\n",
+ " church 50 0.62 0.9\n",
+ " movie theater 50 0.58 0.88\n",
+ " cleaver 50 0.32 0.62\n",
+ " cliff dwelling 50 0.88 1\n",
+ " cloak 50 0.32 0.64\n",
+ " clogs 50 0.58 0.88\n",
+ " cocktail shaker 50 0.62 0.7\n",
+ " coffee mug 50 0.44 0.72\n",
+ " coffeemaker 50 0.64 0.92\n",
+ " coil 50 0.66 0.84\n",
+ " combination lock 50 0.64 0.84\n",
+ " computer keyboard 50 0.7 0.82\n",
+ " confectionery store 50 0.54 0.86\n",
+ " container ship 50 0.82 0.98\n",
+ " convertible 50 0.78 0.98\n",
+ " corkscrew 50 0.82 0.92\n",
+ " cornet 50 0.46 0.88\n",
+ " cowboy boot 50 0.64 0.8\n",
+ " cowboy hat 50 0.64 0.82\n",
+ " cradle 50 0.38 0.8\n",
+ " crane (machine) 50 0.78 0.94\n",
+ " crash helmet 50 0.92 0.96\n",
+ " crate 50 0.52 0.82\n",
+ " infant bed 50 0.74 1\n",
+ " Crock Pot 50 0.78 0.9\n",
+ " croquet ball 50 0.9 0.96\n",
+ " crutch 50 0.46 0.7\n",
+ " cuirass 50 0.54 0.86\n",
+ " dam 50 0.74 0.92\n",
+ " desk 50 0.6 0.86\n",
+ " desktop computer 50 0.54 0.94\n",
+ " rotary dial telephone 50 0.88 0.94\n",
+ " diaper 50 0.68 0.84\n",
+ " digital clock 50 0.54 0.76\n",
+ " digital watch 50 0.58 0.86\n",
+ " dining table 50 0.76 0.9\n",
+ " dishcloth 50 0.94 1\n",
+ " dishwasher 50 0.44 0.78\n",
+ " disc brake 50 0.98 1\n",
+ " dock 50 0.54 0.94\n",
+ " dog sled 50 0.84 1\n",
+ " dome 50 0.72 0.92\n",
+ " doormat 50 0.56 0.82\n",
+ " drilling rig 50 0.84 0.96\n",
+ " drum 50 0.38 0.68\n",
+ " drumstick 50 0.56 0.72\n",
+ " dumbbell 50 0.62 0.9\n",
+ " Dutch oven 50 0.7 0.84\n",
+ " electric fan 50 0.82 0.86\n",
+ " electric guitar 50 0.62 0.84\n",
+ " electric locomotive 50 0.92 0.98\n",
+ " entertainment center 50 0.9 0.98\n",
+ " envelope 50 0.44 0.86\n",
+ " espresso machine 50 0.72 0.94\n",
+ " face powder 50 0.7 0.92\n",
+ " feather boa 50 0.7 0.84\n",
+ " filing cabinet 50 0.88 0.98\n",
+ " fireboat 50 0.94 0.98\n",
+ " fire engine 50 0.84 0.9\n",
+ " fire screen sheet 50 0.62 0.76\n",
+ " flagpole 50 0.74 0.88\n",
+ " flute 50 0.36 0.72\n",
+ " folding chair 50 0.62 0.84\n",
+ " football helmet 50 0.86 0.94\n",
+ " forklift 50 0.8 0.92\n",
+ " fountain 50 0.84 0.94\n",
+ " fountain pen 50 0.76 0.92\n",
+ " four-poster bed 50 0.78 0.94\n",
+ " freight car 50 0.96 1\n",
+ " French horn 50 0.76 0.92\n",
+ " frying pan 50 0.36 0.78\n",
+ " fur coat 50 0.84 0.96\n",
+ " garbage truck 50 0.9 0.98\n",
+ " gas mask 50 0.84 0.92\n",
+ " gas pump 50 0.9 0.98\n",
+ " goblet 50 0.68 0.82\n",
+ " go-kart 50 0.9 1\n",
+ " golf ball 50 0.84 0.9\n",
+ " golf cart 50 0.78 0.86\n",
+ " gondola 50 0.98 0.98\n",
+ " gong 50 0.74 0.92\n",
+ " gown 50 0.62 0.96\n",
+ " grand piano 50 0.7 0.96\n",
+ " greenhouse 50 0.8 0.98\n",
+ " grille 50 0.72 0.9\n",
+ " grocery store 50 0.66 0.94\n",
+ " guillotine 50 0.86 0.92\n",
+ " barrette 50 0.52 0.66\n",
+ " hair spray 50 0.5 0.74\n",
+ " half-track 50 0.78 0.9\n",
+ " hammer 50 0.56 0.76\n",
+ " hamper 50 0.64 0.84\n",
+ " hair dryer 50 0.56 0.74\n",
+ " hand-held computer 50 0.42 0.86\n",
+ " handkerchief 50 0.78 0.94\n",
+ " hard disk drive 50 0.76 0.84\n",
+ " harmonica 50 0.7 0.88\n",
+ " harp 50 0.88 0.96\n",
+ " harvester 50 0.78 1\n",
+ " hatchet 50 0.54 0.74\n",
+ " holster 50 0.66 0.84\n",
+ " home theater 50 0.64 0.94\n",
+ " honeycomb 50 0.56 0.88\n",
+ " hook 50 0.3 0.6\n",
+ " hoop skirt 50 0.64 0.86\n",
+ " horizontal bar 50 0.68 0.98\n",
+ " horse-drawn vehicle 50 0.88 0.94\n",
+ " hourglass 50 0.88 0.96\n",
+ " iPod 50 0.76 0.94\n",
+ " clothes iron 50 0.82 0.88\n",
+ " jack-o'-lantern 50 0.98 0.98\n",
+ " jeans 50 0.68 0.84\n",
+ " jeep 50 0.72 0.9\n",
+ " T-shirt 50 0.72 0.96\n",
+ " jigsaw puzzle 50 0.84 0.94\n",
+ " pulled rickshaw 50 0.86 0.94\n",
+ " joystick 50 0.8 0.9\n",
+ " kimono 50 0.84 0.96\n",
+ " knee pad 50 0.62 0.88\n",
+ " knot 50 0.66 0.8\n",
+ " lab coat 50 0.8 0.96\n",
+ " ladle 50 0.36 0.64\n",
+ " lampshade 50 0.48 0.84\n",
+ " laptop computer 50 0.26 0.88\n",
+ " lawn mower 50 0.78 0.96\n",
+ " lens cap 50 0.46 0.72\n",
+ " paper knife 50 0.26 0.5\n",
+ " library 50 0.54 0.9\n",
+ " lifeboat 50 0.92 0.98\n",
+ " lighter 50 0.56 0.78\n",
+ " limousine 50 0.76 0.92\n",
+ " ocean liner 50 0.88 0.94\n",
+ " lipstick 50 0.74 0.9\n",
+ " slip-on shoe 50 0.74 0.92\n",
+ " lotion 50 0.5 0.86\n",
+ " speaker 50 0.52 0.68\n",
+ " loupe 50 0.32 0.52\n",
+ " sawmill 50 0.72 0.9\n",
+ " magnetic compass 50 0.52 0.82\n",
+ " mail bag 50 0.68 0.92\n",
+ " mailbox 50 0.82 0.92\n",
+ " tights 50 0.22 0.94\n",
+ " tank suit 50 0.24 0.9\n",
+ " manhole cover 50 0.96 0.98\n",
+ " maraca 50 0.74 0.9\n",
+ " marimba 50 0.84 0.94\n",
+ " mask 50 0.44 0.82\n",
+ " match 50 0.66 0.9\n",
+ " maypole 50 0.96 1\n",
+ " maze 50 0.8 0.96\n",
+ " measuring cup 50 0.54 0.76\n",
+ " medicine chest 50 0.6 0.84\n",
+ " megalith 50 0.8 0.92\n",
+ " microphone 50 0.52 0.7\n",
+ " microwave oven 50 0.48 0.72\n",
+ " military uniform 50 0.62 0.84\n",
+ " milk can 50 0.68 0.82\n",
+ " minibus 50 0.7 1\n",
+ " miniskirt 50 0.46 0.76\n",
+ " minivan 50 0.38 0.8\n",
+ " missile 50 0.4 0.84\n",
+ " mitten 50 0.76 0.88\n",
+ " mixing bowl 50 0.8 0.92\n",
+ " mobile home 50 0.54 0.78\n",
+ " Model T 50 0.92 0.96\n",
+ " modem 50 0.58 0.86\n",
+ " monastery 50 0.44 0.9\n",
+ " monitor 50 0.4 0.86\n",
+ " moped 50 0.56 0.94\n",
+ " mortar 50 0.68 0.94\n",
+ " square academic cap 50 0.5 0.84\n",
+ " mosque 50 0.9 1\n",
+ " mosquito net 50 0.9 0.98\n",
+ " scooter 50 0.9 0.98\n",
+ " mountain bike 50 0.78 0.96\n",
+ " tent 50 0.88 0.96\n",
+ " computer mouse 50 0.42 0.82\n",
+ " mousetrap 50 0.76 0.88\n",
+ " moving van 50 0.4 0.72\n",
+ " muzzle 50 0.5 0.72\n",
+ " nail 50 0.68 0.74\n",
+ " neck brace 50 0.56 0.68\n",
+ " necklace 50 0.86 1\n",
+ " nipple 50 0.7 0.88\n",
+ " notebook computer 50 0.34 0.84\n",
+ " obelisk 50 0.8 0.92\n",
+ " oboe 50 0.6 0.84\n",
+ " ocarina 50 0.8 0.86\n",
+ " odometer 50 0.96 1\n",
+ " oil filter 50 0.58 0.82\n",
+ " organ 50 0.82 0.9\n",
+ " oscilloscope 50 0.9 0.96\n",
+ " overskirt 50 0.2 0.7\n",
+ " bullock cart 50 0.7 0.94\n",
+ " oxygen mask 50 0.46 0.84\n",
+ " packet 50 0.5 0.78\n",
+ " paddle 50 0.56 0.94\n",
+ " paddle wheel 50 0.86 0.96\n",
+ " padlock 50 0.74 0.78\n",
+ " paintbrush 50 0.62 0.8\n",
+ " pajamas 50 0.56 0.92\n",
+ " palace 50 0.64 0.96\n",
+ " pan flute 50 0.84 0.86\n",
+ " paper towel 50 0.66 0.84\n",
+ " parachute 50 0.92 0.94\n",
+ " parallel bars 50 0.62 0.96\n",
+ " park bench 50 0.74 0.9\n",
+ " parking meter 50 0.84 0.92\n",
+ " passenger car 50 0.5 0.82\n",
+ " patio 50 0.58 0.84\n",
+ " payphone 50 0.74 0.92\n",
+ " pedestal 50 0.52 0.9\n",
+ " pencil case 50 0.64 0.92\n",
+ " pencil sharpener 50 0.52 0.78\n",
+ " perfume 50 0.7 0.9\n",
+ " Petri dish 50 0.6 0.8\n",
+ " photocopier 50 0.88 0.98\n",
+ " plectrum 50 0.7 0.84\n",
+ " Pickelhaube 50 0.72 0.86\n",
+ " picket fence 50 0.84 0.94\n",
+ " pickup truck 50 0.64 0.92\n",
+ " pier 50 0.52 0.82\n",
+ " piggy bank 50 0.82 0.94\n",
+ " pill bottle 50 0.76 0.86\n",
+ " pillow 50 0.76 0.9\n",
+ " ping-pong ball 50 0.84 0.88\n",
+ " pinwheel 50 0.76 0.88\n",
+ " pirate ship 50 0.76 0.94\n",
+ " pitcher 50 0.46 0.84\n",
+ " hand plane 50 0.84 0.94\n",
+ " planetarium 50 0.88 0.98\n",
+ " plastic bag 50 0.36 0.62\n",
+ " plate rack 50 0.52 0.78\n",
+ " plow 50 0.78 0.88\n",
+ " plunger 50 0.42 0.7\n",
+ " Polaroid camera 50 0.84 0.92\n",
+ " pole 50 0.38 0.74\n",
+ " police van 50 0.76 0.94\n",
+ " poncho 50 0.58 0.86\n",
+ " billiard table 50 0.8 0.88\n",
+ " soda bottle 50 0.56 0.94\n",
+ " pot 50 0.78 0.92\n",
+ " potter's wheel 50 0.9 0.94\n",
+ " power drill 50 0.42 0.72\n",
+ " prayer rug 50 0.7 0.86\n",
+ " printer 50 0.54 0.86\n",
+ " prison 50 0.7 0.9\n",
+ " projectile 50 0.28 0.9\n",
+ " projector 50 0.62 0.84\n",
+ " hockey puck 50 0.92 0.96\n",
+ " punching bag 50 0.6 0.68\n",
+ " purse 50 0.42 0.78\n",
+ " quill 50 0.68 0.84\n",
+ " quilt 50 0.64 0.9\n",
+ " race car 50 0.72 0.92\n",
+ " racket 50 0.72 0.9\n",
+ " radiator 50 0.66 0.76\n",
+ " radio 50 0.64 0.92\n",
+ " radio telescope 50 0.9 0.96\n",
+ " rain barrel 50 0.8 0.98\n",
+ " recreational vehicle 50 0.84 0.94\n",
+ " reel 50 0.72 0.82\n",
+ " reflex camera 50 0.72 0.92\n",
+ " refrigerator 50 0.7 0.9\n",
+ " remote control 50 0.7 0.88\n",
+ " restaurant 50 0.5 0.66\n",
+ " revolver 50 0.82 1\n",
+ " rifle 50 0.38 0.7\n",
+ " rocking chair 50 0.62 0.84\n",
+ " rotisserie 50 0.88 0.92\n",
+ " eraser 50 0.54 0.76\n",
+ " rugby ball 50 0.86 0.94\n",
+ " ruler 50 0.68 0.86\n",
+ " running shoe 50 0.78 0.94\n",
+ " safe 50 0.82 0.92\n",
+ " safety pin 50 0.4 0.62\n",
+ " salt shaker 50 0.66 0.9\n",
+ " sandal 50 0.66 0.86\n",
+ " sarong 50 0.64 0.86\n",
+ " saxophone 50 0.66 0.88\n",
+ " scabbard 50 0.76 0.92\n",
+ " weighing scale 50 0.58 0.78\n",
+ " school bus 50 0.92 1\n",
+ " schooner 50 0.84 1\n",
+ " scoreboard 50 0.9 0.96\n",
+ " CRT screen 50 0.14 0.7\n",
+ " screw 50 0.9 0.98\n",
+ " screwdriver 50 0.3 0.58\n",
+ " seat belt 50 0.88 0.94\n",
+ " sewing machine 50 0.76 0.9\n",
+ " shield 50 0.56 0.82\n",
+ " shoe store 50 0.78 0.96\n",
+ " shoji 50 0.8 0.92\n",
+ " shopping basket 50 0.52 0.88\n",
+ " shopping cart 50 0.76 0.92\n",
+ " shovel 50 0.62 0.84\n",
+ " shower cap 50 0.7 0.84\n",
+ " shower curtain 50 0.64 0.82\n",
+ " ski 50 0.74 0.92\n",
+ " ski mask 50 0.72 0.88\n",
+ " sleeping bag 50 0.68 0.8\n",
+ " slide rule 50 0.72 0.88\n",
+ " sliding door 50 0.44 0.78\n",
+ " slot machine 50 0.94 0.98\n",
+ " snorkel 50 0.86 0.98\n",
+ " snowmobile 50 0.88 1\n",
+ " snowplow 50 0.84 0.98\n",
+ " soap dispenser 50 0.56 0.86\n",
+ " soccer ball 50 0.86 0.96\n",
+ " sock 50 0.62 0.76\n",
+ " solar thermal collector 50 0.72 0.96\n",
+ " sombrero 50 0.6 0.84\n",
+ " soup bowl 50 0.56 0.94\n",
+ " space bar 50 0.34 0.88\n",
+ " space heater 50 0.52 0.74\n",
+ " space shuttle 50 0.82 0.96\n",
+ " spatula 50 0.3 0.6\n",
+ " motorboat 50 0.86 1\n",
+ " spider web 50 0.7 0.9\n",
+ " spindle 50 0.86 0.98\n",
+ " sports car 50 0.6 0.94\n",
+ " spotlight 50 0.26 0.6\n",
+ " stage 50 0.68 0.86\n",
+ " steam locomotive 50 0.94 1\n",
+ " through arch bridge 50 0.84 0.96\n",
+ " steel drum 50 0.82 0.9\n",
+ " stethoscope 50 0.6 0.82\n",
+ " scarf 50 0.5 0.92\n",
+ " stone wall 50 0.76 0.9\n",
+ " stopwatch 50 0.58 0.9\n",
+ " stove 50 0.46 0.74\n",
+ " strainer 50 0.64 0.84\n",
+ " tram 50 0.88 0.96\n",
+ " stretcher 50 0.6 0.8\n",
+ " couch 50 0.8 0.96\n",
+ " stupa 50 0.88 0.88\n",
+ " submarine 50 0.72 0.92\n",
+ " suit 50 0.4 0.78\n",
+ " sundial 50 0.58 0.74\n",
+ " sunglass 50 0.14 0.58\n",
+ " sunglasses 50 0.28 0.58\n",
+ " sunscreen 50 0.32 0.7\n",
+ " suspension bridge 50 0.6 0.94\n",
+ " mop 50 0.74 0.92\n",
+ " sweatshirt 50 0.28 0.66\n",
+ " swimsuit 50 0.52 0.82\n",
+ " swing 50 0.76 0.84\n",
+ " switch 50 0.56 0.76\n",
+ " syringe 50 0.62 0.82\n",
+ " table lamp 50 0.6 0.88\n",
+ " tank 50 0.8 0.96\n",
+ " tape player 50 0.46 0.76\n",
+ " teapot 50 0.84 1\n",
+ " teddy bear 50 0.82 0.94\n",
+ " television 50 0.6 0.9\n",
+ " tennis ball 50 0.7 0.94\n",
+ " thatched roof 50 0.88 0.9\n",
+ " front curtain 50 0.8 0.92\n",
+ " thimble 50 0.6 0.8\n",
+ " threshing machine 50 0.56 0.88\n",
+ " throne 50 0.72 0.82\n",
+ " tile roof 50 0.72 0.94\n",
+ " toaster 50 0.66 0.84\n",
+ " tobacco shop 50 0.42 0.7\n",
+ " toilet seat 50 0.62 0.88\n",
+ " torch 50 0.64 0.84\n",
+ " totem pole 50 0.92 0.98\n",
+ " tow truck 50 0.62 0.88\n",
+ " toy store 50 0.6 0.94\n",
+ " tractor 50 0.76 0.98\n",
+ " semi-trailer truck 50 0.78 0.92\n",
+ " tray 50 0.46 0.64\n",
+ " trench coat 50 0.54 0.72\n",
+ " tricycle 50 0.72 0.94\n",
+ " trimaran 50 0.7 0.98\n",
+ " tripod 50 0.58 0.86\n",
+ " triumphal arch 50 0.92 0.98\n",
+ " trolleybus 50 0.9 1\n",
+ " trombone 50 0.54 0.88\n",
+ " tub 50 0.24 0.82\n",
+ " turnstile 50 0.84 0.94\n",
+ " typewriter keyboard 50 0.68 0.98\n",
+ " umbrella 50 0.52 0.7\n",
+ " unicycle 50 0.74 0.96\n",
+ " upright piano 50 0.76 0.9\n",
+ " vacuum cleaner 50 0.62 0.9\n",
+ " vase 50 0.5 0.78\n",
+ " vault 50 0.76 0.92\n",
+ " velvet 50 0.2 0.42\n",
+ " vending machine 50 0.9 1\n",
+ " vestment 50 0.54 0.82\n",
+ " viaduct 50 0.78 0.86\n",
+ " violin 50 0.68 0.78\n",
+ " volleyball 50 0.86 1\n",
+ " waffle iron 50 0.72 0.88\n",
+ " wall clock 50 0.54 0.88\n",
+ " wallet 50 0.52 0.9\n",
+ " wardrobe 50 0.68 0.88\n",
+ " military aircraft 50 0.9 0.98\n",
+ " sink 50 0.72 0.96\n",
+ " washing machine 50 0.78 0.94\n",
+ " water bottle 50 0.54 0.74\n",
+ " water jug 50 0.22 0.74\n",
+ " water tower 50 0.9 0.96\n",
+ " whiskey jug 50 0.64 0.74\n",
+ " whistle 50 0.72 0.84\n",
+ " wig 50 0.84 0.9\n",
+ " window screen 50 0.68 0.8\n",
+ " window shade 50 0.52 0.76\n",
+ " Windsor tie 50 0.22 0.66\n",
+ " wine bottle 50 0.42 0.82\n",
+ " wing 50 0.54 0.96\n",
+ " wok 50 0.46 0.82\n",
+ " wooden spoon 50 0.58 0.8\n",
+ " wool 50 0.32 0.82\n",
+ " split-rail fence 50 0.74 0.9\n",
+ " shipwreck 50 0.84 0.96\n",
+ " yawl 50 0.78 0.96\n",
+ " yurt 50 0.84 1\n",
+ " website 50 0.98 1\n",
+ " comic book 50 0.62 0.9\n",
+ " crossword 50 0.84 0.88\n",
+ " traffic sign 50 0.78 0.9\n",
+ " traffic light 50 0.8 0.94\n",
+ " dust jacket 50 0.72 0.94\n",
+ " menu 50 0.82 0.96\n",
+ " plate 50 0.44 0.88\n",
+ " guacamole 50 0.8 0.92\n",
+ " consomme 50 0.54 0.88\n",
+ " hot pot 50 0.86 0.98\n",
+ " trifle 50 0.92 0.98\n",
+ " ice cream 50 0.68 0.94\n",
+ " ice pop 50 0.62 0.84\n",
+ " baguette 50 0.62 0.88\n",
+ " bagel 50 0.64 0.92\n",
+ " pretzel 50 0.72 0.88\n",
+ " cheeseburger 50 0.9 1\n",
+ " hot dog 50 0.74 0.94\n",
+ " mashed potato 50 0.74 0.9\n",
+ " cabbage 50 0.84 0.96\n",
+ " broccoli 50 0.9 0.96\n",
+ " cauliflower 50 0.82 1\n",
+ " zucchini 50 0.74 0.9\n",
+ " spaghetti squash 50 0.8 0.96\n",
+ " acorn squash 50 0.82 0.96\n",
+ " butternut squash 50 0.7 0.94\n",
+ " cucumber 50 0.6 0.96\n",
+ " artichoke 50 0.84 0.94\n",
+ " bell pepper 50 0.84 0.98\n",
+ " cardoon 50 0.88 0.94\n",
+ " mushroom 50 0.38 0.92\n",
+ " Granny Smith 50 0.9 0.96\n",
+ " strawberry 50 0.6 0.88\n",
+ " orange 50 0.7 0.92\n",
+ " lemon 50 0.78 0.98\n",
+ " fig 50 0.82 0.96\n",
+ " pineapple 50 0.86 0.96\n",
+ " banana 50 0.84 0.96\n",
+ " jackfruit 50 0.9 0.98\n",
+ " custard apple 50 0.86 0.96\n",
+ " pomegranate 50 0.82 0.98\n",
+ " hay 50 0.8 0.92\n",
+ " carbonara 50 0.88 0.94\n",
+ " chocolate syrup 50 0.46 0.84\n",
+ " dough 50 0.4 0.6\n",
+ " meatloaf 50 0.58 0.84\n",
+ " pizza 50 0.84 0.96\n",
+ " pot pie 50 0.68 0.9\n",
+ " burrito 50 0.8 0.98\n",
+ " red wine 50 0.54 0.82\n",
+ " espresso 50 0.64 0.88\n",
+ " cup 50 0.38 0.7\n",
+ " eggnog 50 0.38 0.7\n",
+ " alp 50 0.54 0.88\n",
+ " bubble 50 0.8 0.96\n",
+ " cliff 50 0.64 1\n",
+ " coral reef 50 0.72 0.96\n",
+ " geyser 50 0.94 1\n",
+ " lakeshore 50 0.54 0.88\n",
+ " promontory 50 0.58 0.94\n",
+ " shoal 50 0.6 0.96\n",
+ " seashore 50 0.44 0.78\n",
+ " valley 50 0.72 0.94\n",
+ " volcano 50 0.78 0.96\n",
+ " baseball player 50 0.72 0.94\n",
+ " bridegroom 50 0.72 0.88\n",
+ " scuba diver 50 0.8 1\n",
+ " rapeseed 50 0.94 0.98\n",
+ " daisy 50 0.96 0.98\n",
+ " yellow lady's slipper 50 1 1\n",
+ " corn 50 0.4 0.88\n",
+ " acorn 50 0.92 0.98\n",
+ " rose hip 50 0.92 0.98\n",
+ " horse chestnut seed 50 0.94 0.98\n",
+ " coral fungus 50 0.96 0.96\n",
+ " agaric 50 0.82 0.94\n",
+ " gyromitra 50 0.98 1\n",
+ " stinkhorn mushroom 50 0.8 0.94\n",
+ " earth star 50 0.98 1\n",
+ " hen-of-the-woods 50 0.8 0.96\n",
+ " bolete 50 0.74 0.94\n",
+ " ear 50 0.48 0.94\n",
+ " toilet paper 50 0.36 0.68\n",
+ "Speed: 0.1ms pre-process, 0.3ms inference, 0.0ms post-process per image at shape (1, 3, 224, 224)\n",
+ "Results saved to \u001b[1mruns/val-cls/exp\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Validate YOLOv5s on Imagenet val\n",
+ "!python classify/val.py --weights yolov5s-cls.pt --data ../datasets/imagenet --img 224 --half"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ZY2VXXXu74w5"
+ },
+ "source": [
+ "# 3. Train\n",
+ "\n",
+ "![](\"https://github.com/ultralytics/assets/raw/main/im/integrations-loop.png\"/)
\n", + "\n", + "Train a YOLOv5s Classification model on the [Imagenette](https://image-net.org/) dataset with `--data imagenet`, starting from pretrained `--pretrained yolov5s-cls.pt`.\n", + "\n", + "- **Pretrained [Models](https://github.com/ultralytics/yolov5/tree/master/models)** are downloaded\n", + "automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases)\n", + "- **Training Results** are saved to `runs/train-cls/` with incrementing run directories, i.e. `runs/train-cls/exp2`, `runs/train-cls/exp3` etc.\n", + "
\n", + "\n", + "A **Mosaic Dataloader** is used for training which combines 4 images into 1 mosaic.\n", + "\n", + "## Train on Custom Data with Roboflow 🌟 NEW\n", + "\n", + "[Roboflow](https://roboflow.com/?ref=ultralytics) enables you to easily **organize, label, and prepare** a high quality dataset with your own custom data. Roboflow also makes it easy to establish an active learning pipeline, collaborate with your team on dataset improvement, and integrate directly into your model building workflow with the `roboflow` pip package.\n", + "\n", + "- Custom Training Example: [https://blog.roboflow.com/train-yolov5-classification-custom-data/](https://blog.roboflow.com/train-yolov5-classification-custom-data/?ref=ultralytics)\n", + "- Custom Training Notebook: [](https://colab.research.google.com/drive/1KZiKUAjtARHAfZCXbJRv14-pOnIsBLPV?usp=sharing)\n", + "
\n", + "\n", + "
![](\"https://user-images.githubusercontent.com/26833433/202802162-92e60571-ab58-4409-948d-b31fddcd3c6f.png\"/)
![\"Comet](\"https://user-images.githubusercontent.com/26833433/202851203-164e94e1-2238-46dd-91f8-de020e9d6b41.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Lay2WsTjNJzP"
+ },
+ "source": [
+ "## ClearML Logging and Automation 🌟 NEW\n",
+ "\n",
+ "[ClearML](https://cutt.ly/yolov5-notebook-clearml) is completely integrated into YOLOv5 to track your experimentation, manage dataset versions and even remotely execute training runs. To enable ClearML (check cells above):\n",
+ "\n",
+ "- `pip install clearml`\n",
+ "- run `clearml-init` to connect to a ClearML server (**deploy your own [open-source server](https://github.com/allegroai/clearml-server)**, or use our [free hosted server](https://cutt.ly/yolov5-notebook-clearml))\n",
+ "\n",
+ "You'll get all the great expected features from an experiment manager: live updates, model upload, experiment comparison etc. but ClearML also tracks uncommitted changes and installed packages for example. Thanks to that ClearML Tasks (which is what we call experiments) are also reproducible on different machines! With only 1 extra line, we can schedule a YOLOv5 training task on a queue to be executed by any number of ClearML Agents (workers).\n",
+ "\n",
+ "You can use ClearML Data to version your dataset and then pass it to YOLOv5 simply using its unique ID. This will help you keep track of your data without adding extra hassle. Explore the [ClearML Tutorial](https://docs.ultralytics.com/yolov5/tutorials/clearml_logging_integration) for details!\n",
+ "\n",
+ "\n",
+ "![\"ClearML](\"https://github.com/thepycoder/clearml_screenshots/raw/main/scalars.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-WPvRbS5Swl6"
+ },
+ "source": [
+ "## Local Logging\n",
+ "\n",
+ "Training results are automatically logged with [Tensorboard](https://www.tensorflow.org/tensorboard) and [CSV](https://github.com/ultralytics/yolov5/pull/4148) loggers to `runs/train`, with a new experiment directory created for each new training as `runs/train/exp2`, `runs/train/exp3`, etc.\n",
+ "\n",
+ "This directory contains train and val statistics, mosaics, labels, predictions and augmentated mosaics, as well as metrics and charts including precision-recall (PR) curves and confusion matrices. \n",
+ "\n",
+ "![\"Local](\"https://user-images.githubusercontent.com/26833433/183222430-e1abd1b7-782c-4cde-b04d-ad52926bf818.jpg\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Zelyeqbyt3GD"
+ },
+ "source": [
+ "# Environments\n",
+ "\n",
+ "YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including [CUDA](https://developer.nvidia.com/cuda)/[CUDNN](https://developer.nvidia.com/cudnn), [Python](https://www.python.org/) and [PyTorch](https://pytorch.org/) preinstalled):\n",
+ "\n",
+ "- **Notebooks** with free GPU: \n",
+ "- **Google Cloud** Deep Learning VM. See [GCP Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/google_cloud_quickstart_tutorial/)\n",
+ "- **Amazon** Deep Learning AMI. See [AWS Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/aws_quickstart_tutorial/)\n",
+ "- **Docker Image**. See [Docker Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/docker_image_quickstart_tutorial/) ![\"Docker](\"https://img.shields.io/docker/pulls/ultralytics/yolov5?logo=docker\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6Qu7Iesl0p54"
+ },
+ "source": [
+ "# Status\n",
+ "\n",
+ "\n",
+ "\n",
+ "If this badge is green, all [YOLOv5 GitHub Actions](https://github.com/ultralytics/yolov5/actions) Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training ([train.py](https://github.com/ultralytics/yolov5/blob/master/train.py)), testing ([val.py](https://github.com/ultralytics/yolov5/blob/master/val.py)), inference ([detect.py](https://github.com/ultralytics/yolov5/blob/master/detect.py)) and export ([export.py](https://github.com/ultralytics/yolov5/blob/master/export.py)) on macOS, Windows, and Ubuntu every 24 hours and on every commit.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IEijrePND_2I"
+ },
+ "source": [
+ "# Appendix\n",
+ "\n",
+ "Additional content below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GMusP4OAxFu6"
+ },
+ "outputs": [],
+ "source": [
+ "# YOLOv5 PyTorch HUB Inference (DetectionModels only)\n",
+ "import torch\n",
+ "\n",
+ "model = torch.hub.load('ultralytics/yolov5', 'yolov5s') # yolov5n - yolov5x6 or custom\n",
+ "im = 'https://ultralytics.com/images/zidane.jpg' # file, Path, PIL.Image, OpenCV, nparray, list\n",
+ "results = model(im) # inference\n",
+ "results.print() # or .show(), .save(), .crop(), .pandas(), etc."
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "name": "YOLOv5 Classification Tutorial",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/TextDetection/classify/val.py b/TextDetection/classify/val.py
new file mode 100644
index 0000000000000000000000000000000000000000..4b92e9f105db9e7af6521b6689c279e948153a11
--- /dev/null
+++ b/TextDetection/classify/val.py
@@ -0,0 +1,170 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Validate a trained YOLOv5 classification model on a classification dataset
+
+Usage:
+ $ bash data/scripts/get_imagenet.sh --val # download ImageNet val split (6.3G, 50000 images)
+ $ python classify/val.py --weights yolov5m-cls.pt --data ../datasets/imagenet --img 224 # validate ImageNet
+
+Usage - formats:
+ $ python classify/val.py --weights yolov5s-cls.pt # PyTorch
+ yolov5s-cls.torchscript # TorchScript
+ yolov5s-cls.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s-cls_openvino_model # OpenVINO
+ yolov5s-cls.engine # TensorRT
+ yolov5s-cls.mlmodel # CoreML (macOS-only)
+ yolov5s-cls_saved_model # TensorFlow SavedModel
+ yolov5s-cls.pb # TensorFlow GraphDef
+ yolov5s-cls.tflite # TensorFlow Lite
+ yolov5s-cls_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s-cls_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+import torch
+from tqdm import tqdm
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[1] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.common import DetectMultiBackend
+from utils.dataloaders import create_classification_dataloader
+from utils.general import (LOGGER, TQDM_BAR_FORMAT, Profile, check_img_size, check_requirements, colorstr,
+ increment_path, print_args)
+from utils.torch_utils import select_device, smart_inference_mode
+
+
+@smart_inference_mode()
+def run(
+ data=ROOT / '../datasets/mnist', # dataset dir
+ weights=ROOT / 'yolov5s-cls.pt', # model.pt path(s)
+ batch_size=128, # batch size
+ imgsz=224, # inference size (pixels)
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ workers=8, # max dataloader workers (per RANK in DDP mode)
+ verbose=False, # verbose output
+ project=ROOT / 'runs/val-cls', # save to project/name
+ name='exp', # save to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ half=False, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ model=None,
+ dataloader=None,
+ criterion=None,
+ pbar=None,
+):
+ # Initialize/load model and set device
+ training = model is not None
+ if training: # called by train.py
+ device, pt, jit, engine = next(model.parameters()).device, True, False, False # get model device, PyTorch model
+ half &= device.type != 'cpu' # half precision only supported on CUDA
+ model.half() if half else model.float()
+ else: # called directly
+ device = select_device(device, batch_size=batch_size)
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ save_dir.mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, fp16=half)
+ stride, pt, jit, engine = model.stride, model.pt, model.jit, model.engine
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+ half = model.fp16 # FP16 supported on limited backends with CUDA
+ if engine:
+ batch_size = model.batch_size
+ else:
+ device = model.device
+ if not (pt or jit):
+ batch_size = 1 # export.py models default to batch-size 1
+ LOGGER.info(f'Forcing --batch-size 1 square inference (1,3,{imgsz},{imgsz}) for non-PyTorch models')
+
+ # Dataloader
+ data = Path(data)
+ test_dir = data / 'test' if (data / 'test').exists() else data / 'val' # data/test or data/val
+ dataloader = create_classification_dataloader(path=test_dir,
+ imgsz=imgsz,
+ batch_size=batch_size,
+ augment=False,
+ rank=-1,
+ workers=workers)
+
+ model.eval()
+ pred, targets, loss, dt = [], [], 0, (Profile(), Profile(), Profile())
+ n = len(dataloader) # number of batches
+ action = 'validating' if dataloader.dataset.root.stem == 'val' else 'testing'
+ desc = f'{pbar.desc[:-36]}{action:>36}' if pbar else f'{action}'
+ bar = tqdm(dataloader, desc, n, not training, bar_format=TQDM_BAR_FORMAT, position=0)
+ with torch.cuda.amp.autocast(enabled=device.type != 'cpu'):
+ for images, labels in bar:
+ with dt[0]:
+ images, labels = images.to(device, non_blocking=True), labels.to(device)
+
+ with dt[1]:
+ y = model(images)
+
+ with dt[2]:
+ pred.append(y.argsort(1, descending=True)[:, :5])
+ targets.append(labels)
+ if criterion:
+ loss += criterion(y, labels)
+
+ loss /= n
+ pred, targets = torch.cat(pred), torch.cat(targets)
+ correct = (targets[:, None] == pred).float()
+ acc = torch.stack((correct[:, 0], correct.max(1).values), dim=1) # (top1, top5) accuracy
+ top1, top5 = acc.mean(0).tolist()
+
+ if pbar:
+ pbar.desc = f'{pbar.desc[:-36]}{loss:>12.3g}{top1:>12.3g}{top5:>12.3g}'
+ if verbose: # all classes
+ LOGGER.info(f"{'Class':>24}{'Images':>12}{'top1_acc':>12}{'top5_acc':>12}")
+ LOGGER.info(f"{'all':>24}{targets.shape[0]:>12}{top1:>12.3g}{top5:>12.3g}")
+ for i, c in model.names.items():
+ acc_i = acc[targets == i]
+ top1i, top5i = acc_i.mean(0).tolist()
+ LOGGER.info(f'{c:>24}{acc_i.shape[0]:>12}{top1i:>12.3g}{top5i:>12.3g}')
+
+ # Print results
+ t = tuple(x.t / len(dataloader.dataset.samples) * 1E3 for x in dt) # speeds per image
+ shape = (1, 3, imgsz, imgsz)
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms post-process per image at shape {shape}' % t)
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}")
+
+ return top1, top5, loss
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--data', type=str, default=ROOT / '../datasets/mnist', help='dataset path')
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s-cls.pt', help='model.pt path(s)')
+ parser.add_argument('--batch-size', type=int, default=128, help='batch size')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=224, help='inference size (pixels)')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--verbose', nargs='?', const=True, default=True, help='verbose output')
+ parser.add_argument('--project', default=ROOT / 'runs/val-cls', help='save to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ opt = parser.parse_args()
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(ROOT / 'requirements.txt', exclude=('tensorboard', 'thop'))
+ run(**vars(opt))
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/detect.py b/TextDetection/detect.py
new file mode 100644
index 0000000000000000000000000000000000000000..6157c587924fe5c633881b2ed70eb62d43670a8b
--- /dev/null
+++ b/TextDetection/detect.py
@@ -0,0 +1,271 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Run YOLOv5 detection inference on images, videos, directories, globs, YouTube, webcam, streams, etc.
+
+Usage - sources:
+ $ python detect.py --weights yolov5s.pt --source 0 # webcam
+ img.jpg # image
+ vid.mp4 # video
+ screen # screenshot
+ path/ # directory
+ list.txt # list of images
+ list.streams # list of streams
+ 'path/*.jpg' # glob
+ 'https://youtu.be/Zgi9g1ksQHc' # YouTube
+ 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream
+
+Usage - formats:
+ $ python detect.py --weights yolov5s.pt # PyTorch
+ yolov5s.torchscript # TorchScript
+ yolov5s.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s_openvino_model # OpenVINO
+ yolov5s.engine # TensorRT
+ yolov5s.mlmodel # CoreML (macOS-only)
+ yolov5s_saved_model # TensorFlow SavedModel
+ yolov5s.pb # TensorFlow GraphDef
+ yolov5s.tflite # TensorFlow Lite
+ yolov5s_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import os
+import platform
+import sys
+from pathlib import Path
+
+import torch
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[0] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.common import DetectMultiBackend
+from utils.dataloaders import IMG_FORMATS, VID_FORMATS, LoadImages, LoadScreenshots, LoadStreams
+from utils.general import (LOGGER, Profile, check_file, check_img_size, check_imshow, check_requirements, colorstr, cv2,
+ increment_path, non_max_suppression, print_args, scale_boxes, strip_optimizer, xyxy2xywh)
+from utils.plots import Annotator, colors, save_one_box
+from utils.torch_utils import select_device, smart_inference_mode
+
+
+@smart_inference_mode()
+def run(
+ weights=ROOT / 'yolov5s.pt', # model path or triton URL
+ source=ROOT / 'data/images', # file/dir/URL/glob/screen/0(webcam)
+ data=ROOT / 'data/coco128.yaml', # dataset.yaml path
+ imgsz=(640, 640), # inference size (height, width)
+ conf_thres=0.25, # confidence threshold
+ iou_thres=0.45, # NMS IOU threshold
+ max_det=1000, # maximum detections per image
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ view_img=False, # show results
+ save_txt=True, # save results to *.txt
+ save_conf=True, # save confidences in --save-txt labels
+ save_crop=False, # save cropped prediction boxes
+ nosave=False, # do not save images/videos
+ classes=None, # filter by class: --class 0, or --class 0 2 3
+ agnostic_nms=False, # class-agnostic NMS
+ augment=False, # augmented inference
+ visualize=False, # visualize features
+ update=False, # update all models
+ project=ROOT / 'runs/detect', # save results to project/name
+ name='exp', # save results to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ line_thickness=3, # bounding box thickness (pixels)
+ hide_labels=False, # hide labels
+ hide_conf=False, # hide confidences
+ half=False, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ vid_stride=1, # video frame-rate stride
+):
+ source = str(source)
+ save_img = not nosave and not source.endswith('.txt') # save inference images
+ is_file = Path(source).suffix[1:] in (IMG_FORMATS + VID_FORMATS)
+ is_url = source.lower().startswith(('rtsp://', 'rtmp://', 'http://', 'https://'))
+ webcam = source.isnumeric() or source.endswith('.streams') or (is_url and not is_file)
+ screenshot = source.lower().startswith('screen')
+ if is_url and is_file:
+ source = check_file(source) # download
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ (save_dir / 'labels' if save_txt else save_dir).mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ device = select_device(device)
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
+ stride, names, pt = model.stride, model.names, model.pt
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+
+ # Dataloader
+ bs = 1 # batch_size
+ if webcam:
+ view_img = check_imshow(warn=True)
+ dataset = LoadStreams(source, img_size=imgsz, stride=stride, auto=pt, vid_stride=vid_stride)
+ bs = len(dataset)
+ elif screenshot:
+ dataset = LoadScreenshots(source, img_size=imgsz, stride=stride, auto=pt)
+ else:
+ dataset = LoadImages(source, img_size=imgsz, stride=stride, auto=pt, vid_stride=vid_stride)
+ vid_path, vid_writer = [None] * bs, [None] * bs
+
+ # Run inference
+ model.warmup(imgsz=(1 if pt or model.triton else bs, 3, *imgsz)) # warmup
+ seen, windows, dt = 0, [], (Profile(), Profile(), Profile())
+ for path, im, im0s, vid_cap, s in dataset:
+ with dt[0]:
+ im = torch.from_numpy(im).to(model.device)
+ im = im.half() if model.fp16 else im.float() # uint8 to fp16/32
+ im /= 255 # 0 - 255 to 0.0 - 1.0
+ if len(im.shape) == 3:
+ im = im[None] # expand for batch dim
+
+ # Inference
+ with dt[1]:
+ visualize = increment_path(save_dir / Path(path).stem, mkdir=True) if visualize else False
+ pred = model(im, augment=augment, visualize=visualize)
+
+ # NMS
+ with dt[2]:
+ pred = non_max_suppression(pred, conf_thres, iou_thres, classes, agnostic_nms, max_det=max_det)
+
+ # Second-stage classifier (optional)
+ # pred = utils.general.apply_classifier(pred, classifier_model, im, im0s)
+
+ # Process predictions
+ for i, det in enumerate(pred): # per image
+ seen += 1
+ if webcam: # batch_size >= 1
+ p, im0, frame = path[i], im0s[i].copy(), dataset.count
+ s += f'{i}: '
+ else:
+ p, im0, frame = path, im0s.copy(), getattr(dataset, 'frame', 0)
+
+ p = Path(p) # to Path
+ save_path = str(save_dir / p.name) # im.jpg
+ txt_path = str(save_dir / 'labels' / p.stem) + ('' if dataset.mode == 'image' else f'_{frame}') # im.txt
+ s += '%gx%g ' % im.shape[2:] # print string
+ gn = torch.tensor(im0.shape)[[1, 0, 1, 0]] # normalization gain whwh
+ imc = im0.copy() if save_crop else im0 # for save_crop
+ annotator = Annotator(im0, line_width=line_thickness, example=str(names))
+ if len(det):
+ # Rescale boxes from img_size to im0 size
+ det[:, :4] = scale_boxes(im.shape[2:], det[:, :4], im0.shape).round()
+
+ # Print results
+ for c in det[:, 5].unique():
+ n = (det[:, 5] == c).sum() # detections per class
+ s += f"{n} {names[int(c)]}{'s' * (n > 1)}, " # add to string
+
+ # Write results
+ for *xyxy, conf, cls in reversed(det):
+ if save_txt: # Write to file
+ xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh
+ line = (cls, *xywh, conf) if save_conf else (cls, *xywh) # label format
+ with open(f'{txt_path}.txt', 'a') as f:
+ f.write(('%g ' * len(line)).rstrip() % line + '\n')
+
+ if save_img or save_crop or view_img: # Add bbox to image
+ c = int(cls) # integer class
+ label = None if hide_labels else (names[c] if hide_conf else f'{names[c]} {conf:.2f}')
+ annotator.box_label(xyxy, label, color=colors(c, True))
+ if save_crop:
+ ###changed part###
+ xydata = str(xyxy).replace("[tensor(", "x")
+ xydata = str(xydata).replace("tensor(", "y",1)
+ xydata = str(xydata).replace("tensor(", "w",1)
+ xydata = str(xydata).replace("tensor(", "h",1)
+ xydata = str(xydata).replace("]", "_",1)
+ xydata = str(xydata).replace(".), ", "")
+ xydata = str(xydata).replace(".)", "")
+ #str.replace("tensor", "_")
+ save_one_box(xyxy, imc, file=save_dir / 'crops' / names[c] / f'{xydata}{p.stem}.jpg', BGR=True)
+ ###changed part###
+
+ # Stream results
+ im0 = annotator.result()
+ if view_img:
+ if platform.system() == 'Linux' and p not in windows:
+ windows.append(p)
+ cv2.namedWindow(str(p), cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO) # allow window resize (Linux)
+ cv2.resizeWindow(str(p), im0.shape[1], im0.shape[0])
+ cv2.imshow(str(p), im0)
+ cv2.waitKey(1) # 1 millisecond
+
+ # Save results (image with detections)
+ if save_img:
+ if dataset.mode == 'image':
+ cv2.imwrite(save_path, im0)
+ else: # 'video' or 'stream'
+ if vid_path[i] != save_path: # new video
+ vid_path[i] = save_path
+ if isinstance(vid_writer[i], cv2.VideoWriter):
+ vid_writer[i].release() # release previous video writer
+ if vid_cap: # video
+ fps = vid_cap.get(cv2.CAP_PROP_FPS)
+ w = int(vid_cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+ h = int(vid_cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+ else: # stream
+ fps, w, h = 30, im0.shape[1], im0.shape[0]
+ save_path = str(Path(save_path).with_suffix('.mp4')) # force *.mp4 suffix on results videos
+ vid_writer[i] = cv2.VideoWriter(save_path, cv2.VideoWriter_fourcc(*'mp4v'), fps, (w, h))
+ vid_writer[i].write(im0)
+
+ # Print time (inference-only)
+ LOGGER.info(f"{s}{'' if len(det) else '(no detections), '}{dt[1].dt * 1E3:.1f}ms")
+
+ # Print results
+ t = tuple(x.t / seen * 1E3 for x in dt) # speeds per image
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {(1, 3, *imgsz)}' % t)
+ if save_txt or save_img:
+ s = f"\n{len(list(save_dir.glob('labels/*.txt')))} labels saved to {save_dir / 'labels'}" if save_txt else ''
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}{s}")
+ if update:
+ strip_optimizer(weights[0]) # update model (to fix SourceChangeWarning)
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model path or triton URL')
+ parser.add_argument('--source', type=str, default=ROOT / 'data/images', help='file/dir/URL/glob/screen/0(webcam)')
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='(optional) dataset.yaml path')
+ parser.add_argument('--imgsz', '--img', '--img-size', nargs='+', type=int, default=[640], help='inference size h,w')
+ parser.add_argument('--conf-thres', type=float, default=0.25, help='confidence threshold')
+ parser.add_argument('--iou-thres', type=float, default=0.45, help='NMS IoU threshold')
+ parser.add_argument('--max-det', type=int, default=1000, help='maximum detections per image')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--view-img', action='store_true', help='show results')
+ parser.add_argument('--save-txt', action='store_true', help='save results to *.txt')
+ parser.add_argument('--save-conf', action='store_true', help='save confidences in --save-txt labels')
+ parser.add_argument('--save-crop', action='store_true', help='save cropped prediction boxes')
+ parser.add_argument('--nosave', action='store_true', help='do not save images/videos')
+ parser.add_argument('--classes', nargs='+', type=int, help='filter by class: --classes 0, or --classes 0 2 3')
+ parser.add_argument('--agnostic-nms', action='store_true', help='class-agnostic NMS')
+ parser.add_argument('--augment', action='store_true', help='augmented inference')
+ parser.add_argument('--visualize', action='store_true', help='visualize features')
+ parser.add_argument('--update', action='store_true', help='update all models')
+ parser.add_argument('--project', default=ROOT / 'runs/detect', help='save results to project/name')
+ parser.add_argument('--name', default='exp', help='save results to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--line-thickness', default=3, type=int, help='bounding box thickness (pixels)')
+ parser.add_argument('--hide-labels', default=False, action='store_true', help='hide labels')
+ parser.add_argument('--hide-conf', default=False, action='store_true', help='hide confidences')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ parser.add_argument('--vid-stride', type=int, default=1, help='video frame-rate stride')
+ opt = parser.parse_args()
+ opt.imgsz *= 2 if len(opt.imgsz) == 1 else 1 # expand
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(ROOT / 'requirements.txt', exclude=('tensorboard', 'thop'))
+ run(**vars(opt))
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/export.py b/TextDetection/export.py
new file mode 100644
index 0000000000000000000000000000000000000000..5755774d2f547f121f18a5af43d735fe32b959a0
--- /dev/null
+++ b/TextDetection/export.py
@@ -0,0 +1,863 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Export a YOLOv5 PyTorch model to other formats. TensorFlow exports authored by https://github.com/zldrobit
+
+Format | `export.py --include` | Model
+--- | --- | ---
+PyTorch | - | yolov5s.pt
+TorchScript | `torchscript` | yolov5s.torchscript
+ONNX | `onnx` | yolov5s.onnx
+OpenVINO | `openvino` | yolov5s_openvino_model/
+TensorRT | `engine` | yolov5s.engine
+CoreML | `coreml` | yolov5s.mlmodel
+TensorFlow SavedModel | `saved_model` | yolov5s_saved_model/
+TensorFlow GraphDef | `pb` | yolov5s.pb
+TensorFlow Lite | `tflite` | yolov5s.tflite
+TensorFlow Edge TPU | `edgetpu` | yolov5s_edgetpu.tflite
+TensorFlow.js | `tfjs` | yolov5s_web_model/
+PaddlePaddle | `paddle` | yolov5s_paddle_model/
+
+Requirements:
+ $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu # CPU
+ $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime-gpu openvino-dev tensorflow # GPU
+
+Usage:
+ $ python export.py --weights yolov5s.pt --include torchscript onnx openvino engine coreml tflite ...
+
+Inference:
+ $ python detect.py --weights yolov5s.pt # PyTorch
+ yolov5s.torchscript # TorchScript
+ yolov5s.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s_openvino_model # OpenVINO
+ yolov5s.engine # TensorRT
+ yolov5s.mlmodel # CoreML (macOS-only)
+ yolov5s_saved_model # TensorFlow SavedModel
+ yolov5s.pb # TensorFlow GraphDef
+ yolov5s.tflite # TensorFlow Lite
+ yolov5s_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s_paddle_model # PaddlePaddle
+
+TensorFlow.js:
+ $ cd .. && git clone https://github.com/zldrobit/tfjs-yolov5-example.git && cd tfjs-yolov5-example
+ $ npm install
+ $ ln -s ../../yolov5/yolov5s_web_model public/yolov5s_web_model
+ $ npm start
+"""
+
+import argparse
+import contextlib
+import json
+import os
+import platform
+import re
+import subprocess
+import sys
+import time
+import warnings
+from pathlib import Path
+
+import pandas as pd
+import torch
+from torch.utils.mobile_optimizer import optimize_for_mobile
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[0] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+if platform.system() != 'Windows':
+ ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.experimental import attempt_load
+from models.yolo import ClassificationModel, Detect, DetectionModel, SegmentationModel
+from utils.dataloaders import LoadImages
+from utils.general import (LOGGER, Profile, check_dataset, check_img_size, check_requirements, check_version,
+ check_yaml, colorstr, file_size, get_default_args, print_args, url2file, yaml_save)
+from utils.torch_utils import select_device, smart_inference_mode
+
+MACOS = platform.system() == 'Darwin' # macOS environment
+
+
+class iOSModel(torch.nn.Module):
+
+ def __init__(self, model, im):
+ super().__init__()
+ b, c, h, w = im.shape # batch, channel, height, width
+ self.model = model
+ self.nc = model.nc # number of classes
+ if w == h:
+ self.normalize = 1. / w
+ else:
+ self.normalize = torch.tensor([1. / w, 1. / h, 1. / w, 1. / h]) # broadcast (slower, smaller)
+ # np = model(im)[0].shape[1] # number of points
+ # self.normalize = torch.tensor([1. / w, 1. / h, 1. / w, 1. / h]).expand(np, 4) # explicit (faster, larger)
+
+ def forward(self, x):
+ xywh, conf, cls = self.model(x)[0].squeeze().split((4, 1, self.nc), 1)
+ return cls * conf, xywh * self.normalize # confidence (3780, 80), coordinates (3780, 4)
+
+
+def export_formats():
+ # YOLOv5 export formats
+ x = [
+ ['PyTorch', '-', '.pt', True, True],
+ ['TorchScript', 'torchscript', '.torchscript', True, True],
+ ['ONNX', 'onnx', '.onnx', True, True],
+ ['OpenVINO', 'openvino', '_openvino_model', True, False],
+ ['TensorRT', 'engine', '.engine', False, True],
+ ['CoreML', 'coreml', '.mlmodel', True, False],
+ ['TensorFlow SavedModel', 'saved_model', '_saved_model', True, True],
+ ['TensorFlow GraphDef', 'pb', '.pb', True, True],
+ ['TensorFlow Lite', 'tflite', '.tflite', True, False],
+ ['TensorFlow Edge TPU', 'edgetpu', '_edgetpu.tflite', False, False],
+ ['TensorFlow.js', 'tfjs', '_web_model', False, False],
+ ['PaddlePaddle', 'paddle', '_paddle_model', True, True], ]
+ return pd.DataFrame(x, columns=['Format', 'Argument', 'Suffix', 'CPU', 'GPU'])
+
+
+def try_export(inner_func):
+ # YOLOv5 export decorator, i..e @try_export
+ inner_args = get_default_args(inner_func)
+
+ def outer_func(*args, **kwargs):
+ prefix = inner_args['prefix']
+ try:
+ with Profile() as dt:
+ f, model = inner_func(*args, **kwargs)
+ LOGGER.info(f'{prefix} export success ✅ {dt.t:.1f}s, saved as {f} ({file_size(f):.1f} MB)')
+ return f, model
+ except Exception as e:
+ LOGGER.info(f'{prefix} export failure ❌ {dt.t:.1f}s: {e}')
+ return None, None
+
+ return outer_func
+
+
+@try_export
+def export_torchscript(model, im, file, optimize, prefix=colorstr('TorchScript:')):
+ # YOLOv5 TorchScript model export
+ LOGGER.info(f'\n{prefix} starting export with torch {torch.__version__}...')
+ f = file.with_suffix('.torchscript')
+
+ ts = torch.jit.trace(model, im, strict=False)
+ d = {'shape': im.shape, 'stride': int(max(model.stride)), 'names': model.names}
+ extra_files = {'config.txt': json.dumps(d)} # torch._C.ExtraFilesMap()
+ if optimize: # https://pytorch.org/tutorials/recipes/mobile_interpreter.html
+ optimize_for_mobile(ts)._save_for_lite_interpreter(str(f), _extra_files=extra_files)
+ else:
+ ts.save(str(f), _extra_files=extra_files)
+ return f, None
+
+
+@try_export
+def export_onnx(model, im, file, opset, dynamic, simplify, prefix=colorstr('ONNX:')):
+ # YOLOv5 ONNX export
+ check_requirements('onnx>=1.12.0')
+ import onnx
+
+ LOGGER.info(f'\n{prefix} starting export with onnx {onnx.__version__}...')
+ f = file.with_suffix('.onnx')
+
+ output_names = ['output0', 'output1'] if isinstance(model, SegmentationModel) else ['output0']
+ if dynamic:
+ dynamic = {'images': {0: 'batch', 2: 'height', 3: 'width'}} # shape(1,3,640,640)
+ if isinstance(model, SegmentationModel):
+ dynamic['output0'] = {0: 'batch', 1: 'anchors'} # shape(1,25200,85)
+ dynamic['output1'] = {0: 'batch', 2: 'mask_height', 3: 'mask_width'} # shape(1,32,160,160)
+ elif isinstance(model, DetectionModel):
+ dynamic['output0'] = {0: 'batch', 1: 'anchors'} # shape(1,25200,85)
+
+ torch.onnx.export(
+ model.cpu() if dynamic else model, # --dynamic only compatible with cpu
+ im.cpu() if dynamic else im,
+ f,
+ verbose=False,
+ opset_version=opset,
+ do_constant_folding=True, # WARNING: DNN inference with torch>=1.12 may require do_constant_folding=False
+ input_names=['images'],
+ output_names=output_names,
+ dynamic_axes=dynamic or None)
+
+ # Checks
+ model_onnx = onnx.load(f) # load onnx model
+ onnx.checker.check_model(model_onnx) # check onnx model
+
+ # Metadata
+ d = {'stride': int(max(model.stride)), 'names': model.names}
+ for k, v in d.items():
+ meta = model_onnx.metadata_props.add()
+ meta.key, meta.value = k, str(v)
+ onnx.save(model_onnx, f)
+
+ # Simplify
+ if simplify:
+ try:
+ cuda = torch.cuda.is_available()
+ check_requirements(('onnxruntime-gpu' if cuda else 'onnxruntime', 'onnx-simplifier>=0.4.1'))
+ import onnxsim
+
+ LOGGER.info(f'{prefix} simplifying with onnx-simplifier {onnxsim.__version__}...')
+ model_onnx, check = onnxsim.simplify(model_onnx)
+ assert check, 'assert check failed'
+ onnx.save(model_onnx, f)
+ except Exception as e:
+ LOGGER.info(f'{prefix} simplifier failure: {e}')
+ return f, model_onnx
+
+
+@try_export
+def export_openvino(file, metadata, half, int8, data, prefix=colorstr('OpenVINO:')):
+ # YOLOv5 OpenVINO export
+ check_requirements('openvino-dev>=2022.3') # requires openvino-dev: https://pypi.org/project/openvino-dev/
+ import openvino.runtime as ov # noqa
+ from openvino.tools import mo # noqa
+
+ LOGGER.info(f'\n{prefix} starting export with openvino {ov.__version__}...')
+ f = str(file).replace(file.suffix, f'_openvino_model{os.sep}')
+ f_onnx = file.with_suffix('.onnx')
+ f_ov = str(Path(f) / file.with_suffix('.xml').name)
+ if int8:
+ check_requirements('nncf')
+ import nncf
+ import numpy as np
+ from openvino.runtime import Core
+
+ from utils.dataloaders import create_dataloader
+ core = Core()
+ onnx_model = core.read_model(f_onnx) # export
+
+ def prepare_input_tensor(image: np.ndarray):
+ input_tensor = image.astype(np.float32) # uint8 to fp16/32
+ input_tensor /= 255.0 # 0 - 255 to 0.0 - 1.0
+
+ if input_tensor.ndim == 3:
+ input_tensor = np.expand_dims(input_tensor, 0)
+ return input_tensor
+
+ def gen_dataloader(yaml_path, task='train', imgsz=640, workers=4):
+ data_yaml = check_yaml(yaml_path)
+ data = check_dataset(data_yaml)
+ dataloader = create_dataloader(data[task],
+ imgsz=imgsz,
+ batch_size=1,
+ stride=32,
+ pad=0.5,
+ single_cls=False,
+ rect=False,
+ workers=workers)[0]
+ return dataloader
+
+ # noqa: F811
+
+ def transform_fn(data_item):
+ """
+ Quantization transform function. Extracts and preprocess input data from dataloader item for quantization.
+ Parameters:
+ data_item: Tuple with data item produced by DataLoader during iteration
+ Returns:
+ input_tensor: Input data for quantization
+ """
+ img = data_item[0].numpy()
+ input_tensor = prepare_input_tensor(img)
+ return input_tensor
+
+ ds = gen_dataloader(data)
+ quantization_dataset = nncf.Dataset(ds, transform_fn)
+ ov_model = nncf.quantize(onnx_model, quantization_dataset, preset=nncf.QuantizationPreset.MIXED)
+ else:
+ ov_model = mo.convert_model(f_onnx, model_name=file.stem, framework='onnx', compress_to_fp16=half) # export
+
+ ov.serialize(ov_model, f_ov) # save
+ yaml_save(Path(f) / file.with_suffix('.yaml').name, metadata) # add metadata.yaml
+ return f, None
+
+
+@try_export
+def export_paddle(model, im, file, metadata, prefix=colorstr('PaddlePaddle:')):
+ # YOLOv5 Paddle export
+ check_requirements(('paddlepaddle', 'x2paddle'))
+ import x2paddle
+ from x2paddle.convert import pytorch2paddle
+
+ LOGGER.info(f'\n{prefix} starting export with X2Paddle {x2paddle.__version__}...')
+ f = str(file).replace('.pt', f'_paddle_model{os.sep}')
+
+ pytorch2paddle(module=model, save_dir=f, jit_type='trace', input_examples=[im]) # export
+ yaml_save(Path(f) / file.with_suffix('.yaml').name, metadata) # add metadata.yaml
+ return f, None
+
+
+@try_export
+def export_coreml(model, im, file, int8, half, nms, prefix=colorstr('CoreML:')):
+ # YOLOv5 CoreML export
+ check_requirements('coremltools')
+ import coremltools as ct
+
+ LOGGER.info(f'\n{prefix} starting export with coremltools {ct.__version__}...')
+ f = file.with_suffix('.mlmodel')
+
+ if nms:
+ model = iOSModel(model, im)
+ ts = torch.jit.trace(model, im, strict=False) # TorchScript model
+ ct_model = ct.convert(ts, inputs=[ct.ImageType('image', shape=im.shape, scale=1 / 255, bias=[0, 0, 0])])
+ bits, mode = (8, 'kmeans_lut') if int8 else (16, 'linear') if half else (32, None)
+ if bits < 32:
+ if MACOS: # quantization only supported on macOS
+ with warnings.catch_warnings():
+ warnings.filterwarnings('ignore', category=DeprecationWarning) # suppress numpy==1.20 float warning
+ ct_model = ct.models.neural_network.quantization_utils.quantize_weights(ct_model, bits, mode)
+ else:
+ print(f'{prefix} quantization only supported on macOS, skipping...')
+ ct_model.save(f)
+ return f, ct_model
+
+
+@try_export
+def export_engine(model, im, file, half, dynamic, simplify, workspace=4, verbose=False, prefix=colorstr('TensorRT:')):
+ # YOLOv5 TensorRT export https://developer.nvidia.com/tensorrt
+ assert im.device.type != 'cpu', 'export running on CPU but must be on GPU, i.e. `python export.py --device 0`'
+ try:
+ import tensorrt as trt
+ except Exception:
+ if platform.system() == 'Linux':
+ check_requirements('nvidia-tensorrt', cmds='-U --index-url https://pypi.ngc.nvidia.com')
+ import tensorrt as trt
+
+ if trt.__version__[0] == '7': # TensorRT 7 handling https://github.com/ultralytics/yolov5/issues/6012
+ grid = model.model[-1].anchor_grid
+ model.model[-1].anchor_grid = [a[..., :1, :1, :] for a in grid]
+ export_onnx(model, im, file, 12, dynamic, simplify) # opset 12
+ model.model[-1].anchor_grid = grid
+ else: # TensorRT >= 8
+ check_version(trt.__version__, '8.0.0', hard=True) # require tensorrt>=8.0.0
+ export_onnx(model, im, file, 12, dynamic, simplify) # opset 12
+ onnx = file.with_suffix('.onnx')
+
+ LOGGER.info(f'\n{prefix} starting export with TensorRT {trt.__version__}...')
+ assert onnx.exists(), f'failed to export ONNX file: {onnx}'
+ f = file.with_suffix('.engine') # TensorRT engine file
+ logger = trt.Logger(trt.Logger.INFO)
+ if verbose:
+ logger.min_severity = trt.Logger.Severity.VERBOSE
+
+ builder = trt.Builder(logger)
+ config = builder.create_builder_config()
+ config.max_workspace_size = workspace * 1 << 30
+ # config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, workspace << 30) # fix TRT 8.4 deprecation notice
+
+ flag = (1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
+ network = builder.create_network(flag)
+ parser = trt.OnnxParser(network, logger)
+ if not parser.parse_from_file(str(onnx)):
+ raise RuntimeError(f'failed to load ONNX file: {onnx}')
+
+ inputs = [network.get_input(i) for i in range(network.num_inputs)]
+ outputs = [network.get_output(i) for i in range(network.num_outputs)]
+ for inp in inputs:
+ LOGGER.info(f'{prefix} input "{inp.name}" with shape{inp.shape} {inp.dtype}')
+ for out in outputs:
+ LOGGER.info(f'{prefix} output "{out.name}" with shape{out.shape} {out.dtype}')
+
+ if dynamic:
+ if im.shape[0] <= 1:
+ LOGGER.warning(f'{prefix} WARNING ⚠️ --dynamic model requires maximum --batch-size argument')
+ profile = builder.create_optimization_profile()
+ for inp in inputs:
+ profile.set_shape(inp.name, (1, *im.shape[1:]), (max(1, im.shape[0] // 2), *im.shape[1:]), im.shape)
+ config.add_optimization_profile(profile)
+
+ LOGGER.info(f'{prefix} building FP{16 if builder.platform_has_fast_fp16 and half else 32} engine as {f}')
+ if builder.platform_has_fast_fp16 and half:
+ config.set_flag(trt.BuilderFlag.FP16)
+ with builder.build_engine(network, config) as engine, open(f, 'wb') as t:
+ t.write(engine.serialize())
+ return f, None
+
+
+@try_export
+def export_saved_model(model,
+ im,
+ file,
+ dynamic,
+ tf_nms=False,
+ agnostic_nms=False,
+ topk_per_class=100,
+ topk_all=100,
+ iou_thres=0.45,
+ conf_thres=0.25,
+ keras=False,
+ prefix=colorstr('TensorFlow SavedModel:')):
+ # YOLOv5 TensorFlow SavedModel export
+ try:
+ import tensorflow as tf
+ except Exception:
+ check_requirements(f"tensorflow{'' if torch.cuda.is_available() else '-macos' if MACOS else '-cpu'}")
+ import tensorflow as tf
+ from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2
+
+ from models.tf import TFModel
+
+ LOGGER.info(f'\n{prefix} starting export with tensorflow {tf.__version__}...')
+ f = str(file).replace('.pt', '_saved_model')
+ batch_size, ch, *imgsz = list(im.shape) # BCHW
+
+ tf_model = TFModel(cfg=model.yaml, model=model, nc=model.nc, imgsz=imgsz)
+ im = tf.zeros((batch_size, *imgsz, ch)) # BHWC order for TensorFlow
+ _ = tf_model.predict(im, tf_nms, agnostic_nms, topk_per_class, topk_all, iou_thres, conf_thres)
+ inputs = tf.keras.Input(shape=(*imgsz, ch), batch_size=None if dynamic else batch_size)
+ outputs = tf_model.predict(inputs, tf_nms, agnostic_nms, topk_per_class, topk_all, iou_thres, conf_thres)
+ keras_model = tf.keras.Model(inputs=inputs, outputs=outputs)
+ keras_model.trainable = False
+ keras_model.summary()
+ if keras:
+ keras_model.save(f, save_format='tf')
+ else:
+ spec = tf.TensorSpec(keras_model.inputs[0].shape, keras_model.inputs[0].dtype)
+ m = tf.function(lambda x: keras_model(x)) # full model
+ m = m.get_concrete_function(spec)
+ frozen_func = convert_variables_to_constants_v2(m)
+ tfm = tf.Module()
+ tfm.__call__ = tf.function(lambda x: frozen_func(x)[:4] if tf_nms else frozen_func(x), [spec])
+ tfm.__call__(im)
+ tf.saved_model.save(tfm,
+ f,
+ options=tf.saved_model.SaveOptions(experimental_custom_gradients=False) if check_version(
+ tf.__version__, '2.6') else tf.saved_model.SaveOptions())
+ return f, keras_model
+
+
+@try_export
+def export_pb(keras_model, file, prefix=colorstr('TensorFlow GraphDef:')):
+ # YOLOv5 TensorFlow GraphDef *.pb export https://github.com/leimao/Frozen_Graph_TensorFlow
+ import tensorflow as tf
+ from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2
+
+ LOGGER.info(f'\n{prefix} starting export with tensorflow {tf.__version__}...')
+ f = file.with_suffix('.pb')
+
+ m = tf.function(lambda x: keras_model(x)) # full model
+ m = m.get_concrete_function(tf.TensorSpec(keras_model.inputs[0].shape, keras_model.inputs[0].dtype))
+ frozen_func = convert_variables_to_constants_v2(m)
+ frozen_func.graph.as_graph_def()
+ tf.io.write_graph(graph_or_graph_def=frozen_func.graph, logdir=str(f.parent), name=f.name, as_text=False)
+ return f, None
+
+
+@try_export
+def export_tflite(keras_model, im, file, int8, data, nms, agnostic_nms, prefix=colorstr('TensorFlow Lite:')):
+ # YOLOv5 TensorFlow Lite export
+ import tensorflow as tf
+
+ LOGGER.info(f'\n{prefix} starting export with tensorflow {tf.__version__}...')
+ batch_size, ch, *imgsz = list(im.shape) # BCHW
+ f = str(file).replace('.pt', '-fp16.tflite')
+
+ converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
+ converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
+ converter.target_spec.supported_types = [tf.float16]
+ converter.optimizations = [tf.lite.Optimize.DEFAULT]
+ if int8:
+ from models.tf import representative_dataset_gen
+ dataset = LoadImages(check_dataset(check_yaml(data))['train'], img_size=imgsz, auto=False)
+ converter.representative_dataset = lambda: representative_dataset_gen(dataset, ncalib=100)
+ converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
+ converter.target_spec.supported_types = []
+ converter.inference_input_type = tf.uint8 # or tf.int8
+ converter.inference_output_type = tf.uint8 # or tf.int8
+ converter.experimental_new_quantizer = True
+ f = str(file).replace('.pt', '-int8.tflite')
+ if nms or agnostic_nms:
+ converter.target_spec.supported_ops.append(tf.lite.OpsSet.SELECT_TF_OPS)
+
+ tflite_model = converter.convert()
+ open(f, 'wb').write(tflite_model)
+ return f, None
+
+
+@try_export
+def export_edgetpu(file, prefix=colorstr('Edge TPU:')):
+ # YOLOv5 Edge TPU export https://coral.ai/docs/edgetpu/models-intro/
+ cmd = 'edgetpu_compiler --version'
+ help_url = 'https://coral.ai/docs/edgetpu/compiler/'
+ assert platform.system() == 'Linux', f'export only supported on Linux. See {help_url}'
+ if subprocess.run(f'{cmd} > /dev/null 2>&1', shell=True).returncode != 0:
+ LOGGER.info(f'\n{prefix} export requires Edge TPU compiler. Attempting install from {help_url}')
+ sudo = subprocess.run('sudo --version >/dev/null', shell=True).returncode == 0 # sudo installed on system
+ for c in (
+ 'curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -',
+ 'echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list',
+ 'sudo apt-get update', 'sudo apt-get install edgetpu-compiler'):
+ subprocess.run(c if sudo else c.replace('sudo ', ''), shell=True, check=True)
+ ver = subprocess.run(cmd, shell=True, capture_output=True, check=True).stdout.decode().split()[-1]
+
+ LOGGER.info(f'\n{prefix} starting export with Edge TPU compiler {ver}...')
+ f = str(file).replace('.pt', '-int8_edgetpu.tflite') # Edge TPU model
+ f_tfl = str(file).replace('.pt', '-int8.tflite') # TFLite model
+
+ subprocess.run([
+ 'edgetpu_compiler',
+ '-s',
+ '-d',
+ '-k',
+ '10',
+ '--out_dir',
+ str(file.parent),
+ f_tfl, ], check=True)
+ return f, None
+
+
+@try_export
+def export_tfjs(file, int8, prefix=colorstr('TensorFlow.js:')):
+ # YOLOv5 TensorFlow.js export
+ check_requirements('tensorflowjs')
+ import tensorflowjs as tfjs
+
+ LOGGER.info(f'\n{prefix} starting export with tensorflowjs {tfjs.__version__}...')
+ f = str(file).replace('.pt', '_web_model') # js dir
+ f_pb = file.with_suffix('.pb') # *.pb path
+ f_json = f'{f}/model.json' # *.json path
+
+ args = [
+ 'tensorflowjs_converter',
+ '--input_format=tf_frozen_model',
+ '--quantize_uint8' if int8 else '',
+ '--output_node_names=Identity,Identity_1,Identity_2,Identity_3',
+ str(f_pb),
+ str(f), ]
+ subprocess.run([arg for arg in args if arg], check=True)
+
+ json = Path(f_json).read_text()
+ with open(f_json, 'w') as j: # sort JSON Identity_* in ascending order
+ subst = re.sub(
+ r'{"outputs": {"Identity.?.?": {"name": "Identity.?.?"}, '
+ r'"Identity.?.?": {"name": "Identity.?.?"}, '
+ r'"Identity.?.?": {"name": "Identity.?.?"}, '
+ r'"Identity.?.?": {"name": "Identity.?.?"}}}', r'{"outputs": {"Identity": {"name": "Identity"}, '
+ r'"Identity_1": {"name": "Identity_1"}, '
+ r'"Identity_2": {"name": "Identity_2"}, '
+ r'"Identity_3": {"name": "Identity_3"}}}', json)
+ j.write(subst)
+ return f, None
+
+
+def add_tflite_metadata(file, metadata, num_outputs):
+ # Add metadata to *.tflite models per https://www.tensorflow.org/lite/models/convert/metadata
+ with contextlib.suppress(ImportError):
+ # check_requirements('tflite_support')
+ from tflite_support import flatbuffers
+ from tflite_support import metadata as _metadata
+ from tflite_support import metadata_schema_py_generated as _metadata_fb
+
+ tmp_file = Path('/tmp/meta.txt')
+ with open(tmp_file, 'w') as meta_f:
+ meta_f.write(str(metadata))
+
+ model_meta = _metadata_fb.ModelMetadataT()
+ label_file = _metadata_fb.AssociatedFileT()
+ label_file.name = tmp_file.name
+ model_meta.associatedFiles = [label_file]
+
+ subgraph = _metadata_fb.SubGraphMetadataT()
+ subgraph.inputTensorMetadata = [_metadata_fb.TensorMetadataT()]
+ subgraph.outputTensorMetadata = [_metadata_fb.TensorMetadataT()] * num_outputs
+ model_meta.subgraphMetadata = [subgraph]
+
+ b = flatbuffers.Builder(0)
+ b.Finish(model_meta.Pack(b), _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
+ metadata_buf = b.Output()
+
+ populator = _metadata.MetadataPopulator.with_model_file(file)
+ populator.load_metadata_buffer(metadata_buf)
+ populator.load_associated_files([str(tmp_file)])
+ populator.populate()
+ tmp_file.unlink()
+
+
+def pipeline_coreml(model, im, file, names, y, prefix=colorstr('CoreML Pipeline:')):
+ # YOLOv5 CoreML pipeline
+ import coremltools as ct
+ from PIL import Image
+
+ print(f'{prefix} starting pipeline with coremltools {ct.__version__}...')
+ batch_size, ch, h, w = list(im.shape) # BCHW
+ t = time.time()
+
+ # YOLOv5 Output shapes
+ spec = model.get_spec()
+ out0, out1 = iter(spec.description.output)
+ if platform.system() == 'Darwin':
+ img = Image.new('RGB', (w, h)) # img(192 width, 320 height)
+ # img = torch.zeros((*opt.img_size, 3)).numpy() # img size(320,192,3) iDetection
+ out = model.predict({'image': img})
+ out0_shape, out1_shape = out[out0.name].shape, out[out1.name].shape
+ else: # linux and windows can not run model.predict(), get sizes from pytorch output y
+ s = tuple(y[0].shape)
+ out0_shape, out1_shape = (s[1], s[2] - 5), (s[1], 4) # (3780, 80), (3780, 4)
+
+ # Checks
+ nx, ny = spec.description.input[0].type.imageType.width, spec.description.input[0].type.imageType.height
+ na, nc = out0_shape
+ # na, nc = out0.type.multiArrayType.shape # number anchors, classes
+ assert len(names) == nc, f'{len(names)} names found for nc={nc}' # check
+
+ # Define output shapes (missing)
+ out0.type.multiArrayType.shape[:] = out0_shape # (3780, 80)
+ out1.type.multiArrayType.shape[:] = out1_shape # (3780, 4)
+ # spec.neuralNetwork.preprocessing[0].featureName = '0'
+
+ # Flexible input shapes
+ # from coremltools.models.neural_network import flexible_shape_utils
+ # s = [] # shapes
+ # s.append(flexible_shape_utils.NeuralNetworkImageSize(320, 192))
+ # s.append(flexible_shape_utils.NeuralNetworkImageSize(640, 384)) # (height, width)
+ # flexible_shape_utils.add_enumerated_image_sizes(spec, feature_name='image', sizes=s)
+ # r = flexible_shape_utils.NeuralNetworkImageSizeRange() # shape ranges
+ # r.add_height_range((192, 640))
+ # r.add_width_range((192, 640))
+ # flexible_shape_utils.update_image_size_range(spec, feature_name='image', size_range=r)
+
+ # Print
+ print(spec.description)
+
+ # Model from spec
+ model = ct.models.MLModel(spec)
+
+ # 3. Create NMS protobuf
+ nms_spec = ct.proto.Model_pb2.Model()
+ nms_spec.specificationVersion = 5
+ for i in range(2):
+ decoder_output = model._spec.description.output[i].SerializeToString()
+ nms_spec.description.input.add()
+ nms_spec.description.input[i].ParseFromString(decoder_output)
+ nms_spec.description.output.add()
+ nms_spec.description.output[i].ParseFromString(decoder_output)
+
+ nms_spec.description.output[0].name = 'confidence'
+ nms_spec.description.output[1].name = 'coordinates'
+
+ output_sizes = [nc, 4]
+ for i in range(2):
+ ma_type = nms_spec.description.output[i].type.multiArrayType
+ ma_type.shapeRange.sizeRanges.add()
+ ma_type.shapeRange.sizeRanges[0].lowerBound = 0
+ ma_type.shapeRange.sizeRanges[0].upperBound = -1
+ ma_type.shapeRange.sizeRanges.add()
+ ma_type.shapeRange.sizeRanges[1].lowerBound = output_sizes[i]
+ ma_type.shapeRange.sizeRanges[1].upperBound = output_sizes[i]
+ del ma_type.shape[:]
+
+ nms = nms_spec.nonMaximumSuppression
+ nms.confidenceInputFeatureName = out0.name # 1x507x80
+ nms.coordinatesInputFeatureName = out1.name # 1x507x4
+ nms.confidenceOutputFeatureName = 'confidence'
+ nms.coordinatesOutputFeatureName = 'coordinates'
+ nms.iouThresholdInputFeatureName = 'iouThreshold'
+ nms.confidenceThresholdInputFeatureName = 'confidenceThreshold'
+ nms.iouThreshold = 0.45
+ nms.confidenceThreshold = 0.25
+ nms.pickTop.perClass = True
+ nms.stringClassLabels.vector.extend(names.values())
+ nms_model = ct.models.MLModel(nms_spec)
+
+ # 4. Pipeline models together
+ pipeline = ct.models.pipeline.Pipeline(input_features=[('image', ct.models.datatypes.Array(3, ny, nx)),
+ ('iouThreshold', ct.models.datatypes.Double()),
+ ('confidenceThreshold', ct.models.datatypes.Double())],
+ output_features=['confidence', 'coordinates'])
+ pipeline.add_model(model)
+ pipeline.add_model(nms_model)
+
+ # Correct datatypes
+ pipeline.spec.description.input[0].ParseFromString(model._spec.description.input[0].SerializeToString())
+ pipeline.spec.description.output[0].ParseFromString(nms_model._spec.description.output[0].SerializeToString())
+ pipeline.spec.description.output[1].ParseFromString(nms_model._spec.description.output[1].SerializeToString())
+
+ # Update metadata
+ pipeline.spec.specificationVersion = 5
+ pipeline.spec.description.metadata.versionString = 'https://github.com/ultralytics/yolov5'
+ pipeline.spec.description.metadata.shortDescription = 'https://github.com/ultralytics/yolov5'
+ pipeline.spec.description.metadata.author = 'glenn.jocher@ultralytics.com'
+ pipeline.spec.description.metadata.license = 'https://github.com/ultralytics/yolov5/blob/master/LICENSE'
+ pipeline.spec.description.metadata.userDefined.update({
+ 'classes': ','.join(names.values()),
+ 'iou_threshold': str(nms.iouThreshold),
+ 'confidence_threshold': str(nms.confidenceThreshold)})
+
+ # Save the model
+ f = file.with_suffix('.mlmodel') # filename
+ model = ct.models.MLModel(pipeline.spec)
+ model.input_description['image'] = 'Input image'
+ model.input_description['iouThreshold'] = f'(optional) IOU Threshold override (default: {nms.iouThreshold})'
+ model.input_description['confidenceThreshold'] = \
+ f'(optional) Confidence Threshold override (default: {nms.confidenceThreshold})'
+ model.output_description['confidence'] = 'Boxes × Class confidence (see user-defined metadata "classes")'
+ model.output_description['coordinates'] = 'Boxes × [x, y, width, height] (relative to image size)'
+ model.save(f) # pipelined
+ print(f'{prefix} pipeline success ({time.time() - t:.2f}s), saved as {f} ({file_size(f):.1f} MB)')
+
+
+@smart_inference_mode()
+def run(
+ data=ROOT / 'data/coco128.yaml', # 'dataset.yaml path'
+ weights=ROOT / 'yolov5s.pt', # weights path
+ imgsz=(640, 640), # image (height, width)
+ batch_size=1, # batch size
+ device='cpu', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ include=('torchscript', 'onnx'), # include formats
+ half=False, # FP16 half-precision export
+ inplace=False, # set YOLOv5 Detect() inplace=True
+ keras=False, # use Keras
+ optimize=False, # TorchScript: optimize for mobile
+ int8=False, # CoreML/TF INT8 quantization
+ dynamic=False, # ONNX/TF/TensorRT: dynamic axes
+ simplify=False, # ONNX: simplify model
+ opset=12, # ONNX: opset version
+ verbose=False, # TensorRT: verbose log
+ workspace=4, # TensorRT: workspace size (GB)
+ nms=False, # TF: add NMS to model
+ agnostic_nms=False, # TF: add agnostic NMS to model
+ topk_per_class=100, # TF.js NMS: topk per class to keep
+ topk_all=100, # TF.js NMS: topk for all classes to keep
+ iou_thres=0.45, # TF.js NMS: IoU threshold
+ conf_thres=0.25, # TF.js NMS: confidence threshold
+):
+ t = time.time()
+ include = [x.lower() for x in include] # to lowercase
+ fmts = tuple(export_formats()['Argument'][1:]) # --include arguments
+ flags = [x in include for x in fmts]
+ assert sum(flags) == len(include), f'ERROR: Invalid --include {include}, valid --include arguments are {fmts}'
+ jit, onnx, xml, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle = flags # export booleans
+ file = Path(url2file(weights) if str(weights).startswith(('http:/', 'https:/')) else weights) # PyTorch weights
+
+ # Load PyTorch model
+ device = select_device(device)
+ if half:
+ assert device.type != 'cpu' or coreml, '--half only compatible with GPU export, i.e. use --device 0'
+ assert not dynamic, '--half not compatible with --dynamic, i.e. use either --half or --dynamic but not both'
+ model = attempt_load(weights, device=device, inplace=True, fuse=True) # load FP32 model
+
+ # Checks
+ imgsz *= 2 if len(imgsz) == 1 else 1 # expand
+ if optimize:
+ assert device.type == 'cpu', '--optimize not compatible with cuda devices, i.e. use --device cpu'
+
+ # Input
+ gs = int(max(model.stride)) # grid size (max stride)
+ imgsz = [check_img_size(x, gs) for x in imgsz] # verify img_size are gs-multiples
+ im = torch.zeros(batch_size, 3, *imgsz).to(device) # image size(1,3,320,192) BCHW iDetection
+
+ # Update model
+ model.eval()
+ for k, m in model.named_modules():
+ if isinstance(m, Detect):
+ m.inplace = inplace
+ m.dynamic = dynamic
+ m.export = True
+
+ for _ in range(2):
+ y = model(im) # dry runs
+ if half and not coreml:
+ im, model = im.half(), model.half() # to FP16
+ shape = tuple((y[0] if isinstance(y, tuple) else y).shape) # model output shape
+ metadata = {'stride': int(max(model.stride)), 'names': model.names} # model metadata
+ LOGGER.info(f"\n{colorstr('PyTorch:')} starting from {file} with output shape {shape} ({file_size(file):.1f} MB)")
+
+ # Exports
+ f = [''] * len(fmts) # exported filenames
+ warnings.filterwarnings(action='ignore', category=torch.jit.TracerWarning) # suppress TracerWarning
+ if jit: # TorchScript
+ f[0], _ = export_torchscript(model, im, file, optimize)
+ if engine: # TensorRT required before ONNX
+ f[1], _ = export_engine(model, im, file, half, dynamic, simplify, workspace, verbose)
+ if onnx or xml: # OpenVINO requires ONNX
+ f[2], _ = export_onnx(model, im, file, opset, dynamic, simplify)
+ if xml: # OpenVINO
+ f[3], _ = export_openvino(file, metadata, half, int8, data)
+ if coreml: # CoreML
+ f[4], ct_model = export_coreml(model, im, file, int8, half, nms)
+ if nms:
+ pipeline_coreml(ct_model, im, file, model.names, y)
+ if any((saved_model, pb, tflite, edgetpu, tfjs)): # TensorFlow formats
+ assert not tflite or not tfjs, 'TFLite and TF.js models must be exported separately, please pass only one type.'
+ assert not isinstance(model, ClassificationModel), 'ClassificationModel export to TF formats not yet supported.'
+ f[5], s_model = export_saved_model(model.cpu(),
+ im,
+ file,
+ dynamic,
+ tf_nms=nms or agnostic_nms or tfjs,
+ agnostic_nms=agnostic_nms or tfjs,
+ topk_per_class=topk_per_class,
+ topk_all=topk_all,
+ iou_thres=iou_thres,
+ conf_thres=conf_thres,
+ keras=keras)
+ if pb or tfjs: # pb prerequisite to tfjs
+ f[6], _ = export_pb(s_model, file)
+ if tflite or edgetpu:
+ f[7], _ = export_tflite(s_model, im, file, int8 or edgetpu, data=data, nms=nms, agnostic_nms=agnostic_nms)
+ if edgetpu:
+ f[8], _ = export_edgetpu(file)
+ add_tflite_metadata(f[8] or f[7], metadata, num_outputs=len(s_model.outputs))
+ if tfjs:
+ f[9], _ = export_tfjs(file, int8)
+ if paddle: # PaddlePaddle
+ f[10], _ = export_paddle(model, im, file, metadata)
+
+ # Finish
+ f = [str(x) for x in f if x] # filter out '' and None
+ if any(f):
+ cls, det, seg = (isinstance(model, x) for x in (ClassificationModel, DetectionModel, SegmentationModel)) # type
+ det &= not seg # segmentation models inherit from SegmentationModel(DetectionModel)
+ dir = Path('segment' if seg else 'classify' if cls else '')
+ h = '--half' if half else '' # --half FP16 inference arg
+ s = '# WARNING ⚠️ ClassificationModel not yet supported for PyTorch Hub AutoShape inference' if cls else \
+ '# WARNING ⚠️ SegmentationModel not yet supported for PyTorch Hub AutoShape inference' if seg else ''
+ LOGGER.info(f'\nExport complete ({time.time() - t:.1f}s)'
+ f"\nResults saved to {colorstr('bold', file.parent.resolve())}"
+ f"\nDetect: python {dir / ('detect.py' if det else 'predict.py')} --weights {f[-1]} {h}"
+ f"\nValidate: python {dir / 'val.py'} --weights {f[-1]} {h}"
+ f"\nPyTorch Hub: model = torch.hub.load('ultralytics/yolov5', 'custom', '{f[-1]}') {s}"
+ f'\nVisualize: https://netron.app')
+ return f # return list of exported files/dirs
+
+
+def parse_opt(known=False):
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='dataset.yaml path')
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model.pt path(s)')
+ parser.add_argument('--imgsz', '--img', '--img-size', nargs='+', type=int, default=[640, 640], help='image (h, w)')
+ parser.add_argument('--batch-size', type=int, default=1, help='batch size')
+ parser.add_argument('--device', default='cpu', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--half', action='store_true', help='FP16 half-precision export')
+ parser.add_argument('--inplace', action='store_true', help='set YOLOv5 Detect() inplace=True')
+ parser.add_argument('--keras', action='store_true', help='TF: use Keras')
+ parser.add_argument('--optimize', action='store_true', help='TorchScript: optimize for mobile')
+ parser.add_argument('--int8', action='store_true', help='CoreML/TF/OpenVINO INT8 quantization')
+ parser.add_argument('--dynamic', action='store_true', help='ONNX/TF/TensorRT: dynamic axes')
+ parser.add_argument('--simplify', action='store_true', help='ONNX: simplify model')
+ parser.add_argument('--opset', type=int, default=17, help='ONNX: opset version')
+ parser.add_argument('--verbose', action='store_true', help='TensorRT: verbose log')
+ parser.add_argument('--workspace', type=int, default=4, help='TensorRT: workspace size (GB)')
+ parser.add_argument('--nms', action='store_true', help='TF: add NMS to model')
+ parser.add_argument('--agnostic-nms', action='store_true', help='TF: add agnostic NMS to model')
+ parser.add_argument('--topk-per-class', type=int, default=100, help='TF.js NMS: topk per class to keep')
+ parser.add_argument('--topk-all', type=int, default=100, help='TF.js NMS: topk for all classes to keep')
+ parser.add_argument('--iou-thres', type=float, default=0.45, help='TF.js NMS: IoU threshold')
+ parser.add_argument('--conf-thres', type=float, default=0.25, help='TF.js NMS: confidence threshold')
+ parser.add_argument(
+ '--include',
+ nargs='+',
+ default=['torchscript'],
+ help='torchscript, onnx, openvino, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle')
+ opt = parser.parse_known_args()[0] if known else parser.parse_args()
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ for opt.weights in (opt.weights if isinstance(opt.weights, list) else [opt.weights]):
+ run(**vars(opt))
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/hubconf.py b/TextDetection/hubconf.py
new file mode 100644
index 0000000000000000000000000000000000000000..f0192698fbe39f463e21a3092230258565cc7e0f
--- /dev/null
+++ b/TextDetection/hubconf.py
@@ -0,0 +1,169 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+PyTorch Hub models https://pytorch.org/hub/ultralytics_yolov5
+
+Usage:
+ import torch
+ model = torch.hub.load('ultralytics/yolov5', 'yolov5s') # official model
+ model = torch.hub.load('ultralytics/yolov5:master', 'yolov5s') # from branch
+ model = torch.hub.load('ultralytics/yolov5', 'custom', 'yolov5s.pt') # custom/local model
+ model = torch.hub.load('.', 'custom', 'yolov5s.pt', source='local') # local repo
+"""
+
+import torch
+
+
+def _create(name, pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None):
+ """Creates or loads a YOLOv5 model
+
+ Arguments:
+ name (str): model name 'yolov5s' or path 'path/to/best.pt'
+ pretrained (bool): load pretrained weights into the model
+ channels (int): number of input channels
+ classes (int): number of model classes
+ autoshape (bool): apply YOLOv5 .autoshape() wrapper to model
+ verbose (bool): print all information to screen
+ device (str, torch.device, None): device to use for model parameters
+
+ Returns:
+ YOLOv5 model
+ """
+ from pathlib import Path
+
+ from models.common import AutoShape, DetectMultiBackend
+ from models.experimental import attempt_load
+ from models.yolo import ClassificationModel, DetectionModel, SegmentationModel
+ from utils.downloads import attempt_download
+ from utils.general import LOGGER, ROOT, check_requirements, intersect_dicts, logging
+ from utils.torch_utils import select_device
+
+ if not verbose:
+ LOGGER.setLevel(logging.WARNING)
+ check_requirements(ROOT / 'requirements.txt', exclude=('opencv-python', 'tensorboard', 'thop'))
+ name = Path(name)
+ path = name.with_suffix('.pt') if name.suffix == '' and not name.is_dir() else name # checkpoint path
+ try:
+ device = select_device(device)
+ if pretrained and channels == 3 and classes == 80:
+ try:
+ model = DetectMultiBackend(path, device=device, fuse=autoshape) # detection model
+ if autoshape:
+ if model.pt and isinstance(model.model, ClassificationModel):
+ LOGGER.warning('WARNING ⚠️ YOLOv5 ClassificationModel is not yet AutoShape compatible. '
+ 'You must pass torch tensors in BCHW to this model, i.e. shape(1,3,224,224).')
+ elif model.pt and isinstance(model.model, SegmentationModel):
+ LOGGER.warning('WARNING ⚠️ YOLOv5 SegmentationModel is not yet AutoShape compatible. '
+ 'You will not be able to run inference with this model.')
+ else:
+ model = AutoShape(model) # for file/URI/PIL/cv2/np inputs and NMS
+ except Exception:
+ model = attempt_load(path, device=device, fuse=False) # arbitrary model
+ else:
+ cfg = list((Path(__file__).parent / 'models').rglob(f'{path.stem}.yaml'))[0] # model.yaml path
+ model = DetectionModel(cfg, channels, classes) # create model
+ if pretrained:
+ ckpt = torch.load(attempt_download(path), map_location=device) # load
+ csd = ckpt['model'].float().state_dict() # checkpoint state_dict as FP32
+ csd = intersect_dicts(csd, model.state_dict(), exclude=['anchors']) # intersect
+ model.load_state_dict(csd, strict=False) # load
+ if len(ckpt['model'].names) == classes:
+ model.names = ckpt['model'].names # set class names attribute
+ if not verbose:
+ LOGGER.setLevel(logging.INFO) # reset to default
+ return model.to(device)
+
+ except Exception as e:
+ help_url = 'https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading'
+ s = f'{e}. Cache may be out of date, try `force_reload=True` or see {help_url} for help.'
+ raise Exception(s) from e
+
+
+def custom(path='path/to/model.pt', autoshape=True, _verbose=True, device=None):
+ # YOLOv5 custom or local model
+ return _create(path, autoshape=autoshape, verbose=_verbose, device=device)
+
+
+def yolov5n(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-nano model https://github.com/ultralytics/yolov5
+ return _create('yolov5n', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5s(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-small model https://github.com/ultralytics/yolov5
+ return _create('yolov5s', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5m(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-medium model https://github.com/ultralytics/yolov5
+ return _create('yolov5m', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5l(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-large model https://github.com/ultralytics/yolov5
+ return _create('yolov5l', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5x(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-xlarge model https://github.com/ultralytics/yolov5
+ return _create('yolov5x', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5n6(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-nano-P6 model https://github.com/ultralytics/yolov5
+ return _create('yolov5n6', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5s6(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-small-P6 model https://github.com/ultralytics/yolov5
+ return _create('yolov5s6', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5m6(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-medium-P6 model https://github.com/ultralytics/yolov5
+ return _create('yolov5m6', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5l6(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-large-P6 model https://github.com/ultralytics/yolov5
+ return _create('yolov5l6', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+def yolov5x6(pretrained=True, channels=3, classes=80, autoshape=True, _verbose=True, device=None):
+ # YOLOv5-xlarge-P6 model https://github.com/ultralytics/yolov5
+ return _create('yolov5x6', pretrained, channels, classes, autoshape, _verbose, device)
+
+
+if __name__ == '__main__':
+ import argparse
+ from pathlib import Path
+
+ import numpy as np
+ from PIL import Image
+
+ from utils.general import cv2, print_args
+
+ # Argparser
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--model', type=str, default='yolov5s', help='model name')
+ opt = parser.parse_args()
+ print_args(vars(opt))
+
+ # Model
+ model = _create(name=opt.model, pretrained=True, channels=3, classes=80, autoshape=True, verbose=True)
+ # model = custom(path='path/to/model.pt') # custom
+
+ # Images
+ imgs = [
+ 'data/images/zidane.jpg', # filename
+ Path('data/images/zidane.jpg'), # Path
+ 'https://ultralytics.com/images/zidane.jpg', # URI
+ cv2.imread('data/images/bus.jpg')[:, :, ::-1], # OpenCV
+ Image.open('data/images/bus.jpg'), # PIL
+ np.zeros((320, 640, 3))] # numpy
+
+ # Inference
+ results = model(imgs, size=320) # batched inference
+
+ # Results
+ results.print()
+ results.save()
diff --git a/TextDetection/requirements.txt b/TextDetection/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..069cafe7e14fa9cbba64a27453120d0e930c3b90
--- /dev/null
+++ b/TextDetection/requirements.txt
@@ -0,0 +1,49 @@
+# YOLOv5 requirements
+# Usage: pip install -r requirements.txt
+
+# Base ------------------------------------------------------------------------
+gitpython>=3.1.30
+matplotlib>=3.3
+numpy>=1.18.5
+opencv-python>=4.1.1
+Pillow>=7.1.2
+psutil # system resources
+PyYAML>=5.3.1
+requests>=2.23.0
+scipy>=1.4.1
+thop>=0.1.1 # FLOPs computation
+torch>=1.7.0 # see https://pytorch.org/get-started/locally (recommended)
+torchvision>=0.8.1
+tqdm>=4.64.0
+ultralytics>=8.0.111
+# protobuf<=3.20.1 # https://github.com/ultralytics/yolov5/issues/8012
+
+# Logging ---------------------------------------------------------------------
+# tensorboard>=2.4.1
+# clearml>=1.2.0
+# comet
+
+# Plotting --------------------------------------------------------------------
+pandas>=1.1.4
+seaborn>=0.11.0
+
+# Export ----------------------------------------------------------------------
+# coremltools>=6.0 # CoreML export
+# onnx>=1.10.0 # ONNX export
+# onnx-simplifier>=0.4.1 # ONNX simplifier
+# nvidia-pyindex # TensorRT export
+# nvidia-tensorrt # TensorRT export
+# scikit-learn<=1.1.2 # CoreML quantization
+# tensorflow>=2.4.0 # TF exports (-cpu, -aarch64, -macos)
+# tensorflowjs>=3.9.0 # TF.js export
+# openvino-dev # OpenVINO export
+
+# Deploy ----------------------------------------------------------------------
+setuptools>=65.5.1 # Snyk vulnerability fix
+# tritonclient[all]~=2.24.0
+
+# Extras ----------------------------------------------------------------------
+# ipython # interactive notebook
+# mss # screenshots
+# albumentations>=1.0.3
+# pycocotools>=2.0.6 # COCO mAP
diff --git a/TextDetection/runs/wordDetection/F1_curve.png b/TextDetection/runs/wordDetection/F1_curve.png
new file mode 100644
index 0000000000000000000000000000000000000000..d9f1a8694fa57ffd19bd3eeefa6709dca5d53d23
Binary files /dev/null and b/TextDetection/runs/wordDetection/F1_curve.png differ
diff --git a/TextDetection/runs/wordDetection/PR_curve.png b/TextDetection/runs/wordDetection/PR_curve.png
new file mode 100644
index 0000000000000000000000000000000000000000..81d8739c44446da6d6dc2937266e1777a049a654
Binary files /dev/null and b/TextDetection/runs/wordDetection/PR_curve.png differ
diff --git a/TextDetection/runs/wordDetection/P_curve.png b/TextDetection/runs/wordDetection/P_curve.png
new file mode 100644
index 0000000000000000000000000000000000000000..df36c188b0f18c4443a7a159e51cfdd64cbd5864
Binary files /dev/null and b/TextDetection/runs/wordDetection/P_curve.png differ
diff --git a/TextDetection/runs/wordDetection/R_curve.png b/TextDetection/runs/wordDetection/R_curve.png
new file mode 100644
index 0000000000000000000000000000000000000000..e2e2b249097384dfb71c99ec8d48df4c932f502b
Binary files /dev/null and b/TextDetection/runs/wordDetection/R_curve.png differ
diff --git a/TextDetection/runs/wordDetection/confusion_matrix.png b/TextDetection/runs/wordDetection/confusion_matrix.png
new file mode 100644
index 0000000000000000000000000000000000000000..21746822c4d90cd6470a76f100947ecfc58d5e11
Binary files /dev/null and b/TextDetection/runs/wordDetection/confusion_matrix.png differ
diff --git a/TextDetection/runs/wordDetection/hyp.yaml b/TextDetection/runs/wordDetection/hyp.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..e14c2f0a09dfcc4f24bf6642399bdf2b64b5d6d0
--- /dev/null
+++ b/TextDetection/runs/wordDetection/hyp.yaml
@@ -0,0 +1,28 @@
+lr0: 0.01
+lrf: 0.1
+momentum: 0.937
+weight_decay: 0.0005
+warmup_epochs: 3.0
+warmup_momentum: 0.8
+warmup_bias_lr: 0.1
+box: 0.05
+cls: 0.3
+cls_pw: 1.0
+obj: 0.7
+obj_pw: 1.0
+iou_t: 0.2
+anchor_t: 4.0
+fl_gamma: 0.0
+hsv_h: 0.015
+hsv_s: 0.7
+hsv_v: 0.4
+degrees: 0.0
+translate: 0.1
+scale: 0.9
+shear: 0.0
+perspective: 0.0
+flipud: 0.0
+fliplr: 0.5
+mosaic: 1.0
+mixup: 0.1
+copy_paste: 0.1
diff --git a/TextDetection/runs/wordDetection/labels.jpg b/TextDetection/runs/wordDetection/labels.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..1562ad76de808883d820001f41e7310f8fc02ee3
Binary files /dev/null and b/TextDetection/runs/wordDetection/labels.jpg differ
diff --git a/TextDetection/runs/wordDetection/labels_correlogram.jpg b/TextDetection/runs/wordDetection/labels_correlogram.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..565b3592572d2060edfaab5d1c85b9612e63df97
Binary files /dev/null and b/TextDetection/runs/wordDetection/labels_correlogram.jpg differ
diff --git a/TextDetection/runs/wordDetection/opt.yaml b/TextDetection/runs/wordDetection/opt.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..296d8ee3b17846ea6a53e4b9b08852eafd8513dc
--- /dev/null
+++ b/TextDetection/runs/wordDetection/opt.yaml
@@ -0,0 +1,68 @@
+weights: yolov5s.pt
+cfg: C:\Users\ParkLab\yolov5\models\yolov5s.yaml
+data: C:\Users\ParkLab\yolov5\data\HCR.yaml
+hyp:
+ lr0: 0.01
+ lrf: 0.1
+ momentum: 0.937
+ weight_decay: 0.0005
+ warmup_epochs: 3.0
+ warmup_momentum: 0.8
+ warmup_bias_lr: 0.1
+ box: 0.05
+ cls: 0.3
+ cls_pw: 1.0
+ obj: 0.7
+ obj_pw: 1.0
+ iou_t: 0.2
+ anchor_t: 4.0
+ fl_gamma: 0.0
+ hsv_h: 0.015
+ hsv_s: 0.7
+ hsv_v: 0.4
+ degrees: 0.0
+ translate: 0.1
+ scale: 0.9
+ shear: 0.0
+ perspective: 0.0
+ flipud: 0.0
+ fliplr: 0.5
+ mosaic: 1.0
+ mixup: 0.1
+ copy_paste: 0.1
+epochs: 100
+batch_size: 32
+imgsz: 640
+rect: false
+resume: false
+nosave: false
+noval: false
+noautoanchor: false
+noplots: false
+evolve: null
+bucket: ''
+cache: null
+image_weights: false
+device: '0'
+multi_scale: false
+single_cls: false
+optimizer: SGD
+sync_bn: false
+workers: 8
+project: runs\train
+name: yolo_word_det
+exist_ok: false
+quad: false
+cos_lr: false
+label_smoothing: 0.0
+patience: 100
+freeze:
+- 0
+save_period: -1
+seed: 0
+local_rank: -1
+entity: null
+upload_dataset: false
+bbox_interval: -1
+artifact_alias: latest
+save_dir: runs\train\yolo_word_det5
diff --git a/TextDetection/runs/wordDetection/results.csv b/TextDetection/runs/wordDetection/results.csv
new file mode 100644
index 0000000000000000000000000000000000000000..b42431f81fd749075f8a0e190ad27a420c716042
--- /dev/null
+++ b/TextDetection/runs/wordDetection/results.csv
@@ -0,0 +1,101 @@
+ epoch, train/box_loss, train/obj_loss, train/cls_loss, metrics/precision, metrics/recall, metrics/mAP_0.5,metrics/mAP_0.5:0.95, val/box_loss, val/obj_loss, val/cls_loss, x/lr0, x/lr1, x/lr2
+ 0, 0.14343, 0.1321, 0, 0.10678, 0.24045, 0.070025, 0.016663, 0.12338, 0.16908, 0, 0.070769, 0.0032479, 0.0032479
+ 1, 0.11814, 0.17616, 0, 0.29756, 0.4878, 0.29039, 0.090169, 0.09548, 0.21864, 0, 0.04071, 0.006522, 0.006522
+ 2, 0.11103, 0.19449, 0, 0.36226, 0.46782, 0.33602, 0.084028, 0.10112, 0.22694, 0, 0.010591, 0.0097361, 0.0097361
+ 3, 0.10865, 0.19592, 0, 0.43876, 0.69566, 0.51856, 0.18541, 0.086634, 0.2838, 0, 0.00973, 0.00973, 0.00973
+ 4, 0.10498, 0.19423, 0, 0.42118, 0.70279, 0.50729, 0.18303, 0.083472, 0.25062, 0, 0.00973, 0.00973, 0.00973
+ 5, 0.10106, 0.1933, 0, 0.75029, 0.74585, 0.7626, 0.36368, 0.0782, 0.20763, 0, 0.00964, 0.00964, 0.00964
+ 6, 0.10067, 0.18856, 0, 0.88406, 0.79332, 0.81448, 0.38753, 0.071593, 0.19628, 0, 0.00955, 0.00955, 0.00955
+ 7, 0.097406, 0.19524, 0, 0.84659, 0.8041, 0.82744, 0.45989, 0.069756, 0.19131, 0, 0.00946, 0.00946, 0.00946
+ 8, 0.096121, 0.19137, 0, 0.87909, 0.80622, 0.83644, 0.48052, 0.067627, 0.18748, 0, 0.00937, 0.00937, 0.00937
+ 9, 0.095513, 0.19479, 0, 0.87642, 0.78766, 0.82577, 0.47358, 0.067359, 0.19059, 0, 0.00928, 0.00928, 0.00928
+ 10, 0.093917, 0.18792, 0, 0.91461, 0.81907, 0.84504, 0.5117, 0.065237, 0.18478, 0, 0.00919, 0.00919, 0.00919
+ 11, 0.093425, 0.19291, 0, 0.90101, 0.80465, 0.83468, 0.474, 0.068537, 0.19139, 0, 0.0091, 0.0091, 0.0091
+ 12, 0.09289, 0.18656, 0, 0.92043, 0.82673, 0.85478, 0.53099, 0.064175, 0.18201, 0, 0.00901, 0.00901, 0.00901
+ 13, 0.090676, 0.18805, 0, 0.92008, 0.82673, 0.85636, 0.51892, 0.064579, 0.18353, 0, 0.00892, 0.00892, 0.00892
+ 14, 0.093122, 0.19202, 0, 0.92253, 0.83725, 0.8624, 0.57188, 0.060162, 0.17608, 0, 0.00883, 0.00883, 0.00883
+ 15, 0.09126, 0.1907, 0, 0.92946, 0.83274, 0.8676, 0.54458, 0.062534, 0.17887, 0, 0.00874, 0.00874, 0.00874
+ 16, 0.090793, 0.18172, 0, 0.93174, 0.83929, 0.86568, 0.58843, 0.059511, 0.17312, 0, 0.00865, 0.00865, 0.00865
+ 17, 0.089946, 0.1857, 0, 0.92914, 0.83106, 0.86364, 0.57621, 0.061277, 0.17707, 0, 0.00856, 0.00856, 0.00856
+ 18, 0.091165, 0.1837, 0, 0.92212, 0.83831, 0.86719, 0.55994, 0.061093, 0.17976, 0, 0.00847, 0.00847, 0.00847
+ 19, 0.088599, 0.18972, 0, 0.93237, 0.84209, 0.87114, 0.62397, 0.057884, 0.16953, 0, 0.00838, 0.00838, 0.00838
+ 20, 0.090061, 0.18562, 0, 0.93445, 0.83685, 0.87553, 0.60871, 0.0585, 0.16933, 0, 0.00829, 0.00829, 0.00829
+ 21, 0.088208, 0.18263, 0, 0.93335, 0.84114, 0.87449, 0.60064, 0.059247, 0.17255, 0, 0.0082, 0.0082, 0.0082
+ 22, 0.0895, 0.18648, 0, 0.93338, 0.84309, 0.87566, 0.61248, 0.057899, 0.16824, 0, 0.00811, 0.00811, 0.00811
+ 23, 0.08849, 0.18464, 0, 0.93093, 0.84751, 0.87541, 0.59865, 0.059204, 0.17046, 0, 0.00802, 0.00802, 0.00802
+ 24, 0.087192, 0.18088, 0, 0.93032, 0.84618, 0.87731, 0.63006, 0.057412, 0.16889, 0, 0.00793, 0.00793, 0.00793
+ 25, 0.088312, 0.1889, 0, 0.93524, 0.84637, 0.88106, 0.63846, 0.056614, 0.16571, 0, 0.00784, 0.00784, 0.00784
+ 26, 0.088479, 0.18287, 0, 0.93518, 0.85074, 0.88419, 0.65769, 0.055262, 0.16252, 0, 0.00775, 0.00775, 0.00775
+ 27, 0.087063, 0.18672, 0, 0.9345, 0.84504, 0.87943, 0.61842, 0.05768, 0.16942, 0, 0.00766, 0.00766, 0.00766
+ 28, 0.086641, 0.18357, 0, 0.9364, 0.84936, 0.88206, 0.64976, 0.05565, 0.16261, 0, 0.00757, 0.00757, 0.00757
+ 29, 0.086989, 0.18444, 0, 0.93528, 0.8431, 0.87314, 0.63409, 0.056122, 0.16428, 0, 0.00748, 0.00748, 0.00748
+ 30, 0.085464, 0.17787, 0, 0.93888, 0.85282, 0.88306, 0.65505, 0.054957, 0.16096, 0, 0.00739, 0.00739, 0.00739
+ 31, 0.086473, 0.18357, 0, 0.93646, 0.84441, 0.88046, 0.63613, 0.05671, 0.16618, 0, 0.0073, 0.0073, 0.0073
+ 32, 0.086182, 0.17859, 0, 0.94104, 0.85224, 0.88425, 0.65941, 0.054651, 0.15962, 0, 0.00721, 0.00721, 0.00721
+ 33, 0.085764, 0.17765, 0, 0.93804, 0.85115, 0.8856, 0.64438, 0.056184, 0.16485, 0, 0.00712, 0.00712, 0.00712
+ 34, 0.08671, 0.17883, 0, 0.93353, 0.85538, 0.88441, 0.64989, 0.055651, 0.16415, 0, 0.00703, 0.00703, 0.00703
+ 35, 0.084131, 0.17841, 0, 0.94215, 0.85281, 0.8878, 0.67065, 0.054301, 0.158, 0, 0.00694, 0.00694, 0.00694
+ 36, 0.085284, 0.17247, 0, 0.94246, 0.85436, 0.88769, 0.67468, 0.054275, 0.15919, 0, 0.00685, 0.00685, 0.00685
+ 37, 0.085696, 0.17306, 0, 0.94052, 0.85661, 0.88989, 0.67156, 0.054833, 0.15942, 0, 0.00676, 0.00676, 0.00676
+ 38, 0.086575, 0.1815, 0, 0.93904, 0.85797, 0.89072, 0.67339, 0.054065, 0.15824, 0, 0.00667, 0.00667, 0.00667
+ 39, 0.084747, 0.17844, 0, 0.94037, 0.85318, 0.88526, 0.66841, 0.054299, 0.15972, 0, 0.00658, 0.00658, 0.00658
+ 40, 0.085733, 0.17424, 0, 0.94515, 0.85003, 0.89003, 0.66812, 0.053869, 0.15828, 0, 0.00649, 0.00649, 0.00649
+ 41, 0.086109, 0.17978, 0, 0.94387, 0.85705, 0.89019, 0.66022, 0.05489, 0.16003, 0, 0.0064, 0.0064, 0.0064
+ 42, 0.084408, 0.17579, 0, 0.93925, 0.84928, 0.88556, 0.66384, 0.054702, 0.16477, 0, 0.00631, 0.00631, 0.00631
+ 43, 0.083645, 0.17836, 0, 0.94362, 0.86298, 0.89463, 0.68944, 0.052935, 0.15475, 0, 0.00622, 0.00622, 0.00622
+ 44, 0.08338, 0.17339, 0, 0.94253, 0.86408, 0.89546, 0.6745, 0.053576, 0.15763, 0, 0.00613, 0.00613, 0.00613
+ 45, 0.084568, 0.17409, 0, 0.92463, 0.84786, 0.87813, 0.66222, 0.054652, 0.16843, 0, 0.00604, 0.00604, 0.00604
+ 46, 0.084783, 0.1667, 0, 0.94382, 0.85541, 0.88957, 0.67704, 0.053853, 0.15688, 0, 0.00595, 0.00595, 0.00595
+ 47, 0.083283, 0.17465, 0, 0.94633, 0.8605, 0.89158, 0.68796, 0.053052, 0.15478, 0, 0.00586, 0.00586, 0.00586
+ 48, 0.084094, 0.17705, 0, 0.94249, 0.86307, 0.89251, 0.69703, 0.052554, 0.15498, 0, 0.00577, 0.00577, 0.00577
+ 49, 0.083676, 0.17547, 0, 0.94565, 0.86156, 0.89494, 0.69568, 0.052399, 0.15305, 0, 0.00568, 0.00568, 0.00568
+ 50, 0.083022, 0.1728, 0, 0.94449, 0.86147, 0.89294, 0.69564, 0.052254, 0.15408, 0, 0.00559, 0.00559, 0.00559
+ 51, 0.081813, 0.1704, 0, 0.94366, 0.86687, 0.89606, 0.69269, 0.052507, 0.15389, 0, 0.0055, 0.0055, 0.0055
+ 52, 0.083914, 0.1764, 0, 0.92865, 0.84273, 0.87447, 0.66104, 0.054271, 0.16671, 0, 0.00541, 0.00541, 0.00541
+ 53, 0.082179, 0.1702, 0, 0.94446, 0.86165, 0.89463, 0.70157, 0.052073, 0.15209, 0, 0.00532, 0.00532, 0.00532
+ 54, 0.081849, 0.17025, 0, 0.9405, 0.84998, 0.89014, 0.68139, 0.053257, 0.15801, 0, 0.00523, 0.00523, 0.00523
+ 55, 0.083674, 0.17681, 0, 0.94171, 0.86368, 0.89523, 0.69899, 0.051968, 0.15354, 0, 0.00514, 0.00514, 0.00514
+ 56, 0.082308, 0.17403, 0, 0.93685, 0.86183, 0.88979, 0.66637, 0.053955, 0.15908, 0, 0.00505, 0.00505, 0.00505
+ 57, 0.082482, 0.17658, 0, 0.94371, 0.86209, 0.89347, 0.67414, 0.052889, 0.15547, 0, 0.00496, 0.00496, 0.00496
+ 58, 0.081728, 0.16741, 0, 0.94564, 0.86112, 0.89675, 0.70912, 0.051474, 0.1514, 0, 0.00487, 0.00487, 0.00487
+ 59, 0.083933, 0.1704, 0, 0.94333, 0.86307, 0.89448, 0.70195, 0.05189, 0.15253, 0, 0.00478, 0.00478, 0.00478
+ 60, 0.082575, 0.17006, 0, 0.94497, 0.86607, 0.89639, 0.70839, 0.051361, 0.15026, 0, 0.00469, 0.00469, 0.00469
+ 61, 0.082538, 0.16962, 0, 0.93968, 0.8643, 0.89563, 0.69941, 0.051753, 0.1522, 0, 0.0046, 0.0046, 0.0046
+ 62, 0.080964, 0.16399, 0, 0.94686, 0.85971, 0.89541, 0.71117, 0.051314, 0.15036, 0, 0.00451, 0.00451, 0.00451
+ 63, 0.081617, 0.17485, 0, 0.9446, 0.85228, 0.89139, 0.68514, 0.052893, 0.15677, 0, 0.00442, 0.00442, 0.00442
+ 64, 0.082902, 0.17403, 0, 0.94468, 0.85752, 0.89338, 0.6903, 0.052549, 0.15499, 0, 0.00433, 0.00433, 0.00433
+ 65, 0.077992, 0.165, 0, 0.94387, 0.8622, 0.89289, 0.69896, 0.05173, 0.15193, 0, 0.00424, 0.00424, 0.00424
+ 66, 0.081977, 0.1678, 0, 0.94592, 0.86432, 0.89541, 0.69135, 0.052076, 0.15315, 0, 0.00415, 0.00415, 0.00415
+ 67, 0.083111, 0.17345, 0, 0.94692, 0.8613, 0.89599, 0.70917, 0.051276, 0.15006, 0, 0.00406, 0.00406, 0.00406
+ 68, 0.081756, 0.17021, 0, 0.94576, 0.86254, 0.89739, 0.71084, 0.051092, 0.15002, 0, 0.00397, 0.00397, 0.00397
+ 69, 0.082199, 0.17096, 0, 0.9463, 0.86643, 0.89746, 0.71133, 0.050999, 0.14945, 0, 0.00388, 0.00388, 0.00388
+ 70, 0.082778, 0.17548, 0, 0.9482, 0.8609, 0.89782, 0.71145, 0.051164, 0.14959, 0, 0.00379, 0.00379, 0.00379
+ 71, 0.080038, 0.16676, 0, 0.94958, 0.86736, 0.90067, 0.70935, 0.0511, 0.15013, 0, 0.0037, 0.0037, 0.0037
+ 72, 0.081567, 0.16679, 0, 0.94337, 0.86651, 0.89831, 0.70053, 0.051656, 0.15132, 0, 0.00361, 0.00361, 0.00361
+ 73, 0.080533, 0.16562, 0, 0.95096, 0.86536, 0.90124, 0.72249, 0.050545, 0.1472, 0, 0.00352, 0.00352, 0.00352
+ 74, 0.081029, 0.16575, 0, 0.94243, 0.85382, 0.89317, 0.6888, 0.052537, 0.15647, 0, 0.00343, 0.00343, 0.00343
+ 75, 0.081182, 0.17176, 0, 0.95107, 0.86266, 0.90062, 0.72069, 0.050538, 0.14724, 0, 0.00334, 0.00334, 0.00334
+ 76, 0.081938, 0.16753, 0, 0.95038, 0.86699, 0.90307, 0.72068, 0.050536, 0.14704, 0, 0.00325, 0.00325, 0.00325
+ 77, 0.081434, 0.16673, 0, 0.95063, 0.86647, 0.90187, 0.72559, 0.050253, 0.14626, 0, 0.00316, 0.00316, 0.00316
+ 78, 0.083186, 0.16854, 0, 0.93933, 0.85865, 0.89181, 0.69442, 0.052013, 0.15513, 0, 0.00307, 0.00307, 0.00307
+ 79, 0.080724, 0.16352, 0, 0.94591, 0.86574, 0.90045, 0.70892, 0.051192, 0.15029, 0, 0.00298, 0.00298, 0.00298
+ 80, 0.082062, 0.16734, 0, 0.94767, 0.86283, 0.899, 0.72162, 0.050446, 0.1479, 0, 0.00289, 0.00289, 0.00289
+ 81, 0.082112, 0.16387, 0, 0.94625, 0.87067, 0.8994, 0.72242, 0.050153, 0.14716, 0, 0.0028, 0.0028, 0.0028
+ 82, 0.081101, 0.16599, 0, 0.94863, 0.86682, 0.90004, 0.7212, 0.050394, 0.14794, 0, 0.00271, 0.00271, 0.00271
+ 83, 0.080418, 0.17192, 0, 0.94982, 0.86881, 0.9021, 0.72215, 0.050213, 0.14673, 0, 0.00262, 0.00262, 0.00262
+ 84, 0.080462, 0.16803, 0, 0.9455, 0.86421, 0.89992, 0.70799, 0.051035, 0.14959, 0, 0.00253, 0.00253, 0.00253
+ 85, 0.081527, 0.17278, 0, 0.94606, 0.86855, 0.89989, 0.72238, 0.050204, 0.14736, 0, 0.00244, 0.00244, 0.00244
+ 86, 0.08107, 0.16729, 0, 0.95121, 0.86766, 0.90192, 0.72732, 0.049951, 0.14585, 0, 0.00235, 0.00235, 0.00235
+ 87, 0.080949, 0.16978, 0, 0.94822, 0.8644, 0.89882, 0.7229, 0.050133, 0.14727, 0, 0.00226, 0.00226, 0.00226
+ 88, 0.082379, 0.17123, 0, 0.94819, 0.86717, 0.90187, 0.71682, 0.050697, 0.14834, 0, 0.00217, 0.00217, 0.00217
+ 89, 0.079732, 0.16071, 0, 0.9487, 0.86813, 0.90238, 0.72628, 0.05002, 0.14618, 0, 0.00208, 0.00208, 0.00208
+ 90, 0.0803, 0.1615, 0, 0.95029, 0.86793, 0.90243, 0.72982, 0.049789, 0.14528, 0, 0.00199, 0.00199, 0.00199
+ 91, 0.080364, 0.16062, 0, 0.94924, 0.86469, 0.90068, 0.72207, 0.050106, 0.14672, 0, 0.0019, 0.0019, 0.0019
+ 92, 0.081735, 0.16763, 0, 0.94764, 0.87037, 0.90141, 0.72725, 0.049953, 0.14589, 0, 0.00181, 0.00181, 0.00181
+ 93, 0.080632, 0.16383, 0, 0.94882, 0.86704, 0.90079, 0.72667, 0.04985, 0.14562, 0, 0.00172, 0.00172, 0.00172
+ 94, 0.08076, 0.15877, 0, 0.94522, 0.87164, 0.90384, 0.72507, 0.050051, 0.14646, 0, 0.00163, 0.00163, 0.00163
+ 95, 0.081695, 0.16942, 0, 0.9482, 0.86917, 0.90346, 0.72735, 0.049871, 0.146, 0, 0.00154, 0.00154, 0.00154
+ 96, 0.080296, 0.16362, 0, 0.94895, 0.86483, 0.90046, 0.72406, 0.050098, 0.14637, 0, 0.00145, 0.00145, 0.00145
+ 97, 0.080617, 0.16571, 0, 0.95251, 0.86722, 0.90244, 0.73023, 0.049788, 0.14505, 0, 0.00136, 0.00136, 0.00136
+ 98, 0.080079, 0.16336, 0, 0.94819, 0.87085, 0.90324, 0.73317, 0.049553, 0.14422, 0, 0.00127, 0.00127, 0.00127
+ 99, 0.079671, 0.16214, 0, 0.95115, 0.86978, 0.90437, 0.73298, 0.049534, 0.14428, 0, 0.00118, 0.00118, 0.00118
diff --git a/TextDetection/runs/wordDetection/results.png b/TextDetection/runs/wordDetection/results.png
new file mode 100644
index 0000000000000000000000000000000000000000..0215dab509da1ce5a99967d923776fac45b60213
Binary files /dev/null and b/TextDetection/runs/wordDetection/results.png differ
diff --git a/TextDetection/runs/wordDetection/weights/best.pt b/TextDetection/runs/wordDetection/weights/best.pt
new file mode 100644
index 0000000000000000000000000000000000000000..88e81242d8ff05a70a58cf903ea95054172591d5
--- /dev/null
+++ b/TextDetection/runs/wordDetection/weights/best.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9039ccbb2733b3a557ec0e28b3fbc53d3a0489812c102cf9bc082ed2b6266868
+size 14385013
diff --git a/TextDetection/runs/wordDetection/weights/last.pt b/TextDetection/runs/wordDetection/weights/last.pt
new file mode 100644
index 0000000000000000000000000000000000000000..2e14f95818fb20bd2f55941989cba7d99177086b
--- /dev/null
+++ b/TextDetection/runs/wordDetection/weights/last.pt
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ef353e8445999f03f471e6667e4559b168213f4b24379b83e007f876754d9d9
+size 14385013
diff --git a/TextDetection/segment/predict.py b/TextDetection/segment/predict.py
new file mode 100644
index 0000000000000000000000000000000000000000..6a4d5eff3fc119d2e8b4a73d070863227a323739
--- /dev/null
+++ b/TextDetection/segment/predict.py
@@ -0,0 +1,284 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Run YOLOv5 segmentation inference on images, videos, directories, streams, etc.
+
+Usage - sources:
+ $ python segment/predict.py --weights yolov5s-seg.pt --source 0 # webcam
+ img.jpg # image
+ vid.mp4 # video
+ screen # screenshot
+ path/ # directory
+ list.txt # list of images
+ list.streams # list of streams
+ 'path/*.jpg' # glob
+ 'https://youtu.be/Zgi9g1ksQHc' # YouTube
+ 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream
+
+Usage - formats:
+ $ python segment/predict.py --weights yolov5s-seg.pt # PyTorch
+ yolov5s-seg.torchscript # TorchScript
+ yolov5s-seg.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s-seg_openvino_model # OpenVINO
+ yolov5s-seg.engine # TensorRT
+ yolov5s-seg.mlmodel # CoreML (macOS-only)
+ yolov5s-seg_saved_model # TensorFlow SavedModel
+ yolov5s-seg.pb # TensorFlow GraphDef
+ yolov5s-seg.tflite # TensorFlow Lite
+ yolov5s-seg_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s-seg_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import os
+import platform
+import sys
+from pathlib import Path
+
+import torch
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[1] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.common import DetectMultiBackend
+from utils.dataloaders import IMG_FORMATS, VID_FORMATS, LoadImages, LoadScreenshots, LoadStreams
+from utils.general import (LOGGER, Profile, check_file, check_img_size, check_imshow, check_requirements, colorstr, cv2,
+ increment_path, non_max_suppression, print_args, scale_boxes, scale_segments,
+ strip_optimizer)
+from utils.plots import Annotator, colors, save_one_box
+from utils.segment.general import masks2segments, process_mask, process_mask_native
+from utils.torch_utils import select_device, smart_inference_mode
+
+
+@smart_inference_mode()
+def run(
+ weights=ROOT / 'yolov5s-seg.pt', # model.pt path(s)
+ source=ROOT / 'data/images', # file/dir/URL/glob/screen/0(webcam)
+ data=ROOT / 'data/coco128.yaml', # dataset.yaml path
+ imgsz=(640, 640), # inference size (height, width)
+ conf_thres=0.25, # confidence threshold
+ iou_thres=0.45, # NMS IOU threshold
+ max_det=1000, # maximum detections per image
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ view_img=False, # show results
+ save_txt=False, # save results to *.txt
+ save_conf=False, # save confidences in --save-txt labels
+ save_crop=False, # save cropped prediction boxes
+ nosave=False, # do not save images/videos
+ classes=None, # filter by class: --class 0, or --class 0 2 3
+ agnostic_nms=False, # class-agnostic NMS
+ augment=False, # augmented inference
+ visualize=False, # visualize features
+ update=False, # update all models
+ project=ROOT / 'runs/predict-seg', # save results to project/name
+ name='exp', # save results to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ line_thickness=3, # bounding box thickness (pixels)
+ hide_labels=False, # hide labels
+ hide_conf=False, # hide confidences
+ half=False, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ vid_stride=1, # video frame-rate stride
+ retina_masks=False,
+):
+ source = str(source)
+ save_img = not nosave and not source.endswith('.txt') # save inference images
+ is_file = Path(source).suffix[1:] in (IMG_FORMATS + VID_FORMATS)
+ is_url = source.lower().startswith(('rtsp://', 'rtmp://', 'http://', 'https://'))
+ webcam = source.isnumeric() or source.endswith('.streams') or (is_url and not is_file)
+ screenshot = source.lower().startswith('screen')
+ if is_url and is_file:
+ source = check_file(source) # download
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ (save_dir / 'labels' if save_txt else save_dir).mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ device = select_device(device)
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
+ stride, names, pt = model.stride, model.names, model.pt
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+
+ # Dataloader
+ bs = 1 # batch_size
+ if webcam:
+ view_img = check_imshow(warn=True)
+ dataset = LoadStreams(source, img_size=imgsz, stride=stride, auto=pt, vid_stride=vid_stride)
+ bs = len(dataset)
+ elif screenshot:
+ dataset = LoadScreenshots(source, img_size=imgsz, stride=stride, auto=pt)
+ else:
+ dataset = LoadImages(source, img_size=imgsz, stride=stride, auto=pt, vid_stride=vid_stride)
+ vid_path, vid_writer = [None] * bs, [None] * bs
+
+ # Run inference
+ model.warmup(imgsz=(1 if pt else bs, 3, *imgsz)) # warmup
+ seen, windows, dt = 0, [], (Profile(), Profile(), Profile())
+ for path, im, im0s, vid_cap, s in dataset:
+ with dt[0]:
+ im = torch.from_numpy(im).to(model.device)
+ im = im.half() if model.fp16 else im.float() # uint8 to fp16/32
+ im /= 255 # 0 - 255 to 0.0 - 1.0
+ if len(im.shape) == 3:
+ im = im[None] # expand for batch dim
+
+ # Inference
+ with dt[1]:
+ visualize = increment_path(save_dir / Path(path).stem, mkdir=True) if visualize else False
+ pred, proto = model(im, augment=augment, visualize=visualize)[:2]
+
+ # NMS
+ with dt[2]:
+ pred = non_max_suppression(pred, conf_thres, iou_thres, classes, agnostic_nms, max_det=max_det, nm=32)
+
+ # Second-stage classifier (optional)
+ # pred = utils.general.apply_classifier(pred, classifier_model, im, im0s)
+
+ # Process predictions
+ for i, det in enumerate(pred): # per image
+ seen += 1
+ if webcam: # batch_size >= 1
+ p, im0, frame = path[i], im0s[i].copy(), dataset.count
+ s += f'{i}: '
+ else:
+ p, im0, frame = path, im0s.copy(), getattr(dataset, 'frame', 0)
+
+ p = Path(p) # to Path
+ save_path = str(save_dir / p.name) # im.jpg
+ txt_path = str(save_dir / 'labels' / p.stem) + ('' if dataset.mode == 'image' else f'_{frame}') # im.txt
+ s += '%gx%g ' % im.shape[2:] # print string
+ imc = im0.copy() if save_crop else im0 # for save_crop
+ annotator = Annotator(im0, line_width=line_thickness, example=str(names))
+ if len(det):
+ if retina_masks:
+ # scale bbox first the crop masks
+ det[:, :4] = scale_boxes(im.shape[2:], det[:, :4], im0.shape).round() # rescale boxes to im0 size
+ masks = process_mask_native(proto[i], det[:, 6:], det[:, :4], im0.shape[:2]) # HWC
+ else:
+ masks = process_mask(proto[i], det[:, 6:], det[:, :4], im.shape[2:], upsample=True) # HWC
+ det[:, :4] = scale_boxes(im.shape[2:], det[:, :4], im0.shape).round() # rescale boxes to im0 size
+
+ # Segments
+ if save_txt:
+ segments = [
+ scale_segments(im0.shape if retina_masks else im.shape[2:], x, im0.shape, normalize=True)
+ for x in reversed(masks2segments(masks))]
+
+ # Print results
+ for c in det[:, 5].unique():
+ n = (det[:, 5] == c).sum() # detections per class
+ s += f"{n} {names[int(c)]}{'s' * (n > 1)}, " # add to string
+
+ # Mask plotting
+ annotator.masks(
+ masks,
+ colors=[colors(x, True) for x in det[:, 5]],
+ im_gpu=torch.as_tensor(im0, dtype=torch.float16).to(device).permute(2, 0, 1).flip(0).contiguous() /
+ 255 if retina_masks else im[i])
+
+ # Write results
+ for j, (*xyxy, conf, cls) in enumerate(reversed(det[:, :6])):
+ if save_txt: # Write to file
+ seg = segments[j].reshape(-1) # (n,2) to (n*2)
+ line = (cls, *seg, conf) if save_conf else (cls, *seg) # label format
+ with open(f'{txt_path}.txt', 'a') as f:
+ f.write(('%g ' * len(line)).rstrip() % line + '\n')
+
+ if save_img or save_crop or view_img: # Add bbox to image
+ c = int(cls) # integer class
+ label = None if hide_labels else (names[c] if hide_conf else f'{names[c]} {conf:.2f}')
+ annotator.box_label(xyxy, label, color=colors(c, True))
+ # annotator.draw.polygon(segments[j], outline=colors(c, True), width=3)
+ if save_crop:
+ save_one_box(xyxy, imc, file=save_dir / 'crops' / names[c] / f'{p.stem}.jpg', BGR=True)
+
+ # Stream results
+ im0 = annotator.result()
+ if view_img:
+ if platform.system() == 'Linux' and p not in windows:
+ windows.append(p)
+ cv2.namedWindow(str(p), cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO) # allow window resize (Linux)
+ cv2.resizeWindow(str(p), im0.shape[1], im0.shape[0])
+ cv2.imshow(str(p), im0)
+ if cv2.waitKey(1) == ord('q'): # 1 millisecond
+ exit()
+
+ # Save results (image with detections)
+ if save_img:
+ if dataset.mode == 'image':
+ cv2.imwrite(save_path, im0)
+ else: # 'video' or 'stream'
+ if vid_path[i] != save_path: # new video
+ vid_path[i] = save_path
+ if isinstance(vid_writer[i], cv2.VideoWriter):
+ vid_writer[i].release() # release previous video writer
+ if vid_cap: # video
+ fps = vid_cap.get(cv2.CAP_PROP_FPS)
+ w = int(vid_cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+ h = int(vid_cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+ else: # stream
+ fps, w, h = 30, im0.shape[1], im0.shape[0]
+ save_path = str(Path(save_path).with_suffix('.mp4')) # force *.mp4 suffix on results videos
+ vid_writer[i] = cv2.VideoWriter(save_path, cv2.VideoWriter_fourcc(*'mp4v'), fps, (w, h))
+ vid_writer[i].write(im0)
+
+ # Print time (inference-only)
+ LOGGER.info(f"{s}{'' if len(det) else '(no detections), '}{dt[1].dt * 1E3:.1f}ms")
+
+ # Print results
+ t = tuple(x.t / seen * 1E3 for x in dt) # speeds per image
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {(1, 3, *imgsz)}' % t)
+ if save_txt or save_img:
+ s = f"\n{len(list(save_dir.glob('labels/*.txt')))} labels saved to {save_dir / 'labels'}" if save_txt else ''
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}{s}")
+ if update:
+ strip_optimizer(weights[0]) # update model (to fix SourceChangeWarning)
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s-seg.pt', help='model path(s)')
+ parser.add_argument('--source', type=str, default=ROOT / 'data/images', help='file/dir/URL/glob/screen/0(webcam)')
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='(optional) dataset.yaml path')
+ parser.add_argument('--imgsz', '--img', '--img-size', nargs='+', type=int, default=[640], help='inference size h,w')
+ parser.add_argument('--conf-thres', type=float, default=0.25, help='confidence threshold')
+ parser.add_argument('--iou-thres', type=float, default=0.45, help='NMS IoU threshold')
+ parser.add_argument('--max-det', type=int, default=1000, help='maximum detections per image')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--view-img', action='store_true', help='show results')
+ parser.add_argument('--save-txt', action='store_true', help='save results to *.txt')
+ parser.add_argument('--save-conf', action='store_true', help='save confidences in --save-txt labels')
+ parser.add_argument('--save-crop', action='store_true', help='save cropped prediction boxes')
+ parser.add_argument('--nosave', action='store_true', help='do not save images/videos')
+ parser.add_argument('--classes', nargs='+', type=int, help='filter by class: --classes 0, or --classes 0 2 3')
+ parser.add_argument('--agnostic-nms', action='store_true', help='class-agnostic NMS')
+ parser.add_argument('--augment', action='store_true', help='augmented inference')
+ parser.add_argument('--visualize', action='store_true', help='visualize features')
+ parser.add_argument('--update', action='store_true', help='update all models')
+ parser.add_argument('--project', default=ROOT / 'runs/predict-seg', help='save results to project/name')
+ parser.add_argument('--name', default='exp', help='save results to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--line-thickness', default=3, type=int, help='bounding box thickness (pixels)')
+ parser.add_argument('--hide-labels', default=False, action='store_true', help='hide labels')
+ parser.add_argument('--hide-conf', default=False, action='store_true', help='hide confidences')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ parser.add_argument('--vid-stride', type=int, default=1, help='video frame-rate stride')
+ parser.add_argument('--retina-masks', action='store_true', help='whether to plot masks in native resolution')
+ opt = parser.parse_args()
+ opt.imgsz *= 2 if len(opt.imgsz) == 1 else 1 # expand
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(ROOT / 'requirements.txt', exclude=('tensorboard', 'thop'))
+ run(**vars(opt))
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/segment/train.py b/TextDetection/segment/train.py
new file mode 100644
index 0000000000000000000000000000000000000000..2ae09c1cbf66843ff885348999f930a5d80f74ac
--- /dev/null
+++ b/TextDetection/segment/train.py
@@ -0,0 +1,666 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Train a YOLOv5 segment model on a segment dataset
+Models and datasets download automatically from the latest YOLOv5 release.
+
+Usage - Single-GPU training:
+ $ python segment/train.py --data coco128-seg.yaml --weights yolov5s-seg.pt --img 640 # from pretrained (recommended)
+ $ python segment/train.py --data coco128-seg.yaml --weights '' --cfg yolov5s-seg.yaml --img 640 # from scratch
+
+Usage - Multi-GPU DDP training:
+ $ python -m torch.distributed.run --nproc_per_node 4 --master_port 1 segment/train.py --data coco128-seg.yaml --weights yolov5s-seg.pt --img 640 --device 0,1,2,3
+
+Models: https://github.com/ultralytics/yolov5/tree/master/models
+Datasets: https://github.com/ultralytics/yolov5/tree/master/data
+Tutorial: https://docs.ultralytics.com/yolov5/tutorials/train_custom_data
+"""
+
+import argparse
+import math
+import os
+import random
+import subprocess
+import sys
+import time
+from copy import deepcopy
+from datetime import datetime
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+import yaml
+from torch.optim import lr_scheduler
+from tqdm import tqdm
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[1] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+import segment.val as validate # for end-of-epoch mAP
+from models.experimental import attempt_load
+from models.yolo import SegmentationModel
+from utils.autoanchor import check_anchors
+from utils.autobatch import check_train_batch_size
+from utils.callbacks import Callbacks
+from utils.downloads import attempt_download, is_url
+from utils.general import (LOGGER, TQDM_BAR_FORMAT, check_amp, check_dataset, check_file, check_git_info,
+ check_git_status, check_img_size, check_requirements, check_suffix, check_yaml, colorstr,
+ get_latest_run, increment_path, init_seeds, intersect_dicts, labels_to_class_weights,
+ labels_to_image_weights, one_cycle, print_args, print_mutation, strip_optimizer, yaml_save)
+from utils.loggers import GenericLogger
+from utils.plots import plot_evolve, plot_labels
+from utils.segment.dataloaders import create_dataloader
+from utils.segment.loss import ComputeLoss
+from utils.segment.metrics import KEYS, fitness
+from utils.segment.plots import plot_images_and_masks, plot_results_with_masks
+from utils.torch_utils import (EarlyStopping, ModelEMA, de_parallel, select_device, smart_DDP, smart_optimizer,
+ smart_resume, torch_distributed_zero_first)
+
+LOCAL_RANK = int(os.getenv('LOCAL_RANK', -1)) # https://pytorch.org/docs/stable/elastic/run.html
+RANK = int(os.getenv('RANK', -1))
+WORLD_SIZE = int(os.getenv('WORLD_SIZE', 1))
+GIT_INFO = check_git_info()
+
+
+def train(hyp, opt, device, callbacks): # hyp is path/to/hyp.yaml or hyp dictionary
+ save_dir, epochs, batch_size, weights, single_cls, evolve, data, cfg, resume, noval, nosave, workers, freeze, mask_ratio = \
+ Path(opt.save_dir), opt.epochs, opt.batch_size, opt.weights, opt.single_cls, opt.evolve, opt.data, opt.cfg, \
+ opt.resume, opt.noval, opt.nosave, opt.workers, opt.freeze, opt.mask_ratio
+ # callbacks.run('on_pretrain_routine_start')
+
+ # Directories
+ w = save_dir / 'weights' # weights dir
+ (w.parent if evolve else w).mkdir(parents=True, exist_ok=True) # make dir
+ last, best = w / 'last.pt', w / 'best.pt'
+
+ # Hyperparameters
+ if isinstance(hyp, str):
+ with open(hyp, errors='ignore') as f:
+ hyp = yaml.safe_load(f) # load hyps dict
+ LOGGER.info(colorstr('hyperparameters: ') + ', '.join(f'{k}={v}' for k, v in hyp.items()))
+ opt.hyp = hyp.copy() # for saving hyps to checkpoints
+
+ # Save run settings
+ if not evolve:
+ yaml_save(save_dir / 'hyp.yaml', hyp)
+ yaml_save(save_dir / 'opt.yaml', vars(opt))
+
+ # Loggers
+ data_dict = None
+ if RANK in {-1, 0}:
+ logger = GenericLogger(opt=opt, console_logger=LOGGER)
+
+ # Config
+ plots = not evolve and not opt.noplots # create plots
+ overlap = not opt.no_overlap
+ cuda = device.type != 'cpu'
+ init_seeds(opt.seed + 1 + RANK, deterministic=True)
+ with torch_distributed_zero_first(LOCAL_RANK):
+ data_dict = data_dict or check_dataset(data) # check if None
+ train_path, val_path = data_dict['train'], data_dict['val']
+ nc = 1 if single_cls else int(data_dict['nc']) # number of classes
+ names = {0: 'item'} if single_cls and len(data_dict['names']) != 1 else data_dict['names'] # class names
+ is_coco = isinstance(val_path, str) and val_path.endswith('coco/val2017.txt') # COCO dataset
+
+ # Model
+ check_suffix(weights, '.pt') # check weights
+ pretrained = weights.endswith('.pt')
+ if pretrained:
+ with torch_distributed_zero_first(LOCAL_RANK):
+ weights = attempt_download(weights) # download if not found locally
+ ckpt = torch.load(weights, map_location='cpu') # load checkpoint to CPU to avoid CUDA memory leak
+ model = SegmentationModel(cfg or ckpt['model'].yaml, ch=3, nc=nc, anchors=hyp.get('anchors')).to(device)
+ exclude = ['anchor'] if (cfg or hyp.get('anchors')) and not resume else [] # exclude keys
+ csd = ckpt['model'].float().state_dict() # checkpoint state_dict as FP32
+ csd = intersect_dicts(csd, model.state_dict(), exclude=exclude) # intersect
+ model.load_state_dict(csd, strict=False) # load
+ LOGGER.info(f'Transferred {len(csd)}/{len(model.state_dict())} items from {weights}') # report
+ else:
+ model = SegmentationModel(cfg, ch=3, nc=nc, anchors=hyp.get('anchors')).to(device) # create
+ amp = check_amp(model) # check AMP
+
+ # Freeze
+ freeze = [f'model.{x}.' for x in (freeze if len(freeze) > 1 else range(freeze[0]))] # layers to freeze
+ for k, v in model.named_parameters():
+ v.requires_grad = True # train all layers
+ # v.register_hook(lambda x: torch.nan_to_num(x)) # NaN to 0 (commented for erratic training results)
+ if any(x in k for x in freeze):
+ LOGGER.info(f'freezing {k}')
+ v.requires_grad = False
+
+ # Image size
+ gs = max(int(model.stride.max()), 32) # grid size (max stride)
+ imgsz = check_img_size(opt.imgsz, gs, floor=gs * 2) # verify imgsz is gs-multiple
+
+ # Batch size
+ if RANK == -1 and batch_size == -1: # single-GPU only, estimate best batch size
+ batch_size = check_train_batch_size(model, imgsz, amp)
+ logger.update_params({'batch_size': batch_size})
+ # loggers.on_params_update({"batch_size": batch_size})
+
+ # Optimizer
+ nbs = 64 # nominal batch size
+ accumulate = max(round(nbs / batch_size), 1) # accumulate loss before optimizing
+ hyp['weight_decay'] *= batch_size * accumulate / nbs # scale weight_decay
+ optimizer = smart_optimizer(model, opt.optimizer, hyp['lr0'], hyp['momentum'], hyp['weight_decay'])
+
+ # Scheduler
+ if opt.cos_lr:
+ lf = one_cycle(1, hyp['lrf'], epochs) # cosine 1->hyp['lrf']
+ else:
+ lf = lambda x: (1 - x / epochs) * (1.0 - hyp['lrf']) + hyp['lrf'] # linear
+ scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lf) # plot_lr_scheduler(optimizer, scheduler, epochs)
+
+ # EMA
+ ema = ModelEMA(model) if RANK in {-1, 0} else None
+
+ # Resume
+ best_fitness, start_epoch = 0.0, 0
+ if pretrained:
+ if resume:
+ best_fitness, start_epoch, epochs = smart_resume(ckpt, optimizer, ema, weights, epochs, resume)
+ del ckpt, csd
+
+ # DP mode
+ if cuda and RANK == -1 and torch.cuda.device_count() > 1:
+ LOGGER.warning(
+ 'WARNING ⚠️ DP not recommended, use torch.distributed.run for best DDP Multi-GPU results.\n'
+ 'See Multi-GPU Tutorial at https://docs.ultralytics.com/yolov5/tutorials/multi_gpu_training to get started.'
+ )
+ model = torch.nn.DataParallel(model)
+
+ # SyncBatchNorm
+ if opt.sync_bn and cuda and RANK != -1:
+ model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model).to(device)
+ LOGGER.info('Using SyncBatchNorm()')
+
+ # Trainloader
+ train_loader, dataset = create_dataloader(
+ train_path,
+ imgsz,
+ batch_size // WORLD_SIZE,
+ gs,
+ single_cls,
+ hyp=hyp,
+ augment=True,
+ cache=None if opt.cache == 'val' else opt.cache,
+ rect=opt.rect,
+ rank=LOCAL_RANK,
+ workers=workers,
+ image_weights=opt.image_weights,
+ quad=opt.quad,
+ prefix=colorstr('train: '),
+ shuffle=True,
+ mask_downsample_ratio=mask_ratio,
+ overlap_mask=overlap,
+ )
+ labels = np.concatenate(dataset.labels, 0)
+ mlc = int(labels[:, 0].max()) # max label class
+ assert mlc < nc, f'Label class {mlc} exceeds nc={nc} in {data}. Possible class labels are 0-{nc - 1}'
+
+ # Process 0
+ if RANK in {-1, 0}:
+ val_loader = create_dataloader(val_path,
+ imgsz,
+ batch_size // WORLD_SIZE * 2,
+ gs,
+ single_cls,
+ hyp=hyp,
+ cache=None if noval else opt.cache,
+ rect=True,
+ rank=-1,
+ workers=workers * 2,
+ pad=0.5,
+ mask_downsample_ratio=mask_ratio,
+ overlap_mask=overlap,
+ prefix=colorstr('val: '))[0]
+
+ if not resume:
+ if not opt.noautoanchor:
+ check_anchors(dataset, model=model, thr=hyp['anchor_t'], imgsz=imgsz) # run AutoAnchor
+ model.half().float() # pre-reduce anchor precision
+
+ if plots:
+ plot_labels(labels, names, save_dir)
+ # callbacks.run('on_pretrain_routine_end', labels, names)
+
+ # DDP mode
+ if cuda and RANK != -1:
+ model = smart_DDP(model)
+
+ # Model attributes
+ nl = de_parallel(model).model[-1].nl # number of detection layers (to scale hyps)
+ hyp['box'] *= 3 / nl # scale to layers
+ hyp['cls'] *= nc / 80 * 3 / nl # scale to classes and layers
+ hyp['obj'] *= (imgsz / 640) ** 2 * 3 / nl # scale to image size and layers
+ hyp['label_smoothing'] = opt.label_smoothing
+ model.nc = nc # attach number of classes to model
+ model.hyp = hyp # attach hyperparameters to model
+ model.class_weights = labels_to_class_weights(dataset.labels, nc).to(device) * nc # attach class weights
+ model.names = names
+
+ # Start training
+ t0 = time.time()
+ nb = len(train_loader) # number of batches
+ nw = max(round(hyp['warmup_epochs'] * nb), 100) # number of warmup iterations, max(3 epochs, 100 iterations)
+ # nw = min(nw, (epochs - start_epoch) / 2 * nb) # limit warmup to < 1/2 of training
+ last_opt_step = -1
+ maps = np.zeros(nc) # mAP per class
+ results = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) # P, R, mAP@.5, mAP@.5-.95, val_loss(box, obj, cls)
+ scheduler.last_epoch = start_epoch - 1 # do not move
+ scaler = torch.cuda.amp.GradScaler(enabled=amp)
+ stopper, stop = EarlyStopping(patience=opt.patience), False
+ compute_loss = ComputeLoss(model, overlap=overlap) # init loss class
+ # callbacks.run('on_train_start')
+ LOGGER.info(f'Image sizes {imgsz} train, {imgsz} val\n'
+ f'Using {train_loader.num_workers * WORLD_SIZE} dataloader workers\n'
+ f"Logging results to {colorstr('bold', save_dir)}\n"
+ f'Starting training for {epochs} epochs...')
+ for epoch in range(start_epoch, epochs): # epoch ------------------------------------------------------------------
+ # callbacks.run('on_train_epoch_start')
+ model.train()
+
+ # Update image weights (optional, single-GPU only)
+ if opt.image_weights:
+ cw = model.class_weights.cpu().numpy() * (1 - maps) ** 2 / nc # class weights
+ iw = labels_to_image_weights(dataset.labels, nc=nc, class_weights=cw) # image weights
+ dataset.indices = random.choices(range(dataset.n), weights=iw, k=dataset.n) # rand weighted idx
+
+ # Update mosaic border (optional)
+ # b = int(random.uniform(0.25 * imgsz, 0.75 * imgsz + gs) // gs * gs)
+ # dataset.mosaic_border = [b - imgsz, -b] # height, width borders
+
+ mloss = torch.zeros(4, device=device) # mean losses
+ if RANK != -1:
+ train_loader.sampler.set_epoch(epoch)
+ pbar = enumerate(train_loader)
+ LOGGER.info(('\n' + '%11s' * 8) %
+ ('Epoch', 'GPU_mem', 'box_loss', 'seg_loss', 'obj_loss', 'cls_loss', 'Instances', 'Size'))
+ if RANK in {-1, 0}:
+ pbar = tqdm(pbar, total=nb, bar_format=TQDM_BAR_FORMAT) # progress bar
+ optimizer.zero_grad()
+ for i, (imgs, targets, paths, _, masks) in pbar: # batch ------------------------------------------------------
+ # callbacks.run('on_train_batch_start')
+ ni = i + nb * epoch # number integrated batches (since train start)
+ imgs = imgs.to(device, non_blocking=True).float() / 255 # uint8 to float32, 0-255 to 0.0-1.0
+
+ # Warmup
+ if ni <= nw:
+ xi = [0, nw] # x interp
+ # compute_loss.gr = np.interp(ni, xi, [0.0, 1.0]) # iou loss ratio (obj_loss = 1.0 or iou)
+ accumulate = max(1, np.interp(ni, xi, [1, nbs / batch_size]).round())
+ for j, x in enumerate(optimizer.param_groups):
+ # bias lr falls from 0.1 to lr0, all other lrs rise from 0.0 to lr0
+ x['lr'] = np.interp(ni, xi, [hyp['warmup_bias_lr'] if j == 0 else 0.0, x['initial_lr'] * lf(epoch)])
+ if 'momentum' in x:
+ x['momentum'] = np.interp(ni, xi, [hyp['warmup_momentum'], hyp['momentum']])
+
+ # Multi-scale
+ if opt.multi_scale:
+ sz = random.randrange(int(imgsz * 0.5), int(imgsz * 1.5) + gs) // gs * gs # size
+ sf = sz / max(imgs.shape[2:]) # scale factor
+ if sf != 1:
+ ns = [math.ceil(x * sf / gs) * gs for x in imgs.shape[2:]] # new shape (stretched to gs-multiple)
+ imgs = nn.functional.interpolate(imgs, size=ns, mode='bilinear', align_corners=False)
+
+ # Forward
+ with torch.cuda.amp.autocast(amp):
+ pred = model(imgs) # forward
+ loss, loss_items = compute_loss(pred, targets.to(device), masks=masks.to(device).float())
+ if RANK != -1:
+ loss *= WORLD_SIZE # gradient averaged between devices in DDP mode
+ if opt.quad:
+ loss *= 4.
+
+ # Backward
+ scaler.scale(loss).backward()
+
+ # Optimize - https://pytorch.org/docs/master/notes/amp_examples.html
+ if ni - last_opt_step >= accumulate:
+ scaler.unscale_(optimizer) # unscale gradients
+ torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=10.0) # clip gradients
+ scaler.step(optimizer) # optimizer.step
+ scaler.update()
+ optimizer.zero_grad()
+ if ema:
+ ema.update(model)
+ last_opt_step = ni
+
+ # Log
+ if RANK in {-1, 0}:
+ mloss = (mloss * i + loss_items) / (i + 1) # update mean losses
+ mem = f'{torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0:.3g}G' # (GB)
+ pbar.set_description(('%11s' * 2 + '%11.4g' * 6) %
+ (f'{epoch}/{epochs - 1}', mem, *mloss, targets.shape[0], imgs.shape[-1]))
+ # callbacks.run('on_train_batch_end', model, ni, imgs, targets, paths)
+ # if callbacks.stop_training:
+ # return
+
+ # Mosaic plots
+ if plots:
+ if ni < 3:
+ plot_images_and_masks(imgs, targets, masks, paths, save_dir / f'train_batch{ni}.jpg')
+ if ni == 10:
+ files = sorted(save_dir.glob('train*.jpg'))
+ logger.log_images(files, 'Mosaics', epoch)
+ # end batch ------------------------------------------------------------------------------------------------
+
+ # Scheduler
+ lr = [x['lr'] for x in optimizer.param_groups] # for loggers
+ scheduler.step()
+
+ if RANK in {-1, 0}:
+ # mAP
+ # callbacks.run('on_train_epoch_end', epoch=epoch)
+ ema.update_attr(model, include=['yaml', 'nc', 'hyp', 'names', 'stride', 'class_weights'])
+ final_epoch = (epoch + 1 == epochs) or stopper.possible_stop
+ if not noval or final_epoch: # Calculate mAP
+ results, maps, _ = validate.run(data_dict,
+ batch_size=batch_size // WORLD_SIZE * 2,
+ imgsz=imgsz,
+ half=amp,
+ model=ema.ema,
+ single_cls=single_cls,
+ dataloader=val_loader,
+ save_dir=save_dir,
+ plots=False,
+ callbacks=callbacks,
+ compute_loss=compute_loss,
+ mask_downsample_ratio=mask_ratio,
+ overlap=overlap)
+
+ # Update best mAP
+ fi = fitness(np.array(results).reshape(1, -1)) # weighted combination of [P, R, mAP@.5, mAP@.5-.95]
+ stop = stopper(epoch=epoch, fitness=fi) # early stop check
+ if fi > best_fitness:
+ best_fitness = fi
+ log_vals = list(mloss) + list(results) + lr
+ # callbacks.run('on_fit_epoch_end', log_vals, epoch, best_fitness, fi)
+ # Log val metrics and media
+ metrics_dict = dict(zip(KEYS, log_vals))
+ logger.log_metrics(metrics_dict, epoch)
+
+ # Save model
+ if (not nosave) or (final_epoch and not evolve): # if save
+ ckpt = {
+ 'epoch': epoch,
+ 'best_fitness': best_fitness,
+ 'model': deepcopy(de_parallel(model)).half(),
+ 'ema': deepcopy(ema.ema).half(),
+ 'updates': ema.updates,
+ 'optimizer': optimizer.state_dict(),
+ 'opt': vars(opt),
+ 'git': GIT_INFO, # {remote, branch, commit} if a git repo
+ 'date': datetime.now().isoformat()}
+
+ # Save last, best and delete
+ torch.save(ckpt, last)
+ if best_fitness == fi:
+ torch.save(ckpt, best)
+ if opt.save_period > 0 and epoch % opt.save_period == 0:
+ torch.save(ckpt, w / f'epoch{epoch}.pt')
+ logger.log_model(w / f'epoch{epoch}.pt')
+ del ckpt
+ # callbacks.run('on_model_save', last, epoch, final_epoch, best_fitness, fi)
+
+ # EarlyStopping
+ if RANK != -1: # if DDP training
+ broadcast_list = [stop if RANK == 0 else None]
+ dist.broadcast_object_list(broadcast_list, 0) # broadcast 'stop' to all ranks
+ if RANK != 0:
+ stop = broadcast_list[0]
+ if stop:
+ break # must break all DDP ranks
+
+ # end epoch ----------------------------------------------------------------------------------------------------
+ # end training -----------------------------------------------------------------------------------------------------
+ if RANK in {-1, 0}:
+ LOGGER.info(f'\n{epoch - start_epoch + 1} epochs completed in {(time.time() - t0) / 3600:.3f} hours.')
+ for f in last, best:
+ if f.exists():
+ strip_optimizer(f) # strip optimizers
+ if f is best:
+ LOGGER.info(f'\nValidating {f}...')
+ results, _, _ = validate.run(
+ data_dict,
+ batch_size=batch_size // WORLD_SIZE * 2,
+ imgsz=imgsz,
+ model=attempt_load(f, device).half(),
+ iou_thres=0.65 if is_coco else 0.60, # best pycocotools at iou 0.65
+ single_cls=single_cls,
+ dataloader=val_loader,
+ save_dir=save_dir,
+ save_json=is_coco,
+ verbose=True,
+ plots=plots,
+ callbacks=callbacks,
+ compute_loss=compute_loss,
+ mask_downsample_ratio=mask_ratio,
+ overlap=overlap) # val best model with plots
+ if is_coco:
+ # callbacks.run('on_fit_epoch_end', list(mloss) + list(results) + lr, epoch, best_fitness, fi)
+ metrics_dict = dict(zip(KEYS, list(mloss) + list(results) + lr))
+ logger.log_metrics(metrics_dict, epoch)
+
+ # callbacks.run('on_train_end', last, best, epoch, results)
+ # on train end callback using genericLogger
+ logger.log_metrics(dict(zip(KEYS[4:16], results)), epochs)
+ if not opt.evolve:
+ logger.log_model(best, epoch)
+ if plots:
+ plot_results_with_masks(file=save_dir / 'results.csv') # save results.png
+ files = ['results.png', 'confusion_matrix.png', *(f'{x}_curve.png' for x in ('F1', 'PR', 'P', 'R'))]
+ files = [(save_dir / f) for f in files if (save_dir / f).exists()] # filter
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}")
+ logger.log_images(files, 'Results', epoch + 1)
+ logger.log_images(sorted(save_dir.glob('val*.jpg')), 'Validation', epoch + 1)
+ torch.cuda.empty_cache()
+ return results
+
+
+def parse_opt(known=False):
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--weights', type=str, default=ROOT / 'yolov5s-seg.pt', help='initial weights path')
+ parser.add_argument('--cfg', type=str, default='', help='model.yaml path')
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128-seg.yaml', help='dataset.yaml path')
+ parser.add_argument('--hyp', type=str, default=ROOT / 'data/hyps/hyp.scratch-low.yaml', help='hyperparameters path')
+ parser.add_argument('--epochs', type=int, default=100, help='total training epochs')
+ parser.add_argument('--batch-size', type=int, default=16, help='total batch size for all GPUs, -1 for autobatch')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='train, val image size (pixels)')
+ parser.add_argument('--rect', action='store_true', help='rectangular training')
+ parser.add_argument('--resume', nargs='?', const=True, default=False, help='resume most recent training')
+ parser.add_argument('--nosave', action='store_true', help='only save final checkpoint')
+ parser.add_argument('--noval', action='store_true', help='only validate final epoch')
+ parser.add_argument('--noautoanchor', action='store_true', help='disable AutoAnchor')
+ parser.add_argument('--noplots', action='store_true', help='save no plot files')
+ parser.add_argument('--evolve', type=int, nargs='?', const=300, help='evolve hyperparameters for x generations')
+ parser.add_argument('--bucket', type=str, default='', help='gsutil bucket')
+ parser.add_argument('--cache', type=str, nargs='?', const='ram', help='image --cache ram/disk')
+ parser.add_argument('--image-weights', action='store_true', help='use weighted image selection for training')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--multi-scale', action='store_true', help='vary img-size +/- 50%%')
+ parser.add_argument('--single-cls', action='store_true', help='train multi-class data as single-class')
+ parser.add_argument('--optimizer', type=str, choices=['SGD', 'Adam', 'AdamW'], default='SGD', help='optimizer')
+ parser.add_argument('--sync-bn', action='store_true', help='use SyncBatchNorm, only available in DDP mode')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--project', default=ROOT / 'runs/train-seg', help='save to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--quad', action='store_true', help='quad dataloader')
+ parser.add_argument('--cos-lr', action='store_true', help='cosine LR scheduler')
+ parser.add_argument('--label-smoothing', type=float, default=0.0, help='Label smoothing epsilon')
+ parser.add_argument('--patience', type=int, default=100, help='EarlyStopping patience (epochs without improvement)')
+ parser.add_argument('--freeze', nargs='+', type=int, default=[0], help='Freeze layers: backbone=10, first3=0 1 2')
+ parser.add_argument('--save-period', type=int, default=-1, help='Save checkpoint every x epochs (disabled if < 1)')
+ parser.add_argument('--seed', type=int, default=0, help='Global training seed')
+ parser.add_argument('--local_rank', type=int, default=-1, help='Automatic DDP Multi-GPU argument, do not modify')
+
+ # Instance Segmentation Args
+ parser.add_argument('--mask-ratio', type=int, default=4, help='Downsample the truth masks to saving memory')
+ parser.add_argument('--no-overlap', action='store_true', help='Overlap masks train faster at slightly less mAP')
+
+ return parser.parse_known_args()[0] if known else parser.parse_args()
+
+
+def main(opt, callbacks=Callbacks()):
+ # Checks
+ if RANK in {-1, 0}:
+ print_args(vars(opt))
+ check_git_status()
+ check_requirements(ROOT / 'requirements.txt')
+
+ # Resume
+ if opt.resume and not opt.evolve: # resume from specified or most recent last.pt
+ last = Path(check_file(opt.resume) if isinstance(opt.resume, str) else get_latest_run())
+ opt_yaml = last.parent.parent / 'opt.yaml' # train options yaml
+ opt_data = opt.data # original dataset
+ if opt_yaml.is_file():
+ with open(opt_yaml, errors='ignore') as f:
+ d = yaml.safe_load(f)
+ else:
+ d = torch.load(last, map_location='cpu')['opt']
+ opt = argparse.Namespace(**d) # replace
+ opt.cfg, opt.weights, opt.resume = '', str(last), True # reinstate
+ if is_url(opt_data):
+ opt.data = check_file(opt_data) # avoid HUB resume auth timeout
+ else:
+ opt.data, opt.cfg, opt.hyp, opt.weights, opt.project = \
+ check_file(opt.data), check_yaml(opt.cfg), check_yaml(opt.hyp), str(opt.weights), str(opt.project) # checks
+ assert len(opt.cfg) or len(opt.weights), 'either --cfg or --weights must be specified'
+ if opt.evolve:
+ if opt.project == str(ROOT / 'runs/train-seg'): # if default project name, rename to runs/evolve-seg
+ opt.project = str(ROOT / 'runs/evolve-seg')
+ opt.exist_ok, opt.resume = opt.resume, False # pass resume to exist_ok and disable resume
+ if opt.name == 'cfg':
+ opt.name = Path(opt.cfg).stem # use model.yaml as name
+ opt.save_dir = str(increment_path(Path(opt.project) / opt.name, exist_ok=opt.exist_ok))
+
+ # DDP mode
+ device = select_device(opt.device, batch_size=opt.batch_size)
+ if LOCAL_RANK != -1:
+ msg = 'is not compatible with YOLOv5 Multi-GPU DDP training'
+ assert not opt.image_weights, f'--image-weights {msg}'
+ assert not opt.evolve, f'--evolve {msg}'
+ assert opt.batch_size != -1, f'AutoBatch with --batch-size -1 {msg}, please pass a valid --batch-size'
+ assert opt.batch_size % WORLD_SIZE == 0, f'--batch-size {opt.batch_size} must be multiple of WORLD_SIZE'
+ assert torch.cuda.device_count() > LOCAL_RANK, 'insufficient CUDA devices for DDP command'
+ torch.cuda.set_device(LOCAL_RANK)
+ device = torch.device('cuda', LOCAL_RANK)
+ dist.init_process_group(backend='nccl' if dist.is_nccl_available() else 'gloo')
+
+ # Train
+ if not opt.evolve:
+ train(opt.hyp, opt, device, callbacks)
+
+ # Evolve hyperparameters (optional)
+ else:
+ # Hyperparameter evolution metadata (mutation scale 0-1, lower_limit, upper_limit)
+ meta = {
+ 'lr0': (1, 1e-5, 1e-1), # initial learning rate (SGD=1E-2, Adam=1E-3)
+ 'lrf': (1, 0.01, 1.0), # final OneCycleLR learning rate (lr0 * lrf)
+ 'momentum': (0.3, 0.6, 0.98), # SGD momentum/Adam beta1
+ 'weight_decay': (1, 0.0, 0.001), # optimizer weight decay
+ 'warmup_epochs': (1, 0.0, 5.0), # warmup epochs (fractions ok)
+ 'warmup_momentum': (1, 0.0, 0.95), # warmup initial momentum
+ 'warmup_bias_lr': (1, 0.0, 0.2), # warmup initial bias lr
+ 'box': (1, 0.02, 0.2), # box loss gain
+ 'cls': (1, 0.2, 4.0), # cls loss gain
+ 'cls_pw': (1, 0.5, 2.0), # cls BCELoss positive_weight
+ 'obj': (1, 0.2, 4.0), # obj loss gain (scale with pixels)
+ 'obj_pw': (1, 0.5, 2.0), # obj BCELoss positive_weight
+ 'iou_t': (0, 0.1, 0.7), # IoU training threshold
+ 'anchor_t': (1, 2.0, 8.0), # anchor-multiple threshold
+ 'anchors': (2, 2.0, 10.0), # anchors per output grid (0 to ignore)
+ 'fl_gamma': (0, 0.0, 2.0), # focal loss gamma (efficientDet default gamma=1.5)
+ 'hsv_h': (1, 0.0, 0.1), # image HSV-Hue augmentation (fraction)
+ 'hsv_s': (1, 0.0, 0.9), # image HSV-Saturation augmentation (fraction)
+ 'hsv_v': (1, 0.0, 0.9), # image HSV-Value augmentation (fraction)
+ 'degrees': (1, 0.0, 45.0), # image rotation (+/- deg)
+ 'translate': (1, 0.0, 0.9), # image translation (+/- fraction)
+ 'scale': (1, 0.0, 0.9), # image scale (+/- gain)
+ 'shear': (1, 0.0, 10.0), # image shear (+/- deg)
+ 'perspective': (0, 0.0, 0.001), # image perspective (+/- fraction), range 0-0.001
+ 'flipud': (1, 0.0, 1.0), # image flip up-down (probability)
+ 'fliplr': (0, 0.0, 1.0), # image flip left-right (probability)
+ 'mosaic': (1, 0.0, 1.0), # image mixup (probability)
+ 'mixup': (1, 0.0, 1.0), # image mixup (probability)
+ 'copy_paste': (1, 0.0, 1.0)} # segment copy-paste (probability)
+
+ with open(opt.hyp, errors='ignore') as f:
+ hyp = yaml.safe_load(f) # load hyps dict
+ if 'anchors' not in hyp: # anchors commented in hyp.yaml
+ hyp['anchors'] = 3
+ if opt.noautoanchor:
+ del hyp['anchors'], meta['anchors']
+ opt.noval, opt.nosave, save_dir = True, True, Path(opt.save_dir) # only val/save final epoch
+ # ei = [isinstance(x, (int, float)) for x in hyp.values()] # evolvable indices
+ evolve_yaml, evolve_csv = save_dir / 'hyp_evolve.yaml', save_dir / 'evolve.csv'
+ if opt.bucket:
+ # download evolve.csv if exists
+ subprocess.run([
+ 'gsutil',
+ 'cp',
+ f'gs://{opt.bucket}/evolve.csv',
+ str(evolve_csv), ])
+
+ for _ in range(opt.evolve): # generations to evolve
+ if evolve_csv.exists(): # if evolve.csv exists: select best hyps and mutate
+ # Select parent(s)
+ parent = 'single' # parent selection method: 'single' or 'weighted'
+ x = np.loadtxt(evolve_csv, ndmin=2, delimiter=',', skiprows=1)
+ n = min(5, len(x)) # number of previous results to consider
+ x = x[np.argsort(-fitness(x))][:n] # top n mutations
+ w = fitness(x) - fitness(x).min() + 1E-6 # weights (sum > 0)
+ if parent == 'single' or len(x) == 1:
+ # x = x[random.randint(0, n - 1)] # random selection
+ x = x[random.choices(range(n), weights=w)[0]] # weighted selection
+ elif parent == 'weighted':
+ x = (x * w.reshape(n, 1)).sum(0) / w.sum() # weighted combination
+
+ # Mutate
+ mp, s = 0.8, 0.2 # mutation probability, sigma
+ npr = np.random
+ npr.seed(int(time.time()))
+ g = np.array([meta[k][0] for k in hyp.keys()]) # gains 0-1
+ ng = len(meta)
+ v = np.ones(ng)
+ while all(v == 1): # mutate until a change occurs (prevent duplicates)
+ v = (g * (npr.random(ng) < mp) * npr.randn(ng) * npr.random() * s + 1).clip(0.3, 3.0)
+ for i, k in enumerate(hyp.keys()): # plt.hist(v.ravel(), 300)
+ hyp[k] = float(x[i + 12] * v[i]) # mutate
+
+ # Constrain to limits
+ for k, v in meta.items():
+ hyp[k] = max(hyp[k], v[1]) # lower limit
+ hyp[k] = min(hyp[k], v[2]) # upper limit
+ hyp[k] = round(hyp[k], 5) # significant digits
+
+ # Train mutation
+ results = train(hyp.copy(), opt, device, callbacks)
+ callbacks = Callbacks()
+ # Write mutation results
+ print_mutation(KEYS[4:16], results, hyp.copy(), save_dir, opt.bucket)
+
+ # Plot results
+ plot_evolve(evolve_csv)
+ LOGGER.info(f'Hyperparameter evolution finished {opt.evolve} generations\n'
+ f"Results saved to {colorstr('bold', save_dir)}\n"
+ f'Usage example: $ python train.py --hyp {evolve_yaml}')
+
+
+def run(**kwargs):
+ # Usage: import train; train.run(data='coco128.yaml', imgsz=320, weights='yolov5m.pt')
+ opt = parse_opt(True)
+ for k, v in kwargs.items():
+ setattr(opt, k, v)
+ main(opt)
+ return opt
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/segment/tutorial.ipynb b/TextDetection/segment/tutorial.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..0ece0f60e4d6538bae86c878482161d583841c71
--- /dev/null
+++ b/TextDetection/segment/tutorial.ipynb
@@ -0,0 +1,595 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "t6MPjfT5NrKQ"
+ },
+ "source": [
+ "\n",
+ "\n",
+ " \n",
+ " \n",
+ "\n",
+ "\n",
+ "
\n", + "\n",
+ " \n",
+ " \n",
+ "
\n", + "\n", + "This YOLOv5 🚀 notebook by Ultralytics presents simple train, validate and predict examples to help start your AI adventure.
See GitHub for community support or contact us for professional support.\n", + "\n", + "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7mGmQbAO5pQb"
+ },
+ "source": [
+ "# Setup\n",
+ "\n",
+ "Clone GitHub [repository](https://github.com/ultralytics/yolov5), install [dependencies](https://github.com/ultralytics/yolov5/blob/master/requirements.txt) and check PyTorch and GPU."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wbvMlHd_QwMG",
+ "outputId": "171b23f0-71b9-4cbf-b666-6fa2ecef70c8"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "YOLOv5 🚀 v7.0-2-gc9d47ae Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n"
+ ]
+ },
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Setup complete ✅ (2 CPUs, 12.7 GB RAM, 22.6/78.2 GB disk)\n"
+ ]
+ }
+ ],
+ "source": [
+ "!git clone https://github.com/ultralytics/yolov5 # clone\n",
+ "%cd yolov5\n",
+ "%pip install -qr requirements.txt comet_ml # install\n",
+ "\n",
+ "import torch\n",
+ "import utils\n",
+ "display = utils.notebook_init() # checks"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4JnkELT0cIJg"
+ },
+ "source": [
+ "# 1. Predict\n",
+ "\n",
+ "`segment/predict.py` runs YOLOv5 instance segmentation inference on a variety of sources, downloading models automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases), and saving results to `runs/predict`. Example inference sources are:\n",
+ "\n",
+ "```shell\n",
+ "python segment/predict.py --source 0 # webcam\n",
+ " img.jpg # image \n",
+ " vid.mp4 # video\n",
+ " screen # screenshot\n",
+ " path/ # directory\n",
+ " 'path/*.jpg' # glob\n",
+ " 'https://youtu.be/Zgi9g1ksQHc' # YouTube\n",
+ " 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "zR9ZbuQCH7FX",
+ "outputId": "3f67f1c7-f15e-4fa5-d251-967c3b77eaad"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[34m\u001b[1msegment/predict: \u001b[0mweights=['yolov5s-seg.pt'], source=data/images, data=data/coco128.yaml, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/predict-seg, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False, vid_stride=1, retina_masks=False\n",
+ "YOLOv5 🚀 v7.0-2-gc9d47ae Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n",
+ "\n",
+ "Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s-seg.pt to yolov5s-seg.pt...\n",
+ "100% 14.9M/14.9M [00:01<00:00, 12.0MB/s]\n",
+ "\n",
+ "Fusing layers... \n",
+ "YOLOv5s-seg summary: 224 layers, 7611485 parameters, 0 gradients, 26.4 GFLOPs\n",
+ "image 1/2 /content/yolov5/data/images/bus.jpg: 640x480 4 persons, 1 bus, 18.2ms\n",
+ "image 2/2 /content/yolov5/data/images/zidane.jpg: 384x640 2 persons, 1 tie, 13.4ms\n",
+ "Speed: 0.5ms pre-process, 15.8ms inference, 18.5ms NMS per image at shape (1, 3, 640, 640)\n",
+ "Results saved to \u001b[1mruns/predict-seg/exp\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python segment/predict.py --weights yolov5s-seg.pt --img 640 --conf 0.25 --source data/images\n",
+ "#display.Image(filename='runs/predict-seg/exp/zidane.jpg', width=600)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hkAzDWJ7cWTr"
+ },
+ "source": [
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n", + "
\n",
+ " \n",
+ " \n",
+ "\n", + "\n", + "This YOLOv5 🚀 notebook by Ultralytics presents simple train, validate and predict examples to help start your AI adventure.
See GitHub for community support or contact us for professional support.\n", + "\n", + "
![](\"https://user-images.githubusercontent.com/26833433/199030123-08c72f8d-6871-4116-8ed3-c373642cf28e.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0eq1SMWl6Sfn"
+ },
+ "source": [
+ "# 2. Validate\n",
+ "Validate a model's accuracy on the [COCO](https://cocodataset.org/#home) dataset's `val` or `test` splits. Models are downloaded automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases). To show results by class use the `--verbose` flag."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "WQPtK1QYVaD_",
+ "outputId": "9d751d8c-bee8-4339-cf30-9854ca530449"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Downloading https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels-segments.zip ...\n",
+ "Downloading http://images.cocodataset.org/zips/val2017.zip ...\n",
+ "######################################################################## 100.0%\n",
+ "######################################################################## 100.0%\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Download COCO val\n",
+ "!bash data/scripts/get_coco.sh --val --segments # download (780M - 5000 images)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "X58w8JLpMnjH",
+ "outputId": "a140d67a-02da-479e-9ddb-7d54bf9e407a"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[34m\u001b[1msegment/val: \u001b[0mdata=/content/yolov5/data/coco.yaml, weights=['yolov5s-seg.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.6, max_det=300, task=val, device=, workers=8, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=False, project=runs/val-seg, name=exp, exist_ok=False, half=True, dnn=False\n",
+ "YOLOv5 🚀 v7.0-2-gc9d47ae Python-3.7.15 torch-1.12.1+cu113 CUDA:0 (Tesla T4, 15110MiB)\n",
+ "\n",
+ "Fusing layers... \n",
+ "YOLOv5s-seg summary: 224 layers, 7611485 parameters, 0 gradients, 26.4 GFLOPs\n",
+ "\u001b[34m\u001b[1mval: \u001b[0mScanning /content/datasets/coco/val2017... 4952 images, 48 backgrounds, 0 corrupt: 100% 5000/5000 [00:03<00:00, 1361.31it/s]\n",
+ "\u001b[34m\u001b[1mval: \u001b[0mNew cache created: /content/datasets/coco/val2017.cache\n",
+ " Class Images Instances Box(P R mAP50 mAP50-95) Mask(P R mAP50 mAP50-95): 100% 157/157 [01:54<00:00, 1.37it/s]\n",
+ " all 5000 36335 0.673 0.517 0.566 0.373 0.672 0.49 0.532 0.319\n",
+ "Speed: 0.6ms pre-process, 4.4ms inference, 2.9ms NMS per image at shape (32, 3, 640, 640)\n",
+ "Results saved to \u001b[1mruns/val-seg/exp\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Validate YOLOv5s-seg on COCO val\n",
+ "!python segment/val.py --weights yolov5s-seg.pt --data coco.yaml --img 640 --half"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ZY2VXXXu74w5"
+ },
+ "source": [
+ "# 3. Train\n",
+ "\n",
+ "
\n", + "\n", + "Train a YOLOv5s-seg model on the [COCO128](https://www.kaggle.com/ultralytics/coco128) dataset with `--data coco128-seg.yaml`, starting from pretrained `--weights yolov5s-seg.pt`, or from randomly initialized `--weights '' --cfg yolov5s-seg.yaml`.\n", + "\n", + "- **Pretrained [Models](https://github.com/ultralytics/yolov5/tree/master/models)** are downloaded\n", + "automatically from the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases)\n", + "- **[Datasets](https://github.com/ultralytics/yolov5/tree/master/data)** available for autodownload include: [COCO](https://github.com/ultralytics/yolov5/blob/master/data/coco.yaml), [COCO128](https://github.com/ultralytics/yolov5/blob/master/data/coco128.yaml), [VOC](https://github.com/ultralytics/yolov5/blob/master/data/VOC.yaml), [Argoverse](https://github.com/ultralytics/yolov5/blob/master/data/Argoverse.yaml), [VisDrone](https://github.com/ultralytics/yolov5/blob/master/data/VisDrone.yaml), [GlobalWheat](https://github.com/ultralytics/yolov5/blob/master/data/GlobalWheat2020.yaml), [xView](https://github.com/ultralytics/yolov5/blob/master/data/xView.yaml), [Objects365](https://github.com/ultralytics/yolov5/blob/master/data/Objects365.yaml), [SKU-110K](https://github.com/ultralytics/yolov5/blob/master/data/SKU-110K.yaml).\n", + "- **Training Results** are saved to `runs/train-seg/` with incrementing run directories, i.e. `runs/train-seg/exp2`, `runs/train-seg/exp3` etc.\n", + "
\n", + "\n", + "A **Mosaic Dataloader** is used for training which combines 4 images into 1 mosaic.\n", + "\n", + "## Train on Custom Data with Roboflow 🌟 NEW\n", + "\n", + "[Roboflow](https://roboflow.com/?ref=ultralytics) enables you to easily **organize, label, and prepare** a high quality dataset with your own custom data. Roboflow also makes it easy to establish an active learning pipeline, collaborate with your team on dataset improvement, and integrate directly into your model building workflow with the `roboflow` pip package.\n", + "\n", + "- Custom Training Example: [https://blog.roboflow.com/train-yolov5-instance-segmentation-custom-dataset/](https://blog.roboflow.com/train-yolov5-instance-segmentation-custom-dataset/?ref=ultralytics)\n", + "- Custom Training Notebook: [](https://colab.research.google.com/drive/1JTz7kpmHsg-5qwVz2d2IH3AaenI1tv0N?usp=sharing)\n", + "
\n", + "\n", + "
![](\"https://robflow-public-assets.s3.amazonaws.com/how-to-train-yolov5-segmentation-annotation.gif\"/)
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Lay2WsTjNJzP"
+ },
+ "source": [
+ "## ClearML Logging and Automation 🌟 NEW\n",
+ "\n",
+ "[ClearML](https://cutt.ly/yolov5-notebook-clearml) is completely integrated into YOLOv5 to track your experimentation, manage dataset versions and even remotely execute training runs. To enable ClearML (check cells above):\n",
+ "\n",
+ "- `pip install clearml`\n",
+ "- run `clearml-init` to connect to a ClearML server (**deploy your own [open-source server](https://github.com/allegroai/clearml-server)**, or use our [free hosted server](https://cutt.ly/yolov5-notebook-clearml))\n",
+ "\n",
+ "You'll get all the great expected features from an experiment manager: live updates, model upload, experiment comparison etc. but ClearML also tracks uncommitted changes and installed packages for example. Thanks to that ClearML Tasks (which is what we call experiments) are also reproducible on different machines! With only 1 extra line, we can schedule a YOLOv5 training task on a queue to be executed by any number of ClearML Agents (workers).\n",
+ "\n",
+ "You can use ClearML Data to version your dataset and then pass it to YOLOv5 simply using its unique ID. This will help you keep track of your data without adding extra hassle. Explore the [ClearML Tutorial](https://docs.ultralytics.com/yolov5/tutorials/clearml_logging_integration) for details!\n",
+ "\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-WPvRbS5Swl6"
+ },
+ "source": [
+ "## Local Logging\n",
+ "\n",
+ "Training results are automatically logged with [Tensorboard](https://www.tensorflow.org/tensorboard) and [CSV](https://github.com/ultralytics/yolov5/pull/4148) loggers to `runs/train`, with a new experiment directory created for each new training as `runs/train/exp2`, `runs/train/exp3`, etc.\n",
+ "\n",
+ "This directory contains train and val statistics, mosaics, labels, predictions and augmentated mosaics, as well as metrics and charts including precision-recall (PR) curves and confusion matrices. \n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Zelyeqbyt3GD"
+ },
+ "source": [
+ "# Environments\n",
+ "\n",
+ "YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including [CUDA](https://developer.nvidia.com/cuda)/[CUDNN](https://developer.nvidia.com/cudnn), [Python](https://www.python.org/) and [PyTorch](https://pytorch.org/) preinstalled):\n",
+ "\n",
+ "- **Notebooks** with free GPU: \n",
+ "- **Google Cloud** Deep Learning VM. See [GCP Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/google_cloud_quickstart_tutorial/)\n",
+ "- **Amazon** Deep Learning AMI. See [AWS Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/aws_quickstart_tutorial/)\n",
+ "- **Docker Image**. See [Docker Quickstart Guide](https://docs.ultralytics.com/yolov5/environments/docker_image_quickstart_tutorial/) \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6Qu7Iesl0p54"
+ },
+ "source": [
+ "# Status\n",
+ "\n",
+ "\n",
+ "\n",
+ "If this badge is green, all [YOLOv5 GitHub Actions](https://github.com/ultralytics/yolov5/actions) Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training ([train.py](https://github.com/ultralytics/yolov5/blob/master/train.py)), testing ([val.py](https://github.com/ultralytics/yolov5/blob/master/val.py)), inference ([detect.py](https://github.com/ultralytics/yolov5/blob/master/detect.py)) and export ([export.py](https://github.com/ultralytics/yolov5/blob/master/export.py)) on macOS, Windows, and Ubuntu every 24 hours and on every commit.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IEijrePND_2I"
+ },
+ "source": [
+ "# Appendix\n",
+ "\n",
+ "Additional content below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GMusP4OAxFu6"
+ },
+ "outputs": [],
+ "source": [
+ "# YOLOv5 PyTorch HUB Inference (DetectionModels only)\n",
+ "import torch\n",
+ "\n",
+ "model = torch.hub.load('ultralytics/yolov5', 'yolov5s-seg') # yolov5n - yolov5x6 or custom\n",
+ "im = 'https://ultralytics.com/images/zidane.jpg' # file, Path, PIL.Image, OpenCV, nparray, list\n",
+ "results = model(im) # inference\n",
+ "results.print() # or .show(), .save(), .crop(), .pandas(), etc."
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "name": "YOLOv5 Segmentation Tutorial",
+ "provenance": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/TextDetection/segment/val.py b/TextDetection/segment/val.py
new file mode 100644
index 0000000000000000000000000000000000000000..dc8081840e3708504edef0a3f42747059a46b086
--- /dev/null
+++ b/TextDetection/segment/val.py
@@ -0,0 +1,473 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Validate a trained YOLOv5 segment model on a segment dataset
+
+Usage:
+ $ bash data/scripts/get_coco.sh --val --segments # download COCO-segments val split (1G, 5000 images)
+ $ python segment/val.py --weights yolov5s-seg.pt --data coco.yaml --img 640 # validate COCO-segments
+
+Usage - formats:
+ $ python segment/val.py --weights yolov5s-seg.pt # PyTorch
+ yolov5s-seg.torchscript # TorchScript
+ yolov5s-seg.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s-seg_openvino_label # OpenVINO
+ yolov5s-seg.engine # TensorRT
+ yolov5s-seg.mlmodel # CoreML (macOS-only)
+ yolov5s-seg_saved_model # TensorFlow SavedModel
+ yolov5s-seg.pb # TensorFlow GraphDef
+ yolov5s-seg.tflite # TensorFlow Lite
+ yolov5s-seg_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s-seg_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from multiprocessing.pool import ThreadPool
+from pathlib import Path
+
+import numpy as np
+import torch
+from tqdm import tqdm
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[1] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+import torch.nn.functional as F
+
+from models.common import DetectMultiBackend
+from models.yolo import SegmentationModel
+from utils.callbacks import Callbacks
+from utils.general import (LOGGER, NUM_THREADS, TQDM_BAR_FORMAT, Profile, check_dataset, check_img_size,
+ check_requirements, check_yaml, coco80_to_coco91_class, colorstr, increment_path,
+ non_max_suppression, print_args, scale_boxes, xywh2xyxy, xyxy2xywh)
+from utils.metrics import ConfusionMatrix, box_iou
+from utils.plots import output_to_target, plot_val_study
+from utils.segment.dataloaders import create_dataloader
+from utils.segment.general import mask_iou, process_mask, process_mask_native, scale_image
+from utils.segment.metrics import Metrics, ap_per_class_box_and_mask
+from utils.segment.plots import plot_images_and_masks
+from utils.torch_utils import de_parallel, select_device, smart_inference_mode
+
+
+def save_one_txt(predn, save_conf, shape, file):
+ # Save one txt result
+ gn = torch.tensor(shape)[[1, 0, 1, 0]] # normalization gain whwh
+ for *xyxy, conf, cls in predn.tolist():
+ xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh
+ line = (cls, *xywh, conf) if save_conf else (cls, *xywh) # label format
+ with open(file, 'a') as f:
+ f.write(('%g ' * len(line)).rstrip() % line + '\n')
+
+
+def save_one_json(predn, jdict, path, class_map, pred_masks):
+ # Save one JSON result {"image_id": 42, "category_id": 18, "bbox": [258.15, 41.29, 348.26, 243.78], "score": 0.236}
+ from pycocotools.mask import encode
+
+ def single_encode(x):
+ rle = encode(np.asarray(x[:, :, None], order='F', dtype='uint8'))[0]
+ rle['counts'] = rle['counts'].decode('utf-8')
+ return rle
+
+ image_id = int(path.stem) if path.stem.isnumeric() else path.stem
+ box = xyxy2xywh(predn[:, :4]) # xywh
+ box[:, :2] -= box[:, 2:] / 2 # xy center to top-left corner
+ pred_masks = np.transpose(pred_masks, (2, 0, 1))
+ with ThreadPool(NUM_THREADS) as pool:
+ rles = pool.map(single_encode, pred_masks)
+ for i, (p, b) in enumerate(zip(predn.tolist(), box.tolist())):
+ jdict.append({
+ 'image_id': image_id,
+ 'category_id': class_map[int(p[5])],
+ 'bbox': [round(x, 3) for x in b],
+ 'score': round(p[4], 5),
+ 'segmentation': rles[i]})
+
+
+def process_batch(detections, labels, iouv, pred_masks=None, gt_masks=None, overlap=False, masks=False):
+ """
+ Return correct prediction matrix
+ Arguments:
+ detections (array[N, 6]), x1, y1, x2, y2, conf, class
+ labels (array[M, 5]), class, x1, y1, x2, y2
+ Returns:
+ correct (array[N, 10]), for 10 IoU levels
+ """
+ if masks:
+ if overlap:
+ nl = len(labels)
+ index = torch.arange(nl, device=gt_masks.device).view(nl, 1, 1) + 1
+ gt_masks = gt_masks.repeat(nl, 1, 1) # shape(1,640,640) -> (n,640,640)
+ gt_masks = torch.where(gt_masks == index, 1.0, 0.0)
+ if gt_masks.shape[1:] != pred_masks.shape[1:]:
+ gt_masks = F.interpolate(gt_masks[None], pred_masks.shape[1:], mode='bilinear', align_corners=False)[0]
+ gt_masks = gt_masks.gt_(0.5)
+ iou = mask_iou(gt_masks.view(gt_masks.shape[0], -1), pred_masks.view(pred_masks.shape[0], -1))
+ else: # boxes
+ iou = box_iou(labels[:, 1:], detections[:, :4])
+
+ correct = np.zeros((detections.shape[0], iouv.shape[0])).astype(bool)
+ correct_class = labels[:, 0:1] == detections[:, 5]
+ for i in range(len(iouv)):
+ x = torch.where((iou >= iouv[i]) & correct_class) # IoU > threshold and classes match
+ if x[0].shape[0]:
+ matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy() # [label, detect, iou]
+ if x[0].shape[0] > 1:
+ matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
+ # matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
+ correct[matches[:, 1].astype(int), i] = True
+ return torch.tensor(correct, dtype=torch.bool, device=iouv.device)
+
+
+@smart_inference_mode()
+def run(
+ data,
+ weights=None, # model.pt path(s)
+ batch_size=32, # batch size
+ imgsz=640, # inference size (pixels)
+ conf_thres=0.001, # confidence threshold
+ iou_thres=0.6, # NMS IoU threshold
+ max_det=300, # maximum detections per image
+ task='val', # train, val, test, speed or study
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ workers=8, # max dataloader workers (per RANK in DDP mode)
+ single_cls=False, # treat as single-class dataset
+ augment=False, # augmented inference
+ verbose=False, # verbose output
+ save_txt=False, # save results to *.txt
+ save_hybrid=False, # save label+prediction hybrid results to *.txt
+ save_conf=False, # save confidences in --save-txt labels
+ save_json=False, # save a COCO-JSON results file
+ project=ROOT / 'runs/val-seg', # save to project/name
+ name='exp', # save to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ half=True, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ model=None,
+ dataloader=None,
+ save_dir=Path(''),
+ plots=True,
+ overlap=False,
+ mask_downsample_ratio=1,
+ compute_loss=None,
+ callbacks=Callbacks(),
+):
+ if save_json:
+ check_requirements('pycocotools>=2.0.6')
+ process = process_mask_native # more accurate
+ else:
+ process = process_mask # faster
+
+ # Initialize/load model and set device
+ training = model is not None
+ if training: # called by train.py
+ device, pt, jit, engine = next(model.parameters()).device, True, False, False # get model device, PyTorch model
+ half &= device.type != 'cpu' # half precision only supported on CUDA
+ model.half() if half else model.float()
+ nm = de_parallel(model).model[-1].nm # number of masks
+ else: # called directly
+ device = select_device(device, batch_size=batch_size)
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ (save_dir / 'labels' if save_txt else save_dir).mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
+ stride, pt, jit, engine = model.stride, model.pt, model.jit, model.engine
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+ half = model.fp16 # FP16 supported on limited backends with CUDA
+ nm = de_parallel(model).model.model[-1].nm if isinstance(model, SegmentationModel) else 32 # number of masks
+ if engine:
+ batch_size = model.batch_size
+ else:
+ device = model.device
+ if not (pt or jit):
+ batch_size = 1 # export.py models default to batch-size 1
+ LOGGER.info(f'Forcing --batch-size 1 square inference (1,3,{imgsz},{imgsz}) for non-PyTorch models')
+
+ # Data
+ data = check_dataset(data) # check
+
+ # Configure
+ model.eval()
+ cuda = device.type != 'cpu'
+ is_coco = isinstance(data.get('val'), str) and data['val'].endswith(f'coco{os.sep}val2017.txt') # COCO dataset
+ nc = 1 if single_cls else int(data['nc']) # number of classes
+ iouv = torch.linspace(0.5, 0.95, 10, device=device) # iou vector for mAP@0.5:0.95
+ niou = iouv.numel()
+
+ # Dataloader
+ if not training:
+ if pt and not single_cls: # check --weights are trained on --data
+ ncm = model.model.nc
+ assert ncm == nc, f'{weights} ({ncm} classes) trained on different --data than what you passed ({nc} ' \
+ f'classes). Pass correct combination of --weights and --data that are trained together.'
+ model.warmup(imgsz=(1 if pt else batch_size, 3, imgsz, imgsz)) # warmup
+ pad, rect = (0.0, False) if task == 'speed' else (0.5, pt) # square inference for benchmarks
+ task = task if task in ('train', 'val', 'test') else 'val' # path to train/val/test images
+ dataloader = create_dataloader(data[task],
+ imgsz,
+ batch_size,
+ stride,
+ single_cls,
+ pad=pad,
+ rect=rect,
+ workers=workers,
+ prefix=colorstr(f'{task}: '),
+ overlap_mask=overlap,
+ mask_downsample_ratio=mask_downsample_ratio)[0]
+
+ seen = 0
+ confusion_matrix = ConfusionMatrix(nc=nc)
+ names = model.names if hasattr(model, 'names') else model.module.names # get class names
+ if isinstance(names, (list, tuple)): # old format
+ names = dict(enumerate(names))
+ class_map = coco80_to_coco91_class() if is_coco else list(range(1000))
+ s = ('%22s' + '%11s' * 10) % ('Class', 'Images', 'Instances', 'Box(P', 'R', 'mAP50', 'mAP50-95)', 'Mask(P', 'R',
+ 'mAP50', 'mAP50-95)')
+ dt = Profile(), Profile(), Profile()
+ metrics = Metrics()
+ loss = torch.zeros(4, device=device)
+ jdict, stats = [], []
+ # callbacks.run('on_val_start')
+ pbar = tqdm(dataloader, desc=s, bar_format=TQDM_BAR_FORMAT) # progress bar
+ for batch_i, (im, targets, paths, shapes, masks) in enumerate(pbar):
+ # callbacks.run('on_val_batch_start')
+ with dt[0]:
+ if cuda:
+ im = im.to(device, non_blocking=True)
+ targets = targets.to(device)
+ masks = masks.to(device)
+ masks = masks.float()
+ im = im.half() if half else im.float() # uint8 to fp16/32
+ im /= 255 # 0 - 255 to 0.0 - 1.0
+ nb, _, height, width = im.shape # batch size, channels, height, width
+
+ # Inference
+ with dt[1]:
+ preds, protos, train_out = model(im) if compute_loss else (*model(im, augment=augment)[:2], None)
+
+ # Loss
+ if compute_loss:
+ loss += compute_loss((train_out, protos), targets, masks)[1] # box, obj, cls
+
+ # NMS
+ targets[:, 2:] *= torch.tensor((width, height, width, height), device=device) # to pixels
+ lb = [targets[targets[:, 0] == i, 1:] for i in range(nb)] if save_hybrid else [] # for autolabelling
+ with dt[2]:
+ preds = non_max_suppression(preds,
+ conf_thres,
+ iou_thres,
+ labels=lb,
+ multi_label=True,
+ agnostic=single_cls,
+ max_det=max_det,
+ nm=nm)
+
+ # Metrics
+ plot_masks = [] # masks for plotting
+ for si, (pred, proto) in enumerate(zip(preds, protos)):
+ labels = targets[targets[:, 0] == si, 1:]
+ nl, npr = labels.shape[0], pred.shape[0] # number of labels, predictions
+ path, shape = Path(paths[si]), shapes[si][0]
+ correct_masks = torch.zeros(npr, niou, dtype=torch.bool, device=device) # init
+ correct_bboxes = torch.zeros(npr, niou, dtype=torch.bool, device=device) # init
+ seen += 1
+
+ if npr == 0:
+ if nl:
+ stats.append((correct_masks, correct_bboxes, *torch.zeros((2, 0), device=device), labels[:, 0]))
+ if plots:
+ confusion_matrix.process_batch(detections=None, labels=labels[:, 0])
+ continue
+
+ # Masks
+ midx = [si] if overlap else targets[:, 0] == si
+ gt_masks = masks[midx]
+ pred_masks = process(proto, pred[:, 6:], pred[:, :4], shape=im[si].shape[1:])
+
+ # Predictions
+ if single_cls:
+ pred[:, 5] = 0
+ predn = pred.clone()
+ scale_boxes(im[si].shape[1:], predn[:, :4], shape, shapes[si][1]) # native-space pred
+
+ # Evaluate
+ if nl:
+ tbox = xywh2xyxy(labels[:, 1:5]) # target boxes
+ scale_boxes(im[si].shape[1:], tbox, shape, shapes[si][1]) # native-space labels
+ labelsn = torch.cat((labels[:, 0:1], tbox), 1) # native-space labels
+ correct_bboxes = process_batch(predn, labelsn, iouv)
+ correct_masks = process_batch(predn, labelsn, iouv, pred_masks, gt_masks, overlap=overlap, masks=True)
+ if plots:
+ confusion_matrix.process_batch(predn, labelsn)
+ stats.append((correct_masks, correct_bboxes, pred[:, 4], pred[:, 5], labels[:, 0])) # (conf, pcls, tcls)
+
+ pred_masks = torch.as_tensor(pred_masks, dtype=torch.uint8)
+ if plots and batch_i < 3:
+ plot_masks.append(pred_masks[:15]) # filter top 15 to plot
+
+ # Save/log
+ if save_txt:
+ save_one_txt(predn, save_conf, shape, file=save_dir / 'labels' / f'{path.stem}.txt')
+ if save_json:
+ pred_masks = scale_image(im[si].shape[1:],
+ pred_masks.permute(1, 2, 0).contiguous().cpu().numpy(), shape, shapes[si][1])
+ save_one_json(predn, jdict, path, class_map, pred_masks) # append to COCO-JSON dictionary
+ # callbacks.run('on_val_image_end', pred, predn, path, names, im[si])
+
+ # Plot images
+ if plots and batch_i < 3:
+ if len(plot_masks):
+ plot_masks = torch.cat(plot_masks, dim=0)
+ plot_images_and_masks(im, targets, masks, paths, save_dir / f'val_batch{batch_i}_labels.jpg', names)
+ plot_images_and_masks(im, output_to_target(preds, max_det=15), plot_masks, paths,
+ save_dir / f'val_batch{batch_i}_pred.jpg', names) # pred
+
+ # callbacks.run('on_val_batch_end')
+
+ # Compute metrics
+ stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)] # to numpy
+ if len(stats) and stats[0].any():
+ results = ap_per_class_box_and_mask(*stats, plot=plots, save_dir=save_dir, names=names)
+ metrics.update(results)
+ nt = np.bincount(stats[4].astype(int), minlength=nc) # number of targets per class
+
+ # Print results
+ pf = '%22s' + '%11i' * 2 + '%11.3g' * 8 # print format
+ LOGGER.info(pf % ('all', seen, nt.sum(), *metrics.mean_results()))
+ if nt.sum() == 0:
+ LOGGER.warning(f'WARNING ⚠️ no labels found in {task} set, can not compute metrics without labels')
+
+ # Print results per class
+ if (verbose or (nc < 50 and not training)) and nc > 1 and len(stats):
+ for i, c in enumerate(metrics.ap_class_index):
+ LOGGER.info(pf % (names[c], seen, nt[c], *metrics.class_result(i)))
+
+ # Print speeds
+ t = tuple(x.t / seen * 1E3 for x in dt) # speeds per image
+ if not training:
+ shape = (batch_size, 3, imgsz, imgsz)
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {shape}' % t)
+
+ # Plots
+ if plots:
+ confusion_matrix.plot(save_dir=save_dir, names=list(names.values()))
+ # callbacks.run('on_val_end')
+
+ mp_bbox, mr_bbox, map50_bbox, map_bbox, mp_mask, mr_mask, map50_mask, map_mask = metrics.mean_results()
+
+ # Save JSON
+ if save_json and len(jdict):
+ w = Path(weights[0] if isinstance(weights, list) else weights).stem if weights is not None else '' # weights
+ anno_json = str(Path('../datasets/coco/annotations/instances_val2017.json')) # annotations
+ pred_json = str(save_dir / f'{w}_predictions.json') # predictions
+ LOGGER.info(f'\nEvaluating pycocotools mAP... saving {pred_json}...')
+ with open(pred_json, 'w') as f:
+ json.dump(jdict, f)
+
+ try: # https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb
+ from pycocotools.coco import COCO
+ from pycocotools.cocoeval import COCOeval
+
+ anno = COCO(anno_json) # init annotations api
+ pred = anno.loadRes(pred_json) # init predictions api
+ results = []
+ for eval in COCOeval(anno, pred, 'bbox'), COCOeval(anno, pred, 'segm'):
+ if is_coco:
+ eval.params.imgIds = [int(Path(x).stem) for x in dataloader.dataset.im_files] # img ID to evaluate
+ eval.evaluate()
+ eval.accumulate()
+ eval.summarize()
+ results.extend(eval.stats[:2]) # update results (mAP@0.5:0.95, mAP@0.5)
+ map_bbox, map50_bbox, map_mask, map50_mask = results
+ except Exception as e:
+ LOGGER.info(f'pycocotools unable to run: {e}')
+
+ # Return results
+ model.float() # for training
+ if not training:
+ s = f"\n{len(list(save_dir.glob('labels/*.txt')))} labels saved to {save_dir / 'labels'}" if save_txt else ''
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}{s}")
+ final_metric = mp_bbox, mr_bbox, map50_bbox, map_bbox, mp_mask, mr_mask, map50_mask, map_mask
+ return (*final_metric, *(loss.cpu() / len(dataloader)).tolist()), metrics.get_maps(nc), t
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128-seg.yaml', help='dataset.yaml path')
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s-seg.pt', help='model path(s)')
+ parser.add_argument('--batch-size', type=int, default=32, help='batch size')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='inference size (pixels)')
+ parser.add_argument('--conf-thres', type=float, default=0.001, help='confidence threshold')
+ parser.add_argument('--iou-thres', type=float, default=0.6, help='NMS IoU threshold')
+ parser.add_argument('--max-det', type=int, default=300, help='maximum detections per image')
+ parser.add_argument('--task', default='val', help='train, val, test, speed or study')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--single-cls', action='store_true', help='treat as single-class dataset')
+ parser.add_argument('--augment', action='store_true', help='augmented inference')
+ parser.add_argument('--verbose', action='store_true', help='report mAP by class')
+ parser.add_argument('--save-txt', action='store_true', help='save results to *.txt')
+ parser.add_argument('--save-hybrid', action='store_true', help='save label+prediction hybrid results to *.txt')
+ parser.add_argument('--save-conf', action='store_true', help='save confidences in --save-txt labels')
+ parser.add_argument('--save-json', action='store_true', help='save a COCO-JSON results file')
+ parser.add_argument('--project', default=ROOT / 'runs/val-seg', help='save results to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ opt = parser.parse_args()
+ opt.data = check_yaml(opt.data) # check YAML
+ # opt.save_json |= opt.data.endswith('coco.yaml')
+ opt.save_txt |= opt.save_hybrid
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(ROOT / 'requirements.txt', exclude=('tensorboard', 'thop'))
+
+ if opt.task in ('train', 'val', 'test'): # run normally
+ if opt.conf_thres > 0.001: # https://github.com/ultralytics/yolov5/issues/1466
+ LOGGER.warning(f'WARNING ⚠️ confidence threshold {opt.conf_thres} > 0.001 produces invalid results')
+ if opt.save_hybrid:
+ LOGGER.warning('WARNING ⚠️ --save-hybrid returns high mAP from hybrid labels, not from predictions alone')
+ run(**vars(opt))
+
+ else:
+ weights = opt.weights if isinstance(opt.weights, list) else [opt.weights]
+ opt.half = torch.cuda.is_available() and opt.device != 'cpu' # FP16 for fastest results
+ if opt.task == 'speed': # speed benchmarks
+ # python val.py --task speed --data coco.yaml --batch 1 --weights yolov5n.pt yolov5s.pt...
+ opt.conf_thres, opt.iou_thres, opt.save_json = 0.25, 0.45, False
+ for opt.weights in weights:
+ run(**vars(opt), plots=False)
+
+ elif opt.task == 'study': # speed vs mAP benchmarks
+ # python val.py --task study --data coco.yaml --iou 0.7 --weights yolov5n.pt yolov5s.pt...
+ for opt.weights in weights:
+ f = f'study_{Path(opt.data).stem}_{Path(opt.weights).stem}.txt' # filename to save to
+ x, y = list(range(256, 1536 + 128, 128)), [] # x axis (image sizes), y axis
+ for opt.imgsz in x: # img-size
+ LOGGER.info(f'\nRunning {f} --imgsz {opt.imgsz}...')
+ r, _, t = run(**vars(opt), plots=False)
+ y.append(r + t) # results and times
+ np.savetxt(f, y, fmt='%10.4g') # save
+ subprocess.run(['zip', '-r', 'study.zip', 'study_*.txt'])
+ plot_val_study(x=x) # plot
+ else:
+ raise NotImplementedError(f'--task {opt.task} not in ("train", "val", "test", "speed", "study")')
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextDetection/setup.cfg b/TextDetection/setup.cfg
new file mode 100644
index 0000000000000000000000000000000000000000..2cde6a494836f93681b73aaa7bcf0d0d487de469
--- /dev/null
+++ b/TextDetection/setup.cfg
@@ -0,0 +1,56 @@
+# Project-wide configuration file, can be used for package metadata and other toll configurations
+# Example usage: global configuration for PEP8 (via flake8) setting or default pytest arguments
+# Local usage: pip install pre-commit, pre-commit run --all-files
+
+[metadata]
+license_files = LICENSE
+description_file = README.md
+
+[tool:pytest]
+norecursedirs =
+ .git
+ dist
+ build
+addopts =
+ --doctest-modules
+ --durations=25
+ --color=yes
+
+[flake8]
+max-line-length = 120
+exclude = .tox,*.egg,build,temp
+select = E,W,F
+doctests = True
+verbose = 2
+# https://pep8.readthedocs.io/en/latest/intro.html#error-codes
+format = pylint
+# see: https://www.flake8rules.com/
+ignore = E731,F405,E402,W504,E501
+ # E731: Do not assign a lambda expression, use a def
+ # F405: name may be undefined, or defined from star imports: module
+ # E402: module level import not at top of file
+ # W504: line break after binary operator
+ # E501: line too long
+ # removed:
+ # F401: module imported but unused
+ # E231: missing whitespace after ‘,’, ‘;’, or ‘:’
+ # E127: continuation line over-indented for visual indent
+ # F403: ‘from module import *’ used; unable to detect undefined names
+
+
+[isort]
+# https://pycqa.github.io/isort/docs/configuration/options.html
+line_length = 120
+# see: https://pycqa.github.io/isort/docs/configuration/multi_line_output_modes.html
+multi_line_output = 0
+
+[yapf]
+based_on_style = pep8
+spaces_before_comment = 2
+COLUMN_LIMIT = 120
+COALESCE_BRACKETS = True
+SPACES_AROUND_POWER_OPERATOR = True
+SPACE_BETWEEN_ENDING_COMMA_AND_CLOSING_BRACKET = True
+SPLIT_BEFORE_CLOSING_BRACKET = False
+SPLIT_BEFORE_FIRST_ARGUMENT = False
+# EACH_DICT_ENTRY_ON_SEPARATE_LINE = False
diff --git a/TextDetection/utils/__init__.py b/TextDetection/utils/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..f849dd851751058dd4e1b1c3f1b4e851fcc29d19
--- /dev/null
+++ b/TextDetection/utils/__init__.py
@@ -0,0 +1,86 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+utils/initialization
+"""
+
+import contextlib
+import platform
+import threading
+
+
+def emojis(str=''):
+ # Return platform-dependent emoji-safe version of string
+ return str.encode().decode('ascii', 'ignore') if platform.system() == 'Windows' else str
+
+
+class TryExcept(contextlib.ContextDecorator):
+ # YOLOv5 TryExcept class. Usage: @TryExcept() decorator or 'with TryExcept():' context manager
+ def __init__(self, msg=''):
+ self.msg = msg
+
+ def __enter__(self):
+ pass
+
+ def __exit__(self, exc_type, value, traceback):
+ if value:
+ print(emojis(f"{self.msg}{': ' if self.msg else ''}{value}"))
+ return True
+
+
+def threaded(func):
+ # Multi-threads a target function and returns thread. Usage: @threaded decorator
+ def wrapper(*args, **kwargs):
+ thread = threading.Thread(target=func, args=args, kwargs=kwargs, daemon=True)
+ thread.start()
+ return thread
+
+ return wrapper
+
+
+def join_threads(verbose=False):
+ # Join all daemon threads, i.e. atexit.register(lambda: join_threads())
+ main_thread = threading.current_thread()
+ for t in threading.enumerate():
+ if t is not main_thread:
+ if verbose:
+ print(f'Joining thread {t.name}')
+ t.join()
+
+
+def notebook_init(verbose=True):
+ # Check system software and hardware
+ print('Checking setup...')
+
+ import os
+ import shutil
+
+ from ultralytics.yolo.utils.checks import check_requirements
+
+ from utils.general import check_font, is_colab
+ from utils.torch_utils import select_device # imports
+
+ check_font()
+
+ import psutil
+
+ if check_requirements('wandb', install=False):
+ os.system('pip uninstall -y wandb') # eliminate unexpected account creation prompt with infinite hang
+ if is_colab():
+ shutil.rmtree('/content/sample_data', ignore_errors=True) # remove colab /sample_data directory
+
+ # System info
+ display = None
+ if verbose:
+ gb = 1 << 30 # bytes to GiB (1024 ** 3)
+ ram = psutil.virtual_memory().total
+ total, used, free = shutil.disk_usage('/')
+ with contextlib.suppress(Exception): # clear display if ipython is installed
+ from IPython import display
+ display.clear_output()
+ s = f'({os.cpu_count()} CPUs, {ram / gb:.1f} GB RAM, {(total - free) / gb:.1f}/{total / gb:.1f} GB disk)'
+ else:
+ s = ''
+
+ select_device(newline=False)
+ print(emojis(f'Setup complete ✅ {s}'))
+ return display
diff --git a/TextDetection/utils/__pycache__/__init__.cpython-310.pyc b/TextDetection/utils/__pycache__/__init__.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..e4921dfde4f55c1c5ed7f5947e59676a6d57a8b2
Binary files /dev/null and b/TextDetection/utils/__pycache__/__init__.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/__init__.cpython-39.pyc b/TextDetection/utils/__pycache__/__init__.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..99af1bdda87ad536c1e57c32b65515b0190d278c
Binary files /dev/null and b/TextDetection/utils/__pycache__/__init__.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/augmentations.cpython-310.pyc b/TextDetection/utils/__pycache__/augmentations.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..c3e8a0fbee4d9205b045a00c9f25c90ac5825ec8
Binary files /dev/null and b/TextDetection/utils/__pycache__/augmentations.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/augmentations.cpython-39.pyc b/TextDetection/utils/__pycache__/augmentations.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..d99e93af2ffe76e82eb381ef6d6e41bf05be5a3c
Binary files /dev/null and b/TextDetection/utils/__pycache__/augmentations.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/autoanchor.cpython-310.pyc b/TextDetection/utils/__pycache__/autoanchor.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..122872b7f628ffa0bba791da6e16cae3cbcf9dc5
Binary files /dev/null and b/TextDetection/utils/__pycache__/autoanchor.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/autoanchor.cpython-39.pyc b/TextDetection/utils/__pycache__/autoanchor.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..b7193e91c36599cd437639530e160e8aae279b00
Binary files /dev/null and b/TextDetection/utils/__pycache__/autoanchor.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/dataloaders.cpython-310.pyc b/TextDetection/utils/__pycache__/dataloaders.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..4b8c5d711a2384d6e5898c1e89872176d1dfe6d2
Binary files /dev/null and b/TextDetection/utils/__pycache__/dataloaders.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/dataloaders.cpython-39.pyc b/TextDetection/utils/__pycache__/dataloaders.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..aaee8fefa2232589b6ae7f4885b8721cde04bde1
Binary files /dev/null and b/TextDetection/utils/__pycache__/dataloaders.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/downloads.cpython-310.pyc b/TextDetection/utils/__pycache__/downloads.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..9ad8e0facc0096b84f83bb266e7e410ea0e882f1
Binary files /dev/null and b/TextDetection/utils/__pycache__/downloads.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/downloads.cpython-39.pyc b/TextDetection/utils/__pycache__/downloads.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..f6dd4ea9ec9f0be0a4407186adb34a45f7af2ace
Binary files /dev/null and b/TextDetection/utils/__pycache__/downloads.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/general.cpython-310.pyc b/TextDetection/utils/__pycache__/general.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..6c4a9912f465176a363a34c2504d7e6261c6a110
Binary files /dev/null and b/TextDetection/utils/__pycache__/general.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/general.cpython-39.pyc b/TextDetection/utils/__pycache__/general.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..b771120492e3ec3d11282d45ddc88167d1f40e80
Binary files /dev/null and b/TextDetection/utils/__pycache__/general.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/metrics.cpython-310.pyc b/TextDetection/utils/__pycache__/metrics.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..207de64d2f550df457ad76099848d27c2757b099
Binary files /dev/null and b/TextDetection/utils/__pycache__/metrics.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/metrics.cpython-39.pyc b/TextDetection/utils/__pycache__/metrics.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..b919ec92ab250818bacaf7d5f84dfa66bca753a9
Binary files /dev/null and b/TextDetection/utils/__pycache__/metrics.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/plots.cpython-310.pyc b/TextDetection/utils/__pycache__/plots.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..f95f079fedd2788cda0da7052cf8bf868a16e7b3
Binary files /dev/null and b/TextDetection/utils/__pycache__/plots.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/plots.cpython-39.pyc b/TextDetection/utils/__pycache__/plots.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..8573d08fcd5ecf3a2903130dc668925eb923f548
Binary files /dev/null and b/TextDetection/utils/__pycache__/plots.cpython-39.pyc differ
diff --git a/TextDetection/utils/__pycache__/torch_utils.cpython-310.pyc b/TextDetection/utils/__pycache__/torch_utils.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..25dfa38ff41cdf1216ca99756ffe34059cef1967
Binary files /dev/null and b/TextDetection/utils/__pycache__/torch_utils.cpython-310.pyc differ
diff --git a/TextDetection/utils/__pycache__/torch_utils.cpython-39.pyc b/TextDetection/utils/__pycache__/torch_utils.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..f71e574e2b85e26a4a97584d4945bf2c6cf75920
Binary files /dev/null and b/TextDetection/utils/__pycache__/torch_utils.cpython-39.pyc differ
diff --git a/TextDetection/utils/activations.py b/TextDetection/utils/activations.py
new file mode 100644
index 0000000000000000000000000000000000000000..e4d4bbde5ec8610a5ff13fe2ef2281721c14ca1a
--- /dev/null
+++ b/TextDetection/utils/activations.py
@@ -0,0 +1,103 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Activation functions
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class SiLU(nn.Module):
+ # SiLU activation https://arxiv.org/pdf/1606.08415.pdf
+ @staticmethod
+ def forward(x):
+ return x * torch.sigmoid(x)
+
+
+class Hardswish(nn.Module):
+ # Hard-SiLU activation
+ @staticmethod
+ def forward(x):
+ # return x * F.hardsigmoid(x) # for TorchScript and CoreML
+ return x * F.hardtanh(x + 3, 0.0, 6.0) / 6.0 # for TorchScript, CoreML and ONNX
+
+
+class Mish(nn.Module):
+ # Mish activation https://github.com/digantamisra98/Mish
+ @staticmethod
+ def forward(x):
+ return x * F.softplus(x).tanh()
+
+
+class MemoryEfficientMish(nn.Module):
+ # Mish activation memory-efficient
+ class F(torch.autograd.Function):
+
+ @staticmethod
+ def forward(ctx, x):
+ ctx.save_for_backward(x)
+ return x.mul(torch.tanh(F.softplus(x))) # x * tanh(ln(1 + exp(x)))
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ x = ctx.saved_tensors[0]
+ sx = torch.sigmoid(x)
+ fx = F.softplus(x).tanh()
+ return grad_output * (fx + x * sx * (1 - fx * fx))
+
+ def forward(self, x):
+ return self.F.apply(x)
+
+
+class FReLU(nn.Module):
+ # FReLU activation https://arxiv.org/abs/2007.11824
+ def __init__(self, c1, k=3): # ch_in, kernel
+ super().__init__()
+ self.conv = nn.Conv2d(c1, c1, k, 1, 1, groups=c1, bias=False)
+ self.bn = nn.BatchNorm2d(c1)
+
+ def forward(self, x):
+ return torch.max(x, self.bn(self.conv(x)))
+
+
+class AconC(nn.Module):
+ r""" ACON activation (activate or not)
+ AconC: (p1*x-p2*x) * sigmoid(beta*(p1*x-p2*x)) + p2*x, beta is a learnable parameter
+ according to "Activate or Not: Learning Customized Activation" 
+
+## About ClearML
+
+[ClearML](https://cutt.ly/yolov5-tutorial-clearml) is an [open-source](https://github.com/allegroai/clearml) toolbox designed to save you time ⏱️.
+
+🔨 Track every YOLOv5 training run in the experiment manager
+
+🔧 Version and easily access your custom training data with the integrated ClearML Data Versioning Tool
+
+🔦 Remotely train and monitor your YOLOv5 training runs using ClearML Agent
+
+🔬 Get the very best mAP using ClearML Hyperparameter Optimization
+
+🔭 Turn your newly trained YOLOv5 model into an API with just a few commands using ClearML Serving
+
++And so much more. It's up to you how many of these tools you want to use, you can stick to the experiment manager, or chain them all together into an impressive pipeline! +
+
+ + + +
+
+ +## 🦾 Setting Things Up + +To keep track of your experiments and/or data, ClearML needs to communicate to a server. You have 2 options to get one: + +Either sign up for free to the [ClearML Hosted Service](https://cutt.ly/yolov5-tutorial-clearml) or you can set up your own server, see [here](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server). Even the server is open-source, so even if you're dealing with sensitive data, you should be good to go! + +1. Install the `clearml` python package: + + ```bash + pip install clearml + ``` + +1. Connect the ClearML SDK to the server by [creating credentials](https://app.clear.ml/settings/workspace-configuration) (go right top to Settings -> Workspace -> Create new credentials), then execute the command below and follow the instructions: + + ```bash + clearml-init + ``` + +That's it! You're done 😎 + +
+ +## 🚀 Training YOLOv5 With ClearML + +To enable ClearML experiment tracking, simply install the ClearML pip package. + +```bash +pip install clearml>=1.2.0 +``` + +This will enable integration with the YOLOv5 training script. Every training run from now on, will be captured and stored by the ClearML experiment manager. + +If you want to change the `project_name` or `task_name`, use the `--project` and `--name` arguments of the `train.py` script, by default the project will be called `YOLOv5` and the task `Training`. +PLEASE NOTE: ClearML uses `/` as a delimiter for subprojects, so be careful when using `/` in your project name! + +```bash +python train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache +``` + +or with custom project and task name: + +```bash +python train.py --project my_project --name my_training --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache +``` + +This will capture: + +- Source code + uncommitted changes +- Installed packages +- (Hyper)parameters +- Model files (use `--save-period n` to save a checkpoint every n epochs) +- Console output +- Scalars (mAP_0.5, mAP_0.5:0.95, precision, recall, losses, learning rates, ...) +- General info such as machine details, runtime, creation date etc. +- All produced plots such as label correlogram and confusion matrix +- Images with bounding boxes per epoch +- Mosaic per epoch +- Validation images per epoch +- ... + +That's a lot right? 🤯 +Now, we can visualize all of this information in the ClearML UI to get an overview of our training progress. Add custom columns to the table view (such as e.g. mAP_0.5) so you can easily sort on the best performing model. Or select multiple experiments and directly compare them! + +There even more we can do with all of this information, like hyperparameter optimization and remote execution, so keep reading if you want to see how that works! + +
+ +## 🔗 Dataset Version Management + +Versioning your data separately from your code is generally a good idea and makes it easy to acquire the latest version too. This repository supports supplying a dataset version ID, and it will make sure to get the data if it's not there yet. Next to that, this workflow also saves the used dataset ID as part of the task parameters, so you will always know for sure which data was used in which experiment! + + + +### Prepare Your Dataset + +The YOLOv5 repository supports a number of different datasets by using yaml files containing their information. By default datasets are downloaded to the `../datasets` folder in relation to the repository root folder. So if you downloaded the `coco128` dataset using the link in the yaml or with the scripts provided by yolov5, you get this folder structure: + +``` +.. +|_ yolov5 +|_ datasets + |_ coco128 + |_ images + |_ labels + |_ LICENSE + |_ README.txt +``` + +But this can be any dataset you wish. Feel free to use your own, as long as you keep to this folder structure. + +Next, ⚠️**copy the corresponding yaml file to the root of the dataset folder**⚠️. This yaml files contains the information ClearML will need to properly use the dataset. You can make this yourself too, of course, just follow the structure of the example yamls. + +Basically we need the following keys: `path`, `train`, `test`, `val`, `nc`, `names`. + +``` +.. +|_ yolov5 +|_ datasets + |_ coco128 + |_ images + |_ labels + |_ coco128.yaml # <---- HERE! + |_ LICENSE + |_ README.txt +``` + +### Upload Your Dataset + +To get this dataset into ClearML as a versioned dataset, go to the dataset root folder and run the following command: + +```bash +cd coco128 +clearml-data sync --project YOLOv5 --name coco128 --folder . +``` + +The command `clearml-data sync` is actually a shorthand command. You could also run these commands one after the other: + +```bash +# Optionally add --parent
+ +## 👀 Hyperparameter Optimization + +Now that we have our experiments and data versioned, it's time to take a look at what we can build on top! + +Using the code information, installed packages and environment details, the experiment itself is now **completely reproducible**. In fact, ClearML allows you to clone an experiment and even change its parameters. We can then just rerun it with these new parameters automatically, this is basically what HPO does! + +To **run hyperparameter optimization locally**, we've included a pre-made script for you. Just make sure a training task has been run at least once, so it is in the ClearML experiment manager, we will essentially clone it and change its hyperparameters. + +You'll need to fill in the ID of this `template task` in the script found at `utils/loggers/clearml/hpo.py` and then just run it :) You can change `task.execute_locally()` to `task.execute()` to put it in a ClearML queue and have a remote agent work on it instead. + +```bash +# To use optuna, install it first, otherwise you can change the optimizer to just be RandomSearch +pip install optuna +python utils/loggers/clearml/hpo.py +``` + + + +## 🤯 Remote Execution (advanced) + +Running HPO locally is really handy, but what if we want to run our experiments on a remote machine instead? Maybe you have access to a very powerful GPU machine on-site, or you have some budget to use cloud GPUs. +This is where the ClearML Agent comes into play. Check out what the agent can do here: + +- [YouTube video](https://youtu.be/MX3BrXnaULs) +- [Documentation](https://clear.ml/docs/latest/docs/clearml_agent) + +In short: every experiment tracked by the experiment manager contains enough information to reproduce it on a different machine (installed packages, uncommitted changes etc.). So a ClearML agent does just that: it listens to a queue for incoming tasks and when it finds one, it recreates the environment and runs it while still reporting scalars, plots etc. to the experiment manager. + +You can turn any machine (a cloud VM, a local GPU machine, your own laptop ... ) into a ClearML agent by simply running: + +```bash +clearml-agent daemon --queue
+
+# YOLOv5 with Comet
+
+This guide will cover how to use YOLOv5 with [Comet](https://bit.ly/yolov5-readme-comet2)
+
+# About Comet
+
+Comet builds tools that help data scientists, engineers, and team leaders accelerate and optimize machine learning and deep learning models.
+
+Track and visualize model metrics in real time, save your hyperparameters, datasets, and model checkpoints, and visualize your model predictions with [Comet Custom Panels](https://www.comet.com/docs/v2/guides/comet-dashboard/code-panels/about-panels/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)!
+Comet makes sure you never lose track of your work and makes it easy to share results and collaborate across teams of all sizes!
+
+# Getting Started
+
+## Install Comet
+
+```shell
+pip install comet_ml
+```
+
+## Configure Comet Credentials
+
+There are two ways to configure Comet with YOLOv5.
+
+You can either set your credentials through environment variables
+
+**Environment Variables**
+
+```shell
+export COMET_API_KEY=
+
+# Try out an Example!
+
+Check out an example of a [completed run here](https://www.comet.com/examples/comet-example-yolov5/a0e29e0e9b984e4a822db2a62d0cb357?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step&utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)
+
+Or better yet, try it out yourself in this Colab Notebook
+
+[](https://colab.research.google.com/github/comet-ml/comet-examples/blob/master/integrations/model-training/yolov5/notebooks/Comet_and_YOLOv5.ipynb)
+
+# Log automatically
+
+By default, Comet will log the following items
+
+## Metrics
+
+- Box Loss, Object Loss, Classification Loss for the training and validation data
+- mAP_0.5, mAP_0.5:0.95 metrics for the validation data.
+- Precision and Recall for the validation data
+
+## Parameters
+
+- Model Hyperparameters
+- All parameters passed through the command line options
+
+## Visualizations
+
+- Confusion Matrix of the model predictions on the validation data
+- Plots for the PR and F1 curves across all classes
+- Correlogram of the Class Labels
+
+# Configure Comet Logging
+
+Comet can be configured to log additional data either through command line flags passed to the training script
+or through environment variables.
+
+```shell
+export COMET_MODE=online # Set whether to run Comet in 'online' or 'offline' mode. Defaults to online
+export COMET_MODEL_NAME=
+
+You can preview the data directly in the Comet UI.
+
+
+Artifacts are versioned and also support adding metadata about the dataset. Comet will automatically log the metadata from your dataset `yaml` file
+
+
+### Using a saved Artifact
+
+If you would like to use a dataset from Comet Artifacts, set the `path` variable in your dataset `yaml` file to point to the following Artifact resource URL.
+
+```
+# contents of artifact.yaml file
+path: "comet://
+
+## Resuming a Training Run
+
+If your training run is interrupted for any reason, e.g. disrupted internet connection, you can resume the run using the `resume` flag and the Comet Run Path.
+
+The Run Path has the following format `comet://
diff --git a/TextDetection/utils/loggers/comet/__init__.py b/TextDetection/utils/loggers/comet/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..ad7fa5521c6d527d0ed23ad5115505ab73688c4c
--- /dev/null
+++ b/TextDetection/utils/loggers/comet/__init__.py
@@ -0,0 +1,519 @@
+import glob
+import json
+import logging
+import os
+import sys
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[3] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+
+try:
+ import comet_ml
+
+ # Project Configuration
+ config = comet_ml.config.get_config()
+ COMET_PROJECT_NAME = config.get_string(os.getenv('COMET_PROJECT_NAME'), 'comet.project_name', default='yolov5')
+except ImportError:
+ comet_ml = None
+ COMET_PROJECT_NAME = None
+
+import PIL
+import torch
+import torchvision.transforms as T
+import yaml
+
+from utils.dataloaders import img2label_paths
+from utils.general import check_dataset, scale_boxes, xywh2xyxy
+from utils.metrics import box_iou
+
+COMET_PREFIX = 'comet://'
+
+COMET_MODE = os.getenv('COMET_MODE', 'online')
+
+# Model Saving Settings
+COMET_MODEL_NAME = os.getenv('COMET_MODEL_NAME', 'yolov5')
+
+# Dataset Artifact Settings
+COMET_UPLOAD_DATASET = os.getenv('COMET_UPLOAD_DATASET', 'false').lower() == 'true'
+
+# Evaluation Settings
+COMET_LOG_CONFUSION_MATRIX = (os.getenv('COMET_LOG_CONFUSION_MATRIX', 'true').lower() == 'true')
+COMET_LOG_PREDICTIONS = os.getenv('COMET_LOG_PREDICTIONS', 'true').lower() == 'true'
+COMET_MAX_IMAGE_UPLOADS = int(os.getenv('COMET_MAX_IMAGE_UPLOADS', 100))
+
+# Confusion Matrix Settings
+CONF_THRES = float(os.getenv('CONF_THRES', 0.001))
+IOU_THRES = float(os.getenv('IOU_THRES', 0.6))
+
+# Batch Logging Settings
+COMET_LOG_BATCH_METRICS = (os.getenv('COMET_LOG_BATCH_METRICS', 'false').lower() == 'true')
+COMET_BATCH_LOGGING_INTERVAL = os.getenv('COMET_BATCH_LOGGING_INTERVAL', 1)
+COMET_PREDICTION_LOGGING_INTERVAL = os.getenv('COMET_PREDICTION_LOGGING_INTERVAL', 1)
+COMET_LOG_PER_CLASS_METRICS = (os.getenv('COMET_LOG_PER_CLASS_METRICS', 'false').lower() == 'true')
+
+RANK = int(os.getenv('RANK', -1))
+
+to_pil = T.ToPILImage()
+
+
+class CometLogger:
+ """Log metrics, parameters, source code, models and much more
+ with Comet
+ """
+
+ def __init__(self, opt, hyp, run_id=None, job_type='Training', **experiment_kwargs) -> None:
+ self.job_type = job_type
+ self.opt = opt
+ self.hyp = hyp
+
+ # Comet Flags
+ self.comet_mode = COMET_MODE
+
+ self.save_model = opt.save_period > -1
+ self.model_name = COMET_MODEL_NAME
+
+ # Batch Logging Settings
+ self.log_batch_metrics = COMET_LOG_BATCH_METRICS
+ self.comet_log_batch_interval = COMET_BATCH_LOGGING_INTERVAL
+
+ # Dataset Artifact Settings
+ self.upload_dataset = self.opt.upload_dataset or COMET_UPLOAD_DATASET
+ self.resume = self.opt.resume
+
+ # Default parameters to pass to Experiment objects
+ self.default_experiment_kwargs = {
+ 'log_code': False,
+ 'log_env_gpu': True,
+ 'log_env_cpu': True,
+ 'project_name': COMET_PROJECT_NAME, }
+ self.default_experiment_kwargs.update(experiment_kwargs)
+ self.experiment = self._get_experiment(self.comet_mode, run_id)
+ self.experiment.set_name(self.opt.name)
+
+ self.data_dict = self.check_dataset(self.opt.data)
+ self.class_names = self.data_dict['names']
+ self.num_classes = self.data_dict['nc']
+
+ self.logged_images_count = 0
+ self.max_images = COMET_MAX_IMAGE_UPLOADS
+
+ if run_id is None:
+ self.experiment.log_other('Created from', 'YOLOv5')
+ if not isinstance(self.experiment, comet_ml.OfflineExperiment):
+ workspace, project_name, experiment_id = self.experiment.url.split('/')[-3:]
+ self.experiment.log_other(
+ 'Run Path',
+ f'{workspace}/{project_name}/{experiment_id}',
+ )
+ self.log_parameters(vars(opt))
+ self.log_parameters(self.opt.hyp)
+ self.log_asset_data(
+ self.opt.hyp,
+ name='hyperparameters.json',
+ metadata={'type': 'hyp-config-file'},
+ )
+ self.log_asset(
+ f'{self.opt.save_dir}/opt.yaml',
+ metadata={'type': 'opt-config-file'},
+ )
+
+ self.comet_log_confusion_matrix = COMET_LOG_CONFUSION_MATRIX
+
+ if hasattr(self.opt, 'conf_thres'):
+ self.conf_thres = self.opt.conf_thres
+ else:
+ self.conf_thres = CONF_THRES
+ if hasattr(self.opt, 'iou_thres'):
+ self.iou_thres = self.opt.iou_thres
+ else:
+ self.iou_thres = IOU_THRES
+
+ self.log_parameters({'val_iou_threshold': self.iou_thres, 'val_conf_threshold': self.conf_thres})
+
+ self.comet_log_predictions = COMET_LOG_PREDICTIONS
+ if self.opt.bbox_interval == -1:
+ self.comet_log_prediction_interval = (1 if self.opt.epochs < 10 else self.opt.epochs // 10)
+ else:
+ self.comet_log_prediction_interval = self.opt.bbox_interval
+
+ if self.comet_log_predictions:
+ self.metadata_dict = {}
+ self.logged_image_names = []
+
+ self.comet_log_per_class_metrics = COMET_LOG_PER_CLASS_METRICS
+
+ self.experiment.log_others({
+ 'comet_mode': COMET_MODE,
+ 'comet_max_image_uploads': COMET_MAX_IMAGE_UPLOADS,
+ 'comet_log_per_class_metrics': COMET_LOG_PER_CLASS_METRICS,
+ 'comet_log_batch_metrics': COMET_LOG_BATCH_METRICS,
+ 'comet_log_confusion_matrix': COMET_LOG_CONFUSION_MATRIX,
+ 'comet_model_name': COMET_MODEL_NAME, })
+
+ # Check if running the Experiment with the Comet Optimizer
+ if hasattr(self.opt, 'comet_optimizer_id'):
+ self.experiment.log_other('optimizer_id', self.opt.comet_optimizer_id)
+ self.experiment.log_other('optimizer_objective', self.opt.comet_optimizer_objective)
+ self.experiment.log_other('optimizer_metric', self.opt.comet_optimizer_metric)
+ self.experiment.log_other('optimizer_parameters', json.dumps(self.hyp))
+
+ def _get_experiment(self, mode, experiment_id=None):
+ if mode == 'offline':
+ if experiment_id is not None:
+ return comet_ml.ExistingOfflineExperiment(
+ previous_experiment=experiment_id,
+ **self.default_experiment_kwargs,
+ )
+
+ return comet_ml.OfflineExperiment(**self.default_experiment_kwargs, )
+
+ else:
+ try:
+ if experiment_id is not None:
+ return comet_ml.ExistingExperiment(
+ previous_experiment=experiment_id,
+ **self.default_experiment_kwargs,
+ )
+
+ return comet_ml.Experiment(**self.default_experiment_kwargs)
+
+ except ValueError:
+ logger.warning('COMET WARNING: '
+ 'Comet credentials have not been set. '
+ 'Comet will default to offline logging. '
+ 'Please set your credentials to enable online logging.')
+ return self._get_experiment('offline', experiment_id)
+
+ return
+
+ def log_metrics(self, log_dict, **kwargs):
+ self.experiment.log_metrics(log_dict, **kwargs)
+
+ def log_parameters(self, log_dict, **kwargs):
+ self.experiment.log_parameters(log_dict, **kwargs)
+
+ def log_asset(self, asset_path, **kwargs):
+ self.experiment.log_asset(asset_path, **kwargs)
+
+ def log_asset_data(self, asset, **kwargs):
+ self.experiment.log_asset_data(asset, **kwargs)
+
+ def log_image(self, img, **kwargs):
+ self.experiment.log_image(img, **kwargs)
+
+ def log_model(self, path, opt, epoch, fitness_score, best_model=False):
+ if not self.save_model:
+ return
+
+ model_metadata = {
+ 'fitness_score': fitness_score[-1],
+ 'epochs_trained': epoch + 1,
+ 'save_period': opt.save_period,
+ 'total_epochs': opt.epochs, }
+
+ model_files = glob.glob(f'{path}/*.pt')
+ for model_path in model_files:
+ name = Path(model_path).name
+
+ self.experiment.log_model(
+ self.model_name,
+ file_or_folder=model_path,
+ file_name=name,
+ metadata=model_metadata,
+ overwrite=True,
+ )
+
+ def check_dataset(self, data_file):
+ with open(data_file) as f:
+ data_config = yaml.safe_load(f)
+
+ path = data_config.get('path')
+ if path and path.startswith(COMET_PREFIX):
+ path = data_config['path'].replace(COMET_PREFIX, '')
+ data_dict = self.download_dataset_artifact(path)
+
+ return data_dict
+
+ self.log_asset(self.opt.data, metadata={'type': 'data-config-file'})
+
+ return check_dataset(data_file)
+
+ def log_predictions(self, image, labelsn, path, shape, predn):
+ if self.logged_images_count >= self.max_images:
+ return
+ detections = predn[predn[:, 4] > self.conf_thres]
+ iou = box_iou(labelsn[:, 1:], detections[:, :4])
+ mask, _ = torch.where(iou > self.iou_thres)
+ if len(mask) == 0:
+ return
+
+ filtered_detections = detections[mask]
+ filtered_labels = labelsn[mask]
+
+ image_id = path.split('/')[-1].split('.')[0]
+ image_name = f'{image_id}_curr_epoch_{self.experiment.curr_epoch}'
+ if image_name not in self.logged_image_names:
+ native_scale_image = PIL.Image.open(path)
+ self.log_image(native_scale_image, name=image_name)
+ self.logged_image_names.append(image_name)
+
+ metadata = []
+ for cls, *xyxy in filtered_labels.tolist():
+ metadata.append({
+ 'label': f'{self.class_names[int(cls)]}-gt',
+ 'score': 100,
+ 'box': {
+ 'x': xyxy[0],
+ 'y': xyxy[1],
+ 'x2': xyxy[2],
+ 'y2': xyxy[3]}, })
+ for *xyxy, conf, cls in filtered_detections.tolist():
+ metadata.append({
+ 'label': f'{self.class_names[int(cls)]}',
+ 'score': conf * 100,
+ 'box': {
+ 'x': xyxy[0],
+ 'y': xyxy[1],
+ 'x2': xyxy[2],
+ 'y2': xyxy[3]}, })
+
+ self.metadata_dict[image_name] = metadata
+ self.logged_images_count += 1
+
+ return
+
+ def preprocess_prediction(self, image, labels, shape, pred):
+ nl, _ = labels.shape[0], pred.shape[0]
+
+ # Predictions
+ if self.opt.single_cls:
+ pred[:, 5] = 0
+
+ predn = pred.clone()
+ scale_boxes(image.shape[1:], predn[:, :4], shape[0], shape[1])
+
+ labelsn = None
+ if nl:
+ tbox = xywh2xyxy(labels[:, 1:5]) # target boxes
+ scale_boxes(image.shape[1:], tbox, shape[0], shape[1]) # native-space labels
+ labelsn = torch.cat((labels[:, 0:1], tbox), 1) # native-space labels
+ scale_boxes(image.shape[1:], predn[:, :4], shape[0], shape[1]) # native-space pred
+
+ return predn, labelsn
+
+ def add_assets_to_artifact(self, artifact, path, asset_path, split):
+ img_paths = sorted(glob.glob(f'{asset_path}/*'))
+ label_paths = img2label_paths(img_paths)
+
+ for image_file, label_file in zip(img_paths, label_paths):
+ image_logical_path, label_logical_path = map(lambda x: os.path.relpath(x, path), [image_file, label_file])
+
+ try:
+ artifact.add(
+ image_file,
+ logical_path=image_logical_path,
+ metadata={'split': split},
+ )
+ artifact.add(
+ label_file,
+ logical_path=label_logical_path,
+ metadata={'split': split},
+ )
+ except ValueError as e:
+ logger.error('COMET ERROR: Error adding file to Artifact. Skipping file.')
+ logger.error(f'COMET ERROR: {e}')
+ continue
+
+ return artifact
+
+ def upload_dataset_artifact(self):
+ dataset_name = self.data_dict.get('dataset_name', 'yolov5-dataset')
+ path = str((ROOT / Path(self.data_dict['path'])).resolve())
+
+ metadata = self.data_dict.copy()
+ for key in ['train', 'val', 'test']:
+ split_path = metadata.get(key)
+ if split_path is not None:
+ metadata[key] = split_path.replace(path, '')
+
+ artifact = comet_ml.Artifact(name=dataset_name, artifact_type='dataset', metadata=metadata)
+ for key in metadata.keys():
+ if key in ['train', 'val', 'test']:
+ if isinstance(self.upload_dataset, str) and (key != self.upload_dataset):
+ continue
+
+ asset_path = self.data_dict.get(key)
+ if asset_path is not None:
+ artifact = self.add_assets_to_artifact(artifact, path, asset_path, key)
+
+ self.experiment.log_artifact(artifact)
+
+ return
+
+ def download_dataset_artifact(self, artifact_path):
+ logged_artifact = self.experiment.get_artifact(artifact_path)
+ artifact_save_dir = str(Path(self.opt.save_dir) / logged_artifact.name)
+ logged_artifact.download(artifact_save_dir)
+
+ metadata = logged_artifact.metadata
+ data_dict = metadata.copy()
+ data_dict['path'] = artifact_save_dir
+
+ metadata_names = metadata.get('names')
+ if type(metadata_names) == dict:
+ data_dict['names'] = {int(k): v for k, v in metadata.get('names').items()}
+ elif type(metadata_names) == list:
+ data_dict['names'] = {int(k): v for k, v in zip(range(len(metadata_names)), metadata_names)}
+ else:
+ raise "Invalid 'names' field in dataset yaml file. Please use a list or dictionary"
+
+ data_dict = self.update_data_paths(data_dict)
+ return data_dict
+
+ def update_data_paths(self, data_dict):
+ path = data_dict.get('path', '')
+
+ for split in ['train', 'val', 'test']:
+ if data_dict.get(split):
+ split_path = data_dict.get(split)
+ data_dict[split] = (f'{path}/{split_path}' if isinstance(split, str) else [
+ f'{path}/{x}' for x in split_path])
+
+ return data_dict
+
+ def on_pretrain_routine_end(self, paths):
+ if self.opt.resume:
+ return
+
+ for path in paths:
+ self.log_asset(str(path))
+
+ if self.upload_dataset:
+ if not self.resume:
+ self.upload_dataset_artifact()
+
+ return
+
+ def on_train_start(self):
+ self.log_parameters(self.hyp)
+
+ def on_train_epoch_start(self):
+ return
+
+ def on_train_epoch_end(self, epoch):
+ self.experiment.curr_epoch = epoch
+
+ return
+
+ def on_train_batch_start(self):
+ return
+
+ def on_train_batch_end(self, log_dict, step):
+ self.experiment.curr_step = step
+ if self.log_batch_metrics and (step % self.comet_log_batch_interval == 0):
+ self.log_metrics(log_dict, step=step)
+
+ return
+
+ def on_train_end(self, files, save_dir, last, best, epoch, results):
+ if self.comet_log_predictions:
+ curr_epoch = self.experiment.curr_epoch
+ self.experiment.log_asset_data(self.metadata_dict, 'image-metadata.json', epoch=curr_epoch)
+
+ for f in files:
+ self.log_asset(f, metadata={'epoch': epoch})
+ self.log_asset(f'{save_dir}/results.csv', metadata={'epoch': epoch})
+
+ if not self.opt.evolve:
+ model_path = str(best if best.exists() else last)
+ name = Path(model_path).name
+ if self.save_model:
+ self.experiment.log_model(
+ self.model_name,
+ file_or_folder=model_path,
+ file_name=name,
+ overwrite=True,
+ )
+
+ # Check if running Experiment with Comet Optimizer
+ if hasattr(self.opt, 'comet_optimizer_id'):
+ metric = results.get(self.opt.comet_optimizer_metric)
+ self.experiment.log_other('optimizer_metric_value', metric)
+
+ self.finish_run()
+
+ def on_val_start(self):
+ return
+
+ def on_val_batch_start(self):
+ return
+
+ def on_val_batch_end(self, batch_i, images, targets, paths, shapes, outputs):
+ if not (self.comet_log_predictions and ((batch_i + 1) % self.comet_log_prediction_interval == 0)):
+ return
+
+ for si, pred in enumerate(outputs):
+ if len(pred) == 0:
+ continue
+
+ image = images[si]
+ labels = targets[targets[:, 0] == si, 1:]
+ shape = shapes[si]
+ path = paths[si]
+ predn, labelsn = self.preprocess_prediction(image, labels, shape, pred)
+ if labelsn is not None:
+ self.log_predictions(image, labelsn, path, shape, predn)
+
+ return
+
+ def on_val_end(self, nt, tp, fp, p, r, f1, ap, ap50, ap_class, confusion_matrix):
+ if self.comet_log_per_class_metrics:
+ if self.num_classes > 1:
+ for i, c in enumerate(ap_class):
+ class_name = self.class_names[c]
+ self.experiment.log_metrics(
+ {
+ 'mAP@.5': ap50[i],
+ 'mAP@.5:.95': ap[i],
+ 'precision': p[i],
+ 'recall': r[i],
+ 'f1': f1[i],
+ 'true_positives': tp[i],
+ 'false_positives': fp[i],
+ 'support': nt[c], },
+ prefix=class_name,
+ )
+
+ if self.comet_log_confusion_matrix:
+ epoch = self.experiment.curr_epoch
+ class_names = list(self.class_names.values())
+ class_names.append('background')
+ num_classes = len(class_names)
+
+ self.experiment.log_confusion_matrix(
+ matrix=confusion_matrix.matrix,
+ max_categories=num_classes,
+ labels=class_names,
+ epoch=epoch,
+ column_label='Actual Category',
+ row_label='Predicted Category',
+ file_name=f'confusion-matrix-epoch-{epoch}.json',
+ )
+
+ def on_fit_epoch_end(self, result, epoch):
+ self.log_metrics(result, epoch=epoch)
+
+ def on_model_save(self, last, epoch, final_epoch, best_fitness, fi):
+ if ((epoch + 1) % self.opt.save_period == 0 and not final_epoch) and self.opt.save_period != -1:
+ self.log_model(last.parent, self.opt, epoch, fi, best_model=best_fitness == fi)
+
+ def on_params_update(self, params):
+ self.log_parameters(params)
+
+ def finish_run(self):
+ self.experiment.end()
diff --git a/TextDetection/utils/loggers/comet/comet_utils.py b/TextDetection/utils/loggers/comet/comet_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..27600761ad2843a6ab66aa22ad06782bb4b7eea7
--- /dev/null
+++ b/TextDetection/utils/loggers/comet/comet_utils.py
@@ -0,0 +1,150 @@
+import logging
+import os
+from urllib.parse import urlparse
+
+try:
+ import comet_ml
+except (ModuleNotFoundError, ImportError):
+ comet_ml = None
+
+import yaml
+
+logger = logging.getLogger(__name__)
+
+COMET_PREFIX = 'comet://'
+COMET_MODEL_NAME = os.getenv('COMET_MODEL_NAME', 'yolov5')
+COMET_DEFAULT_CHECKPOINT_FILENAME = os.getenv('COMET_DEFAULT_CHECKPOINT_FILENAME', 'last.pt')
+
+
+def download_model_checkpoint(opt, experiment):
+ model_dir = f'{opt.project}/{experiment.name}'
+ os.makedirs(model_dir, exist_ok=True)
+
+ model_name = COMET_MODEL_NAME
+ model_asset_list = experiment.get_model_asset_list(model_name)
+
+ if len(model_asset_list) == 0:
+ logger.error(f'COMET ERROR: No checkpoints found for model name : {model_name}')
+ return
+
+ model_asset_list = sorted(
+ model_asset_list,
+ key=lambda x: x['step'],
+ reverse=True,
+ )
+ logged_checkpoint_map = {asset['fileName']: asset['assetId'] for asset in model_asset_list}
+
+ resource_url = urlparse(opt.weights)
+ checkpoint_filename = resource_url.query
+
+ if checkpoint_filename:
+ asset_id = logged_checkpoint_map.get(checkpoint_filename)
+ else:
+ asset_id = logged_checkpoint_map.get(COMET_DEFAULT_CHECKPOINT_FILENAME)
+ checkpoint_filename = COMET_DEFAULT_CHECKPOINT_FILENAME
+
+ if asset_id is None:
+ logger.error(f'COMET ERROR: Checkpoint {checkpoint_filename} not found in the given Experiment')
+ return
+
+ try:
+ logger.info(f'COMET INFO: Downloading checkpoint {checkpoint_filename}')
+ asset_filename = checkpoint_filename
+
+ model_binary = experiment.get_asset(asset_id, return_type='binary', stream=False)
+ model_download_path = f'{model_dir}/{asset_filename}'
+ with open(model_download_path, 'wb') as f:
+ f.write(model_binary)
+
+ opt.weights = model_download_path
+
+ except Exception as e:
+ logger.warning('COMET WARNING: Unable to download checkpoint from Comet')
+ logger.exception(e)
+
+
+def set_opt_parameters(opt, experiment):
+ """Update the opts Namespace with parameters
+ from Comet's ExistingExperiment when resuming a run
+
+ Args:
+ opt (argparse.Namespace): Namespace of command line options
+ experiment (comet_ml.APIExperiment): Comet API Experiment object
+ """
+ asset_list = experiment.get_asset_list()
+ resume_string = opt.resume
+
+ for asset in asset_list:
+ if asset['fileName'] == 'opt.yaml':
+ asset_id = asset['assetId']
+ asset_binary = experiment.get_asset(asset_id, return_type='binary', stream=False)
+ opt_dict = yaml.safe_load(asset_binary)
+ for key, value in opt_dict.items():
+ setattr(opt, key, value)
+ opt.resume = resume_string
+
+ # Save hyperparameters to YAML file
+ # Necessary to pass checks in training script
+ save_dir = f'{opt.project}/{experiment.name}'
+ os.makedirs(save_dir, exist_ok=True)
+
+ hyp_yaml_path = f'{save_dir}/hyp.yaml'
+ with open(hyp_yaml_path, 'w') as f:
+ yaml.dump(opt.hyp, f)
+ opt.hyp = hyp_yaml_path
+
+
+def check_comet_weights(opt):
+ """Downloads model weights from Comet and updates the
+ weights path to point to saved weights location
+
+ Args:
+ opt (argparse.Namespace): Command Line arguments passed
+ to YOLOv5 training script
+
+ Returns:
+ None/bool: Return True if weights are successfully downloaded
+ else return None
+ """
+ if comet_ml is None:
+ return
+
+ if isinstance(opt.weights, str):
+ if opt.weights.startswith(COMET_PREFIX):
+ api = comet_ml.API()
+ resource = urlparse(opt.weights)
+ experiment_path = f'{resource.netloc}{resource.path}'
+ experiment = api.get(experiment_path)
+ download_model_checkpoint(opt, experiment)
+ return True
+
+ return None
+
+
+def check_comet_resume(opt):
+ """Restores run parameters to its original state based on the model checkpoint
+ and logged Experiment parameters.
+
+ Args:
+ opt (argparse.Namespace): Command Line arguments passed
+ to YOLOv5 training script
+
+ Returns:
+ None/bool: Return True if the run is restored successfully
+ else return None
+ """
+ if comet_ml is None:
+ return
+
+ if isinstance(opt.resume, str):
+ if opt.resume.startswith(COMET_PREFIX):
+ api = comet_ml.API()
+ resource = urlparse(opt.resume)
+ experiment_path = f'{resource.netloc}{resource.path}'
+ experiment = api.get(experiment_path)
+ set_opt_parameters(opt, experiment)
+ download_model_checkpoint(opt, experiment)
+
+ return True
+
+ return None
diff --git a/TextDetection/utils/loggers/comet/hpo.py b/TextDetection/utils/loggers/comet/hpo.py
new file mode 100644
index 0000000000000000000000000000000000000000..fc49115c13581554bebe1ddddaf3d5e10caaae07
--- /dev/null
+++ b/TextDetection/utils/loggers/comet/hpo.py
@@ -0,0 +1,118 @@
+import argparse
+import json
+import logging
+import os
+import sys
+from pathlib import Path
+
+import comet_ml
+
+logger = logging.getLogger(__name__)
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[3] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+
+from train import train
+from utils.callbacks import Callbacks
+from utils.general import increment_path
+from utils.torch_utils import select_device
+
+# Project Configuration
+config = comet_ml.config.get_config()
+COMET_PROJECT_NAME = config.get_string(os.getenv('COMET_PROJECT_NAME'), 'comet.project_name', default='yolov5')
+
+
+def get_args(known=False):
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--weights', type=str, default=ROOT / 'yolov5s.pt', help='initial weights path')
+ parser.add_argument('--cfg', type=str, default='', help='model.yaml path')
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='dataset.yaml path')
+ parser.add_argument('--hyp', type=str, default=ROOT / 'data/hyps/hyp.scratch-low.yaml', help='hyperparameters path')
+ parser.add_argument('--epochs', type=int, default=300, help='total training epochs')
+ parser.add_argument('--batch-size', type=int, default=16, help='total batch size for all GPUs, -1 for autobatch')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='train, val image size (pixels)')
+ parser.add_argument('--rect', action='store_true', help='rectangular training')
+ parser.add_argument('--resume', nargs='?', const=True, default=False, help='resume most recent training')
+ parser.add_argument('--nosave', action='store_true', help='only save final checkpoint')
+ parser.add_argument('--noval', action='store_true', help='only validate final epoch')
+ parser.add_argument('--noautoanchor', action='store_true', help='disable AutoAnchor')
+ parser.add_argument('--noplots', action='store_true', help='save no plot files')
+ parser.add_argument('--evolve', type=int, nargs='?', const=300, help='evolve hyperparameters for x generations')
+ parser.add_argument('--bucket', type=str, default='', help='gsutil bucket')
+ parser.add_argument('--cache', type=str, nargs='?', const='ram', help='--cache images in "ram" (default) or "disk"')
+ parser.add_argument('--image-weights', action='store_true', help='use weighted image selection for training')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--multi-scale', action='store_true', help='vary img-size +/- 50%%')
+ parser.add_argument('--single-cls', action='store_true', help='train multi-class data as single-class')
+ parser.add_argument('--optimizer', type=str, choices=['SGD', 'Adam', 'AdamW'], default='SGD', help='optimizer')
+ parser.add_argument('--sync-bn', action='store_true', help='use SyncBatchNorm, only available in DDP mode')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--project', default=ROOT / 'runs/train', help='save to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--quad', action='store_true', help='quad dataloader')
+ parser.add_argument('--cos-lr', action='store_true', help='cosine LR scheduler')
+ parser.add_argument('--label-smoothing', type=float, default=0.0, help='Label smoothing epsilon')
+ parser.add_argument('--patience', type=int, default=100, help='EarlyStopping patience (epochs without improvement)')
+ parser.add_argument('--freeze', nargs='+', type=int, default=[0], help='Freeze layers: backbone=10, first3=0 1 2')
+ parser.add_argument('--save-period', type=int, default=-1, help='Save checkpoint every x epochs (disabled if < 1)')
+ parser.add_argument('--seed', type=int, default=0, help='Global training seed')
+ parser.add_argument('--local_rank', type=int, default=-1, help='Automatic DDP Multi-GPU argument, do not modify')
+
+ # Weights & Biases arguments
+ parser.add_argument('--entity', default=None, help='W&B: Entity')
+ parser.add_argument('--upload_dataset', nargs='?', const=True, default=False, help='W&B: Upload data, "val" option')
+ parser.add_argument('--bbox_interval', type=int, default=-1, help='W&B: Set bounding-box image logging interval')
+ parser.add_argument('--artifact_alias', type=str, default='latest', help='W&B: Version of dataset artifact to use')
+
+ # Comet Arguments
+ parser.add_argument('--comet_optimizer_config', type=str, help='Comet: Path to a Comet Optimizer Config File.')
+ parser.add_argument('--comet_optimizer_id', type=str, help='Comet: ID of the Comet Optimizer sweep.')
+ parser.add_argument('--comet_optimizer_objective', type=str, help="Comet: Set to 'minimize' or 'maximize'.")
+ parser.add_argument('--comet_optimizer_metric', type=str, help='Comet: Metric to Optimize.')
+ parser.add_argument('--comet_optimizer_workers',
+ type=int,
+ default=1,
+ help='Comet: Number of Parallel Workers to use with the Comet Optimizer.')
+
+ return parser.parse_known_args()[0] if known else parser.parse_args()
+
+
+def run(parameters, opt):
+ hyp_dict = {k: v for k, v in parameters.items() if k not in ['epochs', 'batch_size']}
+
+ opt.save_dir = str(increment_path(Path(opt.project) / opt.name, exist_ok=opt.exist_ok or opt.evolve))
+ opt.batch_size = parameters.get('batch_size')
+ opt.epochs = parameters.get('epochs')
+
+ device = select_device(opt.device, batch_size=opt.batch_size)
+ train(hyp_dict, opt, device, callbacks=Callbacks())
+
+
+if __name__ == '__main__':
+ opt = get_args(known=True)
+
+ opt.weights = str(opt.weights)
+ opt.cfg = str(opt.cfg)
+ opt.data = str(opt.data)
+ opt.project = str(opt.project)
+
+ optimizer_id = os.getenv('COMET_OPTIMIZER_ID')
+ if optimizer_id is None:
+ with open(opt.comet_optimizer_config) as f:
+ optimizer_config = json.load(f)
+ optimizer = comet_ml.Optimizer(optimizer_config)
+ else:
+ optimizer = comet_ml.Optimizer(optimizer_id)
+
+ opt.comet_optimizer_id = optimizer.id
+ status = optimizer.status()
+
+ opt.comet_optimizer_objective = status['spec']['objective']
+ opt.comet_optimizer_metric = status['spec']['metric']
+
+ logger.info('COMET INFO: Starting Hyperparameter Sweep')
+ for parameter in optimizer.get_parameters():
+ run(parameter['parameters'], opt)
diff --git a/TextDetection/utils/loggers/comet/optimizer_config.json b/TextDetection/utils/loggers/comet/optimizer_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..83ddddab6f2084b4bdf84dca1e61696de200d1b8
--- /dev/null
+++ b/TextDetection/utils/loggers/comet/optimizer_config.json
@@ -0,0 +1,209 @@
+{
+ "algorithm": "random",
+ "parameters": {
+ "anchor_t": {
+ "type": "discrete",
+ "values": [
+ 2,
+ 8
+ ]
+ },
+ "batch_size": {
+ "type": "discrete",
+ "values": [
+ 16,
+ 32,
+ 64
+ ]
+ },
+ "box": {
+ "type": "discrete",
+ "values": [
+ 0.02,
+ 0.2
+ ]
+ },
+ "cls": {
+ "type": "discrete",
+ "values": [
+ 0.2
+ ]
+ },
+ "cls_pw": {
+ "type": "discrete",
+ "values": [
+ 0.5
+ ]
+ },
+ "copy_paste": {
+ "type": "discrete",
+ "values": [
+ 1
+ ]
+ },
+ "degrees": {
+ "type": "discrete",
+ "values": [
+ 0,
+ 45
+ ]
+ },
+ "epochs": {
+ "type": "discrete",
+ "values": [
+ 5
+ ]
+ },
+ "fl_gamma": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "fliplr": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "flipud": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "hsv_h": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "hsv_s": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "hsv_v": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "iou_t": {
+ "type": "discrete",
+ "values": [
+ 0.7
+ ]
+ },
+ "lr0": {
+ "type": "discrete",
+ "values": [
+ 1e-05,
+ 0.1
+ ]
+ },
+ "lrf": {
+ "type": "discrete",
+ "values": [
+ 0.01,
+ 1
+ ]
+ },
+ "mixup": {
+ "type": "discrete",
+ "values": [
+ 1
+ ]
+ },
+ "momentum": {
+ "type": "discrete",
+ "values": [
+ 0.6
+ ]
+ },
+ "mosaic": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "obj": {
+ "type": "discrete",
+ "values": [
+ 0.2
+ ]
+ },
+ "obj_pw": {
+ "type": "discrete",
+ "values": [
+ 0.5
+ ]
+ },
+ "optimizer": {
+ "type": "categorical",
+ "values": [
+ "SGD",
+ "Adam",
+ "AdamW"
+ ]
+ },
+ "perspective": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "scale": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "shear": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "translate": {
+ "type": "discrete",
+ "values": [
+ 0
+ ]
+ },
+ "warmup_bias_lr": {
+ "type": "discrete",
+ "values": [
+ 0,
+ 0.2
+ ]
+ },
+ "warmup_epochs": {
+ "type": "discrete",
+ "values": [
+ 5
+ ]
+ },
+ "warmup_momentum": {
+ "type": "discrete",
+ "values": [
+ 0,
+ 0.95
+ ]
+ },
+ "weight_decay": {
+ "type": "discrete",
+ "values": [
+ 0,
+ 0.001
+ ]
+ }
+ },
+ "spec": {
+ "maxCombo": 0,
+ "metric": "metrics/mAP_0.5",
+ "objective": "maximize"
+ },
+ "trials": 1
+}
diff --git a/TextDetection/utils/loggers/wandb/__init__.py b/TextDetection/utils/loggers/wandb/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/TextDetection/utils/loggers/wandb/wandb_utils.py b/TextDetection/utils/loggers/wandb/wandb_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..4ea32b1d4c6ec62920a9e90af085346d0f7a5f2c
--- /dev/null
+++ b/TextDetection/utils/loggers/wandb/wandb_utils.py
@@ -0,0 +1,193 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+
+# WARNING ⚠️ wandb is deprecated and will be removed in future release.
+# See supported integrations at https://github.com/ultralytics/yolov5#integrations
+
+import logging
+import os
+import sys
+from contextlib import contextmanager
+from pathlib import Path
+
+from utils.general import LOGGER, colorstr
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[3] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+RANK = int(os.getenv('RANK', -1))
+DEPRECATION_WARNING = f"{colorstr('wandb')}: WARNING ⚠️ wandb is deprecated and will be removed in a future release. " \
+ f'See supported integrations at https://github.com/ultralytics/yolov5#integrations.'
+
+try:
+ import wandb
+
+ assert hasattr(wandb, '__version__') # verify package import not local dir
+ LOGGER.warning(DEPRECATION_WARNING)
+except (ImportError, AssertionError):
+ wandb = None
+
+
+class WandbLogger():
+ """Log training runs, datasets, models, and predictions to Weights & Biases.
+
+ This logger sends information to W&B at wandb.ai. By default, this information
+ includes hyperparameters, system configuration and metrics, model metrics,
+ and basic data metrics and analyses.
+
+ By providing additional command line arguments to train.py, datasets,
+ models and predictions can also be logged.
+
+ For more on how this logger is used, see the Weights & Biases documentation:
+ https://docs.wandb.com/guides/integrations/yolov5
+ """
+
+ def __init__(self, opt, run_id=None, job_type='Training'):
+ """
+ - Initialize WandbLogger instance
+ - Upload dataset if opt.upload_dataset is True
+ - Setup training processes if job_type is 'Training'
+
+ arguments:
+ opt (namespace) -- Commandline arguments for this run
+ run_id (str) -- Run ID of W&B run to be resumed
+ job_type (str) -- To set the job_type for this run
+
+ """
+ # Pre-training routine --
+ self.job_type = job_type
+ self.wandb, self.wandb_run = wandb, wandb.run if wandb else None
+ self.val_artifact, self.train_artifact = None, None
+ self.train_artifact_path, self.val_artifact_path = None, None
+ self.result_artifact = None
+ self.val_table, self.result_table = None, None
+ self.max_imgs_to_log = 16
+ self.data_dict = None
+ if self.wandb:
+ self.wandb_run = wandb.init(config=opt,
+ resume='allow',
+ project='YOLOv5' if opt.project == 'runs/train' else Path(opt.project).stem,
+ entity=opt.entity,
+ name=opt.name if opt.name != 'exp' else None,
+ job_type=job_type,
+ id=run_id,
+ allow_val_change=True) if not wandb.run else wandb.run
+
+ if self.wandb_run:
+ if self.job_type == 'Training':
+ if isinstance(opt.data, dict):
+ # This means another dataset manager has already processed the dataset info (e.g. ClearML)
+ # and they will have stored the already processed dict in opt.data
+ self.data_dict = opt.data
+ self.setup_training(opt)
+
+ def setup_training(self, opt):
+ """
+ Setup the necessary processes for training YOLO models:
+ - Attempt to download model checkpoint and dataset artifacts if opt.resume stats with WANDB_ARTIFACT_PREFIX
+ - Update data_dict, to contain info of previous run if resumed and the paths of dataset artifact if downloaded
+ - Setup log_dict, initialize bbox_interval
+
+ arguments:
+ opt (namespace) -- commandline arguments for this run
+
+ """
+ self.log_dict, self.current_epoch = {}, 0
+ self.bbox_interval = opt.bbox_interval
+ if isinstance(opt.resume, str):
+ model_dir, _ = self.download_model_artifact(opt)
+ if model_dir:
+ self.weights = Path(model_dir) / 'last.pt'
+ config = self.wandb_run.config
+ opt.weights, opt.save_period, opt.batch_size, opt.bbox_interval, opt.epochs, opt.hyp, opt.imgsz = str(
+ self.weights), config.save_period, config.batch_size, config.bbox_interval, config.epochs, \
+ config.hyp, config.imgsz
+
+ if opt.bbox_interval == -1:
+ self.bbox_interval = opt.bbox_interval = (opt.epochs // 10) if opt.epochs > 10 else 1
+ if opt.evolve or opt.noplots:
+ self.bbox_interval = opt.bbox_interval = opt.epochs + 1 # disable bbox_interval
+
+ def log_model(self, path, opt, epoch, fitness_score, best_model=False):
+ """
+ Log the model checkpoint as W&B artifact
+
+ arguments:
+ path (Path) -- Path of directory containing the checkpoints
+ opt (namespace) -- Command line arguments for this run
+ epoch (int) -- Current epoch number
+ fitness_score (float) -- fitness score for current epoch
+ best_model (boolean) -- Boolean representing if the current checkpoint is the best yet.
+ """
+ model_artifact = wandb.Artifact('run_' + wandb.run.id + '_model',
+ type='model',
+ metadata={
+ 'original_url': str(path),
+ 'epochs_trained': epoch + 1,
+ 'save period': opt.save_period,
+ 'project': opt.project,
+ 'total_epochs': opt.epochs,
+ 'fitness_score': fitness_score})
+ model_artifact.add_file(str(path / 'last.pt'), name='last.pt')
+ wandb.log_artifact(model_artifact,
+ aliases=['latest', 'last', 'epoch ' + str(self.current_epoch), 'best' if best_model else ''])
+ LOGGER.info(f'Saving model artifact on epoch {epoch + 1}')
+
+ def val_one_image(self, pred, predn, path, names, im):
+ pass
+
+ def log(self, log_dict):
+ """
+ save the metrics to the logging dictionary
+
+ arguments:
+ log_dict (Dict) -- metrics/media to be logged in current step
+ """
+ if self.wandb_run:
+ for key, value in log_dict.items():
+ self.log_dict[key] = value
+
+ def end_epoch(self):
+ """
+ commit the log_dict, model artifacts and Tables to W&B and flush the log_dict.
+
+ arguments:
+ best_result (boolean): Boolean representing if the result of this evaluation is best or not
+ """
+ if self.wandb_run:
+ with all_logging_disabled():
+ try:
+ wandb.log(self.log_dict)
+ except BaseException as e:
+ LOGGER.info(
+ f'An error occurred in wandb logger. The training will proceed without interruption. More info\n{e}'
+ )
+ self.wandb_run.finish()
+ self.wandb_run = None
+ self.log_dict = {}
+
+ def finish_run(self):
+ """
+ Log metrics if any and finish the current W&B run
+ """
+ if self.wandb_run:
+ if self.log_dict:
+ with all_logging_disabled():
+ wandb.log(self.log_dict)
+ wandb.run.finish()
+ LOGGER.warning(DEPRECATION_WARNING)
+
+
+@contextmanager
+def all_logging_disabled(highest_level=logging.CRITICAL):
+ """ source - https://gist.github.com/simon-weber/7853144
+ A context manager that will prevent any logging messages triggered during the body from being processed.
+ :param highest_level: the maximum logging level in use.
+ This would only need to be changed if a custom level greater than CRITICAL is defined.
+ """
+ previous_level = logging.root.manager.disable
+ logging.disable(highest_level)
+ try:
+ yield
+ finally:
+ logging.disable(previous_level)
diff --git a/TextDetection/utils/loss.py b/TextDetection/utils/loss.py
new file mode 100644
index 0000000000000000000000000000000000000000..26cca8797315a425b26d1c8c083bd321d7b52fff
--- /dev/null
+++ b/TextDetection/utils/loss.py
@@ -0,0 +1,234 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Loss functions
+"""
+
+import torch
+import torch.nn as nn
+
+from utils.metrics import bbox_iou
+from utils.torch_utils import de_parallel
+
+
+def smooth_BCE(eps=0.1): # https://github.com/ultralytics/yolov3/issues/238#issuecomment-598028441
+ # return positive, negative label smoothing BCE targets
+ return 1.0 - 0.5 * eps, 0.5 * eps
+
+
+class BCEBlurWithLogitsLoss(nn.Module):
+ # BCEwithLogitLoss() with reduced missing label effects.
+ def __init__(self, alpha=0.05):
+ super().__init__()
+ self.loss_fcn = nn.BCEWithLogitsLoss(reduction='none') # must be nn.BCEWithLogitsLoss()
+ self.alpha = alpha
+
+ def forward(self, pred, true):
+ loss = self.loss_fcn(pred, true)
+ pred = torch.sigmoid(pred) # prob from logits
+ dx = pred - true # reduce only missing label effects
+ # dx = (pred - true).abs() # reduce missing label and false label effects
+ alpha_factor = 1 - torch.exp((dx - 1) / (self.alpha + 1e-4))
+ loss *= alpha_factor
+ return loss.mean()
+
+
+class FocalLoss(nn.Module):
+ # Wraps focal loss around existing loss_fcn(), i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5)
+ def __init__(self, loss_fcn, gamma=1.5, alpha=0.25):
+ super().__init__()
+ self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss()
+ self.gamma = gamma
+ self.alpha = alpha
+ self.reduction = loss_fcn.reduction
+ self.loss_fcn.reduction = 'none' # required to apply FL to each element
+
+ def forward(self, pred, true):
+ loss = self.loss_fcn(pred, true)
+ # p_t = torch.exp(-loss)
+ # loss *= self.alpha * (1.000001 - p_t) ** self.gamma # non-zero power for gradient stability
+
+ # TF implementation https://github.com/tensorflow/addons/blob/v0.7.1/tensorflow_addons/losses/focal_loss.py
+ pred_prob = torch.sigmoid(pred) # prob from logits
+ p_t = true * pred_prob + (1 - true) * (1 - pred_prob)
+ alpha_factor = true * self.alpha + (1 - true) * (1 - self.alpha)
+ modulating_factor = (1.0 - p_t) ** self.gamma
+ loss *= alpha_factor * modulating_factor
+
+ if self.reduction == 'mean':
+ return loss.mean()
+ elif self.reduction == 'sum':
+ return loss.sum()
+ else: # 'none'
+ return loss
+
+
+class QFocalLoss(nn.Module):
+ # Wraps Quality focal loss around existing loss_fcn(), i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5)
+ def __init__(self, loss_fcn, gamma=1.5, alpha=0.25):
+ super().__init__()
+ self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss()
+ self.gamma = gamma
+ self.alpha = alpha
+ self.reduction = loss_fcn.reduction
+ self.loss_fcn.reduction = 'none' # required to apply FL to each element
+
+ def forward(self, pred, true):
+ loss = self.loss_fcn(pred, true)
+
+ pred_prob = torch.sigmoid(pred) # prob from logits
+ alpha_factor = true * self.alpha + (1 - true) * (1 - self.alpha)
+ modulating_factor = torch.abs(true - pred_prob) ** self.gamma
+ loss *= alpha_factor * modulating_factor
+
+ if self.reduction == 'mean':
+ return loss.mean()
+ elif self.reduction == 'sum':
+ return loss.sum()
+ else: # 'none'
+ return loss
+
+
+class ComputeLoss:
+ sort_obj_iou = False
+
+ # Compute losses
+ def __init__(self, model, autobalance=False):
+ device = next(model.parameters()).device # get model device
+ h = model.hyp # hyperparameters
+
+ # Define criteria
+ BCEcls = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['cls_pw']], device=device))
+ BCEobj = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['obj_pw']], device=device))
+
+ # Class label smoothing https://arxiv.org/pdf/1902.04103.pdf eqn 3
+ self.cp, self.cn = smooth_BCE(eps=h.get('label_smoothing', 0.0)) # positive, negative BCE targets
+
+ # Focal loss
+ g = h['fl_gamma'] # focal loss gamma
+ if g > 0:
+ BCEcls, BCEobj = FocalLoss(BCEcls, g), FocalLoss(BCEobj, g)
+
+ m = de_parallel(model).model[-1] # Detect() module
+ self.balance = {3: [4.0, 1.0, 0.4]}.get(m.nl, [4.0, 1.0, 0.25, 0.06, 0.02]) # P3-P7
+ self.ssi = list(m.stride).index(16) if autobalance else 0 # stride 16 index
+ self.BCEcls, self.BCEobj, self.gr, self.hyp, self.autobalance = BCEcls, BCEobj, 1.0, h, autobalance
+ self.na = m.na # number of anchors
+ self.nc = m.nc # number of classes
+ self.nl = m.nl # number of layers
+ self.anchors = m.anchors
+ self.device = device
+
+ def __call__(self, p, targets): # predictions, targets
+ lcls = torch.zeros(1, device=self.device) # class loss
+ lbox = torch.zeros(1, device=self.device) # box loss
+ lobj = torch.zeros(1, device=self.device) # object loss
+ tcls, tbox, indices, anchors = self.build_targets(p, targets) # targets
+
+ # Losses
+ for i, pi in enumerate(p): # layer index, layer predictions
+ b, a, gj, gi = indices[i] # image, anchor, gridy, gridx
+ tobj = torch.zeros(pi.shape[:4], dtype=pi.dtype, device=self.device) # target obj
+
+ n = b.shape[0] # number of targets
+ if n:
+ # pxy, pwh, _, pcls = pi[b, a, gj, gi].tensor_split((2, 4, 5), dim=1) # faster, requires torch 1.8.0
+ pxy, pwh, _, pcls = pi[b, a, gj, gi].split((2, 2, 1, self.nc), 1) # target-subset of predictions
+
+ # Regression
+ pxy = pxy.sigmoid() * 2 - 0.5
+ pwh = (pwh.sigmoid() * 2) ** 2 * anchors[i]
+ pbox = torch.cat((pxy, pwh), 1) # predicted box
+ iou = bbox_iou(pbox, tbox[i], CIoU=True).squeeze() # iou(prediction, target)
+ lbox += (1.0 - iou).mean() # iou loss
+
+ # Objectness
+ iou = iou.detach().clamp(0).type(tobj.dtype)
+ if self.sort_obj_iou:
+ j = iou.argsort()
+ b, a, gj, gi, iou = b[j], a[j], gj[j], gi[j], iou[j]
+ if self.gr < 1:
+ iou = (1.0 - self.gr) + self.gr * iou
+ tobj[b, a, gj, gi] = iou # iou ratio
+
+ # Classification
+ if self.nc > 1: # cls loss (only if multiple classes)
+ t = torch.full_like(pcls, self.cn, device=self.device) # targets
+ t[range(n), tcls[i]] = self.cp
+ lcls += self.BCEcls(pcls, t) # BCE
+
+ # Append targets to text file
+ # with open('targets.txt', 'a') as file:
+ # [file.write('%11.5g ' * 4 % tuple(x) + '\n') for x in torch.cat((txy[i], twh[i]), 1)]
+
+ obji = self.BCEobj(pi[..., 4], tobj)
+ lobj += obji * self.balance[i] # obj loss
+ if self.autobalance:
+ self.balance[i] = self.balance[i] * 0.9999 + 0.0001 / obji.detach().item()
+
+ if self.autobalance:
+ self.balance = [x / self.balance[self.ssi] for x in self.balance]
+ lbox *= self.hyp['box']
+ lobj *= self.hyp['obj']
+ lcls *= self.hyp['cls']
+ bs = tobj.shape[0] # batch size
+
+ return (lbox + lobj + lcls) * bs, torch.cat((lbox, lobj, lcls)).detach()
+
+ def build_targets(self, p, targets):
+ # Build targets for compute_loss(), input targets(image,class,x,y,w,h)
+ na, nt = self.na, targets.shape[0] # number of anchors, targets
+ tcls, tbox, indices, anch = [], [], [], []
+ gain = torch.ones(7, device=self.device) # normalized to gridspace gain
+ ai = torch.arange(na, device=self.device).float().view(na, 1).repeat(1, nt) # same as .repeat_interleave(nt)
+ targets = torch.cat((targets.repeat(na, 1, 1), ai[..., None]), 2) # append anchor indices
+
+ g = 0.5 # bias
+ off = torch.tensor(
+ [
+ [0, 0],
+ [1, 0],
+ [0, 1],
+ [-1, 0],
+ [0, -1], # j,k,l,m
+ # [1, 1], [1, -1], [-1, 1], [-1, -1], # jk,jm,lk,lm
+ ],
+ device=self.device).float() * g # offsets
+
+ for i in range(self.nl):
+ anchors, shape = self.anchors[i], p[i].shape
+ gain[2:6] = torch.tensor(shape)[[3, 2, 3, 2]] # xyxy gain
+
+ # Match targets to anchors
+ t = targets * gain # shape(3,n,7)
+ if nt:
+ # Matches
+ r = t[..., 4:6] / anchors[:, None] # wh ratio
+ j = torch.max(r, 1 / r).max(2)[0] < self.hyp['anchor_t'] # compare
+ # j = wh_iou(anchors, t[:, 4:6]) > model.hyp['iou_t'] # iou(3,n)=wh_iou(anchors(3,2), gwh(n,2))
+ t = t[j] # filter
+
+ # Offsets
+ gxy = t[:, 2:4] # grid xy
+ gxi = gain[[2, 3]] - gxy # inverse
+ j, k = ((gxy % 1 < g) & (gxy > 1)).T
+ l, m = ((gxi % 1 < g) & (gxi > 1)).T
+ j = torch.stack((torch.ones_like(j), j, k, l, m))
+ t = t.repeat((5, 1, 1))[j]
+ offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j]
+ else:
+ t = targets[0]
+ offsets = 0
+
+ # Define
+ bc, gxy, gwh, a = t.chunk(4, 1) # (image, class), grid xy, grid wh, anchors
+ a, (b, c) = a.long().view(-1), bc.long().T # anchors, image, class
+ gij = (gxy - offsets).long()
+ gi, gj = gij.T # grid indices
+
+ # Append
+ indices.append((b, a, gj.clamp_(0, shape[2] - 1), gi.clamp_(0, shape[3] - 1))) # image, anchor, grid
+ tbox.append(torch.cat((gxy - gij, gwh), 1)) # box
+ anch.append(anchors[a]) # anchors
+ tcls.append(c) # class
+
+ return tcls, tbox, indices, anch
diff --git a/TextDetection/utils/metrics.py b/TextDetection/utils/metrics.py
new file mode 100644
index 0000000000000000000000000000000000000000..5646f40e9860f90648e1dc8d074277de9b827b97
--- /dev/null
+++ b/TextDetection/utils/metrics.py
@@ -0,0 +1,360 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Model validation metrics
+"""
+
+import math
+import warnings
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+import torch
+
+from utils import TryExcept, threaded
+
+
+def fitness(x):
+ # Model fitness as a weighted combination of metrics
+ w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]
+ return (x[:, :4] * w).sum(1)
+
+
+def smooth(y, f=0.05):
+ # Box filter of fraction f
+ nf = round(len(y) * f * 2) // 2 + 1 # number of filter elements (must be odd)
+ p = np.ones(nf // 2) # ones padding
+ yp = np.concatenate((p * y[0], y, p * y[-1]), 0) # y padded
+ return np.convolve(yp, np.ones(nf) / nf, mode='valid') # y-smoothed
+
+
+def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, save_dir='.', names=(), eps=1e-16, prefix=''):
+ """ Compute the average precision, given the recall and precision curves.
+ Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
+ # Arguments
+ tp: True positives (nparray, nx1 or nx10).
+ conf: Objectness value from 0-1 (nparray).
+ pred_cls: Predicted object classes (nparray).
+ target_cls: True object classes (nparray).
+ plot: Plot precision-recall curve at mAP@0.5
+ save_dir: Plot save directory
+ # Returns
+ The average precision as computed in py-faster-rcnn.
+ """
+
+ # Sort by objectness
+ i = np.argsort(-conf)
+ tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
+
+ # Find unique classes
+ unique_classes, nt = np.unique(target_cls, return_counts=True)
+ nc = unique_classes.shape[0] # number of classes, number of detections
+
+ # Create Precision-Recall curve and compute AP for each class
+ px, py = np.linspace(0, 1, 1000), [] # for plotting
+ ap, p, r = np.zeros((nc, tp.shape[1])), np.zeros((nc, 1000)), np.zeros((nc, 1000))
+ for ci, c in enumerate(unique_classes):
+ i = pred_cls == c
+ n_l = nt[ci] # number of labels
+ n_p = i.sum() # number of predictions
+ if n_p == 0 or n_l == 0:
+ continue
+
+ # Accumulate FPs and TPs
+ fpc = (1 - tp[i]).cumsum(0)
+ tpc = tp[i].cumsum(0)
+
+ # Recall
+ recall = tpc / (n_l + eps) # recall curve
+ r[ci] = np.interp(-px, -conf[i], recall[:, 0], left=0) # negative x, xp because xp decreases
+
+ # Precision
+ precision = tpc / (tpc + fpc) # precision curve
+ p[ci] = np.interp(-px, -conf[i], precision[:, 0], left=1) # p at pr_score
+
+ # AP from recall-precision curve
+ for j in range(tp.shape[1]):
+ ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
+ if plot and j == 0:
+ py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5
+
+ # Compute F1 (harmonic mean of precision and recall)
+ f1 = 2 * p * r / (p + r + eps)
+ names = [v for k, v in names.items() if k in unique_classes] # list: only classes that have data
+ names = dict(enumerate(names)) # to dict
+ if plot:
+ plot_pr_curve(px, py, ap, Path(save_dir) / f'{prefix}PR_curve.png', names)
+ plot_mc_curve(px, f1, Path(save_dir) / f'{prefix}F1_curve.png', names, ylabel='F1')
+ plot_mc_curve(px, p, Path(save_dir) / f'{prefix}P_curve.png', names, ylabel='Precision')
+ plot_mc_curve(px, r, Path(save_dir) / f'{prefix}R_curve.png', names, ylabel='Recall')
+
+ i = smooth(f1.mean(0), 0.1).argmax() # max F1 index
+ p, r, f1 = p[:, i], r[:, i], f1[:, i]
+ tp = (r * nt).round() # true positives
+ fp = (tp / (p + eps) - tp).round() # false positives
+ return tp, fp, p, r, f1, ap, unique_classes.astype(int)
+
+
+def compute_ap(recall, precision):
+ """ Compute the average precision, given the recall and precision curves
+ # Arguments
+ recall: The recall curve (list)
+ precision: The precision curve (list)
+ # Returns
+ Average precision, precision curve, recall curve
+ """
+
+ # Append sentinel values to beginning and end
+ mrec = np.concatenate(([0.0], recall, [1.0]))
+ mpre = np.concatenate(([1.0], precision, [0.0]))
+
+ # Compute the precision envelope
+ mpre = np.flip(np.maximum.accumulate(np.flip(mpre)))
+
+ # Integrate area under curve
+ method = 'interp' # methods: 'continuous', 'interp'
+ if method == 'interp':
+ x = np.linspace(0, 1, 101) # 101-point interp (COCO)
+ ap = np.trapz(np.interp(x, mrec, mpre), x) # integrate
+ else: # 'continuous'
+ i = np.where(mrec[1:] != mrec[:-1])[0] # points where x axis (recall) changes
+ ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1]) # area under curve
+
+ return ap, mpre, mrec
+
+
+class ConfusionMatrix:
+ # Updated version of https://github.com/kaanakan/object_detection_confusion_matrix
+ def __init__(self, nc, conf=0.25, iou_thres=0.45):
+ self.matrix = np.zeros((nc + 1, nc + 1))
+ self.nc = nc # number of classes
+ self.conf = conf
+ self.iou_thres = iou_thres
+
+ def process_batch(self, detections, labels):
+ """
+ Return intersection-over-union (Jaccard index) of boxes.
+ Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
+ Arguments:
+ detections (Array[N, 6]), x1, y1, x2, y2, conf, class
+ labels (Array[M, 5]), class, x1, y1, x2, y2
+ Returns:
+ None, updates confusion matrix accordingly
+ """
+ if detections is None:
+ gt_classes = labels.int()
+ for gc in gt_classes:
+ self.matrix[self.nc, gc] += 1 # background FN
+ return
+
+ detections = detections[detections[:, 4] > self.conf]
+ gt_classes = labels[:, 0].int()
+ detection_classes = detections[:, 5].int()
+ iou = box_iou(labels[:, 1:], detections[:, :4])
+
+ x = torch.where(iou > self.iou_thres)
+ if x[0].shape[0]:
+ matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy()
+ if x[0].shape[0] > 1:
+ matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
+ matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
+ else:
+ matches = np.zeros((0, 3))
+
+ n = matches.shape[0] > 0
+ m0, m1, _ = matches.transpose().astype(int)
+ for i, gc in enumerate(gt_classes):
+ j = m0 == i
+ if n and sum(j) == 1:
+ self.matrix[detection_classes[m1[j]], gc] += 1 # correct
+ else:
+ self.matrix[self.nc, gc] += 1 # true background
+
+ if n:
+ for i, dc in enumerate(detection_classes):
+ if not any(m1 == i):
+ self.matrix[dc, self.nc] += 1 # predicted background
+
+ def tp_fp(self):
+ tp = self.matrix.diagonal() # true positives
+ fp = self.matrix.sum(1) - tp # false positives
+ # fn = self.matrix.sum(0) - tp # false negatives (missed detections)
+ return tp[:-1], fp[:-1] # remove background class
+
+ @TryExcept('WARNING ⚠️ ConfusionMatrix plot failure')
+ def plot(self, normalize=True, save_dir='', names=()):
+ import seaborn as sn
+
+ array = self.matrix / ((self.matrix.sum(0).reshape(1, -1) + 1E-9) if normalize else 1) # normalize columns
+ array[array < 0.005] = np.nan # don't annotate (would appear as 0.00)
+
+ fig, ax = plt.subplots(1, 1, figsize=(12, 9), tight_layout=True)
+ nc, nn = self.nc, len(names) # number of classes, names
+ sn.set(font_scale=1.0 if nc < 50 else 0.8) # for label size
+ labels = (0 < nn < 99) and (nn == nc) # apply names to ticklabels
+ ticklabels = (names + ['background']) if labels else 'auto'
+ with warnings.catch_warnings():
+ warnings.simplefilter('ignore') # suppress empty matrix RuntimeWarning: All-NaN slice encountered
+ sn.heatmap(array,
+ ax=ax,
+ annot=nc < 30,
+ annot_kws={
+ 'size': 8},
+ cmap='Blues',
+ fmt='.2f',
+ square=True,
+ vmin=0.0,
+ xticklabels=ticklabels,
+ yticklabels=ticklabels).set_facecolor((1, 1, 1))
+ ax.set_xlabel('True')
+ ax.set_ylabel('Predicted')
+ ax.set_title('Confusion Matrix')
+ fig.savefig(Path(save_dir) / 'confusion_matrix.png', dpi=250)
+ plt.close(fig)
+
+ def print(self):
+ for i in range(self.nc + 1):
+ print(' '.join(map(str, self.matrix[i])))
+
+
+def bbox_iou(box1, box2, xywh=True, GIoU=False, DIoU=False, CIoU=False, eps=1e-7):
+ # Returns Intersection over Union (IoU) of box1(1,4) to box2(n,4)
+
+ # Get the coordinates of bounding boxes
+ if xywh: # transform from xywh to xyxy
+ (x1, y1, w1, h1), (x2, y2, w2, h2) = box1.chunk(4, -1), box2.chunk(4, -1)
+ w1_, h1_, w2_, h2_ = w1 / 2, h1 / 2, w2 / 2, h2 / 2
+ b1_x1, b1_x2, b1_y1, b1_y2 = x1 - w1_, x1 + w1_, y1 - h1_, y1 + h1_
+ b2_x1, b2_x2, b2_y1, b2_y2 = x2 - w2_, x2 + w2_, y2 - h2_, y2 + h2_
+ else: # x1, y1, x2, y2 = box1
+ b1_x1, b1_y1, b1_x2, b1_y2 = box1.chunk(4, -1)
+ b2_x1, b2_y1, b2_x2, b2_y2 = box2.chunk(4, -1)
+ w1, h1 = b1_x2 - b1_x1, (b1_y2 - b1_y1).clamp(eps)
+ w2, h2 = b2_x2 - b2_x1, (b2_y2 - b2_y1).clamp(eps)
+
+ # Intersection area
+ inter = (b1_x2.minimum(b2_x2) - b1_x1.maximum(b2_x1)).clamp(0) * \
+ (b1_y2.minimum(b2_y2) - b1_y1.maximum(b2_y1)).clamp(0)
+
+ # Union Area
+ union = w1 * h1 + w2 * h2 - inter + eps
+
+ # IoU
+ iou = inter / union
+ if CIoU or DIoU or GIoU:
+ cw = b1_x2.maximum(b2_x2) - b1_x1.minimum(b2_x1) # convex (smallest enclosing box) width
+ ch = b1_y2.maximum(b2_y2) - b1_y1.minimum(b2_y1) # convex height
+ if CIoU or DIoU: # Distance or Complete IoU https://arxiv.org/abs/1911.08287v1
+ c2 = cw ** 2 + ch ** 2 + eps # convex diagonal squared
+ rho2 = ((b2_x1 + b2_x2 - b1_x1 - b1_x2) ** 2 + (b2_y1 + b2_y2 - b1_y1 - b1_y2) ** 2) / 4 # center dist ** 2
+ if CIoU: # https://github.com/Zzh-tju/DIoU-SSD-pytorch/blob/master/utils/box/box_utils.py#L47
+ v = (4 / math.pi ** 2) * (torch.atan(w2 / h2) - torch.atan(w1 / h1)).pow(2)
+ with torch.no_grad():
+ alpha = v / (v - iou + (1 + eps))
+ return iou - (rho2 / c2 + v * alpha) # CIoU
+ return iou - rho2 / c2 # DIoU
+ c_area = cw * ch + eps # convex area
+ return iou - (c_area - union) / c_area # GIoU https://arxiv.org/pdf/1902.09630.pdf
+ return iou # IoU
+
+
+def box_iou(box1, box2, eps=1e-7):
+ # https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py
+ """
+ Return intersection-over-union (Jaccard index) of boxes.
+ Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
+ Arguments:
+ box1 (Tensor[N, 4])
+ box2 (Tensor[M, 4])
+ Returns:
+ iou (Tensor[N, M]): the NxM matrix containing the pairwise
+ IoU values for every element in boxes1 and boxes2
+ """
+
+ # inter(N,M) = (rb(N,M,2) - lt(N,M,2)).clamp(0).prod(2)
+ (a1, a2), (b1, b2) = box1.unsqueeze(1).chunk(2, 2), box2.unsqueeze(0).chunk(2, 2)
+ inter = (torch.min(a2, b2) - torch.max(a1, b1)).clamp(0).prod(2)
+
+ # IoU = inter / (area1 + area2 - inter)
+ return inter / ((a2 - a1).prod(2) + (b2 - b1).prod(2) - inter + eps)
+
+
+def bbox_ioa(box1, box2, eps=1e-7):
+ """ Returns the intersection over box2 area given box1, box2. Boxes are x1y1x2y2
+ box1: np.array of shape(4)
+ box2: np.array of shape(nx4)
+ returns: np.array of shape(n)
+ """
+
+ # Get the coordinates of bounding boxes
+ b1_x1, b1_y1, b1_x2, b1_y2 = box1
+ b2_x1, b2_y1, b2_x2, b2_y2 = box2.T
+
+ # Intersection area
+ inter_area = (np.minimum(b1_x2, b2_x2) - np.maximum(b1_x1, b2_x1)).clip(0) * \
+ (np.minimum(b1_y2, b2_y2) - np.maximum(b1_y1, b2_y1)).clip(0)
+
+ # box2 area
+ box2_area = (b2_x2 - b2_x1) * (b2_y2 - b2_y1) + eps
+
+ # Intersection over box2 area
+ return inter_area / box2_area
+
+
+def wh_iou(wh1, wh2, eps=1e-7):
+ # Returns the nxm IoU matrix. wh1 is nx2, wh2 is mx2
+ wh1 = wh1[:, None] # [N,1,2]
+ wh2 = wh2[None] # [1,M,2]
+ inter = torch.min(wh1, wh2).prod(2) # [N,M]
+ return inter / (wh1.prod(2) + wh2.prod(2) - inter + eps) # iou = inter / (area1 + area2 - inter)
+
+
+# Plots ----------------------------------------------------------------------------------------------------------------
+
+
+@threaded
+def plot_pr_curve(px, py, ap, save_dir=Path('pr_curve.png'), names=()):
+ # Precision-recall curve
+ fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)
+ py = np.stack(py, axis=1)
+
+ if 0 < len(names) < 21: # display per-class legend if < 21 classes
+ for i, y in enumerate(py.T):
+ ax.plot(px, y, linewidth=1, label=f'{names[i]} {ap[i, 0]:.3f}') # plot(recall, precision)
+ else:
+ ax.plot(px, py, linewidth=1, color='grey') # plot(recall, precision)
+
+ ax.plot(px, py.mean(1), linewidth=3, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())
+ ax.set_xlabel('Recall')
+ ax.set_ylabel('Precision')
+ ax.set_xlim(0, 1)
+ ax.set_ylim(0, 1)
+ ax.legend(bbox_to_anchor=(1.04, 1), loc='upper left')
+ ax.set_title('Precision-Recall Curve')
+ fig.savefig(save_dir, dpi=250)
+ plt.close(fig)
+
+
+@threaded
+def plot_mc_curve(px, py, save_dir=Path('mc_curve.png'), names=(), xlabel='Confidence', ylabel='Metric'):
+ # Metric-confidence curve
+ fig, ax = plt.subplots(1, 1, figsize=(9, 6), tight_layout=True)
+
+ if 0 < len(names) < 21: # display per-class legend if < 21 classes
+ for i, y in enumerate(py):
+ ax.plot(px, y, linewidth=1, label=f'{names[i]}') # plot(confidence, metric)
+ else:
+ ax.plot(px, py.T, linewidth=1, color='grey') # plot(confidence, metric)
+
+ y = smooth(py.mean(0), 0.05)
+ ax.plot(px, y, linewidth=3, color='blue', label=f'all classes {y.max():.2f} at {px[y.argmax()]:.3f}')
+ ax.set_xlabel(xlabel)
+ ax.set_ylabel(ylabel)
+ ax.set_xlim(0, 1)
+ ax.set_ylim(0, 1)
+ ax.legend(bbox_to_anchor=(1.04, 1), loc='upper left')
+ ax.set_title(f'{ylabel}-Confidence Curve')
+ fig.savefig(save_dir, dpi=250)
+ plt.close(fig)
diff --git a/TextDetection/utils/plots.py b/TextDetection/utils/plots.py
new file mode 100644
index 0000000000000000000000000000000000000000..754fabb84e37d21cb2e88352337bf1db3f610a47
--- /dev/null
+++ b/TextDetection/utils/plots.py
@@ -0,0 +1,562 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Plotting utils
+"""
+
+import contextlib
+import math
+import os
+from copy import copy
+from pathlib import Path
+from urllib.error import URLError
+
+import cv2
+import matplotlib
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import seaborn as sn
+import torch
+from PIL import Image, ImageDraw, ImageFont
+from scipy.ndimage.filters import gaussian_filter1d
+
+from utils import TryExcept, threaded
+from utils.general import (CONFIG_DIR, FONT, LOGGER, check_font, check_requirements, clip_boxes, increment_path,
+ is_ascii, xywh2xyxy, xyxy2xywh)
+from utils.metrics import fitness
+from utils.segment.general import scale_image
+
+# Settings
+RANK = int(os.getenv('RANK', -1))
+matplotlib.rc('font', **{'size': 11})
+matplotlib.use('Agg') # for writing to files only
+
+
+class Colors:
+ # Ultralytics color palette https://ultralytics.com/
+ def __init__(self):
+ # hex = matplotlib.colors.TABLEAU_COLORS.values()
+ hexs = ('FF3838', 'FF9D97', 'FF701F', 'FFB21D', 'CFD231', '48F90A', '92CC17', '3DDB86', '1A9334', '00D4BB',
+ '2C99A8', '00C2FF', '344593', '6473FF', '0018EC', '8438FF', '520085', 'CB38FF', 'FF95C8', 'FF37C7')
+ self.palette = [self.hex2rgb(f'#{c}') for c in hexs]
+ self.n = len(self.palette)
+
+ def __call__(self, i, bgr=False):
+ c = self.palette[int(i) % self.n]
+ return (c[2], c[1], c[0]) if bgr else c
+
+ @staticmethod
+ def hex2rgb(h): # rgb order (PIL)
+ return tuple(int(h[1 + i:1 + i + 2], 16) for i in (0, 2, 4))
+
+
+colors = Colors() # create instance for 'from utils.plots import colors'
+
+
+def check_pil_font(font=FONT, size=10):
+ # Return a PIL TrueType Font, downloading to CONFIG_DIR if necessary
+ font = Path(font)
+ font = font if font.exists() else (CONFIG_DIR / font.name)
+ try:
+ return ImageFont.truetype(str(font) if font.exists() else font.name, size)
+ except Exception: # download if missing
+ try:
+ check_font(font)
+ return ImageFont.truetype(str(font), size)
+ except TypeError:
+ check_requirements('Pillow>=8.4.0') # known issue https://github.com/ultralytics/yolov5/issues/5374
+ except URLError: # not online
+ return ImageFont.load_default()
+
+
+class Annotator:
+ # YOLOv5 Annotator for train/val mosaics and jpgs and detect/hub inference annotations
+ def __init__(self, im, line_width=None, font_size=None, font='Arial.ttf', pil=False, example='abc'):
+ assert im.data.contiguous, 'Image not contiguous. Apply np.ascontiguousarray(im) to Annotator() input images.'
+ non_ascii = not is_ascii(example) # non-latin labels, i.e. asian, arabic, cyrillic
+ self.pil = pil or non_ascii
+ if self.pil: # use PIL
+ self.im = im if isinstance(im, Image.Image) else Image.fromarray(im)
+ self.draw = ImageDraw.Draw(self.im)
+ self.font = check_pil_font(font='Arial.Unicode.ttf' if non_ascii else font,
+ size=font_size or max(round(sum(self.im.size) / 2 * 0.035), 12))
+ else: # use cv2
+ self.im = im
+ self.lw = line_width or max(round(sum(im.shape) / 2 * 0.003), 2) # line width
+
+ def box_label(self, box, label='', color=(128, 128, 128), txt_color=(255, 255, 255)):
+ # Add one xyxy box to image with label
+ if self.pil or not is_ascii(label):
+ self.draw.rectangle(box, width=self.lw, outline=color) # box
+ if label:
+ w, h = self.font.getsize(label) # text width, height (WARNING: deprecated) in 9.2.0
+ # _, _, w, h = self.font.getbbox(label) # text width, height (New)
+ outside = box[1] - h >= 0 # label fits outside box
+ self.draw.rectangle(
+ (box[0], box[1] - h if outside else box[1], box[0] + w + 1,
+ box[1] + 1 if outside else box[1] + h + 1),
+ fill=color,
+ )
+ # self.draw.text((box[0], box[1]), label, fill=txt_color, font=self.font, anchor='ls') # for PIL>8.0
+ self.draw.text((box[0], box[1] - h if outside else box[1]), label, fill=txt_color, font=self.font)
+ else: # cv2
+ p1, p2 = (int(box[0]), int(box[1])), (int(box[2]), int(box[3]))
+ cv2.rectangle(self.im, p1, p2, color, thickness=self.lw, lineType=cv2.LINE_AA)
+ if label:
+ tf = max(self.lw - 1, 1) # font thickness
+ w, h = cv2.getTextSize(label, 0, fontScale=self.lw / 3, thickness=tf)[0] # text width, height
+ outside = p1[1] - h >= 3
+ p2 = p1[0] + w, p1[1] - h - 3 if outside else p1[1] + h + 3
+ cv2.rectangle(self.im, p1, p2, color, -1, cv2.LINE_AA) # filled
+ cv2.putText(self.im,
+ label, (p1[0], p1[1] - 2 if outside else p1[1] + h + 2),
+ 0,
+ self.lw / 3,
+ txt_color,
+ thickness=tf,
+ lineType=cv2.LINE_AA)
+
+ def masks(self, masks, colors, im_gpu, alpha=0.5, retina_masks=False):
+ """Plot masks at once.
+ Args:
+ masks (tensor): predicted masks on cuda, shape: [n, h, w]
+ colors (List[List[Int]]): colors for predicted masks, [[r, g, b] * n]
+ im_gpu (tensor): img is in cuda, shape: [3, h, w], range: [0, 1]
+ alpha (float): mask transparency: 0.0 fully transparent, 1.0 opaque
+ """
+ if self.pil:
+ # convert to numpy first
+ self.im = np.asarray(self.im).copy()
+ if len(masks) == 0:
+ self.im[:] = im_gpu.permute(1, 2, 0).contiguous().cpu().numpy() * 255
+ colors = torch.tensor(colors, device=im_gpu.device, dtype=torch.float32) / 255.0
+ colors = colors[:, None, None] # shape(n,1,1,3)
+ masks = masks.unsqueeze(3) # shape(n,h,w,1)
+ masks_color = masks * (colors * alpha) # shape(n,h,w,3)
+
+ inv_alph_masks = (1 - masks * alpha).cumprod(0) # shape(n,h,w,1)
+ mcs = (masks_color * inv_alph_masks).sum(0) * 2 # mask color summand shape(n,h,w,3)
+
+ im_gpu = im_gpu.flip(dims=[0]) # flip channel
+ im_gpu = im_gpu.permute(1, 2, 0).contiguous() # shape(h,w,3)
+ im_gpu = im_gpu * inv_alph_masks[-1] + mcs
+ im_mask = (im_gpu * 255).byte().cpu().numpy()
+ self.im[:] = im_mask if retina_masks else scale_image(im_gpu.shape, im_mask, self.im.shape)
+ if self.pil:
+ # convert im back to PIL and update draw
+ self.fromarray(self.im)
+
+ def rectangle(self, xy, fill=None, outline=None, width=1):
+ # Add rectangle to image (PIL-only)
+ self.draw.rectangle(xy, fill, outline, width)
+
+ def text(self, xy, text, txt_color=(255, 255, 255), anchor='top'):
+ # Add text to image (PIL-only)
+ if anchor == 'bottom': # start y from font bottom
+ w, h = self.font.getsize(text) # text width, height
+ xy[1] += 1 - h
+ self.draw.text(xy, text, fill=txt_color, font=self.font)
+
+ def fromarray(self, im):
+ # Update self.im from a numpy array
+ self.im = im if isinstance(im, Image.Image) else Image.fromarray(im)
+ self.draw = ImageDraw.Draw(self.im)
+
+ def result(self):
+ # Return annotated image as array
+ return np.asarray(self.im)
+
+
+def feature_visualization(x, module_type, stage, n=32, save_dir=Path('runs/detect/exp')):
+ """
+ x: Features to be visualized
+ module_type: Module type
+ stage: Module stage within model
+ n: Maximum number of feature maps to plot
+ save_dir: Directory to save results
+ """
+ if 'Detect' not in module_type:
+ batch, channels, height, width = x.shape # batch, channels, height, width
+ if height > 1 and width > 1:
+ f = save_dir / f"stage{stage}_{module_type.split('.')[-1]}_features.png" # filename
+
+ blocks = torch.chunk(x[0].cpu(), channels, dim=0) # select batch index 0, block by channels
+ n = min(n, channels) # number of plots
+ fig, ax = plt.subplots(math.ceil(n / 8), 8, tight_layout=True) # 8 rows x n/8 cols
+ ax = ax.ravel()
+ plt.subplots_adjust(wspace=0.05, hspace=0.05)
+ for i in range(n):
+ ax[i].imshow(blocks[i].squeeze()) # cmap='gray'
+ ax[i].axis('off')
+
+ LOGGER.info(f'Saving {f}... ({n}/{channels})')
+ plt.savefig(f, dpi=300, bbox_inches='tight')
+ plt.close()
+ np.save(str(f.with_suffix('.npy')), x[0].cpu().numpy()) # npy save
+
+
+def hist2d(x, y, n=100):
+ # 2d histogram used in labels.png and evolve.png
+ xedges, yedges = np.linspace(x.min(), x.max(), n), np.linspace(y.min(), y.max(), n)
+ hist, xedges, yedges = np.histogram2d(x, y, (xedges, yedges))
+ xidx = np.clip(np.digitize(x, xedges) - 1, 0, hist.shape[0] - 1)
+ yidx = np.clip(np.digitize(y, yedges) - 1, 0, hist.shape[1] - 1)
+ return np.log(hist[xidx, yidx])
+
+
+def butter_lowpass_filtfilt(data, cutoff=1500, fs=50000, order=5):
+ from scipy.signal import butter, filtfilt
+
+ # https://stackoverflow.com/questions/28536191/how-to-filter-smooth-with-scipy-numpy
+ def butter_lowpass(cutoff, fs, order):
+ nyq = 0.5 * fs
+ normal_cutoff = cutoff / nyq
+ return butter(order, normal_cutoff, btype='low', analog=False)
+
+ b, a = butter_lowpass(cutoff, fs, order=order)
+ return filtfilt(b, a, data) # forward-backward filter
+
+
+def output_to_target(output, max_det=300):
+ # Convert model output to target format [batch_id, class_id, x, y, w, h, conf] for plotting
+ targets = []
+ for i, o in enumerate(output):
+ box, conf, cls = o[:max_det, :6].cpu().split((4, 1, 1), 1)
+ j = torch.full((conf.shape[0], 1), i)
+ targets.append(torch.cat((j, cls, xyxy2xywh(box), conf), 1))
+ return torch.cat(targets, 0).numpy()
+
+
+@threaded
+def plot_images(images, targets, paths=None, fname='images.jpg', names=None):
+ # Plot image grid with labels
+ if isinstance(images, torch.Tensor):
+ images = images.cpu().float().numpy()
+ if isinstance(targets, torch.Tensor):
+ targets = targets.cpu().numpy()
+
+ max_size = 1920 # max image size
+ max_subplots = 16 # max image subplots, i.e. 4x4
+ bs, _, h, w = images.shape # batch size, _, height, width
+ bs = min(bs, max_subplots) # limit plot images
+ ns = np.ceil(bs ** 0.5) # number of subplots (square)
+ if np.max(images[0]) <= 1:
+ images *= 255 # de-normalise (optional)
+
+ # Build Image
+ mosaic = np.full((int(ns * h), int(ns * w), 3), 255, dtype=np.uint8) # init
+ for i, im in enumerate(images):
+ if i == max_subplots: # if last batch has fewer images than we expect
+ break
+ x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
+ im = im.transpose(1, 2, 0)
+ mosaic[y:y + h, x:x + w, :] = im
+
+ # Resize (optional)
+ scale = max_size / ns / max(h, w)
+ if scale < 1:
+ h = math.ceil(scale * h)
+ w = math.ceil(scale * w)
+ mosaic = cv2.resize(mosaic, tuple(int(x * ns) for x in (w, h)))
+
+ # Annotate
+ fs = int((h + w) * ns * 0.01) # font size
+ annotator = Annotator(mosaic, line_width=round(fs / 10), font_size=fs, pil=True, example=names)
+ for i in range(i + 1):
+ x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
+ annotator.rectangle([x, y, x + w, y + h], None, (255, 255, 255), width=2) # borders
+ if paths:
+ annotator.text((x + 5, y + 5), text=Path(paths[i]).name[:40], txt_color=(220, 220, 220)) # filenames
+ if len(targets) > 0:
+ ti = targets[targets[:, 0] == i] # image targets
+ boxes = xywh2xyxy(ti[:, 2:6]).T
+ classes = ti[:, 1].astype('int')
+ labels = ti.shape[1] == 6 # labels if no conf column
+ conf = None if labels else ti[:, 6] # check for confidence presence (label vs pred)
+
+ if boxes.shape[1]:
+ if boxes.max() <= 1.01: # if normalized with tolerance 0.01
+ boxes[[0, 2]] *= w # scale to pixels
+ boxes[[1, 3]] *= h
+ elif scale < 1: # absolute coords need scale if image scales
+ boxes *= scale
+ boxes[[0, 2]] += x
+ boxes[[1, 3]] += y
+ for j, box in enumerate(boxes.T.tolist()):
+ cls = classes[j]
+ color = colors(cls)
+ cls = names[cls] if names else cls
+ if labels or conf[j] > 0.25: # 0.25 conf thresh
+ label = f'{cls}' if labels else f'{cls} {conf[j]:.1f}'
+ annotator.box_label(box, label, color=color)
+ annotator.im.save(fname) # save
+
+
+def plot_lr_scheduler(optimizer, scheduler, epochs=300, save_dir=''):
+ # Plot LR simulating training for full epochs
+ optimizer, scheduler = copy(optimizer), copy(scheduler) # do not modify originals
+ y = []
+ for _ in range(epochs):
+ scheduler.step()
+ y.append(optimizer.param_groups[0]['lr'])
+ plt.plot(y, '.-', label='LR')
+ plt.xlabel('epoch')
+ plt.ylabel('LR')
+ plt.grid()
+ plt.xlim(0, epochs)
+ plt.ylim(0)
+ plt.savefig(Path(save_dir) / 'LR.png', dpi=200)
+ plt.close()
+
+
+def plot_val_txt(): # from utils.plots import *; plot_val()
+ # Plot val.txt histograms
+ x = np.loadtxt('val.txt', dtype=np.float32)
+ box = xyxy2xywh(x[:, :4])
+ cx, cy = box[:, 0], box[:, 1]
+
+ fig, ax = plt.subplots(1, 1, figsize=(6, 6), tight_layout=True)
+ ax.hist2d(cx, cy, bins=600, cmax=10, cmin=0)
+ ax.set_aspect('equal')
+ plt.savefig('hist2d.png', dpi=300)
+
+ fig, ax = plt.subplots(1, 2, figsize=(12, 6), tight_layout=True)
+ ax[0].hist(cx, bins=600)
+ ax[1].hist(cy, bins=600)
+ plt.savefig('hist1d.png', dpi=200)
+
+
+def plot_targets_txt(): # from utils.plots import *; plot_targets_txt()
+ # Plot targets.txt histograms
+ x = np.loadtxt('targets.txt', dtype=np.float32).T
+ s = ['x targets', 'y targets', 'width targets', 'height targets']
+ fig, ax = plt.subplots(2, 2, figsize=(8, 8), tight_layout=True)
+ ax = ax.ravel()
+ for i in range(4):
+ ax[i].hist(x[i], bins=100, label=f'{x[i].mean():.3g} +/- {x[i].std():.3g}')
+ ax[i].legend()
+ ax[i].set_title(s[i])
+ plt.savefig('targets.jpg', dpi=200)
+
+
+def plot_val_study(file='', dir='', x=None): # from utils.plots import *; plot_val_study()
+ # Plot file=study.txt generated by val.py (or plot all study*.txt in dir)
+ save_dir = Path(file).parent if file else Path(dir)
+ plot2 = False # plot additional results
+ if plot2:
+ ax = plt.subplots(2, 4, figsize=(10, 6), tight_layout=True)[1].ravel()
+
+ fig2, ax2 = plt.subplots(1, 1, figsize=(8, 4), tight_layout=True)
+ # for f in [save_dir / f'study_coco_{x}.txt' for x in ['yolov5n6', 'yolov5s6', 'yolov5m6', 'yolov5l6', 'yolov5x6']]:
+ for f in sorted(save_dir.glob('study*.txt')):
+ y = np.loadtxt(f, dtype=np.float32, usecols=[0, 1, 2, 3, 7, 8, 9], ndmin=2).T
+ x = np.arange(y.shape[1]) if x is None else np.array(x)
+ if plot2:
+ s = ['P', 'R', 'mAP@.5', 'mAP@.5:.95', 't_preprocess (ms/img)', 't_inference (ms/img)', 't_NMS (ms/img)']
+ for i in range(7):
+ ax[i].plot(x, y[i], '.-', linewidth=2, markersize=8)
+ ax[i].set_title(s[i])
+
+ j = y[3].argmax() + 1
+ ax2.plot(y[5, 1:j],
+ y[3, 1:j] * 1E2,
+ '.-',
+ linewidth=2,
+ markersize=8,
+ label=f.stem.replace('study_coco_', '').replace('yolo', 'YOLO'))
+
+ ax2.plot(1E3 / np.array([209, 140, 97, 58, 35, 18]), [34.6, 40.5, 43.0, 47.5, 49.7, 51.5],
+ 'k.-',
+ linewidth=2,
+ markersize=8,
+ alpha=.25,
+ label='EfficientDet')
+
+ ax2.grid(alpha=0.2)
+ ax2.set_yticks(np.arange(20, 60, 5))
+ ax2.set_xlim(0, 57)
+ ax2.set_ylim(25, 55)
+ ax2.set_xlabel('GPU Speed (ms/img)')
+ ax2.set_ylabel('COCO AP val')
+ ax2.legend(loc='lower right')
+ f = save_dir / 'study.png'
+ print(f'Saving {f}...')
+ plt.savefig(f, dpi=300)
+
+
+@TryExcept() # known issue https://github.com/ultralytics/yolov5/issues/5395
+def plot_labels(labels, names=(), save_dir=Path('')):
+ # plot dataset labels
+ LOGGER.info(f"Plotting labels to {save_dir / 'labels.jpg'}... ")
+ c, b = labels[:, 0], labels[:, 1:].transpose() # classes, boxes
+ nc = int(c.max() + 1) # number of classes
+ x = pd.DataFrame(b.transpose(), columns=['x', 'y', 'width', 'height'])
+
+ # seaborn correlogram
+ sn.pairplot(x, corner=True, diag_kind='auto', kind='hist', diag_kws=dict(bins=50), plot_kws=dict(pmax=0.9))
+ plt.savefig(save_dir / 'labels_correlogram.jpg', dpi=200)
+ plt.close()
+
+ # matplotlib labels
+ matplotlib.use('svg') # faster
+ ax = plt.subplots(2, 2, figsize=(8, 8), tight_layout=True)[1].ravel()
+ y = ax[0].hist(c, bins=np.linspace(0, nc, nc + 1) - 0.5, rwidth=0.8)
+ with contextlib.suppress(Exception): # color histogram bars by class
+ [y[2].patches[i].set_color([x / 255 for x in colors(i)]) for i in range(nc)] # known issue #3195
+ ax[0].set_ylabel('instances')
+ if 0 < len(names) < 30:
+ ax[0].set_xticks(range(len(names)))
+ ax[0].set_xticklabels(list(names.values()), rotation=90, fontsize=10)
+ else:
+ ax[0].set_xlabel('classes')
+ sn.histplot(x, x='x', y='y', ax=ax[2], bins=50, pmax=0.9)
+ sn.histplot(x, x='width', y='height', ax=ax[3], bins=50, pmax=0.9)
+
+ # rectangles
+ labels[:, 1:3] = 0.5 # center
+ labels[:, 1:] = xywh2xyxy(labels[:, 1:]) * 2000
+ img = Image.fromarray(np.ones((2000, 2000, 3), dtype=np.uint8) * 255)
+ for cls, *box in labels[:1000]:
+ ImageDraw.Draw(img).rectangle(box, width=1, outline=colors(cls)) # plot
+ ax[1].imshow(img)
+ ax[1].axis('off')
+
+ for a in [0, 1, 2, 3]:
+ for s in ['top', 'right', 'left', 'bottom']:
+ ax[a].spines[s].set_visible(False)
+
+ plt.savefig(save_dir / 'labels.jpg', dpi=200)
+ matplotlib.use('Agg')
+ plt.close()
+
+
+def imshow_cls(im, labels=None, pred=None, names=None, nmax=25, verbose=False, f=Path('images.jpg')):
+ # Show classification image grid with labels (optional) and predictions (optional)
+ from utils.augmentations import denormalize
+
+ names = names or [f'class{i}' for i in range(1000)]
+ blocks = torch.chunk(denormalize(im.clone()).cpu().float(), len(im),
+ dim=0) # select batch index 0, block by channels
+ n = min(len(blocks), nmax) # number of plots
+ m = min(8, round(n ** 0.5)) # 8 x 8 default
+ fig, ax = plt.subplots(math.ceil(n / m), m) # 8 rows x n/8 cols
+ ax = ax.ravel() if m > 1 else [ax]
+ # plt.subplots_adjust(wspace=0.05, hspace=0.05)
+ for i in range(n):
+ ax[i].imshow(blocks[i].squeeze().permute((1, 2, 0)).numpy().clip(0.0, 1.0))
+ ax[i].axis('off')
+ if labels is not None:
+ s = names[labels[i]] + (f'—{names[pred[i]]}' if pred is not None else '')
+ ax[i].set_title(s, fontsize=8, verticalalignment='top')
+ plt.savefig(f, dpi=300, bbox_inches='tight')
+ plt.close()
+ if verbose:
+ LOGGER.info(f'Saving {f}')
+ if labels is not None:
+ LOGGER.info('True: ' + ' '.join(f'{names[i]:3s}' for i in labels[:nmax]))
+ if pred is not None:
+ LOGGER.info('Predicted:' + ' '.join(f'{names[i]:3s}' for i in pred[:nmax]))
+ return f
+
+
+def plot_evolve(evolve_csv='path/to/evolve.csv'): # from utils.plots import *; plot_evolve()
+ # Plot evolve.csv hyp evolution results
+ evolve_csv = Path(evolve_csv)
+ data = pd.read_csv(evolve_csv)
+ keys = [x.strip() for x in data.columns]
+ x = data.values
+ f = fitness(x)
+ j = np.argmax(f) # max fitness index
+ plt.figure(figsize=(10, 12), tight_layout=True)
+ matplotlib.rc('font', **{'size': 8})
+ print(f'Best results from row {j} of {evolve_csv}:')
+ for i, k in enumerate(keys[7:]):
+ v = x[:, 7 + i]
+ mu = v[j] # best single result
+ plt.subplot(6, 5, i + 1)
+ plt.scatter(v, f, c=hist2d(v, f, 20), cmap='viridis', alpha=.8, edgecolors='none')
+ plt.plot(mu, f.max(), 'k+', markersize=15)
+ plt.title(f'{k} = {mu:.3g}', fontdict={'size': 9}) # limit to 40 characters
+ if i % 5 != 0:
+ plt.yticks([])
+ print(f'{k:>15}: {mu:.3g}')
+ f = evolve_csv.with_suffix('.png') # filename
+ plt.savefig(f, dpi=200)
+ plt.close()
+ print(f'Saved {f}')
+
+
+def plot_results(file='path/to/results.csv', dir=''):
+ # Plot training results.csv. Usage: from utils.plots import *; plot_results('path/to/results.csv')
+ save_dir = Path(file).parent if file else Path(dir)
+ fig, ax = plt.subplots(2, 5, figsize=(12, 6), tight_layout=True)
+ ax = ax.ravel()
+ files = list(save_dir.glob('results*.csv'))
+ assert len(files), f'No results.csv files found in {save_dir.resolve()}, nothing to plot.'
+ for f in files:
+ try:
+ data = pd.read_csv(f)
+ s = [x.strip() for x in data.columns]
+ x = data.values[:, 0]
+ for i, j in enumerate([1, 2, 3, 4, 5, 8, 9, 10, 6, 7]):
+ y = data.values[:, j].astype('float')
+ # y[y == 0] = np.nan # don't show zero values
+ ax[i].plot(x, y, marker='.', label=f.stem, linewidth=2, markersize=8) # actual results
+ ax[i].plot(x, gaussian_filter1d(y, sigma=3), ':', label='smooth', linewidth=2) # smoothing line
+ ax[i].set_title(s[j], fontsize=12)
+ # if j in [8, 9, 10]: # share train and val loss y axes
+ # ax[i].get_shared_y_axes().join(ax[i], ax[i - 5])
+ except Exception as e:
+ LOGGER.info(f'Warning: Plotting error for {f}: {e}')
+ ax[1].legend()
+ fig.savefig(save_dir / 'results.png', dpi=200)
+ plt.close()
+
+
+def profile_idetection(start=0, stop=0, labels=(), save_dir=''):
+ # Plot iDetection '*.txt' per-image logs. from utils.plots import *; profile_idetection()
+ ax = plt.subplots(2, 4, figsize=(12, 6), tight_layout=True)[1].ravel()
+ s = ['Images', 'Free Storage (GB)', 'RAM Usage (GB)', 'Battery', 'dt_raw (ms)', 'dt_smooth (ms)', 'real-world FPS']
+ files = list(Path(save_dir).glob('frames*.txt'))
+ for fi, f in enumerate(files):
+ try:
+ results = np.loadtxt(f, ndmin=2).T[:, 90:-30] # clip first and last rows
+ n = results.shape[1] # number of rows
+ x = np.arange(start, min(stop, n) if stop else n)
+ results = results[:, x]
+ t = (results[0] - results[0].min()) # set t0=0s
+ results[0] = x
+ for i, a in enumerate(ax):
+ if i < len(results):
+ label = labels[fi] if len(labels) else f.stem.replace('frames_', '')
+ a.plot(t, results[i], marker='.', label=label, linewidth=1, markersize=5)
+ a.set_title(s[i])
+ a.set_xlabel('time (s)')
+ # if fi == len(files) - 1:
+ # a.set_ylim(bottom=0)
+ for side in ['top', 'right']:
+ a.spines[side].set_visible(False)
+ else:
+ a.remove()
+ except Exception as e:
+ print(f'Warning: Plotting error for {f}; {e}')
+ ax[1].legend()
+ plt.savefig(Path(save_dir) / 'idetection_profile.png', dpi=200)
+
+
+def save_one_box(xyxy, im, file=Path('im.jpg'), gain=1.02, pad=10, square=False, BGR=False, save=True):
+ # Save image crop as {file} with crop size multiple {gain} and {pad} pixels. Save and/or return crop
+ xyxy = torch.tensor(xyxy).view(-1, 4)
+ b = xyxy2xywh(xyxy) # boxes
+ if square:
+ b[:, 2:] = b[:, 2:].max(1)[0].unsqueeze(1) # attempt rectangle to square
+ b[:, 2:] = b[:, 2:] * gain + pad # box wh * gain + pad
+ xyxy = xywh2xyxy(b).long()
+ clip_boxes(xyxy, im.shape)
+ crop = im[int(xyxy[0, 1]):int(xyxy[0, 3]), int(xyxy[0, 0]):int(xyxy[0, 2]), ::(1 if BGR else -1)]
+ if save:
+ file.parent.mkdir(parents=True, exist_ok=True) # make directory
+ f = str(increment_path(file).with_suffix('.jpg'))
+ # cv2.imwrite(f, crop) # save BGR, https://github.com/ultralytics/yolov5/issues/7007 chroma subsampling issue
+ Image.fromarray(crop[..., ::-1]).save(f, quality=95, subsampling=0) # save RGB
+ return crop
diff --git a/TextDetection/utils/segment/__init__.py b/TextDetection/utils/segment/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/TextDetection/utils/segment/__pycache__/__init__.cpython-310.pyc b/TextDetection/utils/segment/__pycache__/__init__.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..7e50cf48a54cce48a1f80d16d48795da34b3e003
Binary files /dev/null and b/TextDetection/utils/segment/__pycache__/__init__.cpython-310.pyc differ
diff --git a/TextDetection/utils/segment/__pycache__/__init__.cpython-39.pyc b/TextDetection/utils/segment/__pycache__/__init__.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..c21b65609d1103f5ecc52266ada8e037d2ca63e0
Binary files /dev/null and b/TextDetection/utils/segment/__pycache__/__init__.cpython-39.pyc differ
diff --git a/TextDetection/utils/segment/__pycache__/general.cpython-310.pyc b/TextDetection/utils/segment/__pycache__/general.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..19f77006fd9c806852bc680525d465a63b5c7649
Binary files /dev/null and b/TextDetection/utils/segment/__pycache__/general.cpython-310.pyc differ
diff --git a/TextDetection/utils/segment/__pycache__/general.cpython-39.pyc b/TextDetection/utils/segment/__pycache__/general.cpython-39.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..5918368b1f507d2d0902d923891374553a4df70f
Binary files /dev/null and b/TextDetection/utils/segment/__pycache__/general.cpython-39.pyc differ
diff --git a/TextDetection/utils/segment/augmentations.py b/TextDetection/utils/segment/augmentations.py
new file mode 100644
index 0000000000000000000000000000000000000000..f8154b834869acd87f80c0152c870b7631a918ba
--- /dev/null
+++ b/TextDetection/utils/segment/augmentations.py
@@ -0,0 +1,104 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Image augmentation functions
+"""
+
+import math
+import random
+
+import cv2
+import numpy as np
+
+from ..augmentations import box_candidates
+from ..general import resample_segments, segment2box
+
+
+def mixup(im, labels, segments, im2, labels2, segments2):
+ # Applies MixUp augmentation https://arxiv.org/pdf/1710.09412.pdf
+ r = np.random.beta(32.0, 32.0) # mixup ratio, alpha=beta=32.0
+ im = (im * r + im2 * (1 - r)).astype(np.uint8)
+ labels = np.concatenate((labels, labels2), 0)
+ segments = np.concatenate((segments, segments2), 0)
+ return im, labels, segments
+
+
+def random_perspective(im,
+ targets=(),
+ segments=(),
+ degrees=10,
+ translate=.1,
+ scale=.1,
+ shear=10,
+ perspective=0.0,
+ border=(0, 0)):
+ # torchvision.transforms.RandomAffine(degrees=(-10, 10), translate=(.1, .1), scale=(.9, 1.1), shear=(-10, 10))
+ # targets = [cls, xyxy]
+
+ height = im.shape[0] + border[0] * 2 # shape(h,w,c)
+ width = im.shape[1] + border[1] * 2
+
+ # Center
+ C = np.eye(3)
+ C[0, 2] = -im.shape[1] / 2 # x translation (pixels)
+ C[1, 2] = -im.shape[0] / 2 # y translation (pixels)
+
+ # Perspective
+ P = np.eye(3)
+ P[2, 0] = random.uniform(-perspective, perspective) # x perspective (about y)
+ P[2, 1] = random.uniform(-perspective, perspective) # y perspective (about x)
+
+ # Rotation and Scale
+ R = np.eye(3)
+ a = random.uniform(-degrees, degrees)
+ # a += random.choice([-180, -90, 0, 90]) # add 90deg rotations to small rotations
+ s = random.uniform(1 - scale, 1 + scale)
+ # s = 2 ** random.uniform(-scale, scale)
+ R[:2] = cv2.getRotationMatrix2D(angle=a, center=(0, 0), scale=s)
+
+ # Shear
+ S = np.eye(3)
+ S[0, 1] = math.tan(random.uniform(-shear, shear) * math.pi / 180) # x shear (deg)
+ S[1, 0] = math.tan(random.uniform(-shear, shear) * math.pi / 180) # y shear (deg)
+
+ # Translation
+ T = np.eye(3)
+ T[0, 2] = (random.uniform(0.5 - translate, 0.5 + translate) * width) # x translation (pixels)
+ T[1, 2] = (random.uniform(0.5 - translate, 0.5 + translate) * height) # y translation (pixels)
+
+ # Combined rotation matrix
+ M = T @ S @ R @ P @ C # order of operations (right to left) is IMPORTANT
+ if (border[0] != 0) or (border[1] != 0) or (M != np.eye(3)).any(): # image changed
+ if perspective:
+ im = cv2.warpPerspective(im, M, dsize=(width, height), borderValue=(114, 114, 114))
+ else: # affine
+ im = cv2.warpAffine(im, M[:2], dsize=(width, height), borderValue=(114, 114, 114))
+
+ # Visualize
+ # import matplotlib.pyplot as plt
+ # ax = plt.subplots(1, 2, figsize=(12, 6))[1].ravel()
+ # ax[0].imshow(im[:, :, ::-1]) # base
+ # ax[1].imshow(im2[:, :, ::-1]) # warped
+
+ # Transform label coordinates
+ n = len(targets)
+ new_segments = []
+ if n:
+ new = np.zeros((n, 4))
+ segments = resample_segments(segments) # upsample
+ for i, segment in enumerate(segments):
+ xy = np.ones((len(segment), 3))
+ xy[:, :2] = segment
+ xy = xy @ M.T # transform
+ xy = (xy[:, :2] / xy[:, 2:3] if perspective else xy[:, :2]) # perspective rescale or affine
+
+ # clip
+ new[i] = segment2box(xy, width, height)
+ new_segments.append(xy)
+
+ # filter candidates
+ i = box_candidates(box1=targets[:, 1:5].T * s, box2=new.T, area_thr=0.01)
+ targets = targets[i]
+ targets[:, 1:5] = new[i]
+ new_segments = np.array(new_segments)[i]
+
+ return im, targets, new_segments
diff --git a/TextDetection/utils/segment/dataloaders.py b/TextDetection/utils/segment/dataloaders.py
new file mode 100644
index 0000000000000000000000000000000000000000..3ee826dba69cb0cda00c48b82710784cd39c5a81
--- /dev/null
+++ b/TextDetection/utils/segment/dataloaders.py
@@ -0,0 +1,332 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Dataloaders
+"""
+
+import os
+import random
+
+import cv2
+import numpy as np
+import torch
+from torch.utils.data import DataLoader, distributed
+
+from ..augmentations import augment_hsv, copy_paste, letterbox
+from ..dataloaders import InfiniteDataLoader, LoadImagesAndLabels, seed_worker
+from ..general import LOGGER, xyn2xy, xywhn2xyxy, xyxy2xywhn
+from ..torch_utils import torch_distributed_zero_first
+from .augmentations import mixup, random_perspective
+
+RANK = int(os.getenv('RANK', -1))
+
+
+def create_dataloader(path,
+ imgsz,
+ batch_size,
+ stride,
+ single_cls=False,
+ hyp=None,
+ augment=False,
+ cache=False,
+ pad=0.0,
+ rect=False,
+ rank=-1,
+ workers=8,
+ image_weights=False,
+ quad=False,
+ prefix='',
+ shuffle=False,
+ mask_downsample_ratio=1,
+ overlap_mask=False,
+ seed=0):
+ if rect and shuffle:
+ LOGGER.warning('WARNING ⚠️ --rect is incompatible with DataLoader shuffle, setting shuffle=False')
+ shuffle = False
+ with torch_distributed_zero_first(rank): # init dataset *.cache only once if DDP
+ dataset = LoadImagesAndLabelsAndMasks(
+ path,
+ imgsz,
+ batch_size,
+ augment=augment, # augmentation
+ hyp=hyp, # hyperparameters
+ rect=rect, # rectangular batches
+ cache_images=cache,
+ single_cls=single_cls,
+ stride=int(stride),
+ pad=pad,
+ image_weights=image_weights,
+ prefix=prefix,
+ downsample_ratio=mask_downsample_ratio,
+ overlap=overlap_mask)
+
+ batch_size = min(batch_size, len(dataset))
+ nd = torch.cuda.device_count() # number of CUDA devices
+ nw = min([os.cpu_count() // max(nd, 1), batch_size if batch_size > 1 else 0, workers]) # number of workers
+ sampler = None if rank == -1 else distributed.DistributedSampler(dataset, shuffle=shuffle)
+ loader = DataLoader if image_weights else InfiniteDataLoader # only DataLoader allows for attribute updates
+ generator = torch.Generator()
+ generator.manual_seed(6148914691236517205 + seed + RANK)
+ return loader(
+ dataset,
+ batch_size=batch_size,
+ shuffle=shuffle and sampler is None,
+ num_workers=nw,
+ sampler=sampler,
+ pin_memory=True,
+ collate_fn=LoadImagesAndLabelsAndMasks.collate_fn4 if quad else LoadImagesAndLabelsAndMasks.collate_fn,
+ worker_init_fn=seed_worker,
+ generator=generator,
+ ), dataset
+
+
+class LoadImagesAndLabelsAndMasks(LoadImagesAndLabels): # for training/testing
+
+ def __init__(
+ self,
+ path,
+ img_size=640,
+ batch_size=16,
+ augment=False,
+ hyp=None,
+ rect=False,
+ image_weights=False,
+ cache_images=False,
+ single_cls=False,
+ stride=32,
+ pad=0,
+ min_items=0,
+ prefix='',
+ downsample_ratio=1,
+ overlap=False,
+ ):
+ super().__init__(path, img_size, batch_size, augment, hyp, rect, image_weights, cache_images, single_cls,
+ stride, pad, min_items, prefix)
+ self.downsample_ratio = downsample_ratio
+ self.overlap = overlap
+
+ def __getitem__(self, index):
+ index = self.indices[index] # linear, shuffled, or image_weights
+
+ hyp = self.hyp
+ mosaic = self.mosaic and random.random() < hyp['mosaic']
+ masks = []
+ if mosaic:
+ # Load mosaic
+ img, labels, segments = self.load_mosaic(index)
+ shapes = None
+
+ # MixUp augmentation
+ if random.random() < hyp['mixup']:
+ img, labels, segments = mixup(img, labels, segments, *self.load_mosaic(random.randint(0, self.n - 1)))
+
+ else:
+ # Load image
+ img, (h0, w0), (h, w) = self.load_image(index)
+
+ # Letterbox
+ shape = self.batch_shapes[self.batch[index]] if self.rect else self.img_size # final letterboxed shape
+ img, ratio, pad = letterbox(img, shape, auto=False, scaleup=self.augment)
+ shapes = (h0, w0), ((h / h0, w / w0), pad) # for COCO mAP rescaling
+
+ labels = self.labels[index].copy()
+ # [array, array, ....], array.shape=(num_points, 2), xyxyxyxy
+ segments = self.segments[index].copy()
+ if len(segments):
+ for i_s in range(len(segments)):
+ segments[i_s] = xyn2xy(
+ segments[i_s],
+ ratio[0] * w,
+ ratio[1] * h,
+ padw=pad[0],
+ padh=pad[1],
+ )
+ if labels.size: # normalized xywh to pixel xyxy format
+ labels[:, 1:] = xywhn2xyxy(labels[:, 1:], ratio[0] * w, ratio[1] * h, padw=pad[0], padh=pad[1])
+
+ if self.augment:
+ img, labels, segments = random_perspective(img,
+ labels,
+ segments=segments,
+ degrees=hyp['degrees'],
+ translate=hyp['translate'],
+ scale=hyp['scale'],
+ shear=hyp['shear'],
+ perspective=hyp['perspective'])
+
+ nl = len(labels) # number of labels
+ if nl:
+ labels[:, 1:5] = xyxy2xywhn(labels[:, 1:5], w=img.shape[1], h=img.shape[0], clip=True, eps=1e-3)
+ if self.overlap:
+ masks, sorted_idx = polygons2masks_overlap(img.shape[:2],
+ segments,
+ downsample_ratio=self.downsample_ratio)
+ masks = masks[None] # (640, 640) -> (1, 640, 640)
+ labels = labels[sorted_idx]
+ else:
+ masks = polygons2masks(img.shape[:2], segments, color=1, downsample_ratio=self.downsample_ratio)
+
+ masks = (torch.from_numpy(masks) if len(masks) else torch.zeros(1 if self.overlap else nl, img.shape[0] //
+ self.downsample_ratio, img.shape[1] //
+ self.downsample_ratio))
+ # TODO: albumentations support
+ if self.augment:
+ # Albumentations
+ # there are some augmentation that won't change boxes and masks,
+ # so just be it for now.
+ img, labels = self.albumentations(img, labels)
+ nl = len(labels) # update after albumentations
+
+ # HSV color-space
+ augment_hsv(img, hgain=hyp['hsv_h'], sgain=hyp['hsv_s'], vgain=hyp['hsv_v'])
+
+ # Flip up-down
+ if random.random() < hyp['flipud']:
+ img = np.flipud(img)
+ if nl:
+ labels[:, 2] = 1 - labels[:, 2]
+ masks = torch.flip(masks, dims=[1])
+
+ # Flip left-right
+ if random.random() < hyp['fliplr']:
+ img = np.fliplr(img)
+ if nl:
+ labels[:, 1] = 1 - labels[:, 1]
+ masks = torch.flip(masks, dims=[2])
+
+ # Cutouts # labels = cutout(img, labels, p=0.5)
+
+ labels_out = torch.zeros((nl, 6))
+ if nl:
+ labels_out[:, 1:] = torch.from_numpy(labels)
+
+ # Convert
+ img = img.transpose((2, 0, 1))[::-1] # HWC to CHW, BGR to RGB
+ img = np.ascontiguousarray(img)
+
+ return (torch.from_numpy(img), labels_out, self.im_files[index], shapes, masks)
+
+ def load_mosaic(self, index):
+ # YOLOv5 4-mosaic loader. Loads 1 image + 3 random images into a 4-image mosaic
+ labels4, segments4 = [], []
+ s = self.img_size
+ yc, xc = (int(random.uniform(-x, 2 * s + x)) for x in self.mosaic_border) # mosaic center x, y
+
+ # 3 additional image indices
+ indices = [index] + random.choices(self.indices, k=3) # 3 additional image indices
+ for i, index in enumerate(indices):
+ # Load image
+ img, _, (h, w) = self.load_image(index)
+
+ # place img in img4
+ if i == 0: # top left
+ img4 = np.full((s * 2, s * 2, img.shape[2]), 114, dtype=np.uint8) # base image with 4 tiles
+ x1a, y1a, x2a, y2a = max(xc - w, 0), max(yc - h, 0), xc, yc # xmin, ymin, xmax, ymax (large image)
+ x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h # xmin, ymin, xmax, ymax (small image)
+ elif i == 1: # top right
+ x1a, y1a, x2a, y2a = xc, max(yc - h, 0), min(xc + w, s * 2), yc
+ x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h
+ elif i == 2: # bottom left
+ x1a, y1a, x2a, y2a = max(xc - w, 0), yc, xc, min(s * 2, yc + h)
+ x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, w, min(y2a - y1a, h)
+ elif i == 3: # bottom right
+ x1a, y1a, x2a, y2a = xc, yc, min(xc + w, s * 2), min(s * 2, yc + h)
+ x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)
+
+ img4[y1a:y2a, x1a:x2a] = img[y1b:y2b, x1b:x2b] # img4[ymin:ymax, xmin:xmax]
+ padw = x1a - x1b
+ padh = y1a - y1b
+
+ labels, segments = self.labels[index].copy(), self.segments[index].copy()
+
+ if labels.size:
+ labels[:, 1:] = xywhn2xyxy(labels[:, 1:], w, h, padw, padh) # normalized xywh to pixel xyxy format
+ segments = [xyn2xy(x, w, h, padw, padh) for x in segments]
+ labels4.append(labels)
+ segments4.extend(segments)
+
+ # Concat/clip labels
+ labels4 = np.concatenate(labels4, 0)
+ for x in (labels4[:, 1:], *segments4):
+ np.clip(x, 0, 2 * s, out=x) # clip when using random_perspective()
+ # img4, labels4 = replicate(img4, labels4) # replicate
+
+ # Augment
+ img4, labels4, segments4 = copy_paste(img4, labels4, segments4, p=self.hyp['copy_paste'])
+ img4, labels4, segments4 = random_perspective(img4,
+ labels4,
+ segments4,
+ degrees=self.hyp['degrees'],
+ translate=self.hyp['translate'],
+ scale=self.hyp['scale'],
+ shear=self.hyp['shear'],
+ perspective=self.hyp['perspective'],
+ border=self.mosaic_border) # border to remove
+ return img4, labels4, segments4
+
+ @staticmethod
+ def collate_fn(batch):
+ img, label, path, shapes, masks = zip(*batch) # transposed
+ batched_masks = torch.cat(masks, 0)
+ for i, l in enumerate(label):
+ l[:, 0] = i # add target image index for build_targets()
+ return torch.stack(img, 0), torch.cat(label, 0), path, shapes, batched_masks
+
+
+def polygon2mask(img_size, polygons, color=1, downsample_ratio=1):
+ """
+ Args:
+ img_size (tuple): The image size.
+ polygons (np.ndarray): [N, M], N is the number of polygons,
+ M is the number of points(Be divided by 2).
+ """
+ mask = np.zeros(img_size, dtype=np.uint8)
+ polygons = np.asarray(polygons)
+ polygons = polygons.astype(np.int32)
+ shape = polygons.shape
+ polygons = polygons.reshape(shape[0], -1, 2)
+ cv2.fillPoly(mask, polygons, color=color)
+ nh, nw = (img_size[0] // downsample_ratio, img_size[1] // downsample_ratio)
+ # NOTE: fillPoly firstly then resize is trying the keep the same way
+ # of loss calculation when mask-ratio=1.
+ mask = cv2.resize(mask, (nw, nh))
+ return mask
+
+
+def polygons2masks(img_size, polygons, color, downsample_ratio=1):
+ """
+ Args:
+ img_size (tuple): The image size.
+ polygons (list[np.ndarray]): each polygon is [N, M],
+ N is the number of polygons,
+ M is the number of points(Be divided by 2).
+ """
+ masks = []
+ for si in range(len(polygons)):
+ mask = polygon2mask(img_size, [polygons[si].reshape(-1)], color, downsample_ratio)
+ masks.append(mask)
+ return np.array(masks)
+
+
+def polygons2masks_overlap(img_size, segments, downsample_ratio=1):
+ """Return a (640, 640) overlap mask."""
+ masks = np.zeros((img_size[0] // downsample_ratio, img_size[1] // downsample_ratio),
+ dtype=np.int32 if len(segments) > 255 else np.uint8)
+ areas = []
+ ms = []
+ for si in range(len(segments)):
+ mask = polygon2mask(
+ img_size,
+ [segments[si].reshape(-1)],
+ downsample_ratio=downsample_ratio,
+ color=1,
+ )
+ ms.append(mask)
+ areas.append(mask.sum())
+ areas = np.asarray(areas)
+ index = np.argsort(-areas)
+ ms = np.array(ms)[index]
+ for i in range(len(segments)):
+ mask = ms[i] * (i + 1)
+ masks = masks + mask
+ masks = np.clip(masks, a_min=0, a_max=i + 1)
+ return masks, index
diff --git a/TextDetection/utils/segment/general.py b/TextDetection/utils/segment/general.py
new file mode 100644
index 0000000000000000000000000000000000000000..f1b2f1dd120ff47eec618e0c25239c28c4d88475
--- /dev/null
+++ b/TextDetection/utils/segment/general.py
@@ -0,0 +1,160 @@
+import cv2
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+
+def crop_mask(masks, boxes):
+ """
+ "Crop" predicted masks by zeroing out everything not in the predicted bbox.
+ Vectorized by Chong (thanks Chong).
+
+ Args:
+ - masks should be a size [n, h, w] tensor of masks
+ - boxes should be a size [n, 4] tensor of bbox coords in relative point form
+ """
+
+ n, h, w = masks.shape
+ x1, y1, x2, y2 = torch.chunk(boxes[:, :, None], 4, 1) # x1 shape(1,1,n)
+ r = torch.arange(w, device=masks.device, dtype=x1.dtype)[None, None, :] # rows shape(1,w,1)
+ c = torch.arange(h, device=masks.device, dtype=x1.dtype)[None, :, None] # cols shape(h,1,1)
+
+ return masks * ((r >= x1) * (r < x2) * (c >= y1) * (c < y2))
+
+
+def process_mask_upsample(protos, masks_in, bboxes, shape):
+ """
+ Crop after upsample.
+ protos: [mask_dim, mask_h, mask_w]
+ masks_in: [n, mask_dim], n is number of masks after nms
+ bboxes: [n, 4], n is number of masks after nms
+ shape: input_image_size, (h, w)
+
+ return: h, w, n
+ """
+
+ c, mh, mw = protos.shape # CHW
+ masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)
+ masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW
+ masks = crop_mask(masks, bboxes) # CHW
+ return masks.gt_(0.5)
+
+
+def process_mask(protos, masks_in, bboxes, shape, upsample=False):
+ """
+ Crop before upsample.
+ proto_out: [mask_dim, mask_h, mask_w]
+ out_masks: [n, mask_dim], n is number of masks after nms
+ bboxes: [n, 4], n is number of masks after nms
+ shape:input_image_size, (h, w)
+
+ return: h, w, n
+ """
+
+ c, mh, mw = protos.shape # CHW
+ ih, iw = shape
+ masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw) # CHW
+
+ downsampled_bboxes = bboxes.clone()
+ downsampled_bboxes[:, 0] *= mw / iw
+ downsampled_bboxes[:, 2] *= mw / iw
+ downsampled_bboxes[:, 3] *= mh / ih
+ downsampled_bboxes[:, 1] *= mh / ih
+
+ masks = crop_mask(masks, downsampled_bboxes) # CHW
+ if upsample:
+ masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW
+ return masks.gt_(0.5)
+
+
+def process_mask_native(protos, masks_in, bboxes, shape):
+ """
+ Crop after upsample.
+ protos: [mask_dim, mask_h, mask_w]
+ masks_in: [n, mask_dim], n is number of masks after nms
+ bboxes: [n, 4], n is number of masks after nms
+ shape: input_image_size, (h, w)
+
+ return: h, w, n
+ """
+ c, mh, mw = protos.shape # CHW
+ masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)
+ gain = min(mh / shape[0], mw / shape[1]) # gain = old / new
+ pad = (mw - shape[1] * gain) / 2, (mh - shape[0] * gain) / 2 # wh padding
+ top, left = int(pad[1]), int(pad[0]) # y, x
+ bottom, right = int(mh - pad[1]), int(mw - pad[0])
+ masks = masks[:, top:bottom, left:right]
+
+ masks = F.interpolate(masks[None], shape, mode='bilinear', align_corners=False)[0] # CHW
+ masks = crop_mask(masks, bboxes) # CHW
+ return masks.gt_(0.5)
+
+
+def scale_image(im1_shape, masks, im0_shape, ratio_pad=None):
+ """
+ img1_shape: model input shape, [h, w]
+ img0_shape: origin pic shape, [h, w, 3]
+ masks: [h, w, num]
+ """
+ # Rescale coordinates (xyxy) from im1_shape to im0_shape
+ if ratio_pad is None: # calculate from im0_shape
+ gain = min(im1_shape[0] / im0_shape[0], im1_shape[1] / im0_shape[1]) # gain = old / new
+ pad = (im1_shape[1] - im0_shape[1] * gain) / 2, (im1_shape[0] - im0_shape[0] * gain) / 2 # wh padding
+ else:
+ pad = ratio_pad[1]
+ top, left = int(pad[1]), int(pad[0]) # y, x
+ bottom, right = int(im1_shape[0] - pad[1]), int(im1_shape[1] - pad[0])
+
+ if len(masks.shape) < 2:
+ raise ValueError(f'"len of masks shape" should be 2 or 3, but got {len(masks.shape)}')
+ masks = masks[top:bottom, left:right]
+ # masks = masks.permute(2, 0, 1).contiguous()
+ # masks = F.interpolate(masks[None], im0_shape[:2], mode='bilinear', align_corners=False)[0]
+ # masks = masks.permute(1, 2, 0).contiguous()
+ masks = cv2.resize(masks, (im0_shape[1], im0_shape[0]))
+
+ if len(masks.shape) == 2:
+ masks = masks[:, :, None]
+ return masks
+
+
+def mask_iou(mask1, mask2, eps=1e-7):
+ """
+ mask1: [N, n] m1 means number of predicted objects
+ mask2: [M, n] m2 means number of gt objects
+ Note: n means image_w x image_h
+
+ return: masks iou, [N, M]
+ """
+ intersection = torch.matmul(mask1, mask2.t()).clamp(0)
+ union = (mask1.sum(1)[:, None] + mask2.sum(1)[None]) - intersection # (area1 + area2) - intersection
+ return intersection / (union + eps)
+
+
+def masks_iou(mask1, mask2, eps=1e-7):
+ """
+ mask1: [N, n] m1 means number of predicted objects
+ mask2: [N, n] m2 means number of gt objects
+ Note: n means image_w x image_h
+
+ return: masks iou, (N, )
+ """
+ intersection = (mask1 * mask2).sum(1).clamp(0) # (N, )
+ union = (mask1.sum(1) + mask2.sum(1))[None] - intersection # (area1 + area2) - intersection
+ return intersection / (union + eps)
+
+
+def masks2segments(masks, strategy='largest'):
+ # Convert masks(n,160,160) into segments(n,xy)
+ segments = []
+ for x in masks.int().cpu().numpy().astype('uint8'):
+ c = cv2.findContours(x, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]
+ if c:
+ if strategy == 'concat': # concatenate all segments
+ c = np.concatenate([x.reshape(-1, 2) for x in c])
+ elif strategy == 'largest': # select largest segment
+ c = np.array(c[np.array([len(x) for x in c]).argmax()]).reshape(-1, 2)
+ else:
+ c = np.zeros((0, 2)) # no segments found
+ segments.append(c.astype('float32'))
+ return segments
diff --git a/TextDetection/utils/segment/loss.py b/TextDetection/utils/segment/loss.py
new file mode 100644
index 0000000000000000000000000000000000000000..caeff3cad586b4367990aa4626ed6c326b04baf3
--- /dev/null
+++ b/TextDetection/utils/segment/loss.py
@@ -0,0 +1,185 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from ..general import xywh2xyxy
+from ..loss import FocalLoss, smooth_BCE
+from ..metrics import bbox_iou
+from ..torch_utils import de_parallel
+from .general import crop_mask
+
+
+class ComputeLoss:
+ # Compute losses
+ def __init__(self, model, autobalance=False, overlap=False):
+ self.sort_obj_iou = False
+ self.overlap = overlap
+ device = next(model.parameters()).device # get model device
+ h = model.hyp # hyperparameters
+
+ # Define criteria
+ BCEcls = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['cls_pw']], device=device))
+ BCEobj = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['obj_pw']], device=device))
+
+ # Class label smoothing https://arxiv.org/pdf/1902.04103.pdf eqn 3
+ self.cp, self.cn = smooth_BCE(eps=h.get('label_smoothing', 0.0)) # positive, negative BCE targets
+
+ # Focal loss
+ g = h['fl_gamma'] # focal loss gamma
+ if g > 0:
+ BCEcls, BCEobj = FocalLoss(BCEcls, g), FocalLoss(BCEobj, g)
+
+ m = de_parallel(model).model[-1] # Detect() module
+ self.balance = {3: [4.0, 1.0, 0.4]}.get(m.nl, [4.0, 1.0, 0.25, 0.06, 0.02]) # P3-P7
+ self.ssi = list(m.stride).index(16) if autobalance else 0 # stride 16 index
+ self.BCEcls, self.BCEobj, self.gr, self.hyp, self.autobalance = BCEcls, BCEobj, 1.0, h, autobalance
+ self.na = m.na # number of anchors
+ self.nc = m.nc # number of classes
+ self.nl = m.nl # number of layers
+ self.nm = m.nm # number of masks
+ self.anchors = m.anchors
+ self.device = device
+
+ def __call__(self, preds, targets, masks): # predictions, targets, model
+ p, proto = preds
+ bs, nm, mask_h, mask_w = proto.shape # batch size, number of masks, mask height, mask width
+ lcls = torch.zeros(1, device=self.device)
+ lbox = torch.zeros(1, device=self.device)
+ lobj = torch.zeros(1, device=self.device)
+ lseg = torch.zeros(1, device=self.device)
+ tcls, tbox, indices, anchors, tidxs, xywhn = self.build_targets(p, targets) # targets
+
+ # Losses
+ for i, pi in enumerate(p): # layer index, layer predictions
+ b, a, gj, gi = indices[i] # image, anchor, gridy, gridx
+ tobj = torch.zeros(pi.shape[:4], dtype=pi.dtype, device=self.device) # target obj
+
+ n = b.shape[0] # number of targets
+ if n:
+ pxy, pwh, _, pcls, pmask = pi[b, a, gj, gi].split((2, 2, 1, self.nc, nm), 1) # subset of predictions
+
+ # Box regression
+ pxy = pxy.sigmoid() * 2 - 0.5
+ pwh = (pwh.sigmoid() * 2) ** 2 * anchors[i]
+ pbox = torch.cat((pxy, pwh), 1) # predicted box
+ iou = bbox_iou(pbox, tbox[i], CIoU=True).squeeze() # iou(prediction, target)
+ lbox += (1.0 - iou).mean() # iou loss
+
+ # Objectness
+ iou = iou.detach().clamp(0).type(tobj.dtype)
+ if self.sort_obj_iou:
+ j = iou.argsort()
+ b, a, gj, gi, iou = b[j], a[j], gj[j], gi[j], iou[j]
+ if self.gr < 1:
+ iou = (1.0 - self.gr) + self.gr * iou
+ tobj[b, a, gj, gi] = iou # iou ratio
+
+ # Classification
+ if self.nc > 1: # cls loss (only if multiple classes)
+ t = torch.full_like(pcls, self.cn, device=self.device) # targets
+ t[range(n), tcls[i]] = self.cp
+ lcls += self.BCEcls(pcls, t) # BCE
+
+ # Mask regression
+ if tuple(masks.shape[-2:]) != (mask_h, mask_w): # downsample
+ masks = F.interpolate(masks[None], (mask_h, mask_w), mode='nearest')[0]
+ marea = xywhn[i][:, 2:].prod(1) # mask width, height normalized
+ mxyxy = xywh2xyxy(xywhn[i] * torch.tensor([mask_w, mask_h, mask_w, mask_h], device=self.device))
+ for bi in b.unique():
+ j = b == bi # matching index
+ if self.overlap:
+ mask_gti = torch.where(masks[bi][None] == tidxs[i][j].view(-1, 1, 1), 1.0, 0.0)
+ else:
+ mask_gti = masks[tidxs[i]][j]
+ lseg += self.single_mask_loss(mask_gti, pmask[j], proto[bi], mxyxy[j], marea[j])
+
+ obji = self.BCEobj(pi[..., 4], tobj)
+ lobj += obji * self.balance[i] # obj loss
+ if self.autobalance:
+ self.balance[i] = self.balance[i] * 0.9999 + 0.0001 / obji.detach().item()
+
+ if self.autobalance:
+ self.balance = [x / self.balance[self.ssi] for x in self.balance]
+ lbox *= self.hyp['box']
+ lobj *= self.hyp['obj']
+ lcls *= self.hyp['cls']
+ lseg *= self.hyp['box'] / bs
+
+ loss = lbox + lobj + lcls + lseg
+ return loss * bs, torch.cat((lbox, lseg, lobj, lcls)).detach()
+
+ def single_mask_loss(self, gt_mask, pred, proto, xyxy, area):
+ # Mask loss for one image
+ pred_mask = (pred @ proto.view(self.nm, -1)).view(-1, *proto.shape[1:]) # (n,32) @ (32,80,80) -> (n,80,80)
+ loss = F.binary_cross_entropy_with_logits(pred_mask, gt_mask, reduction='none')
+ return (crop_mask(loss, xyxy).mean(dim=(1, 2)) / area).mean()
+
+ def build_targets(self, p, targets):
+ # Build targets for compute_loss(), input targets(image,class,x,y,w,h)
+ na, nt = self.na, targets.shape[0] # number of anchors, targets
+ tcls, tbox, indices, anch, tidxs, xywhn = [], [], [], [], [], []
+ gain = torch.ones(8, device=self.device) # normalized to gridspace gain
+ ai = torch.arange(na, device=self.device).float().view(na, 1).repeat(1, nt) # same as .repeat_interleave(nt)
+ if self.overlap:
+ batch = p[0].shape[0]
+ ti = []
+ for i in range(batch):
+ num = (targets[:, 0] == i).sum() # find number of targets of each image
+ ti.append(torch.arange(num, device=self.device).float().view(1, num).repeat(na, 1) + 1) # (na, num)
+ ti = torch.cat(ti, 1) # (na, nt)
+ else:
+ ti = torch.arange(nt, device=self.device).float().view(1, nt).repeat(na, 1)
+ targets = torch.cat((targets.repeat(na, 1, 1), ai[..., None], ti[..., None]), 2) # append anchor indices
+
+ g = 0.5 # bias
+ off = torch.tensor(
+ [
+ [0, 0],
+ [1, 0],
+ [0, 1],
+ [-1, 0],
+ [0, -1], # j,k,l,m
+ # [1, 1], [1, -1], [-1, 1], [-1, -1], # jk,jm,lk,lm
+ ],
+ device=self.device).float() * g # offsets
+
+ for i in range(self.nl):
+ anchors, shape = self.anchors[i], p[i].shape
+ gain[2:6] = torch.tensor(shape)[[3, 2, 3, 2]] # xyxy gain
+
+ # Match targets to anchors
+ t = targets * gain # shape(3,n,7)
+ if nt:
+ # Matches
+ r = t[..., 4:6] / anchors[:, None] # wh ratio
+ j = torch.max(r, 1 / r).max(2)[0] < self.hyp['anchor_t'] # compare
+ # j = wh_iou(anchors, t[:, 4:6]) > model.hyp['iou_t'] # iou(3,n)=wh_iou(anchors(3,2), gwh(n,2))
+ t = t[j] # filter
+
+ # Offsets
+ gxy = t[:, 2:4] # grid xy
+ gxi = gain[[2, 3]] - gxy # inverse
+ j, k = ((gxy % 1 < g) & (gxy > 1)).T
+ l, m = ((gxi % 1 < g) & (gxi > 1)).T
+ j = torch.stack((torch.ones_like(j), j, k, l, m))
+ t = t.repeat((5, 1, 1))[j]
+ offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j]
+ else:
+ t = targets[0]
+ offsets = 0
+
+ # Define
+ bc, gxy, gwh, at = t.chunk(4, 1) # (image, class), grid xy, grid wh, anchors
+ (a, tidx), (b, c) = at.long().T, bc.long().T # anchors, image, class
+ gij = (gxy - offsets).long()
+ gi, gj = gij.T # grid indices
+
+ # Append
+ indices.append((b, a, gj.clamp_(0, shape[2] - 1), gi.clamp_(0, shape[3] - 1))) # image, anchor, grid
+ tbox.append(torch.cat((gxy - gij, gwh), 1)) # box
+ anch.append(anchors[a]) # anchors
+ tcls.append(c) # class
+ tidxs.append(tidx)
+ xywhn.append(torch.cat((gxy, gwh), 1) / gain[2:6]) # xywh normalized
+
+ return tcls, tbox, indices, anch, tidxs, xywhn
diff --git a/TextDetection/utils/segment/metrics.py b/TextDetection/utils/segment/metrics.py
new file mode 100644
index 0000000000000000000000000000000000000000..787961bee1bf00731274ae87cf04e1bc49248e64
--- /dev/null
+++ b/TextDetection/utils/segment/metrics.py
@@ -0,0 +1,210 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Model validation metrics
+"""
+
+import numpy as np
+
+from ..metrics import ap_per_class
+
+
+def fitness(x):
+ # Model fitness as a weighted combination of metrics
+ w = [0.0, 0.0, 0.1, 0.9, 0.0, 0.0, 0.1, 0.9]
+ return (x[:, :8] * w).sum(1)
+
+
+def ap_per_class_box_and_mask(
+ tp_m,
+ tp_b,
+ conf,
+ pred_cls,
+ target_cls,
+ plot=False,
+ save_dir='.',
+ names=(),
+):
+ """
+ Args:
+ tp_b: tp of boxes.
+ tp_m: tp of masks.
+ other arguments see `func: ap_per_class`.
+ """
+ results_boxes = ap_per_class(tp_b,
+ conf,
+ pred_cls,
+ target_cls,
+ plot=plot,
+ save_dir=save_dir,
+ names=names,
+ prefix='Box')[2:]
+ results_masks = ap_per_class(tp_m,
+ conf,
+ pred_cls,
+ target_cls,
+ plot=plot,
+ save_dir=save_dir,
+ names=names,
+ prefix='Mask')[2:]
+
+ results = {
+ 'boxes': {
+ 'p': results_boxes[0],
+ 'r': results_boxes[1],
+ 'ap': results_boxes[3],
+ 'f1': results_boxes[2],
+ 'ap_class': results_boxes[4]},
+ 'masks': {
+ 'p': results_masks[0],
+ 'r': results_masks[1],
+ 'ap': results_masks[3],
+ 'f1': results_masks[2],
+ 'ap_class': results_masks[4]}}
+ return results
+
+
+class Metric:
+
+ def __init__(self) -> None:
+ self.p = [] # (nc, )
+ self.r = [] # (nc, )
+ self.f1 = [] # (nc, )
+ self.all_ap = [] # (nc, 10)
+ self.ap_class_index = [] # (nc, )
+
+ @property
+ def ap50(self):
+ """AP@0.5 of all classes.
+ Return:
+ (nc, ) or [].
+ """
+ return self.all_ap[:, 0] if len(self.all_ap) else []
+
+ @property
+ def ap(self):
+ """AP@0.5:0.95
+ Return:
+ (nc, ) or [].
+ """
+ return self.all_ap.mean(1) if len(self.all_ap) else []
+
+ @property
+ def mp(self):
+ """mean precision of all classes.
+ Return:
+ float.
+ """
+ return self.p.mean() if len(self.p) else 0.0
+
+ @property
+ def mr(self):
+ """mean recall of all classes.
+ Return:
+ float.
+ """
+ return self.r.mean() if len(self.r) else 0.0
+
+ @property
+ def map50(self):
+ """Mean AP@0.5 of all classes.
+ Return:
+ float.
+ """
+ return self.all_ap[:, 0].mean() if len(self.all_ap) else 0.0
+
+ @property
+ def map(self):
+ """Mean AP@0.5:0.95 of all classes.
+ Return:
+ float.
+ """
+ return self.all_ap.mean() if len(self.all_ap) else 0.0
+
+ def mean_results(self):
+ """Mean of results, return mp, mr, map50, map"""
+ return (self.mp, self.mr, self.map50, self.map)
+
+ def class_result(self, i):
+ """class-aware result, return p[i], r[i], ap50[i], ap[i]"""
+ return (self.p[i], self.r[i], self.ap50[i], self.ap[i])
+
+ def get_maps(self, nc):
+ maps = np.zeros(nc) + self.map
+ for i, c in enumerate(self.ap_class_index):
+ maps[c] = self.ap[i]
+ return maps
+
+ def update(self, results):
+ """
+ Args:
+ results: tuple(p, r, ap, f1, ap_class)
+ """
+ p, r, all_ap, f1, ap_class_index = results
+ self.p = p
+ self.r = r
+ self.all_ap = all_ap
+ self.f1 = f1
+ self.ap_class_index = ap_class_index
+
+
+class Metrics:
+ """Metric for boxes and masks."""
+
+ def __init__(self) -> None:
+ self.metric_box = Metric()
+ self.metric_mask = Metric()
+
+ def update(self, results):
+ """
+ Args:
+ results: Dict{'boxes': Dict{}, 'masks': Dict{}}
+ """
+ self.metric_box.update(list(results['boxes'].values()))
+ self.metric_mask.update(list(results['masks'].values()))
+
+ def mean_results(self):
+ return self.metric_box.mean_results() + self.metric_mask.mean_results()
+
+ def class_result(self, i):
+ return self.metric_box.class_result(i) + self.metric_mask.class_result(i)
+
+ def get_maps(self, nc):
+ return self.metric_box.get_maps(nc) + self.metric_mask.get_maps(nc)
+
+ @property
+ def ap_class_index(self):
+ # boxes and masks have the same ap_class_index
+ return self.metric_box.ap_class_index
+
+
+KEYS = [
+ 'train/box_loss',
+ 'train/seg_loss', # train loss
+ 'train/obj_loss',
+ 'train/cls_loss',
+ 'metrics/precision(B)',
+ 'metrics/recall(B)',
+ 'metrics/mAP_0.5(B)',
+ 'metrics/mAP_0.5:0.95(B)', # metrics
+ 'metrics/precision(M)',
+ 'metrics/recall(M)',
+ 'metrics/mAP_0.5(M)',
+ 'metrics/mAP_0.5:0.95(M)', # metrics
+ 'val/box_loss',
+ 'val/seg_loss', # val loss
+ 'val/obj_loss',
+ 'val/cls_loss',
+ 'x/lr0',
+ 'x/lr1',
+ 'x/lr2', ]
+
+BEST_KEYS = [
+ 'best/epoch',
+ 'best/precision(B)',
+ 'best/recall(B)',
+ 'best/mAP_0.5(B)',
+ 'best/mAP_0.5:0.95(B)',
+ 'best/precision(M)',
+ 'best/recall(M)',
+ 'best/mAP_0.5(M)',
+ 'best/mAP_0.5:0.95(M)', ]
diff --git a/TextDetection/utils/segment/plots.py b/TextDetection/utils/segment/plots.py
new file mode 100644
index 0000000000000000000000000000000000000000..1b22ec838ac93220187b60f5bdaf50eae19d7397
--- /dev/null
+++ b/TextDetection/utils/segment/plots.py
@@ -0,0 +1,143 @@
+import contextlib
+import math
+from pathlib import Path
+
+import cv2
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import torch
+
+from .. import threaded
+from ..general import xywh2xyxy
+from ..plots import Annotator, colors
+
+
+@threaded
+def plot_images_and_masks(images, targets, masks, paths=None, fname='images.jpg', names=None):
+ # Plot image grid with labels
+ if isinstance(images, torch.Tensor):
+ images = images.cpu().float().numpy()
+ if isinstance(targets, torch.Tensor):
+ targets = targets.cpu().numpy()
+ if isinstance(masks, torch.Tensor):
+ masks = masks.cpu().numpy().astype(int)
+
+ max_size = 1920 # max image size
+ max_subplots = 16 # max image subplots, i.e. 4x4
+ bs, _, h, w = images.shape # batch size, _, height, width
+ bs = min(bs, max_subplots) # limit plot images
+ ns = np.ceil(bs ** 0.5) # number of subplots (square)
+ if np.max(images[0]) <= 1:
+ images *= 255 # de-normalise (optional)
+
+ # Build Image
+ mosaic = np.full((int(ns * h), int(ns * w), 3), 255, dtype=np.uint8) # init
+ for i, im in enumerate(images):
+ if i == max_subplots: # if last batch has fewer images than we expect
+ break
+ x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
+ im = im.transpose(1, 2, 0)
+ mosaic[y:y + h, x:x + w, :] = im
+
+ # Resize (optional)
+ scale = max_size / ns / max(h, w)
+ if scale < 1:
+ h = math.ceil(scale * h)
+ w = math.ceil(scale * w)
+ mosaic = cv2.resize(mosaic, tuple(int(x * ns) for x in (w, h)))
+
+ # Annotate
+ fs = int((h + w) * ns * 0.01) # font size
+ annotator = Annotator(mosaic, line_width=round(fs / 10), font_size=fs, pil=True, example=names)
+ for i in range(i + 1):
+ x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
+ annotator.rectangle([x, y, x + w, y + h], None, (255, 255, 255), width=2) # borders
+ if paths:
+ annotator.text((x + 5, y + 5), text=Path(paths[i]).name[:40], txt_color=(220, 220, 220)) # filenames
+ if len(targets) > 0:
+ idx = targets[:, 0] == i
+ ti = targets[idx] # image targets
+
+ boxes = xywh2xyxy(ti[:, 2:6]).T
+ classes = ti[:, 1].astype('int')
+ labels = ti.shape[1] == 6 # labels if no conf column
+ conf = None if labels else ti[:, 6] # check for confidence presence (label vs pred)
+
+ if boxes.shape[1]:
+ if boxes.max() <= 1.01: # if normalized with tolerance 0.01
+ boxes[[0, 2]] *= w # scale to pixels
+ boxes[[1, 3]] *= h
+ elif scale < 1: # absolute coords need scale if image scales
+ boxes *= scale
+ boxes[[0, 2]] += x
+ boxes[[1, 3]] += y
+ for j, box in enumerate(boxes.T.tolist()):
+ cls = classes[j]
+ color = colors(cls)
+ cls = names[cls] if names else cls
+ if labels or conf[j] > 0.25: # 0.25 conf thresh
+ label = f'{cls}' if labels else f'{cls} {conf[j]:.1f}'
+ annotator.box_label(box, label, color=color)
+
+ # Plot masks
+ if len(masks):
+ if masks.max() > 1.0: # mean that masks are overlap
+ image_masks = masks[[i]] # (1, 640, 640)
+ nl = len(ti)
+ index = np.arange(nl).reshape(nl, 1, 1) + 1
+ image_masks = np.repeat(image_masks, nl, axis=0)
+ image_masks = np.where(image_masks == index, 1.0, 0.0)
+ else:
+ image_masks = masks[idx]
+
+ im = np.asarray(annotator.im).copy()
+ for j, box in enumerate(boxes.T.tolist()):
+ if labels or conf[j] > 0.25: # 0.25 conf thresh
+ color = colors(classes[j])
+ mh, mw = image_masks[j].shape
+ if mh != h or mw != w:
+ mask = image_masks[j].astype(np.uint8)
+ mask = cv2.resize(mask, (w, h))
+ mask = mask.astype(bool)
+ else:
+ mask = image_masks[j].astype(bool)
+ with contextlib.suppress(Exception):
+ im[y:y + h, x:x + w, :][mask] = im[y:y + h, x:x + w, :][mask] * 0.4 + np.array(color) * 0.6
+ annotator.fromarray(im)
+ annotator.im.save(fname) # save
+
+
+def plot_results_with_masks(file='path/to/results.csv', dir='', best=True):
+ # Plot training results.csv. Usage: from utils.plots import *; plot_results('path/to/results.csv')
+ save_dir = Path(file).parent if file else Path(dir)
+ fig, ax = plt.subplots(2, 8, figsize=(18, 6), tight_layout=True)
+ ax = ax.ravel()
+ files = list(save_dir.glob('results*.csv'))
+ assert len(files), f'No results.csv files found in {save_dir.resolve()}, nothing to plot.'
+ for f in files:
+ try:
+ data = pd.read_csv(f)
+ index = np.argmax(0.9 * data.values[:, 8] + 0.1 * data.values[:, 7] + 0.9 * data.values[:, 12] +
+ 0.1 * data.values[:, 11])
+ s = [x.strip() for x in data.columns]
+ x = data.values[:, 0]
+ for i, j in enumerate([1, 2, 3, 4, 5, 6, 9, 10, 13, 14, 15, 16, 7, 8, 11, 12]):
+ y = data.values[:, j]
+ # y[y == 0] = np.nan # don't show zero values
+ ax[i].plot(x, y, marker='.', label=f.stem, linewidth=2, markersize=2)
+ if best:
+ # best
+ ax[i].scatter(index, y[index], color='r', label=f'best:{index}', marker='*', linewidth=3)
+ ax[i].set_title(s[j] + f'\n{round(y[index], 5)}')
+ else:
+ # last
+ ax[i].scatter(x[-1], y[-1], color='r', label='last', marker='*', linewidth=3)
+ ax[i].set_title(s[j] + f'\n{round(y[-1], 5)}')
+ # if j in [8, 9, 10]: # share train and val loss y axes
+ # ax[i].get_shared_y_axes().join(ax[i], ax[i - 5])
+ except Exception as e:
+ print(f'Warning: Plotting error for {f}: {e}')
+ ax[1].legend()
+ fig.savefig(save_dir / 'results.png', dpi=200)
+ plt.close()
diff --git a/TextDetection/utils/torch_utils.py b/TextDetection/utils/torch_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..13a356f3238c53356907153e8ded9598c2a4a448
--- /dev/null
+++ b/TextDetection/utils/torch_utils.py
@@ -0,0 +1,432 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+PyTorch utils
+"""
+
+import math
+import os
+import platform
+import subprocess
+import time
+import warnings
+from contextlib import contextmanager
+from copy import deepcopy
+from pathlib import Path
+
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+from utils.general import LOGGER, check_version, colorstr, file_date, git_describe
+
+LOCAL_RANK = int(os.getenv('LOCAL_RANK', -1)) # https://pytorch.org/docs/stable/elastic/run.html
+RANK = int(os.getenv('RANK', -1))
+WORLD_SIZE = int(os.getenv('WORLD_SIZE', 1))
+
+try:
+ import thop # for FLOPs computation
+except ImportError:
+ thop = None
+
+# Suppress PyTorch warnings
+warnings.filterwarnings('ignore', message='User provided device_type of \'cuda\', but CUDA is not available. Disabling')
+warnings.filterwarnings('ignore', category=UserWarning)
+
+
+def smart_inference_mode(torch_1_9=check_version(torch.__version__, '1.9.0')):
+ # Applies torch.inference_mode() decorator if torch>=1.9.0 else torch.no_grad() decorator
+ def decorate(fn):
+ return (torch.inference_mode if torch_1_9 else torch.no_grad)()(fn)
+
+ return decorate
+
+
+def smartCrossEntropyLoss(label_smoothing=0.0):
+ # Returns nn.CrossEntropyLoss with label smoothing enabled for torch>=1.10.0
+ if check_version(torch.__version__, '1.10.0'):
+ return nn.CrossEntropyLoss(label_smoothing=label_smoothing)
+ if label_smoothing > 0:
+ LOGGER.warning(f'WARNING ⚠️ label smoothing {label_smoothing} requires torch>=1.10.0')
+ return nn.CrossEntropyLoss()
+
+
+def smart_DDP(model):
+ # Model DDP creation with checks
+ assert not check_version(torch.__version__, '1.12.0', pinned=True), \
+ 'torch==1.12.0 torchvision==0.13.0 DDP training is not supported due to a known issue. ' \
+ 'Please upgrade or downgrade torch to use DDP. See https://github.com/ultralytics/yolov5/issues/8395'
+ if check_version(torch.__version__, '1.11.0'):
+ return DDP(model, device_ids=[LOCAL_RANK], output_device=LOCAL_RANK, static_graph=True)
+ else:
+ return DDP(model, device_ids=[LOCAL_RANK], output_device=LOCAL_RANK)
+
+
+def reshape_classifier_output(model, n=1000):
+ # Update a TorchVision classification model to class count 'n' if required
+ from models.common import Classify
+ name, m = list((model.model if hasattr(model, 'model') else model).named_children())[-1] # last module
+ if isinstance(m, Classify): # YOLOv5 Classify() head
+ if m.linear.out_features != n:
+ m.linear = nn.Linear(m.linear.in_features, n)
+ elif isinstance(m, nn.Linear): # ResNet, EfficientNet
+ if m.out_features != n:
+ setattr(model, name, nn.Linear(m.in_features, n))
+ elif isinstance(m, nn.Sequential):
+ types = [type(x) for x in m]
+ if nn.Linear in types:
+ i = types.index(nn.Linear) # nn.Linear index
+ if m[i].out_features != n:
+ m[i] = nn.Linear(m[i].in_features, n)
+ elif nn.Conv2d in types:
+ i = types.index(nn.Conv2d) # nn.Conv2d index
+ if m[i].out_channels != n:
+ m[i] = nn.Conv2d(m[i].in_channels, n, m[i].kernel_size, m[i].stride, bias=m[i].bias is not None)
+
+
+@contextmanager
+def torch_distributed_zero_first(local_rank: int):
+ # Decorator to make all processes in distributed training wait for each local_master to do something
+ if local_rank not in [-1, 0]:
+ dist.barrier(device_ids=[local_rank])
+ yield
+ if local_rank == 0:
+ dist.barrier(device_ids=[0])
+
+
+def device_count():
+ # Returns number of CUDA devices available. Safe version of torch.cuda.device_count(). Supports Linux and Windows
+ assert platform.system() in ('Linux', 'Windows'), 'device_count() only supported on Linux or Windows'
+ try:
+ cmd = 'nvidia-smi -L | wc -l' if platform.system() == 'Linux' else 'nvidia-smi -L | find /c /v ""' # Windows
+ return int(subprocess.run(cmd, shell=True, capture_output=True, check=True).stdout.decode().split()[-1])
+ except Exception:
+ return 0
+
+
+def select_device(device='', batch_size=0, newline=True):
+ # device = None or 'cpu' or 0 or '0' or '0,1,2,3'
+ s = f'YOLOv5 🚀 {git_describe() or file_date()} Python-{platform.python_version()} torch-{torch.__version__} '
+ device = str(device).strip().lower().replace('cuda:', '').replace('none', '') # to string, 'cuda:0' to '0'
+ cpu = device == 'cpu'
+ mps = device == 'mps' # Apple Metal Performance Shaders (MPS)
+ if cpu or mps:
+ os.environ['CUDA_VISIBLE_DEVICES'] = '-1' # force torch.cuda.is_available() = False
+ elif device: # non-cpu device requested
+ os.environ['CUDA_VISIBLE_DEVICES'] = device # set environment variable - must be before assert is_available()
+ assert torch.cuda.is_available() and torch.cuda.device_count() >= len(device.replace(',', '')), \
+ f"Invalid CUDA '--device {device}' requested, use '--device cpu' or pass valid CUDA device(s)"
+
+ if not cpu and not mps and torch.cuda.is_available(): # prefer GPU if available
+ devices = device.split(',') if device else '0' # range(torch.cuda.device_count()) # i.e. 0,1,6,7
+ n = len(devices) # device count
+ if n > 1 and batch_size > 0: # check batch_size is divisible by device_count
+ assert batch_size % n == 0, f'batch-size {batch_size} not multiple of GPU count {n}'
+ space = ' ' * (len(s) + 1)
+ for i, d in enumerate(devices):
+ p = torch.cuda.get_device_properties(i)
+ s += f"{'' if i == 0 else space}CUDA:{d} ({p.name}, {p.total_memory / (1 << 20):.0f}MiB)\n" # bytes to MB
+ arg = 'cuda:0'
+ elif mps and getattr(torch, 'has_mps', False) and torch.backends.mps.is_available(): # prefer MPS if available
+ s += 'MPS\n'
+ arg = 'mps'
+ else: # revert to CPU
+ s += 'CPU\n'
+ arg = 'cpu'
+
+ if not newline:
+ s = s.rstrip()
+ LOGGER.info(s)
+ return torch.device(arg)
+
+
+def time_sync():
+ # PyTorch-accurate time
+ if torch.cuda.is_available():
+ torch.cuda.synchronize()
+ return time.time()
+
+
+def profile(input, ops, n=10, device=None):
+ """ YOLOv5 speed/memory/FLOPs profiler
+ Usage:
+ input = torch.randn(16, 3, 640, 640)
+ m1 = lambda x: x * torch.sigmoid(x)
+ m2 = nn.SiLU()
+ profile(input, [m1, m2], n=100) # profile over 100 iterations
+ """
+ results = []
+ if not isinstance(device, torch.device):
+ device = select_device(device)
+ print(f"{'Params':>12s}{'GFLOPs':>12s}{'GPU_mem (GB)':>14s}{'forward (ms)':>14s}{'backward (ms)':>14s}"
+ f"{'input':>24s}{'output':>24s}")
+
+ for x in input if isinstance(input, list) else [input]:
+ x = x.to(device)
+ x.requires_grad = True
+ for m in ops if isinstance(ops, list) else [ops]:
+ m = m.to(device) if hasattr(m, 'to') else m # device
+ m = m.half() if hasattr(m, 'half') and isinstance(x, torch.Tensor) and x.dtype is torch.float16 else m
+ tf, tb, t = 0, 0, [0, 0, 0] # dt forward, backward
+ try:
+ flops = thop.profile(m, inputs=(x, ), verbose=False)[0] / 1E9 * 2 # GFLOPs
+ except Exception:
+ flops = 0
+
+ try:
+ for _ in range(n):
+ t[0] = time_sync()
+ y = m(x)
+ t[1] = time_sync()
+ try:
+ _ = (sum(yi.sum() for yi in y) if isinstance(y, list) else y).sum().backward()
+ t[2] = time_sync()
+ except Exception: # no backward method
+ # print(e) # for debug
+ t[2] = float('nan')
+ tf += (t[1] - t[0]) * 1000 / n # ms per op forward
+ tb += (t[2] - t[1]) * 1000 / n # ms per op backward
+ mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0 # (GB)
+ s_in, s_out = (tuple(x.shape) if isinstance(x, torch.Tensor) else 'list' for x in (x, y)) # shapes
+ p = sum(x.numel() for x in m.parameters()) if isinstance(m, nn.Module) else 0 # parameters
+ print(f'{p:12}{flops:12.4g}{mem:>14.3f}{tf:14.4g}{tb:14.4g}{str(s_in):>24s}{str(s_out):>24s}')
+ results.append([p, flops, mem, tf, tb, s_in, s_out])
+ except Exception as e:
+ print(e)
+ results.append(None)
+ torch.cuda.empty_cache()
+ return results
+
+
+def is_parallel(model):
+ # Returns True if model is of type DP or DDP
+ return type(model) in (nn.parallel.DataParallel, nn.parallel.DistributedDataParallel)
+
+
+def de_parallel(model):
+ # De-parallelize a model: returns single-GPU model if model is of type DP or DDP
+ return model.module if is_parallel(model) else model
+
+
+def initialize_weights(model):
+ for m in model.modules():
+ t = type(m)
+ if t is nn.Conv2d:
+ pass # nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
+ elif t is nn.BatchNorm2d:
+ m.eps = 1e-3
+ m.momentum = 0.03
+ elif t in [nn.Hardswish, nn.LeakyReLU, nn.ReLU, nn.ReLU6, nn.SiLU]:
+ m.inplace = True
+
+
+def find_modules(model, mclass=nn.Conv2d):
+ # Finds layer indices matching module class 'mclass'
+ return [i for i, m in enumerate(model.module_list) if isinstance(m, mclass)]
+
+
+def sparsity(model):
+ # Return global model sparsity
+ a, b = 0, 0
+ for p in model.parameters():
+ a += p.numel()
+ b += (p == 0).sum()
+ return b / a
+
+
+def prune(model, amount=0.3):
+ # Prune model to requested global sparsity
+ import torch.nn.utils.prune as prune
+ for name, m in model.named_modules():
+ if isinstance(m, nn.Conv2d):
+ prune.l1_unstructured(m, name='weight', amount=amount) # prune
+ prune.remove(m, 'weight') # make permanent
+ LOGGER.info(f'Model pruned to {sparsity(model):.3g} global sparsity')
+
+
+def fuse_conv_and_bn(conv, bn):
+ # Fuse Conv2d() and BatchNorm2d() layers https://tehnokv.com/posts/fusing-batchnorm-and-conv/
+ fusedconv = nn.Conv2d(conv.in_channels,
+ conv.out_channels,
+ kernel_size=conv.kernel_size,
+ stride=conv.stride,
+ padding=conv.padding,
+ dilation=conv.dilation,
+ groups=conv.groups,
+ bias=True).requires_grad_(False).to(conv.weight.device)
+
+ # Prepare filters
+ w_conv = conv.weight.clone().view(conv.out_channels, -1)
+ w_bn = torch.diag(bn.weight.div(torch.sqrt(bn.eps + bn.running_var)))
+ fusedconv.weight.copy_(torch.mm(w_bn, w_conv).view(fusedconv.weight.shape))
+
+ # Prepare spatial bias
+ b_conv = torch.zeros(conv.weight.size(0), device=conv.weight.device) if conv.bias is None else conv.bias
+ b_bn = bn.bias - bn.weight.mul(bn.running_mean).div(torch.sqrt(bn.running_var + bn.eps))
+ fusedconv.bias.copy_(torch.mm(w_bn, b_conv.reshape(-1, 1)).reshape(-1) + b_bn)
+
+ return fusedconv
+
+
+def model_info(model, verbose=False, imgsz=640):
+ # Model information. img_size may be int or list, i.e. img_size=640 or img_size=[640, 320]
+ n_p = sum(x.numel() for x in model.parameters()) # number parameters
+ n_g = sum(x.numel() for x in model.parameters() if x.requires_grad) # number gradients
+ if verbose:
+ print(f"{'layer':>5} {'name':>40} {'gradient':>9} {'parameters':>12} {'shape':>20} {'mu':>10} {'sigma':>10}")
+ for i, (name, p) in enumerate(model.named_parameters()):
+ name = name.replace('module_list.', '')
+ print('%5g %40s %9s %12g %20s %10.3g %10.3g' %
+ (i, name, p.requires_grad, p.numel(), list(p.shape), p.mean(), p.std()))
+
+ try: # FLOPs
+ p = next(model.parameters())
+ stride = max(int(model.stride.max()), 32) if hasattr(model, 'stride') else 32 # max stride
+ im = torch.empty((1, p.shape[1], stride, stride), device=p.device) # input image in BCHW format
+ flops = thop.profile(deepcopy(model), inputs=(im, ), verbose=False)[0] / 1E9 * 2 # stride GFLOPs
+ imgsz = imgsz if isinstance(imgsz, list) else [imgsz, imgsz] # expand if int/float
+ fs = f', {flops * imgsz[0] / stride * imgsz[1] / stride:.1f} GFLOPs' # 640x640 GFLOPs
+ except Exception:
+ fs = ''
+
+ name = Path(model.yaml_file).stem.replace('yolov5', 'YOLOv5') if hasattr(model, 'yaml_file') else 'Model'
+ LOGGER.info(f'{name} summary: {len(list(model.modules()))} layers, {n_p} parameters, {n_g} gradients{fs}')
+
+
+def scale_img(img, ratio=1.0, same_shape=False, gs=32): # img(16,3,256,416)
+ # Scales img(bs,3,y,x) by ratio constrained to gs-multiple
+ if ratio == 1.0:
+ return img
+ h, w = img.shape[2:]
+ s = (int(h * ratio), int(w * ratio)) # new size
+ img = F.interpolate(img, size=s, mode='bilinear', align_corners=False) # resize
+ if not same_shape: # pad/crop img
+ h, w = (math.ceil(x * ratio / gs) * gs for x in (h, w))
+ return F.pad(img, [0, w - s[1], 0, h - s[0]], value=0.447) # value = imagenet mean
+
+
+def copy_attr(a, b, include=(), exclude=()):
+ # Copy attributes from b to a, options to only include [...] and to exclude [...]
+ for k, v in b.__dict__.items():
+ if (len(include) and k not in include) or k.startswith('_') or k in exclude:
+ continue
+ else:
+ setattr(a, k, v)
+
+
+def smart_optimizer(model, name='Adam', lr=0.001, momentum=0.9, decay=1e-5):
+ # YOLOv5 3-param group optimizer: 0) weights with decay, 1) weights no decay, 2) biases no decay
+ g = [], [], [] # optimizer parameter groups
+ bn = tuple(v for k, v in nn.__dict__.items() if 'Norm' in k) # normalization layers, i.e. BatchNorm2d()
+ for v in model.modules():
+ for p_name, p in v.named_parameters(recurse=0):
+ if p_name == 'bias': # bias (no decay)
+ g[2].append(p)
+ elif p_name == 'weight' and isinstance(v, bn): # weight (no decay)
+ g[1].append(p)
+ else:
+ g[0].append(p) # weight (with decay)
+
+ if name == 'Adam':
+ optimizer = torch.optim.Adam(g[2], lr=lr, betas=(momentum, 0.999)) # adjust beta1 to momentum
+ elif name == 'AdamW':
+ optimizer = torch.optim.AdamW(g[2], lr=lr, betas=(momentum, 0.999), weight_decay=0.0)
+ elif name == 'RMSProp':
+ optimizer = torch.optim.RMSprop(g[2], lr=lr, momentum=momentum)
+ elif name == 'SGD':
+ optimizer = torch.optim.SGD(g[2], lr=lr, momentum=momentum, nesterov=True)
+ else:
+ raise NotImplementedError(f'Optimizer {name} not implemented.')
+
+ optimizer.add_param_group({'params': g[0], 'weight_decay': decay}) # add g0 with weight_decay
+ optimizer.add_param_group({'params': g[1], 'weight_decay': 0.0}) # add g1 (BatchNorm2d weights)
+ LOGGER.info(f"{colorstr('optimizer:')} {type(optimizer).__name__}(lr={lr}) with parameter groups "
+ f'{len(g[1])} weight(decay=0.0), {len(g[0])} weight(decay={decay}), {len(g[2])} bias')
+ return optimizer
+
+
+def smart_hub_load(repo='ultralytics/yolov5', model='yolov5s', **kwargs):
+ # YOLOv5 torch.hub.load() wrapper with smart error/issue handling
+ if check_version(torch.__version__, '1.9.1'):
+ kwargs['skip_validation'] = True # validation causes GitHub API rate limit errors
+ if check_version(torch.__version__, '1.12.0'):
+ kwargs['trust_repo'] = True # argument required starting in torch 0.12
+ try:
+ return torch.hub.load(repo, model, **kwargs)
+ except Exception:
+ return torch.hub.load(repo, model, force_reload=True, **kwargs)
+
+
+def smart_resume(ckpt, optimizer, ema=None, weights='yolov5s.pt', epochs=300, resume=True):
+ # Resume training from a partially trained checkpoint
+ best_fitness = 0.0
+ start_epoch = ckpt['epoch'] + 1
+ if ckpt['optimizer'] is not None:
+ optimizer.load_state_dict(ckpt['optimizer']) # optimizer
+ best_fitness = ckpt['best_fitness']
+ if ema and ckpt.get('ema'):
+ ema.ema.load_state_dict(ckpt['ema'].float().state_dict()) # EMA
+ ema.updates = ckpt['updates']
+ if resume:
+ assert start_epoch > 0, f'{weights} training to {epochs} epochs is finished, nothing to resume.\n' \
+ f"Start a new training without --resume, i.e. 'python train.py --weights {weights}'"
+ LOGGER.info(f'Resuming training from {weights} from epoch {start_epoch} to {epochs} total epochs')
+ if epochs < start_epoch:
+ LOGGER.info(f"{weights} has been trained for {ckpt['epoch']} epochs. Fine-tuning for {epochs} more epochs.")
+ epochs += ckpt['epoch'] # finetune additional epochs
+ return best_fitness, start_epoch, epochs
+
+
+class EarlyStopping:
+ # YOLOv5 simple early stopper
+ def __init__(self, patience=30):
+ self.best_fitness = 0.0 # i.e. mAP
+ self.best_epoch = 0
+ self.patience = patience or float('inf') # epochs to wait after fitness stops improving to stop
+ self.possible_stop = False # possible stop may occur next epoch
+
+ def __call__(self, epoch, fitness):
+ if fitness >= self.best_fitness: # >= 0 to allow for early zero-fitness stage of training
+ self.best_epoch = epoch
+ self.best_fitness = fitness
+ delta = epoch - self.best_epoch # epochs without improvement
+ self.possible_stop = delta >= (self.patience - 1) # possible stop may occur next epoch
+ stop = delta >= self.patience # stop training if patience exceeded
+ if stop:
+ LOGGER.info(f'Stopping training early as no improvement observed in last {self.patience} epochs. '
+ f'Best results observed at epoch {self.best_epoch}, best model saved as best.pt.\n'
+ f'To update EarlyStopping(patience={self.patience}) pass a new patience value, '
+ f'i.e. `python train.py --patience 300` or use `--patience 0` to disable EarlyStopping.')
+ return stop
+
+
+class ModelEMA:
+ """ Updated Exponential Moving Average (EMA) from https://github.com/rwightman/pytorch-image-models
+ Keeps a moving average of everything in the model state_dict (parameters and buffers)
+ For EMA details see https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage
+ """
+
+ def __init__(self, model, decay=0.9999, tau=2000, updates=0):
+ # Create EMA
+ self.ema = deepcopy(de_parallel(model)).eval() # FP32 EMA
+ self.updates = updates # number of EMA updates
+ self.decay = lambda x: decay * (1 - math.exp(-x / tau)) # decay exponential ramp (to help early epochs)
+ for p in self.ema.parameters():
+ p.requires_grad_(False)
+
+ def update(self, model):
+ # Update EMA parameters
+ self.updates += 1
+ d = self.decay(self.updates)
+
+ msd = de_parallel(model).state_dict() # model state_dict
+ for k, v in self.ema.state_dict().items():
+ if v.dtype.is_floating_point: # true for FP16 and FP32
+ v *= d
+ v += (1 - d) * msd[k].detach()
+ # assert v.dtype == msd[k].dtype == torch.float32, f'{k}: EMA {v.dtype} and model {msd[k].dtype} must be FP32'
+
+ def update_attr(self, model, include=(), exclude=('process_group', 'reducer')):
+ # Update EMA attributes
+ copy_attr(self.ema, model, include, exclude)
diff --git a/TextDetection/utils/triton.py b/TextDetection/utils/triton.py
new file mode 100644
index 0000000000000000000000000000000000000000..b5153dad940ddeceda4d8e39ac3d90e3efa66448
--- /dev/null
+++ b/TextDetection/utils/triton.py
@@ -0,0 +1,85 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+""" Utils to interact with the Triton Inference Server
+"""
+
+import typing
+from urllib.parse import urlparse
+
+import torch
+
+
+class TritonRemoteModel:
+ """ A wrapper over a model served by the Triton Inference Server. It can
+ be configured to communicate over GRPC or HTTP. It accepts Torch Tensors
+ as input and returns them as outputs.
+ """
+
+ def __init__(self, url: str):
+ """
+ Keyword arguments:
+ url: Fully qualified address of the Triton server - for e.g. grpc://localhost:8000
+ """
+
+ parsed_url = urlparse(url)
+ if parsed_url.scheme == 'grpc':
+ from tritonclient.grpc import InferenceServerClient, InferInput
+
+ self.client = InferenceServerClient(parsed_url.netloc) # Triton GRPC client
+ model_repository = self.client.get_model_repository_index()
+ self.model_name = model_repository.models[0].name
+ self.metadata = self.client.get_model_metadata(self.model_name, as_json=True)
+
+ def create_input_placeholders() -> typing.List[InferInput]:
+ return [
+ InferInput(i['name'], [int(s) for s in i['shape']], i['datatype']) for i in self.metadata['inputs']]
+
+ else:
+ from tritonclient.http import InferenceServerClient, InferInput
+
+ self.client = InferenceServerClient(parsed_url.netloc) # Triton HTTP client
+ model_repository = self.client.get_model_repository_index()
+ self.model_name = model_repository[0]['name']
+ self.metadata = self.client.get_model_metadata(self.model_name)
+
+ def create_input_placeholders() -> typing.List[InferInput]:
+ return [
+ InferInput(i['name'], [int(s) for s in i['shape']], i['datatype']) for i in self.metadata['inputs']]
+
+ self._create_input_placeholders_fn = create_input_placeholders
+
+ @property
+ def runtime(self):
+ """Returns the model runtime"""
+ return self.metadata.get('backend', self.metadata.get('platform'))
+
+ def __call__(self, *args, **kwargs) -> typing.Union[torch.Tensor, typing.Tuple[torch.Tensor, ...]]:
+ """ Invokes the model. Parameters can be provided via args or kwargs.
+ args, if provided, are assumed to match the order of inputs of the model.
+ kwargs are matched with the model input names.
+ """
+ inputs = self._create_inputs(*args, **kwargs)
+ response = self.client.infer(model_name=self.model_name, inputs=inputs)
+ result = []
+ for output in self.metadata['outputs']:
+ tensor = torch.as_tensor(response.as_numpy(output['name']))
+ result.append(tensor)
+ return result[0] if len(result) == 1 else result
+
+ def _create_inputs(self, *args, **kwargs):
+ args_len, kwargs_len = len(args), len(kwargs)
+ if not args_len and not kwargs_len:
+ raise RuntimeError('No inputs provided.')
+ if args_len and kwargs_len:
+ raise RuntimeError('Cannot specify args and kwargs at the same time')
+
+ placeholders = self._create_input_placeholders_fn()
+ if args_len:
+ if args_len != len(placeholders):
+ raise RuntimeError(f'Expected {len(placeholders)} inputs, got {args_len}.')
+ for input, value in zip(placeholders, args):
+ input.set_data_from_numpy(value.cpu().numpy())
+ else:
+ for input in placeholders:
+ value = kwargs[input.name]
+ input.set_data_from_numpy(value.cpu().numpy())
+ return placeholders
diff --git a/TextDetection/val.py b/TextDetection/val.py
new file mode 100644
index 0000000000000000000000000000000000000000..8da3ef7667aaeaa3519594f043c127007354fe06
--- /dev/null
+++ b/TextDetection/val.py
@@ -0,0 +1,411 @@
+# YOLOv5 🚀 by Ultralytics, AGPL-3.0 license
+"""
+Validate a trained YOLOv5 detection model on a detection dataset
+
+Usage:
+ $ python val.py --weights yolov5s.pt --data coco128.yaml --img 640
+
+Usage - formats:
+ $ python val.py --weights yolov5s.pt # PyTorch
+ yolov5s.torchscript # TorchScript
+ yolov5s.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s_openvino_model # OpenVINO
+ yolov5s.engine # TensorRT
+ yolov5s.mlmodel # CoreML (macOS-only)
+ yolov5s_saved_model # TensorFlow SavedModel
+ yolov5s.pb # TensorFlow GraphDef
+ yolov5s.tflite # TensorFlow Lite
+ yolov5s_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+from tqdm import tqdm
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[0] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.common import DetectMultiBackend
+from utils.callbacks import Callbacks
+from utils.dataloaders import create_dataloader
+from utils.general import (LOGGER, TQDM_BAR_FORMAT, Profile, check_dataset, check_img_size, check_requirements,
+ check_yaml, coco80_to_coco91_class, colorstr, increment_path, non_max_suppression,
+ print_args, scale_boxes, xywh2xyxy, xyxy2xywh)
+from utils.metrics import ConfusionMatrix, ap_per_class, box_iou
+from utils.plots import output_to_target, plot_images, plot_val_study
+from utils.torch_utils import select_device, smart_inference_mode
+
+
+def save_one_txt(predn, save_conf, shape, file):
+ # Save one txt result
+ gn = torch.tensor(shape)[[1, 0, 1, 0]] # normalization gain whwh
+ for *xyxy, conf, cls in predn.tolist():
+ xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh
+ line = (cls, *xywh, conf) if save_conf else (cls, *xywh) # label format
+ with open(file, 'a') as f:
+ f.write(('%g ' * len(line)).rstrip() % line + '\n')
+
+
+def save_one_json(predn, jdict, path, class_map):
+ # Save one JSON result {"image_id": 42, "category_id": 18, "bbox": [258.15, 41.29, 348.26, 243.78], "score": 0.236}
+ image_id = int(path.stem) if path.stem.isnumeric() else path.stem
+ box = xyxy2xywh(predn[:, :4]) # xywh
+ box[:, :2] -= box[:, 2:] / 2 # xy center to top-left corner
+ for p, b in zip(predn.tolist(), box.tolist()):
+ jdict.append({
+ 'image_id': image_id,
+ 'category_id': class_map[int(p[5])],
+ 'bbox': [round(x, 3) for x in b],
+ 'score': round(p[4], 5)})
+
+
+def process_batch(detections, labels, iouv):
+ """
+ Return correct prediction matrix
+ Arguments:
+ detections (array[N, 6]), x1, y1, x2, y2, conf, class
+ labels (array[M, 5]), class, x1, y1, x2, y2
+ Returns:
+ correct (array[N, 10]), for 10 IoU levels
+ """
+ correct = np.zeros((detections.shape[0], iouv.shape[0])).astype(bool)
+ iou = box_iou(labels[:, 1:], detections[:, :4])
+ correct_class = labels[:, 0:1] == detections[:, 5]
+ for i in range(len(iouv)):
+ x = torch.where((iou >= iouv[i]) & correct_class) # IoU > threshold and classes match
+ if x[0].shape[0]:
+ matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy() # [label, detect, iou]
+ if x[0].shape[0] > 1:
+ matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
+ # matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
+ correct[matches[:, 1].astype(int), i] = True
+ return torch.tensor(correct, dtype=torch.bool, device=iouv.device)
+
+
+@smart_inference_mode()
+def run(
+ data,
+ weights=None, # model.pt path(s)
+ batch_size=32, # batch size
+ imgsz=640, # inference size (pixels)
+ conf_thres=0.001, # confidence threshold
+ iou_thres=0.6, # NMS IoU threshold
+ max_det=300, # maximum detections per image
+ task='val', # train, val, test, speed or study
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ workers=8, # max dataloader workers (per RANK in DDP mode)
+ single_cls=False, # treat as single-class dataset
+ augment=False, # augmented inference
+ verbose=False, # verbose output
+ save_txt=False, # save results to *.txt
+ save_hybrid=False, # save label+prediction hybrid results to *.txt
+ save_conf=False, # save confidences in --save-txt labels
+ save_json=False, # save a COCO-JSON results file
+ project=ROOT / 'runs/val', # save to project/name
+ name='exp', # save to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ half=True, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ model=None,
+ dataloader=None,
+ save_dir=Path(''),
+ plots=True,
+ callbacks=Callbacks(),
+ compute_loss=None,
+):
+ # Initialize/load model and set device
+ training = model is not None
+ if training: # called by train.py
+ device, pt, jit, engine = next(model.parameters()).device, True, False, False # get model device, PyTorch model
+ half &= device.type != 'cpu' # half precision only supported on CUDA
+ model.half() if half else model.float()
+ else: # called directly
+ device = select_device(device, batch_size=batch_size)
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ (save_dir / 'labels' if save_txt else save_dir).mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
+ stride, pt, jit, engine = model.stride, model.pt, model.jit, model.engine
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+ half = model.fp16 # FP16 supported on limited backends with CUDA
+ if engine:
+ batch_size = model.batch_size
+ else:
+ device = model.device
+ if not (pt or jit):
+ batch_size = 1 # export.py models default to batch-size 1
+ LOGGER.info(f'Forcing --batch-size 1 square inference (1,3,{imgsz},{imgsz}) for non-PyTorch models')
+
+ # Data
+ data = check_dataset(data) # check
+
+ # Configure
+ model.eval()
+ cuda = device.type != 'cpu'
+ is_coco = isinstance(data.get('val'), str) and data['val'].endswith(f'coco{os.sep}val2017.txt') # COCO dataset
+ nc = 1 if single_cls else int(data['nc']) # number of classes
+ iouv = torch.linspace(0.5, 0.95, 10, device=device) # iou vector for mAP@0.5:0.95
+ niou = iouv.numel()
+
+ # Dataloader
+ if not training:
+ if pt and not single_cls: # check --weights are trained on --data
+ ncm = model.model.nc
+ assert ncm == nc, f'{weights} ({ncm} classes) trained on different --data than what you passed ({nc} ' \
+ f'classes). Pass correct combination of --weights and --data that are trained together.'
+ model.warmup(imgsz=(1 if pt else batch_size, 3, imgsz, imgsz)) # warmup
+ pad, rect = (0.0, False) if task == 'speed' else (0.5, pt) # square inference for benchmarks
+ task = task if task in ('train', 'val', 'test') else 'val' # path to train/val/test images
+ dataloader = create_dataloader(data[task],
+ imgsz,
+ batch_size,
+ stride,
+ single_cls,
+ pad=pad,
+ rect=rect,
+ workers=workers,
+ prefix=colorstr(f'{task}: '))[0]
+
+ seen = 0
+ confusion_matrix = ConfusionMatrix(nc=nc)
+ names = model.names if hasattr(model, 'names') else model.module.names # get class names
+ if isinstance(names, (list, tuple)): # old format
+ names = dict(enumerate(names))
+ class_map = coco80_to_coco91_class() if is_coco else list(range(1000))
+ s = ('%22s' + '%11s' * 6) % ('Class', 'Images', 'Instances', 'P', 'R', 'mAP50', 'mAP50-95')
+ tp, fp, p, r, f1, mp, mr, map50, ap50, map = 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
+ dt = Profile(), Profile(), Profile() # profiling times
+ loss = torch.zeros(3, device=device)
+ jdict, stats, ap, ap_class = [], [], [], []
+ callbacks.run('on_val_start')
+ pbar = tqdm(dataloader, desc=s, bar_format=TQDM_BAR_FORMAT) # progress bar
+ for batch_i, (im, targets, paths, shapes) in enumerate(pbar):
+ callbacks.run('on_val_batch_start')
+ with dt[0]:
+ if cuda:
+ im = im.to(device, non_blocking=True)
+ targets = targets.to(device)
+ im = im.half() if half else im.float() # uint8 to fp16/32
+ im /= 255 # 0 - 255 to 0.0 - 1.0
+ nb, _, height, width = im.shape # batch size, channels, height, width
+
+ # Inference
+ with dt[1]:
+ preds, train_out = model(im) if compute_loss else (model(im, augment=augment), None)
+
+ # Loss
+ if compute_loss:
+ loss += compute_loss(train_out, targets)[1] # box, obj, cls
+
+ # NMS
+ targets[:, 2:] *= torch.tensor((width, height, width, height), device=device) # to pixels
+ lb = [targets[targets[:, 0] == i, 1:] for i in range(nb)] if save_hybrid else [] # for autolabelling
+ with dt[2]:
+ preds = non_max_suppression(preds,
+ conf_thres,
+ iou_thres,
+ labels=lb,
+ multi_label=True,
+ agnostic=single_cls,
+ max_det=max_det)
+
+ # Metrics
+ for si, pred in enumerate(preds):
+ labels = targets[targets[:, 0] == si, 1:]
+ nl, npr = labels.shape[0], pred.shape[0] # number of labels, predictions
+ path, shape = Path(paths[si]), shapes[si][0]
+ correct = torch.zeros(npr, niou, dtype=torch.bool, device=device) # init
+ seen += 1
+
+ if npr == 0:
+ if nl:
+ stats.append((correct, *torch.zeros((2, 0), device=device), labels[:, 0]))
+ if plots:
+ confusion_matrix.process_batch(detections=None, labels=labels[:, 0])
+ continue
+
+ # Predictions
+ if single_cls:
+ pred[:, 5] = 0
+ predn = pred.clone()
+ scale_boxes(im[si].shape[1:], predn[:, :4], shape, shapes[si][1]) # native-space pred
+
+ # Evaluate
+ if nl:
+ tbox = xywh2xyxy(labels[:, 1:5]) # target boxes
+ scale_boxes(im[si].shape[1:], tbox, shape, shapes[si][1]) # native-space labels
+ labelsn = torch.cat((labels[:, 0:1], tbox), 1) # native-space labels
+ correct = process_batch(predn, labelsn, iouv)
+ if plots:
+ confusion_matrix.process_batch(predn, labelsn)
+ stats.append((correct, pred[:, 4], pred[:, 5], labels[:, 0])) # (correct, conf, pcls, tcls)
+
+ # Save/log
+ if save_txt:
+ save_one_txt(predn, save_conf, shape, file=save_dir / 'labels' / f'{path.stem}.txt')
+ if save_json:
+ save_one_json(predn, jdict, path, class_map) # append to COCO-JSON dictionary
+ callbacks.run('on_val_image_end', pred, predn, path, names, im[si])
+
+ # Plot images
+ if plots and batch_i < 3:
+ plot_images(im, targets, paths, save_dir / f'val_batch{batch_i}_labels.jpg', names) # labels
+ plot_images(im, output_to_target(preds), paths, save_dir / f'val_batch{batch_i}_pred.jpg', names) # pred
+
+ callbacks.run('on_val_batch_end', batch_i, im, targets, paths, shapes, preds)
+
+ # Compute metrics
+ stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)] # to numpy
+ if len(stats) and stats[0].any():
+ tp, fp, p, r, f1, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)
+ ap50, ap = ap[:, 0], ap.mean(1) # AP@0.5, AP@0.5:0.95
+ mp, mr, map50, map = p.mean(), r.mean(), ap50.mean(), ap.mean()
+ nt = np.bincount(stats[3].astype(int), minlength=nc) # number of targets per class
+
+ # Print results
+ pf = '%22s' + '%11i' * 2 + '%11.3g' * 4 # print format
+ LOGGER.info(pf % ('all', seen, nt.sum(), mp, mr, map50, map))
+ if nt.sum() == 0:
+ LOGGER.warning(f'WARNING ⚠️ no labels found in {task} set, can not compute metrics without labels')
+
+ # Print results per class
+ if (verbose or (nc < 50 and not training)) and nc > 1 and len(stats):
+ for i, c in enumerate(ap_class):
+ LOGGER.info(pf % (names[c], seen, nt[c], p[i], r[i], ap50[i], ap[i]))
+
+ # Print speeds
+ t = tuple(x.t / seen * 1E3 for x in dt) # speeds per image
+ if not training:
+ shape = (batch_size, 3, imgsz, imgsz)
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {shape}' % t)
+
+ # Plots
+ if plots:
+ confusion_matrix.plot(save_dir=save_dir, names=list(names.values()))
+ callbacks.run('on_val_end', nt, tp, fp, p, r, f1, ap, ap50, ap_class, confusion_matrix)
+
+ # Save JSON
+ if save_json and len(jdict):
+ w = Path(weights[0] if isinstance(weights, list) else weights).stem if weights is not None else '' # weights
+ anno_json = str(Path('../datasets/coco/annotations/instances_val2017.json')) # annotations
+ if not os.path.exists(anno_json):
+ anno_json = os.path.join(data['path'], 'annotations', 'instances_val2017.json')
+ pred_json = str(save_dir / f'{w}_predictions.json') # predictions
+ LOGGER.info(f'\nEvaluating pycocotools mAP... saving {pred_json}...')
+ with open(pred_json, 'w') as f:
+ json.dump(jdict, f)
+
+ try: # https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb
+ check_requirements('pycocotools>=2.0.6')
+ from pycocotools.coco import COCO
+ from pycocotools.cocoeval import COCOeval
+
+ anno = COCO(anno_json) # init annotations api
+ pred = anno.loadRes(pred_json) # init predictions api
+ eval = COCOeval(anno, pred, 'bbox')
+ if is_coco:
+ eval.params.imgIds = [int(Path(x).stem) for x in dataloader.dataset.im_files] # image IDs to evaluate
+ eval.evaluate()
+ eval.accumulate()
+ eval.summarize()
+ map, map50 = eval.stats[:2] # update results (mAP@0.5:0.95, mAP@0.5)
+ except Exception as e:
+ LOGGER.info(f'pycocotools unable to run: {e}')
+
+ # Return results
+ model.float() # for training
+ if not training:
+ s = f"\n{len(list(save_dir.glob('labels/*.txt')))} labels saved to {save_dir / 'labels'}" if save_txt else ''
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}{s}")
+ maps = np.zeros(nc) + map
+ for i, c in enumerate(ap_class):
+ maps[c] = ap[i]
+ return (mp, mr, map50, map, *(loss.cpu() / len(dataloader)).tolist()), maps, t
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='dataset.yaml path')
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model path(s)')
+ parser.add_argument('--batch-size', type=int, default=32, help='batch size')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='inference size (pixels)')
+ parser.add_argument('--conf-thres', type=float, default=0.001, help='confidence threshold')
+ parser.add_argument('--iou-thres', type=float, default=0.6, help='NMS IoU threshold')
+ parser.add_argument('--max-det', type=int, default=300, help='maximum detections per image')
+ parser.add_argument('--task', default='val', help='train, val, test, speed or study')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--single-cls', action='store_true', help='treat as single-class dataset')
+ parser.add_argument('--augment', action='store_true', help='augmented inference')
+ parser.add_argument('--verbose', action='store_true', help='report mAP by class')
+ parser.add_argument('--save-txt', action='store_true', help='save results to *.txt')
+ parser.add_argument('--save-hybrid', action='store_true', help='save label+prediction hybrid results to *.txt')
+ parser.add_argument('--save-conf', action='store_true', help='save confidences in --save-txt labels')
+ parser.add_argument('--save-json', action='store_true', help='save a COCO-JSON results file')
+ parser.add_argument('--project', default=ROOT / 'runs/val', help='save to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ opt = parser.parse_args()
+ opt.data = check_yaml(opt.data) # check YAML
+ opt.save_json |= opt.data.endswith('coco.yaml')
+ opt.save_txt |= opt.save_hybrid
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(ROOT / 'requirements.txt', exclude=('tensorboard', 'thop'))
+
+ if opt.task in ('train', 'val', 'test'): # run normally
+ if opt.conf_thres > 0.001: # https://github.com/ultralytics/yolov5/issues/1466
+ LOGGER.info(f'WARNING ⚠️ confidence threshold {opt.conf_thres} > 0.001 produces invalid results')
+ if opt.save_hybrid:
+ LOGGER.info('WARNING ⚠️ --save-hybrid will return high mAP from hybrid labels, not from predictions alone')
+ run(**vars(opt))
+
+ else:
+ weights = opt.weights if isinstance(opt.weights, list) else [opt.weights]
+ opt.half = torch.cuda.is_available() and opt.device != 'cpu' # FP16 for fastest results
+ if opt.task == 'speed': # speed benchmarks
+ # python val.py --task speed --data coco.yaml --batch 1 --weights yolov5n.pt yolov5s.pt...
+ opt.conf_thres, opt.iou_thres, opt.save_json = 0.25, 0.45, False
+ for opt.weights in weights:
+ run(**vars(opt), plots=False)
+
+ elif opt.task == 'study': # speed vs mAP benchmarks
+ # python val.py --task study --data coco.yaml --iou 0.7 --weights yolov5n.pt yolov5s.pt...
+ for opt.weights in weights:
+ f = f'study_{Path(opt.data).stem}_{Path(opt.weights).stem}.txt' # filename to save to
+ x, y = list(range(256, 1536 + 128, 128)), [] # x axis (image sizes), y axis
+ for opt.imgsz in x: # img-size
+ LOGGER.info(f'\nRunning {f} --imgsz {opt.imgsz}...')
+ r, _, t = run(**vars(opt), plots=False)
+ y.append(r + t) # results and times
+ np.savetxt(f, y, fmt='%10.4g') # save
+ subprocess.run(['zip', '-r', 'study.zip', 'study_*.txt'])
+ plot_val_study(x=x) # plot
+ else:
+ raise NotImplementedError(f'--task {opt.task} not in ("train", "val", "test", "speed", "study")')
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)
diff --git a/TextRearrange/DBSCAN.py b/TextRearrange/DBSCAN.py
new file mode 100644
index 0000000000000000000000000000000000000000..2f8001fbd4b3b982d4e08388a5906ecd76e11886
--- /dev/null
+++ b/TextRearrange/DBSCAN.py
@@ -0,0 +1,180 @@
+import os
+import glob
+import time
+import numpy as np
+from os import path
+import pandas as pd
+import seaborn as sns
+import matplotlib.pyplot as plt
+from sklearn.cluster import DBSCAN
+from sklearn.preprocessing import StandardScaler
+import lib.arrange as DbscanArrange
+from lib import directories as Dir
+
+"""
+
+이전 프로세스:
+ image detection을 통해 handwritten text image에서
+ 각각의 word image를 추출했으나,
+ 순서를 알지 못하게 단어만 확인이 되어
+ 원래 글의 의미를 잃어버리는 문제가 있었음.
+ 다만 추출 시에 단어의 좌표값을 포함시킬 수 있었기에
+ 각 단어의 이름에 x,y좌표값과 w,h 정보를 포함하여
+ 단어 이미지 파일을 생성
+
+
+현재 프로세스:
+ yolo model을 통해 crop된 이미지들은
+ 파일 이름에 crop되기 전 raw image에서의 좌표 정보를 포함합니다.
+ x89y147w199h184.jpg
+ x10y148w157h184.jpg
+ x28y149w108h180.jpg
+
+ raw image파일에서 같은 line에 있던 word들은
+ 가까운 y값을 갖고있을 것이므로,
+ 밀도기반 군집화 알고리즘을 사용하여
+ 같은 line에 있을 것이라고 예상하는 word값들은
+ 동일 클러스터에 속하게 됨.
+ 이렇게 동일 클러스터에 속하는 y값들의 평균을 구하여
+ word의 새로운 y값으로 label.
+ 단어 이미지 파일 rename할 때, y값이 먼저 오게 한다.
+ yyyy_xxxx.jpg
+
+ y값에 대한 정렬이 되고
+ x값에 대해 오름차순으로 정렬됨.
+ 0148_0010.jpg
+ 0148_0028.jpg
+ 0148_0089.jpg
+ word file의 순서가 자동적으로 원래 text의 의미를 가지게 된다.
+
+코드 설명:
+ extract_text_from_filename(),
+ get_folder_contents_with_text() 함수를 이용해
+ 파일 이름에서 x값과 y값을 추출.
+ StandardScaler로 표준화.
+ Get Clustered y values using DBSCAN.
+ rename_file()을 이용해서 rename image files with y-clustered values.
+
+"""
+# 사용 예시
+# Get cropped word images
+# folder_path = "C:/Users/ban/TEXTAI/yolov5/runs/detect/yujin_paper/crops/word" -> cropped word folder
+
+x_texts, y_texts, name_jpg = DbscanArrange.get_folder_contents_with_text(Dir.folder_path)
+file_name = pd.DataFrame(name_jpg)
+file_name.columns=['file_name']
+
+#더미 리스트 생성, 넘파이 변환 후 reshape (-1, 1)
+zero_list = [0 for _ in range(len(y_texts))] #2차원 맞추기 위한 zero 리스트
+zero_list = np.array([zero_list]).reshape(-1, 1)
+y_text = np.array([y_texts]).reshape(-1, 1)
+#print('y_text\n', y_text)
+#print('zero_list\n', zero_list)
+
+
+#####################################
+# 표준화
+scalerX = StandardScaler() # 스케일 함수 가져와서
+scalerX.fit(y_text.data) # 스케일
+std_y_text = scalerX.transform(y_text.data).reshape(-1, 1) #스케일 완료
+
+feature = pd.DataFrame(std_y_text) # 스케일된 넘파이 자료형을 데이터프레임 형태로 변환
+feature.columns=['feature']
+
+data_list = [std_y_text,zero_list] # 표준화된 데이터와 더미 삽입
+data = pd.DataFrame(data_list[0]) #
+labels = pd.DataFrame(data_list[1])
+
+# 데이터 컬럼명 설정
+labels.columns=['labels']
+data.columns=['y']
+# 두 데이터열 병합
+datadf = pd.concat([data,labels],axis=1)
+
+
+###################################
+# create model and prediction
+model = DBSCAN(eps=0.04,min_samples=2)
+predict = pd.DataFrame(model.fit_predict(feature))
+predict.columns=['predict']
+
+# file_name, feature, predict 병합
+r = pd.concat([file_name,feature,predict],axis=1)
+
+#r.to_csv('C:\\Users\\ban\\Desktop\\predict_final.csv')
+###########################################################
+
+
+
+###########################################################
+r = r.sort_values(by=['predict'])
+#print(type(set(r['predict']))[0])
+predict_list = list(set(r['predict'])) # predict 의 숫자들을 수집
+unknown_words = []
+same_line = []
+whole_word_map = []
+final_result = {'y_mean' : [],
+ 'x_value' : [],
+ 'file_name' : []}
+df_final_result = pd.DataFrame(final_result)
+whole_word_map_df = pd.DataFrame(whole_word_map)
+
+for _,line in enumerate(predict_list): # 하나씩 추출 -1, 0, 1, 2, ...진짜 힘들다
+ if line >= 0: # predict 값이 0 이상이면,
+ y_list = [] # 클러스팅된 y값들의 평균을 구하기 위한 리스트
+ print(type(r['predict']))
+ same_line = r[r['predict'] == line] # r 데이터프레임에서 X인 predict를 가지고 있는 열을 가져와서
+ file_num = 0
+ y_mean_column = []
+ total_word_map = []
+ total_word_map_df = pd.DataFrame(total_word_map)
+ for filename in same_line['file_name']: # 같은 클러스터에서 파일 하나씩 뽑아서
+ x_data, y_data = DbscanArrange.extract_text_from_filename(filename) # 해당 파일의 x, y값을 뽑아서
+ y_list.append(int(y_data)) # y값 평균을 위한 y값 리스트에 삽입
+ #x_file = {x_data:filename} # key값에 x값, value값에 file_name, n_line 데이터프레임에 삽입
+ file_num += 1
+ word_map = {'x_value' : [int(x_data)],
+ 'file_name' : [filename]}
+
+ word_map_df = pd.DataFrame(word_map)
+ total_word_map_df = pd.concat([total_word_map_df, word_map_df])
+ total_word_map_df = total_word_map_df.sort_values(by=['x_value'])
+
+ y_mean = int(np.mean(y_list)) # 한 줄에 대한 평균값 얻음 #y_mean 열에 삽입
+ total_word_map_df['y_mean'] = y_mean # 추가된 행 수만큼 y_mean 열 추가
+
+ else:
+ total_word_map = []
+ same_line = r[r['predict'] == line]
+ for filename in same_line['file_name']:
+ x_data, y_data = DbscanArrange.extract_text_from_filename(filename)
+ unknown_words = [r['predict']==line]
+ y_mean = int(y_data)
+ word_map = {'y_mean' : [y_mean],
+ 'x_value' : [x_data],
+ 'file_name' : [filename]}
+ word_map_df = pd.DataFrame(word_map)
+ total_word_map_df = pd.concat([total_word_map_df, word_map_df])
+
+ whole_word_map_df = pd.concat([whole_word_map_df,total_word_map_df])
+
+
+file_name = list(whole_word_map_df['file_name'])
+x_value = list(whole_word_map_df['x_value'])
+y_mean = list(whole_word_map_df['y_mean'])
+whole_list = [file_name,x_value,y_mean]
+
+
+cnum = 0
+timestr = time.strftime("%Y%m%d%H%M%S")
+
+#전체 데이터 셋에 대해서 하나씩 추출하여 패딩 후 이름 변환
+for i in range(len(file_name)): #_, x_value, file_name, y_mean
+ old_path = str(Dir.folder_path) +"/"+ str(file_name[i])
+ new_path = str(Dir.folder_path) +"/"+ str(y_mean[i]).zfill(4) +"_"+ str(x_value[i]).zfill(4) + ".jpg"
+
+ DbscanArrange.rename_file(old_path, new_path)
+
+if os.listdir(Dir.folder_path) == True:
+ folder_contents = os.listdir(Dir.folder_path)
+ print('UNKNOWN WORDS: \n',folder_contents)
diff --git a/TextRearrange/arrange.py b/TextRearrange/arrange.py
new file mode 100644
index 0000000000000000000000000000000000000000..e3d710e2e76dbcf4768f409de8c91c9d7466192e
--- /dev/null
+++ b/TextRearrange/arrange.py
@@ -0,0 +1,72 @@
+import os
+from os import path
+
+def rename_file(old_path, new_path):
+ """
+ 파일 이름을 수정하는 함수
+
+ Parameters:
+ old_path (str): 파일 경로 + 기존 파일 이름
+ new_path (str): 파일 경로 + 수정할 파일 이름
+
+ Changes:
+ old_path > new_path
+ """
+ if path.exists(new_path)==False:
+ if path.exists(old_path):
+ # 파일이 있는 디렉토리 경로 얻기
+ src = path.realpath(old_path)
+ # 원본파일 파일명 변경
+ os.rename(old_path,new_path)
+ else:
+ print("{old_path} doesn't exist")
+
+def extract_text_from_filename(filename):
+ """
+ 파일 이름에서 'x'와 'y' 사이의 텍스트와 'y'와 'w' 사이의 텍스트를 추출하는 함수
+ ex) x10y148w157h184.jpg
+
+ Parameters:
+ filename (str): 파일 이름
+
+ Returns:
+ tuple: 'x'와 'y' 사이의 텍스트, 'y'와 'w' 사이의 텍스트
+ """
+ start_x = filename.index('x')
+ end_x = filename.index('y')
+
+ start_y = end_x + 1
+ end_y = filename.index('w')
+
+ x_text = filename[start_x+1:end_x]
+ y_text = filename[start_y:end_y]
+
+ return x_text, y_text
+
+def get_folder_contents_with_text(folder_path, file_extension=".jpg"):
+ """
+ 폴더 내부의 파일들의 이름에서 'x'와 'y' 사이의 텍스트를 리스트 x에,
+ 'y'와 'w' 사이의 텍스트를 리스트 y에 각각 저장하는 함수
+
+ Parameters:
+ folder_path (str): 폴더 경로
+ file_extension (str, optional): 가져올 파일의 확장자 (기본값: ".jpg")
+
+ Returns:
+ list, list: 'x'와 'y' 사이의 텍스트를 저장한 리스트 x,
+ 'y'와 'w' 사이의 텍스트를 저장한 리스트 y
+ """
+ x_list = []
+ y_list = []
+
+ folder_contents = os.listdir(folder_path)
+ print(folder_contents)
+ jpg_files = [filename for filename in folder_contents if filename.endswith('jpg')]
+
+ for jpg_file in jpg_files:
+
+ x_text, y_text = extract_text_from_filename(jpg_file)
+ x_list.append(x_text)
+ y_list.append(y_text)
+
+ return x_list, y_list, jpg_files
\ No newline at end of file
diff --git a/TextRecognition/best_accuracy_s/best_accuracy_s.pth b/TextRecognition/best_accuracy_s/best_accuracy_s.pth
new file mode 100644
index 0000000000000000000000000000000000000000..d3025de75b5733d5052d9472e017b7d4ba0a2bad
--- /dev/null
+++ b/TextRecognition/best_accuracy_s/best_accuracy_s.pth
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83792da2b46efe4be91ff1e92377085124235fd82ba0040d59d7cf1e5cc878e7
+size 198716005
diff --git a/TextRecognition/best_accuracy_s/best_norm_ED_s.pth b/TextRecognition/best_accuracy_s/best_norm_ED_s.pth
new file mode 100644
index 0000000000000000000000000000000000000000..d3025de75b5733d5052d9472e017b7d4ba0a2bad
--- /dev/null
+++ b/TextRecognition/best_accuracy_s/best_norm_ED_s.pth
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83792da2b46efe4be91ff1e92377085124235fd82ba0040d59d7cf1e5cc878e7
+size 198716005
diff --git a/TextRecognition/dataset.py b/TextRecognition/dataset.py
new file mode 100644
index 0000000000000000000000000000000000000000..42ede25c16e785a3810376cdcb92032d877fd2db
--- /dev/null
+++ b/TextRecognition/dataset.py
@@ -0,0 +1,339 @@
+import os
+import sys
+import re
+import six
+import math
+import lmdb
+import torch
+
+from natsort import natsorted
+from PIL import Image
+import numpy as np
+from torch.utils.data import Dataset, ConcatDataset, Subset
+from torch._utils import _accumulate
+import torchvision.transforms as transforms
+
+
+class Batch_Balanced_Dataset(object):
+
+ def __init__(self, opt):
+ """
+ Modulate the data ratio in the batch.
+ For example, when select_data is "MJ-ST" and batch_ratio is "0.5-0.5",
+ the 50% of the batch is filled with MJ and the other 50% of the batch is filled with ST.
+ """
+ log = open(f'./saved_models/{opt.exp_name}/log_dataset.txt', 'a')
+ dashed_line = '-' * 80
+ print(dashed_line)
+ log.write(dashed_line + '\n')
+ print(f'dataset_root: {opt.train_data}\nopt.select_data: {opt.select_data}\nopt.batch_ratio: {opt.batch_ratio}')
+ log.write(f'dataset_root: {opt.train_data}\nopt.select_data: {opt.select_data}\nopt.batch_ratio: {opt.batch_ratio}\n')
+ assert len(opt.select_data) == len(opt.batch_ratio)
+
+ _AlignCollate = AlignCollate(imgH=opt.imgH, imgW=opt.imgW, keep_ratio_with_pad=opt.PAD)
+ self.data_loader_list = []
+ self.dataloader_iter_list = []
+ batch_size_list = []
+ Total_batch_size = 0
+ for selected_d, batch_ratio_d in zip(opt.select_data, opt.batch_ratio):
+ _batch_size = max(round(opt.batch_size * float(batch_ratio_d)), 1)
+ print(dashed_line)
+ log.write(dashed_line + '\n')
+ _dataset, _dataset_log = hierarchical_dataset(root=opt.train_data, opt=opt, select_data=[selected_d])
+ total_number_dataset = len(_dataset)
+ log.write(_dataset_log)
+
+ """
+ The total number of data can be modified with opt.total_data_usage_ratio.
+ ex) opt.total_data_usage_ratio = 1 indicates 100% usage, and 0.2 indicates 20% usage.
+ See 4.2 section in our paper.
+ """
+ number_dataset = int(total_number_dataset * float(opt.total_data_usage_ratio))
+ dataset_split = [number_dataset, total_number_dataset - number_dataset]
+ indices = range(total_number_dataset)
+ _dataset, _ = [Subset(_dataset, indices[offset - length:offset])
+ for offset, length in zip(_accumulate(dataset_split), dataset_split)]
+ selected_d_log = f'num total samples of {selected_d}: {total_number_dataset} x {opt.total_data_usage_ratio} (total_data_usage_ratio) = {len(_dataset)}\n'
+ selected_d_log += f'num samples of {selected_d} per batch: {opt.batch_size} x {float(batch_ratio_d)} (batch_ratio) = {_batch_size}'
+ print(selected_d_log)
+ log.write(selected_d_log + '\n')
+ batch_size_list.append(str(_batch_size))
+ Total_batch_size += _batch_size
+
+ _data_loader = torch.utils.data.DataLoader(
+ _dataset, batch_size=_batch_size,
+ shuffle=True,
+ num_workers=int(opt.workers),
+ collate_fn=_AlignCollate, pin_memory=True)
+ self.data_loader_list.append(_data_loader)
+ self.dataloader_iter_list.append(iter(_data_loader))
+
+ Total_batch_size_log = f'{dashed_line}\n'
+ batch_size_sum = '+'.join(batch_size_list)
+ Total_batch_size_log += f'Total_batch_size: {batch_size_sum} = {Total_batch_size}\n'
+ Total_batch_size_log += f'{dashed_line}'
+ opt.batch_size = Total_batch_size
+
+ print(Total_batch_size_log)
+ log.write(Total_batch_size_log + '\n')
+ log.close()
+
+ def get_batch(self):
+ balanced_batch_images = []
+ balanced_batch_texts = []
+
+ for i, data_loader_iter in enumerate(self.dataloader_iter_list):
+ try:
+ image, text = next(data_loader_iter)
+ balanced_batch_images.append(image)
+ balanced_batch_texts += text
+ except StopIteration:
+ self.dataloader_iter_list[i] = iter(self.data_loader_list[i])
+ image, text = next(self.dataloader_iter_list[i])
+ balanced_batch_images.append(image)
+ balanced_batch_texts += text
+ except ValueError:
+ pass
+
+ balanced_batch_images = torch.cat(balanced_batch_images, 0)
+
+ return balanced_batch_images, balanced_batch_texts
+
+
+def hierarchical_dataset(root, opt, select_data='/'):
+ """ select_data='/' contains all sub-directory of root directory """
+ dataset_list = []
+ dataset_log = f'dataset_root: {root}\t dataset: {select_data[0]}'
+ print(dataset_log)
+ dataset_log += '\n'
+ for dirpath, dirnames, filenames in os.walk(root+'/'):
+ if not dirnames:
+ select_flag = False
+ for selected_d in select_data:
+ if selected_d in dirpath:
+ select_flag = True
+ break
+
+ if select_flag:
+ dataset = LmdbDataset(dirpath, opt)
+ sub_dataset_log = f'sub-directory:\t/{os.path.relpath(dirpath, root)}\t num samples: {len(dataset)}'
+ print(sub_dataset_log)
+ dataset_log += f'{sub_dataset_log}\n'
+ dataset_list.append(dataset)
+
+ concatenated_dataset = ConcatDataset(dataset_list)
+
+ return concatenated_dataset, dataset_log
+
+
+class LmdbDataset(Dataset):
+
+ def __init__(self, root, opt):
+
+ self.root = root
+ self.opt = opt
+ self.env = lmdb.open(root, max_readers=32, readonly=True, lock=False, readahead=False, meminit=False)
+ if not self.env:
+ print('cannot create lmdb from %s' % (root))
+ sys.exit(0)
+
+ with self.env.begin(write=False) as txn:
+ nSamples = int(txn.get('num-samples'.encode()))
+ self.nSamples = nSamples
+
+ if self.opt.data_filtering_off:
+ # for fast check or benchmark evaluation with no filtering
+ self.filtered_index_list = [index + 1 for index in range(self.nSamples)]
+ else:
+ """ Filtering part
+ If you want to evaluate IC15-2077 & CUTE datasets which have special character labels,
+ use --data_filtering_off and only evaluate on alphabets and digits.
+ see https://github.com/clovaai/deep-text-recognition-benchmark/blob/6593928855fb7abb999a99f428b3e4477d4ae356/dataset.py#L190-L192
+
+ And if you want to evaluate them with the model trained with --sensitive option,
+ use --sensitive and --data_filtering_off,
+ see https://github.com/clovaai/deep-text-recognition-benchmark/blob/dff844874dbe9e0ec8c5a52a7bd08c7f20afe704/test.py#L137-L144
+ """
+ self.filtered_index_list = []
+ for index in range(self.nSamples):
+ index += 1 # lmdb starts with 1
+ label_key = 'label-%09d'.encode() % index
+ label = txn.get(label_key).decode('utf-8')
+
+ if len(label) > self.opt.batch_max_length:
+ # print(f'The length of the label is longer than max_length: length
+ # {len(label)}, {label} in dataset {self.root}')
+ continue
+
+ # By default, images containing characters which are not in opt.character are filtered.
+ # You can add [UNK] token to `opt.character` in utils.py instead of this filtering.
+ out_of_char = f'[^{self.opt.character}]'
+ if re.search(out_of_char, label.lower()):
+ continue
+
+ self.filtered_index_list.append(index)
+
+ self.nSamples = len(self.filtered_index_list)
+
+ def __len__(self):
+ return self.nSamples
+
+ def __getitem__(self, index):
+ assert index <= len(self), 'index range error'
+ index = self.filtered_index_list[index]
+
+ with self.env.begin(write=False) as txn:
+ label_key = 'label-%09d'.encode() % index
+ label = txn.get(label_key).decode('utf-8')
+ img_key = 'image-%09d'.encode() % index
+ imgbuf = txn.get(img_key)
+
+ buf = six.BytesIO()
+ buf.write(imgbuf)
+ buf.seek(0)
+ try:
+ if self.opt.rgb:
+ img = Image.open(buf).convert('RGB') # for color image
+ else:
+ img = Image.open(buf).convert('L')
+
+ except IOError:
+ print(f'Corrupted image for {index}')
+ # make dummy image and dummy label for corrupted image.
+ if self.opt.rgb:
+ img = Image.new('RGB', (self.opt.imgW, self.opt.imgH))
+ else:
+ img = Image.new('L', (self.opt.imgW, self.opt.imgH))
+ label = '[dummy_label]'
+
+ if not self.opt.sensitive:
+ label = label.lower()
+
+ # We only train and evaluate on alphanumerics (or pre-defined character set in train.py)
+ out_of_char = f'[^{self.opt.character}]'
+ label = re.sub(out_of_char, '', label)
+
+ return (img, label)
+
+
+class RawDataset(Dataset):
+
+ def __init__(self, root, opt):
+ self.opt = opt
+ self.image_path_list = []
+ for dirpath, dirnames, filenames in os.walk(root):
+ for name in filenames:
+ _, ext = os.path.splitext(name)
+ ext = ext.lower()
+ if ext == '.jpg' or ext == '.jpeg' or ext == '.png':
+ self.image_path_list.append(os.path.join(dirpath, name))
+
+ self.image_path_list = natsorted(self.image_path_list)
+ self.nSamples = len(self.image_path_list)
+
+ def __len__(self):
+ return self.nSamples
+
+ def __getitem__(self, index):
+
+ try:
+ if self.opt.rgb:
+ img = Image.open(self.image_path_list[index]).convert('RGB') # for color image
+ else:
+ img = Image.open(self.image_path_list[index]).convert('L')
+
+ except IOError:
+ print(f'Corrupted image for {index}')
+ # make dummy image and dummy label for corrupted image.
+ if self.opt.rgb:
+ img = Image.new('RGB', (self.opt.imgW, self.opt.imgH))
+ else:
+ img = Image.new('L', (self.opt.imgW, self.opt.imgH))
+
+ return (img, self.image_path_list[index])
+
+
+class ResizeNormalize(object):
+
+ def __init__(self, size, interpolation=Image.BICUBIC):
+ self.size = size
+ self.interpolation = interpolation
+ self.toTensor = transforms.ToTensor()
+
+ def __call__(self, img):
+ img = img.resize(self.size, self.interpolation)
+ img = self.toTensor(img)
+ img.sub_(0.5).div_(0.5)
+ return img
+
+
+class NormalizePAD(object):
+
+ def __init__(self, max_size, PAD_type='right'):
+ self.toTensor = transforms.ToTensor()
+ self.max_size = max_size
+ self.max_width_half = math.floor(max_size[2] / 2)
+ self.PAD_type = PAD_type
+
+ def __call__(self, img):
+ img = self.toTensor(img)
+ img.sub_(0.5).div_(0.5)
+ c, h, w = img.size()
+ Pad_img = torch.FloatTensor(*self.max_size).fill_(0)
+ Pad_img[:, :, :w] = img # right pad
+ if self.max_size[2] != w: # add border Pad
+ Pad_img[:, :, w:] = img[:, :, w - 1].unsqueeze(2).expand(c, h, self.max_size[2] - w)
+
+ return Pad_img
+
+
+class AlignCollate(object):
+
+ def __init__(self, imgH=32, imgW=100, keep_ratio_with_pad=False):
+ self.imgH = imgH
+ self.imgW = imgW
+ self.keep_ratio_with_pad = keep_ratio_with_pad
+
+ def __call__(self, batch):
+ batch = filter(lambda x: x is not None, batch)
+ images, labels = zip(*batch)
+
+ if self.keep_ratio_with_pad: # same concept with 'Rosetta' paper
+ resized_max_w = self.imgW
+ input_channel = 3 if images[0].mode == 'RGB' else 1
+ transform = NormalizePAD((input_channel, self.imgH, resized_max_w))
+
+ resized_images = []
+ for image in images:
+ w, h = image.size
+ ratio = w / float(h)
+ if math.ceil(self.imgH * ratio) > self.imgW:
+ resized_w = self.imgW
+ else:
+ resized_w = math.ceil(self.imgH * ratio)
+
+ resized_image = image.resize((resized_w, self.imgH), Image.BICUBIC)
+ resized_images.append(transform(resized_image))
+ # resized_image.save('./image_test/%d_test.jpg' % w)
+
+ image_tensors = torch.cat([t.unsqueeze(0) for t in resized_images], 0)
+
+ else:
+ transform = ResizeNormalize((self.imgW, self.imgH))
+ image_tensors = [transform(image) for image in images]
+ image_tensors = torch.cat([t.unsqueeze(0) for t in image_tensors], 0)
+
+ return image_tensors, labels
+
+
+def tensor2im(image_tensor, imtype=np.uint8):
+ image_numpy = image_tensor.cpu().float().numpy()
+ if image_numpy.shape[0] == 1:
+ image_numpy = np.tile(image_numpy, (3, 1, 1))
+ image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + 1) / 2.0 * 255.0
+ return image_numpy.astype(imtype)
+
+
+def save_image(image_numpy, image_path):
+ image_pil = Image.fromarray(image_numpy)
+ image_pil.save(image_path)
diff --git a/TextRecognition/demo.py b/TextRecognition/demo.py
new file mode 100644
index 0000000000000000000000000000000000000000..5314b4e8c96db8fb4798a581217105ebd378dda1
--- /dev/null
+++ b/TextRecognition/demo.py
@@ -0,0 +1,129 @@
+import string
+import argparse
+
+import torch
+import torch.backends.cudnn as cudnn
+import torch.utils.data
+import torch.nn.functional as F
+
+from utils import CTCLabelConverter, AttnLabelConverter
+from dataset import RawDataset, AlignCollate
+from model import Model
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+
+def demo(opt):
+ """ model configuration """
+ if 'CTC' in opt.Prediction:
+ converter = CTCLabelConverter(opt.character)
+ else:
+ converter = AttnLabelConverter(opt.character)
+ opt.num_class = len(converter.character)
+
+ if opt.rgb:
+ opt.input_channel = 3
+ model = Model(opt)
+ print('model input parameters', opt.imgH, opt.imgW, opt.num_fiducial, opt.input_channel, opt.output_channel,
+ opt.hidden_size, opt.num_class, opt.batch_max_length, opt.Transformation, opt.FeatureExtraction,
+ opt.SequenceModeling, opt.Prediction)
+ model = torch.nn.DataParallel(model).to(device)
+
+ # load model
+ print('loading pretrained model from %s' % opt.saved_model)
+ model.load_state_dict(torch.load(opt.saved_model, map_location=device))
+
+ # prepare data. two demo images from https://github.com/bgshih/crnn#run-demo
+ AlignCollate_demo = AlignCollate(imgH=opt.imgH, imgW=opt.imgW, keep_ratio_with_pad=opt.PAD)
+ demo_data = RawDataset(root=opt.image_folder, opt=opt) # use RawDataset
+ demo_loader = torch.utils.data.DataLoader(
+ demo_data, batch_size=opt.batch_size,
+ shuffle=False,
+ num_workers=int(opt.workers),
+ collate_fn=AlignCollate_demo, pin_memory=True)
+
+ # predict
+ model.eval()
+ with torch.no_grad():
+ for image_tensors, image_path_list in demo_loader:
+ batch_size = image_tensors.size(0)
+ image = image_tensors.to(device)
+ # For max length prediction
+ length_for_pred = torch.IntTensor([opt.batch_max_length] * batch_size).to(device)
+ text_for_pred = torch.LongTensor(batch_size, opt.batch_max_length + 1).fill_(0).to(device)
+
+ if 'CTC' in opt.Prediction:
+ preds = model(image, text_for_pred)
+
+ # Select max probabilty (greedy decoding) then decode index to character
+ preds_size = torch.IntTensor([preds.size(1)] * batch_size)
+ _, preds_index = preds.max(2)
+ # preds_index = preds_index.view(-1)
+ preds_str = converter.decode(preds_index, preds_size)
+
+ else:
+ preds = model(image, text_for_pred, is_train=False)
+
+ # select max probabilty (greedy decoding) then decode index to character
+ _, preds_index = preds.max(2)
+ preds_str = converter.decode(preds_index, length_for_pred)
+
+
+ log = open(f'./log_demo_result.txt', 'a')
+ dashed_line = '-' * 80
+ head = f'{"image_path":25s}\t{"predicted_labels":25s}\tconfidence score'
+
+ print(f'{dashed_line}\n{head}\n{dashed_line}')
+ log.write(f'{dashed_line}\n{head}\n{dashed_line}\n')
+
+ preds_prob = F.softmax(preds, dim=2)
+ preds_max_prob, _ = preds_prob.max(dim=2)
+ for img_name, pred, pred_max_prob in zip(image_path_list, preds_str, preds_max_prob):
+ if 'Attn' in opt.Prediction:
+ pred_EOS = pred.find('[s]')
+ pred = pred[:pred_EOS] # prune after "end of sentence" token ([s])
+ pred_max_prob = pred_max_prob[:pred_EOS]
+
+ # calculate confidence score (= multiply of pred_max_prob)
+ confidence_score = pred_max_prob.cumprod(dim=0)[-1]
+
+ print(f'{img_name:25s}\t{pred:25s}\t{confidence_score:0.4f}')
+ log.write(f'{img_name:25s}\t{pred:25s}\t{confidence_score:0.4f}\n')
+
+ log.close()
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--image_folder', required=True, help='path to image_folder which contains text images')
+ parser.add_argument('--workers', type=int, help='number of data loading workers', default=4)
+ parser.add_argument('--batch_size', type=int, default=192, help='input batch size')
+ parser.add_argument('--saved_model', required=True, help="path to saved_model to evaluation")
+ """ Data processing """
+ parser.add_argument('--batch_max_length', type=int, default=25, help='maximum-label-length')
+ parser.add_argument('--imgH', type=int, default=32, help='the height of the input image')
+ parser.add_argument('--imgW', type=int, default=100, help='the width of the input image')
+ parser.add_argument('--rgb', action='store_true', help='use rgb input')
+ parser.add_argument('--character', type=str, default='0123456789abcdefghijklmnopqrstuvwxyz', help='character label')
+ parser.add_argument('--sensitive', action='store_true', help='for sensitive character mode')
+ parser.add_argument('--PAD', action='store_true', help='whether to keep ratio then pad for image resize')
+ """ Model Architecture """
+ parser.add_argument('--Transformation', type=str, required=True, help='Transformation stage. None|TPS')
+ parser.add_argument('--FeatureExtraction', type=str, required=True, help='FeatureExtraction stage. VGG|RCNN|ResNet')
+ parser.add_argument('--SequenceModeling', type=str, required=True, help='SequenceModeling stage. None|BiLSTM')
+ parser.add_argument('--Prediction', type=str, required=True, help='Prediction stage. CTC|Attn')
+ parser.add_argument('--num_fiducial', type=int, default=20, help='number of fiducial points of TPS-STN')
+ parser.add_argument('--input_channel', type=int, default=1, help='the number of input channel of Feature extractor')
+ parser.add_argument('--output_channel', type=int, default=512,
+ help='the number of output channel of Feature extractor')
+ parser.add_argument('--hidden_size', type=int, default=256, help='the size of the LSTM hidden state')
+
+ opt = parser.parse_args()
+
+ """ vocab / character number configuration """
+ if opt.sensitive:
+ opt.character = string.printable[:-6] # same with ASTER setting (use 94 char).
+
+ cudnn.benchmark = True
+ cudnn.deterministic = True
+ opt.num_gpu = torch.cuda.device_count()
+
+ demo(opt)
diff --git a/TextRecognition/model.py b/TextRecognition/model.py
new file mode 100644
index 0000000000000000000000000000000000000000..be0c612b7eca400ede9684a646af17b985373e31
--- /dev/null
+++ b/TextRecognition/model.py
@@ -0,0 +1,92 @@
+"""
+Copyright (c) 2019-present NAVER Corp.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import torch.nn as nn
+
+from modules.transformation import TPS_SpatialTransformerNetwork
+from modules.feature_extraction import VGG_FeatureExtractor, RCNN_FeatureExtractor, ResNet_FeatureExtractor
+from modules.sequence_modeling import BidirectionalLSTM
+from modules.prediction import Attention
+
+
+class Model(nn.Module):
+
+ def __init__(self, opt):
+ super(Model, self).__init__()
+ self.opt = opt
+ self.stages = {'Trans': opt.Transformation, 'Feat': opt.FeatureExtraction,
+ 'Seq': opt.SequenceModeling, 'Pred': opt.Prediction}
+
+ """ Transformation """
+ if opt.Transformation == 'TPS':
+ self.Transformation = TPS_SpatialTransformerNetwork(
+ F=opt.num_fiducial, I_size=(opt.imgH, opt.imgW), I_r_size=(opt.imgH, opt.imgW), I_channel_num=opt.input_channel)
+ else:
+ print('No Transformation module specified')
+
+ """ FeatureExtraction """
+ if opt.FeatureExtraction == 'VGG':
+ self.FeatureExtraction = VGG_FeatureExtractor(opt.input_channel, opt.output_channel)
+ elif opt.FeatureExtraction == 'RCNN':
+ self.FeatureExtraction = RCNN_FeatureExtractor(opt.input_channel, opt.output_channel)
+ elif opt.FeatureExtraction == 'ResNet':
+ self.FeatureExtraction = ResNet_FeatureExtractor(opt.input_channel, opt.output_channel)
+ else:
+ raise Exception('No FeatureExtraction module specified')
+ self.FeatureExtraction_output = opt.output_channel # int(imgH/16-1) * 512
+ self.AdaptiveAvgPool = nn.AdaptiveAvgPool2d((None, 1)) # Transform final (imgH/16-1) -> 1
+
+ """ Sequence modeling"""
+ if opt.SequenceModeling == 'BiLSTM':
+ self.SequenceModeling = nn.Sequential(
+ BidirectionalLSTM(self.FeatureExtraction_output, opt.hidden_size, opt.hidden_size),
+ BidirectionalLSTM(opt.hidden_size, opt.hidden_size, opt.hidden_size))
+ self.SequenceModeling_output = opt.hidden_size
+ else:
+ print('No SequenceModeling module specified')
+ self.SequenceModeling_output = self.FeatureExtraction_output
+
+ """ Prediction """
+ if opt.Prediction == 'CTC':
+ self.Prediction = nn.Linear(self.SequenceModeling_output, opt.num_class)
+ elif opt.Prediction == 'Attn':
+ self.Prediction = Attention(self.SequenceModeling_output, opt.hidden_size, opt.num_class)
+ else:
+ raise Exception('Prediction is neither CTC or Attn')
+
+ def forward(self, input, text, is_train=True):
+ """ Transformation stage """
+ if not self.stages['Trans'] == "None":
+ input = self.Transformation(input)
+
+ """ Feature extraction stage """
+ visual_feature = self.FeatureExtraction(input)
+ visual_feature = self.AdaptiveAvgPool(visual_feature.permute(0, 3, 1, 2)) # [b, c, h, w] -> [b, w, c, h]
+ visual_feature = visual_feature.squeeze(3)
+
+ """ Sequence modeling stage """
+ if self.stages['Seq'] == 'BiLSTM':
+ contextual_feature = self.SequenceModeling(visual_feature)
+ else:
+ contextual_feature = visual_feature # for convenience. this is NOT contextually modeled by BiLSTM
+
+ """ Prediction stage """
+ if self.stages['Pred'] == 'CTC':
+ prediction = self.Prediction(contextual_feature.contiguous())
+ else:
+ prediction = self.Prediction(contextual_feature.contiguous(), text, is_train, batch_max_length=self.opt.batch_max_length)
+
+ return prediction
diff --git a/TextRecognition/modules/__pycache__/feature_extraction.cpython-310.pyc b/TextRecognition/modules/__pycache__/feature_extraction.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..7b6e3e2ede33c3f6f51cbff2b02c9bf25960606b
Binary files /dev/null and b/TextRecognition/modules/__pycache__/feature_extraction.cpython-310.pyc differ
diff --git a/TextRecognition/modules/__pycache__/prediction.cpython-310.pyc b/TextRecognition/modules/__pycache__/prediction.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..51895f12ab1f76e3255ef4d0d36d25c7ca7739f3
Binary files /dev/null and b/TextRecognition/modules/__pycache__/prediction.cpython-310.pyc differ
diff --git a/TextRecognition/modules/__pycache__/sequence_modeling.cpython-310.pyc b/TextRecognition/modules/__pycache__/sequence_modeling.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..232741c0c470aae023b7c50843908b554d2a15ad
Binary files /dev/null and b/TextRecognition/modules/__pycache__/sequence_modeling.cpython-310.pyc differ
diff --git a/TextRecognition/modules/__pycache__/transformation.cpython-310.pyc b/TextRecognition/modules/__pycache__/transformation.cpython-310.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..4b48f489ecfe763643f917e84e425552b1c2b5da
Binary files /dev/null and b/TextRecognition/modules/__pycache__/transformation.cpython-310.pyc differ
diff --git a/TextRecognition/modules/feature_extraction.py b/TextRecognition/modules/feature_extraction.py
new file mode 100644
index 0000000000000000000000000000000000000000..b5f3004202f175db2ea1494807ea2dbab1aaec76
--- /dev/null
+++ b/TextRecognition/modules/feature_extraction.py
@@ -0,0 +1,246 @@
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class VGG_FeatureExtractor(nn.Module):
+ """ FeatureExtractor of CRNN (https://arxiv.org/pdf/1507.05717.pdf) """
+
+ def __init__(self, input_channel, output_channel=512):
+ super(VGG_FeatureExtractor, self).__init__()
+ self.output_channel = [int(output_channel / 8), int(output_channel / 4),
+ int(output_channel / 2), output_channel] # [64, 128, 256, 512]
+ self.ConvNet = nn.Sequential(
+ nn.Conv2d(input_channel, self.output_channel[0], 3, 1, 1), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # 64x16x50
+ nn.Conv2d(self.output_channel[0], self.output_channel[1], 3, 1, 1), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # 128x8x25
+ nn.Conv2d(self.output_channel[1], self.output_channel[2], 3, 1, 1), nn.ReLU(True), # 256x8x25
+ nn.Conv2d(self.output_channel[2], self.output_channel[2], 3, 1, 1), nn.ReLU(True),
+ nn.MaxPool2d((2, 1), (2, 1)), # 256x4x25
+ nn.Conv2d(self.output_channel[2], self.output_channel[3], 3, 1, 1, bias=False),
+ nn.BatchNorm2d(self.output_channel[3]), nn.ReLU(True), # 512x4x25
+ nn.Conv2d(self.output_channel[3], self.output_channel[3], 3, 1, 1, bias=False),
+ nn.BatchNorm2d(self.output_channel[3]), nn.ReLU(True),
+ nn.MaxPool2d((2, 1), (2, 1)), # 512x2x25
+ nn.Conv2d(self.output_channel[3], self.output_channel[3], 2, 1, 0), nn.ReLU(True)) # 512x1x24
+
+ def forward(self, input):
+ return self.ConvNet(input)
+
+
+class RCNN_FeatureExtractor(nn.Module):
+ """ FeatureExtractor of GRCNN (https://papers.nips.cc/paper/6637-gated-recurrent-convolution-neural-network-for-ocr.pdf) """
+
+ def __init__(self, input_channel, output_channel=512):
+ super(RCNN_FeatureExtractor, self).__init__()
+ self.output_channel = [int(output_channel / 8), int(output_channel / 4),
+ int(output_channel / 2), output_channel] # [64, 128, 256, 512]
+ self.ConvNet = nn.Sequential(
+ nn.Conv2d(input_channel, self.output_channel[0], 3, 1, 1), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # 64 x 16 x 50
+ GRCL(self.output_channel[0], self.output_channel[0], num_iteration=5, kernel_size=3, pad=1),
+ nn.MaxPool2d(2, 2), # 64 x 8 x 25
+ GRCL(self.output_channel[0], self.output_channel[1], num_iteration=5, kernel_size=3, pad=1),
+ nn.MaxPool2d(2, (2, 1), (0, 1)), # 128 x 4 x 26
+ GRCL(self.output_channel[1], self.output_channel[2], num_iteration=5, kernel_size=3, pad=1),
+ nn.MaxPool2d(2, (2, 1), (0, 1)), # 256 x 2 x 27
+ nn.Conv2d(self.output_channel[2], self.output_channel[3], 2, 1, 0, bias=False),
+ nn.BatchNorm2d(self.output_channel[3]), nn.ReLU(True)) # 512 x 1 x 26
+
+ def forward(self, input):
+ return self.ConvNet(input)
+
+
+class ResNet_FeatureExtractor(nn.Module):
+ """ FeatureExtractor of FAN (http://openaccess.thecvf.com/content_ICCV_2017/papers/Cheng_Focusing_Attention_Towards_ICCV_2017_paper.pdf) """
+
+ def __init__(self, input_channel, output_channel=512):
+ super(ResNet_FeatureExtractor, self).__init__()
+ self.ConvNet = ResNet(input_channel, output_channel, BasicBlock, [1, 2, 5, 3])
+
+ def forward(self, input):
+ return self.ConvNet(input)
+
+
+# For Gated RCNN
+class GRCL(nn.Module):
+
+ def __init__(self, input_channel, output_channel, num_iteration, kernel_size, pad):
+ super(GRCL, self).__init__()
+ self.wgf_u = nn.Conv2d(input_channel, output_channel, 1, 1, 0, bias=False)
+ self.wgr_x = nn.Conv2d(output_channel, output_channel, 1, 1, 0, bias=False)
+ self.wf_u = nn.Conv2d(input_channel, output_channel, kernel_size, 1, pad, bias=False)
+ self.wr_x = nn.Conv2d(output_channel, output_channel, kernel_size, 1, pad, bias=False)
+
+ self.BN_x_init = nn.BatchNorm2d(output_channel)
+
+ self.num_iteration = num_iteration
+ self.GRCL = [GRCL_unit(output_channel) for _ in range(num_iteration)]
+ self.GRCL = nn.Sequential(*self.GRCL)
+
+ def forward(self, input):
+ """ The input of GRCL is consistant over time t, which is denoted by u(0)
+ thus wgf_u / wf_u is also consistant over time t.
+ """
+ wgf_u = self.wgf_u(input)
+ wf_u = self.wf_u(input)
+ x = F.relu(self.BN_x_init(wf_u))
+
+ for i in range(self.num_iteration):
+ x = self.GRCL[i](wgf_u, self.wgr_x(x), wf_u, self.wr_x(x))
+
+ return x
+
+
+class GRCL_unit(nn.Module):
+
+ def __init__(self, output_channel):
+ super(GRCL_unit, self).__init__()
+ self.BN_gfu = nn.BatchNorm2d(output_channel)
+ self.BN_grx = nn.BatchNorm2d(output_channel)
+ self.BN_fu = nn.BatchNorm2d(output_channel)
+ self.BN_rx = nn.BatchNorm2d(output_channel)
+ self.BN_Gx = nn.BatchNorm2d(output_channel)
+
+ def forward(self, wgf_u, wgr_x, wf_u, wr_x):
+ G_first_term = self.BN_gfu(wgf_u)
+ G_second_term = self.BN_grx(wgr_x)
+ G = F.sigmoid(G_first_term + G_second_term)
+
+ x_first_term = self.BN_fu(wf_u)
+ x_second_term = self.BN_Gx(self.BN_rx(wr_x) * G)
+ x = F.relu(x_first_term + x_second_term)
+
+ return x
+
+
+class BasicBlock(nn.Module):
+ expansion = 1
+
+ def __init__(self, inplanes, planes, stride=1, downsample=None):
+ super(BasicBlock, self).__init__()
+ self.conv1 = self._conv3x3(inplanes, planes)
+ self.bn1 = nn.BatchNorm2d(planes)
+ self.conv2 = self._conv3x3(planes, planes)
+ self.bn2 = nn.BatchNorm2d(planes)
+ self.relu = nn.ReLU(inplace=True)
+ self.downsample = downsample
+ self.stride = stride
+
+ def _conv3x3(self, in_planes, out_planes, stride=1):
+ "3x3 convolution with padding"
+ return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
+ padding=1, bias=False)
+
+ def forward(self, x):
+ residual = x
+
+ out = self.conv1(x)
+ out = self.bn1(out)
+ out = self.relu(out)
+
+ out = self.conv2(out)
+ out = self.bn2(out)
+
+ if self.downsample is not None:
+ residual = self.downsample(x)
+ out += residual
+ out = self.relu(out)
+
+ return out
+
+
+class ResNet(nn.Module):
+
+ def __init__(self, input_channel, output_channel, block, layers):
+ super(ResNet, self).__init__()
+
+ self.output_channel_block = [int(output_channel / 4), int(output_channel / 2), output_channel, output_channel]
+
+ self.inplanes = int(output_channel / 8)
+ self.conv0_1 = nn.Conv2d(input_channel, int(output_channel / 16),
+ kernel_size=3, stride=1, padding=1, bias=False)
+ self.bn0_1 = nn.BatchNorm2d(int(output_channel / 16))
+ self.conv0_2 = nn.Conv2d(int(output_channel / 16), self.inplanes,
+ kernel_size=3, stride=1, padding=1, bias=False)
+ self.bn0_2 = nn.BatchNorm2d(self.inplanes)
+ self.relu = nn.ReLU(inplace=True)
+
+ self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
+ self.layer1 = self._make_layer(block, self.output_channel_block[0], layers[0])
+ self.conv1 = nn.Conv2d(self.output_channel_block[0], self.output_channel_block[
+ 0], kernel_size=3, stride=1, padding=1, bias=False)
+ self.bn1 = nn.BatchNorm2d(self.output_channel_block[0])
+
+ self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
+ self.layer2 = self._make_layer(block, self.output_channel_block[1], layers[1], stride=1)
+ self.conv2 = nn.Conv2d(self.output_channel_block[1], self.output_channel_block[
+ 1], kernel_size=3, stride=1, padding=1, bias=False)
+ self.bn2 = nn.BatchNorm2d(self.output_channel_block[1])
+
+ self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=(2, 1), padding=(0, 1))
+ self.layer3 = self._make_layer(block, self.output_channel_block[2], layers[2], stride=1)
+ self.conv3 = nn.Conv2d(self.output_channel_block[2], self.output_channel_block[
+ 2], kernel_size=3, stride=1, padding=1, bias=False)
+ self.bn3 = nn.BatchNorm2d(self.output_channel_block[2])
+
+ self.layer4 = self._make_layer(block, self.output_channel_block[3], layers[3], stride=1)
+ self.conv4_1 = nn.Conv2d(self.output_channel_block[3], self.output_channel_block[
+ 3], kernel_size=2, stride=(2, 1), padding=(0, 1), bias=False)
+ self.bn4_1 = nn.BatchNorm2d(self.output_channel_block[3])
+ self.conv4_2 = nn.Conv2d(self.output_channel_block[3], self.output_channel_block[
+ 3], kernel_size=2, stride=1, padding=0, bias=False)
+ self.bn4_2 = nn.BatchNorm2d(self.output_channel_block[3])
+
+ def _make_layer(self, block, planes, blocks, stride=1):
+ downsample = None
+ if stride != 1 or self.inplanes != planes * block.expansion:
+ downsample = nn.Sequential(
+ nn.Conv2d(self.inplanes, planes * block.expansion,
+ kernel_size=1, stride=stride, bias=False),
+ nn.BatchNorm2d(planes * block.expansion),
+ )
+
+ layers = []
+ layers.append(block(self.inplanes, planes, stride, downsample))
+ self.inplanes = planes * block.expansion
+ for i in range(1, blocks):
+ layers.append(block(self.inplanes, planes))
+
+ return nn.Sequential(*layers)
+
+ def forward(self, x):
+ x = self.conv0_1(x)
+ x = self.bn0_1(x)
+ x = self.relu(x)
+ x = self.conv0_2(x)
+ x = self.bn0_2(x)
+ x = self.relu(x)
+
+ x = self.maxpool1(x)
+ x = self.layer1(x)
+ x = self.conv1(x)
+ x = self.bn1(x)
+ x = self.relu(x)
+
+ x = self.maxpool2(x)
+ x = self.layer2(x)
+ x = self.conv2(x)
+ x = self.bn2(x)
+ x = self.relu(x)
+
+ x = self.maxpool3(x)
+ x = self.layer3(x)
+ x = self.conv3(x)
+ x = self.bn3(x)
+ x = self.relu(x)
+
+ x = self.layer4(x)
+ x = self.conv4_1(x)
+ x = self.bn4_1(x)
+ x = self.relu(x)
+ x = self.conv4_2(x)
+ x = self.bn4_2(x)
+ x = self.relu(x)
+
+ return x
diff --git a/TextRecognition/modules/prediction.py b/TextRecognition/modules/prediction.py
new file mode 100644
index 0000000000000000000000000000000000000000..b6c3cb32541ea4ea5f22377ce5062e5ea34ff2f7
--- /dev/null
+++ b/TextRecognition/modules/prediction.py
@@ -0,0 +1,81 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+
+class Attention(nn.Module):
+
+ def __init__(self, input_size, hidden_size, num_classes):
+ super(Attention, self).__init__()
+ self.attention_cell = AttentionCell(input_size, hidden_size, num_classes)
+ self.hidden_size = hidden_size
+ self.num_classes = num_classes
+ self.generator = nn.Linear(hidden_size, num_classes)
+
+ def _char_to_onehot(self, input_char, onehot_dim=38):
+ input_char = input_char.unsqueeze(1)
+ batch_size = input_char.size(0)
+ one_hot = torch.FloatTensor(batch_size, onehot_dim).zero_().to(device)
+ one_hot = one_hot.scatter_(1, input_char, 1)
+ return one_hot
+
+ def forward(self, batch_H, text, is_train=True, batch_max_length=25):
+ """
+ input:
+ batch_H : contextual_feature H = hidden state of encoder. [batch_size x num_steps x contextual_feature_channels]
+ text : the text-index of each image. [batch_size x (max_length+1)]. +1 for [GO] token. text[:, 0] = [GO].
+ output: probability distribution at each step [batch_size x num_steps x num_classes]
+ """
+ batch_size = batch_H.size(0)
+ num_steps = batch_max_length + 1 # +1 for [s] at end of sentence.
+
+ output_hiddens = torch.FloatTensor(batch_size, num_steps, self.hidden_size).fill_(0).to(device)
+ hidden = (torch.FloatTensor(batch_size, self.hidden_size).fill_(0).to(device),
+ torch.FloatTensor(batch_size, self.hidden_size).fill_(0).to(device))
+
+ if is_train:
+ for i in range(num_steps):
+ # one-hot vectors for a i-th char. in a batch
+ char_onehots = self._char_to_onehot(text[:, i], onehot_dim=self.num_classes)
+ # hidden : decoder's hidden s_{t-1}, batch_H : encoder's hidden H, char_onehots : one-hot(y_{t-1})
+ hidden, alpha = self.attention_cell(hidden, batch_H, char_onehots)
+ output_hiddens[:, i, :] = hidden[0] # LSTM hidden index (0: hidden, 1: Cell)
+ probs = self.generator(output_hiddens)
+
+ else:
+ targets = torch.LongTensor(batch_size).fill_(0).to(device) # [GO] token
+ probs = torch.FloatTensor(batch_size, num_steps, self.num_classes).fill_(0).to(device)
+
+ for i in range(num_steps):
+ char_onehots = self._char_to_onehot(targets, onehot_dim=self.num_classes)
+ hidden, alpha = self.attention_cell(hidden, batch_H, char_onehots)
+ probs_step = self.generator(hidden[0])
+ probs[:, i, :] = probs_step
+ _, next_input = probs_step.max(1)
+ targets = next_input
+
+ return probs # batch_size x num_steps x num_classes
+
+
+class AttentionCell(nn.Module):
+
+ def __init__(self, input_size, hidden_size, num_embeddings):
+ super(AttentionCell, self).__init__()
+ self.i2h = nn.Linear(input_size, hidden_size, bias=False)
+ self.h2h = nn.Linear(hidden_size, hidden_size) # either i2i or h2h should have bias
+ self.score = nn.Linear(hidden_size, 1, bias=False)
+ self.rnn = nn.LSTMCell(input_size + num_embeddings, hidden_size)
+ self.hidden_size = hidden_size
+
+ def forward(self, prev_hidden, batch_H, char_onehots):
+ # [batch_size x num_encoder_step x num_channel] -> [batch_size x num_encoder_step x hidden_size]
+ batch_H_proj = self.i2h(batch_H)
+ prev_hidden_proj = self.h2h(prev_hidden[0]).unsqueeze(1)
+ e = self.score(torch.tanh(batch_H_proj + prev_hidden_proj)) # batch_size x num_encoder_step * 1
+
+ alpha = F.softmax(e, dim=1)
+ context = torch.bmm(alpha.permute(0, 2, 1), batch_H).squeeze(1) # batch_size x num_channel
+ concat_context = torch.cat([context, char_onehots], 1) # batch_size x (num_channel + num_embedding)
+ cur_hidden = self.rnn(concat_context, prev_hidden)
+ return cur_hidden, alpha
diff --git a/TextRecognition/modules/sequence_modeling.py b/TextRecognition/modules/sequence_modeling.py
new file mode 100644
index 0000000000000000000000000000000000000000..af32c59b2cc981be1b43412ddf4ac853d0611210
--- /dev/null
+++ b/TextRecognition/modules/sequence_modeling.py
@@ -0,0 +1,19 @@
+import torch.nn as nn
+
+
+class BidirectionalLSTM(nn.Module):
+
+ def __init__(self, input_size, hidden_size, output_size):
+ super(BidirectionalLSTM, self).__init__()
+ self.rnn = nn.LSTM(input_size, hidden_size, bidirectional=True, batch_first=True)
+ self.linear = nn.Linear(hidden_size * 2, output_size)
+
+ def forward(self, input):
+ """
+ input : visual feature [batch_size x T x input_size]
+ output : contextual feature [batch_size x T x output_size]
+ """
+ self.rnn.flatten_parameters()
+ recurrent, _ = self.rnn(input) # batch_size x T x input_size -> batch_size x T x (2*hidden_size)
+ output = self.linear(recurrent) # batch_size x T x output_size
+ return output
diff --git a/TextRecognition/modules/transformation.py b/TextRecognition/modules/transformation.py
new file mode 100644
index 0000000000000000000000000000000000000000..875d1ae96ec31a186c3782c070886d100326fcf6
--- /dev/null
+++ b/TextRecognition/modules/transformation.py
@@ -0,0 +1,164 @@
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+
+class TPS_SpatialTransformerNetwork(nn.Module):
+ """ Rectification Network of RARE, namely TPS based STN """
+
+ def __init__(self, F, I_size, I_r_size, I_channel_num=1):
+ """ Based on RARE TPS
+ input:
+ batch_I: Batch Input Image [batch_size x I_channel_num x I_height x I_width]
+ I_size : (height, width) of the input image I
+ I_r_size : (height, width) of the rectified image I_r
+ I_channel_num : the number of channels of the input image I
+ output:
+ batch_I_r: rectified image [batch_size x I_channel_num x I_r_height x I_r_width]
+ """
+ super(TPS_SpatialTransformerNetwork, self).__init__()
+ self.F = F
+ self.I_size = I_size
+ self.I_r_size = I_r_size # = (I_r_height, I_r_width)
+ self.I_channel_num = I_channel_num
+ self.LocalizationNetwork = LocalizationNetwork(self.F, self.I_channel_num)
+ self.GridGenerator = GridGenerator(self.F, self.I_r_size)
+
+ def forward(self, batch_I):
+ batch_C_prime = self.LocalizationNetwork(batch_I) # batch_size x K x 2
+ build_P_prime = self.GridGenerator.build_P_prime(batch_C_prime) # batch_size x n (= I_r_width x I_r_height) x 2
+ build_P_prime_reshape = build_P_prime.reshape([build_P_prime.size(0), self.I_r_size[0], self.I_r_size[1], 2])
+
+ if torch.__version__ > "1.2.0":
+ batch_I_r = F.grid_sample(batch_I, build_P_prime_reshape, padding_mode='border', align_corners=True)
+ else:
+ batch_I_r = F.grid_sample(batch_I, build_P_prime_reshape, padding_mode='border')
+
+ return batch_I_r
+
+
+class LocalizationNetwork(nn.Module):
+ """ Localization Network of RARE, which predicts C' (K x 2) from I (I_width x I_height) """
+
+ def __init__(self, F, I_channel_num):
+ super(LocalizationNetwork, self).__init__()
+ self.F = F
+ self.I_channel_num = I_channel_num
+ self.conv = nn.Sequential(
+ nn.Conv2d(in_channels=self.I_channel_num, out_channels=64, kernel_size=3, stride=1, padding=1,
+ bias=False), nn.BatchNorm2d(64), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # batch_size x 64 x I_height/2 x I_width/2
+ nn.Conv2d(64, 128, 3, 1, 1, bias=False), nn.BatchNorm2d(128), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # batch_size x 128 x I_height/4 x I_width/4
+ nn.Conv2d(128, 256, 3, 1, 1, bias=False), nn.BatchNorm2d(256), nn.ReLU(True),
+ nn.MaxPool2d(2, 2), # batch_size x 256 x I_height/8 x I_width/8
+ nn.Conv2d(256, 512, 3, 1, 1, bias=False), nn.BatchNorm2d(512), nn.ReLU(True),
+ nn.AdaptiveAvgPool2d(1) # batch_size x 512
+ )
+
+ self.localization_fc1 = nn.Sequential(nn.Linear(512, 256), nn.ReLU(True))
+ self.localization_fc2 = nn.Linear(256, self.F * 2)
+
+ # Init fc2 in LocalizationNetwork
+ self.localization_fc2.weight.data.fill_(0)
+ """ see RARE paper Fig. 6 (a) """
+ ctrl_pts_x = np.linspace(-1.0, 1.0, int(F / 2))
+ ctrl_pts_y_top = np.linspace(0.0, -1.0, num=int(F / 2))
+ ctrl_pts_y_bottom = np.linspace(1.0, 0.0, num=int(F / 2))
+ ctrl_pts_top = np.stack([ctrl_pts_x, ctrl_pts_y_top], axis=1)
+ ctrl_pts_bottom = np.stack([ctrl_pts_x, ctrl_pts_y_bottom], axis=1)
+ initial_bias = np.concatenate([ctrl_pts_top, ctrl_pts_bottom], axis=0)
+ self.localization_fc2.bias.data = torch.from_numpy(initial_bias).float().view(-1)
+
+ def forward(self, batch_I):
+ """
+ input: batch_I : Batch Input Image [batch_size x I_channel_num x I_height x I_width]
+ output: batch_C_prime : Predicted coordinates of fiducial points for input batch [batch_size x F x 2]
+ """
+ batch_size = batch_I.size(0)
+ features = self.conv(batch_I).view(batch_size, -1)
+ batch_C_prime = self.localization_fc2(self.localization_fc1(features)).view(batch_size, self.F, 2)
+ return batch_C_prime
+
+
+class GridGenerator(nn.Module):
+ """ Grid Generator of RARE, which produces P_prime by multipling T with P """
+
+ def __init__(self, F, I_r_size):
+ """ Generate P_hat and inv_delta_C for later """
+ super(GridGenerator, self).__init__()
+ self.eps = 1e-6
+ self.I_r_height, self.I_r_width = I_r_size
+ self.F = F
+ self.C = self._build_C(self.F) # F x 2
+ self.P = self._build_P(self.I_r_width, self.I_r_height)
+ ## for multi-gpu, you need register buffer
+ self.register_buffer("inv_delta_C", torch.tensor(self._build_inv_delta_C(self.F, self.C)).float()) # F+3 x F+3
+ self.register_buffer("P_hat", torch.tensor(self._build_P_hat(self.F, self.C, self.P)).float()) # n x F+3
+ ## for fine-tuning with different image width, you may use below instead of self.register_buffer
+ #self.inv_delta_C = torch.tensor(self._build_inv_delta_C(self.F, self.C)).float().cuda() # F+3 x F+3
+ #self.P_hat = torch.tensor(self._build_P_hat(self.F, self.C, self.P)).float().cuda() # n x F+3
+
+ def _build_C(self, F):
+ """ Return coordinates of fiducial points in I_r; C """
+ ctrl_pts_x = np.linspace(-1.0, 1.0, int(F / 2))
+ ctrl_pts_y_top = -1 * np.ones(int(F / 2))
+ ctrl_pts_y_bottom = np.ones(int(F / 2))
+ ctrl_pts_top = np.stack([ctrl_pts_x, ctrl_pts_y_top], axis=1)
+ ctrl_pts_bottom = np.stack([ctrl_pts_x, ctrl_pts_y_bottom], axis=1)
+ C = np.concatenate([ctrl_pts_top, ctrl_pts_bottom], axis=0)
+ return C # F x 2
+
+ def _build_inv_delta_C(self, F, C):
+ """ Return inv_delta_C which is needed to calculate T """
+ hat_C = np.zeros((F, F), dtype=float) # F x F
+ for i in range(0, F):
+ for j in range(i, F):
+ r = np.linalg.norm(C[i] - C[j])
+ hat_C[i, j] = r
+ hat_C[j, i] = r
+ np.fill_diagonal(hat_C, 1)
+ hat_C = (hat_C ** 2) * np.log(hat_C)
+ # print(C.shape, hat_C.shape)
+ delta_C = np.concatenate( # F+3 x F+3
+ [
+ np.concatenate([np.ones((F, 1)), C, hat_C], axis=1), # F x F+3
+ np.concatenate([np.zeros((2, 3)), np.transpose(C)], axis=1), # 2 x F+3
+ np.concatenate([np.zeros((1, 3)), np.ones((1, F))], axis=1) # 1 x F+3
+ ],
+ axis=0
+ )
+ inv_delta_C = np.linalg.inv(delta_C)
+ return inv_delta_C # F+3 x F+3
+
+ def _build_P(self, I_r_width, I_r_height):
+ I_r_grid_x = (np.arange(-I_r_width, I_r_width, 2) + 1.0) / I_r_width # self.I_r_width
+ I_r_grid_y = (np.arange(-I_r_height, I_r_height, 2) + 1.0) / I_r_height # self.I_r_height
+ P = np.stack( # self.I_r_width x self.I_r_height x 2
+ np.meshgrid(I_r_grid_x, I_r_grid_y),
+ axis=2
+ )
+ return P.reshape([-1, 2]) # n (= self.I_r_width x self.I_r_height) x 2
+
+ def _build_P_hat(self, F, C, P):
+ n = P.shape[0] # n (= self.I_r_width x self.I_r_height)
+ P_tile = np.tile(np.expand_dims(P, axis=1), (1, F, 1)) # n x 2 -> n x 1 x 2 -> n x F x 2
+ C_tile = np.expand_dims(C, axis=0) # 1 x F x 2
+ P_diff = P_tile - C_tile # n x F x 2
+ rbf_norm = np.linalg.norm(P_diff, ord=2, axis=2, keepdims=False) # n x F
+ rbf = np.multiply(np.square(rbf_norm), np.log(rbf_norm + self.eps)) # n x F
+ P_hat = np.concatenate([np.ones((n, 1)), P, rbf], axis=1)
+ return P_hat # n x F+3
+
+ def build_P_prime(self, batch_C_prime):
+ """ Generate Grid from batch_C_prime [batch_size x F x 2] """
+ batch_size = batch_C_prime.size(0)
+ batch_inv_delta_C = self.inv_delta_C.repeat(batch_size, 1, 1)
+ batch_P_hat = self.P_hat.repeat(batch_size, 1, 1)
+ batch_C_prime_with_zeros = torch.cat((batch_C_prime, torch.zeros(
+ batch_size, 3, 2).float().to(device)), dim=1) # batch_size x F+3 x 2
+ batch_T = torch.bmm(batch_inv_delta_C, batch_C_prime_with_zeros) # batch_size x F+3 x 2
+ batch_P_prime = torch.bmm(batch_P_hat, batch_T) # batch_size x n x 2
+ return batch_P_prime # batch_size x n x 2
diff --git a/TextRecognition/utils.py b/TextRecognition/utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..e576358418b844d2371aa99673f76c459ca780d9
--- /dev/null
+++ b/TextRecognition/utils.py
@@ -0,0 +1,169 @@
+import torch
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+
+class CTCLabelConverter(object):
+ """ Convert between text-label and text-index """
+
+ def __init__(self, character):
+ # character (str): set of the possible characters.
+ dict_character = list(character)
+
+ self.dict = {}
+ for i, char in enumerate(dict_character):
+ # NOTE: 0 is reserved for 'CTCblank' token required by CTCLoss
+ self.dict[char] = i + 1
+
+ self.character = ['[CTCblank]'] + dict_character # dummy '[CTCblank]' token for CTCLoss (index 0)
+
+ def encode(self, text, batch_max_length=25):
+ """convert text-label into text-index.
+ input:
+ text: text labels of each image. [batch_size]
+ batch_max_length: max length of text label in the batch. 25 by default
+
+ output:
+ text: text index for CTCLoss. [batch_size, batch_max_length]
+ length: length of each text. [batch_size]
+ """
+ length = [len(s) for s in text]
+
+ # The index used for padding (=0) would not affect the CTC loss calculation.
+ batch_text = torch.LongTensor(len(text), batch_max_length).fill_(0)
+ for i, t in enumerate(text):
+ text = list(t)
+ text = [self.dict[char] for char in text]
+ batch_text[i][:len(text)] = torch.LongTensor(text)
+ return (batch_text.to(device), torch.IntTensor(length).to(device))
+
+ def decode(self, text_index, length):
+ """ convert text-index into text-label. """
+ texts = []
+ for index, l in enumerate(length):
+ t = text_index[index, :]
+
+ char_list = []
+ for i in range(l):
+ if t[i] != 0 and (not (i > 0 and t[i - 1] == t[i])): # removing repeated characters and blank.
+ char_list.append(self.character[t[i]])
+ text = ''.join(char_list)
+
+ texts.append(text)
+ return texts
+
+
+class CTCLabelConverterForBaiduWarpctc(object):
+ """ Convert between text-label and text-index for baidu warpctc """
+
+ def __init__(self, character):
+ # character (str): set of the possible characters.
+ dict_character = list(character)
+
+ self.dict = {}
+ for i, char in enumerate(dict_character):
+ # NOTE: 0 is reserved for 'CTCblank' token required by CTCLoss
+ self.dict[char] = i + 1
+
+ self.character = ['[CTCblank]'] + dict_character # dummy '[CTCblank]' token for CTCLoss (index 0)
+
+ def encode(self, text, batch_max_length=25):
+ """convert text-label into text-index.
+ input:
+ text: text labels of each image. [batch_size]
+ output:
+ text: concatenated text index for CTCLoss.
+ [sum(text_lengths)] = [text_index_0 + text_index_1 + ... + text_index_(n - 1)]
+ length: length of each text. [batch_size]
+ """
+ length = [len(s) for s in text]
+ text = ''.join(text)
+ text = [self.dict[char] for char in text]
+
+ return (torch.IntTensor(text), torch.IntTensor(length))
+
+ def decode(self, text_index, length):
+ """ convert text-index into text-label. """
+ texts = []
+ index = 0
+ for l in length:
+ t = text_index[index:index + l]
+
+ char_list = []
+ for i in range(l):
+ if t[i] != 0 and (not (i > 0 and t[i - 1] == t[i])): # removing repeated characters and blank.
+ char_list.append(self.character[t[i]])
+ text = ''.join(char_list)
+
+ texts.append(text)
+ index += l
+ return texts
+
+
+class AttnLabelConverter(object):
+ """ Convert between text-label and text-index """
+
+ def __init__(self, character):
+ # character (str): set of the possible characters.
+ # [GO] for the start token of the attention decoder. [s] for end-of-sentence token.
+ list_token = ['[GO]', '[s]'] # ['[s]','[UNK]','[PAD]','[GO]']
+ list_character = list(character)
+ self.character = list_token + list_character
+
+ self.dict = {}
+ for i, char in enumerate(self.character):
+ # print(i, char)
+ self.dict[char] = i
+
+ def encode(self, text, batch_max_length=25):
+ """ convert text-label into text-index.
+ input:
+ text: text labels of each image. [batch_size]
+ batch_max_length: max length of text label in the batch. 25 by default
+
+ output:
+ text : the input of attention decoder. [batch_size x (max_length+2)] +1 for [GO] token and +1 for [s] token.
+ text[:, 0] is [GO] token and text is padded with [GO] token after [s] token.
+ length : the length of output of attention decoder, which count [s] token also. [3, 7, ....] [batch_size]
+ """
+ length = [len(s) + 1 for s in text] # +1 for [s] at end of sentence.
+ # batch_max_length = max(length) # this is not allowed for multi-gpu setting
+ batch_max_length += 1
+ # additional +1 for [GO] at first step. batch_text is padded with [GO] token after [s] token.
+ batch_text = torch.LongTensor(len(text), batch_max_length + 1).fill_(0)
+ for i, t in enumerate(text):
+ text = list(t)
+ text.append('[s]')
+ text = [self.dict[char] for char in text]
+ batch_text[i][1:1 + len(text)] = torch.LongTensor(text) # batch_text[:, 0] = [GO] token
+ return (batch_text.to(device), torch.IntTensor(length).to(device))
+
+ def decode(self, text_index, length):
+ """ convert text-index into text-label. """
+ texts = []
+ for index, l in enumerate(length):
+ text = ''.join([self.character[i] for i in text_index[index, :]])
+ texts.append(text)
+ return texts
+
+
+class Averager(object):
+ """Compute average for torch.Tensor, used for loss average."""
+
+ def __init__(self):
+ self.reset()
+
+ def add(self, v):
+ count = v.data.numel()
+ v = v.data.sum()
+ self.n_count += count
+ self.sum += v
+
+ def reset(self):
+ self.n_count = 0
+ self.sum = 0
+
+ def val(self):
+ res = 0
+ if self.n_count != 0:
+ res = self.sum / float(self.n_count)
+ return res
diff --git a/app.py b/app.py
new file mode 100644
index 0000000000000000000000000000000000000000..e213137f4f0ca3759d0a66e16c293e11fff9d8f7
--- /dev/null
+++ b/app.py
@@ -0,0 +1,30 @@
+import gradio as gr
+import subprocess
+
+import directories as Dir
+import process as ps
+
+
+def HCR(im):
+ ps.textDetection(im)
+ ps.textRearrange()
+ ps.textRecognition()
+ hcr = ps.getHcrResult(Dir.txt_file_path)
+ subprocess.call('cd'+ Dir.yolo_dir, shell=True)
+ #clearDir()
+
+ return hcr
+
+title = "HCR"
+description = "[Postech X KITA] Elite Undergradute AI Strategy Program | AI Project Competition Team 2 OCR Part"
+article = "Handwritten Text Character Recognition Task " \ + "Text Detection + Text Rearrangement + Text Recognition" \ + "For More Information" \ + "GitBlog |" \ + "Source code
" + +#Source code +#examples = [['zidane.jpg'], ['bus.jpg']] +#examples=examples, +gr.Interface(fn=HCR, inputs="image", outputs = "text", title=title, description=description, article=article, analytics_enabled=False).launch( + debug=True) \ No newline at end of file diff --git a/directories.py b/directories.py new file mode 100644 index 0000000000000000000000000000000000000000..8f4bdf94940a981a6f02a8324b96b0dd7abd13a9 --- /dev/null +++ b/directories.py @@ -0,0 +1,11 @@ +home_dir = "/HCR" +yolo_dir = "/HCR/TextDetection/" +input_img = "/HCR/TextDetection/cookie/user_input.jpg" +detect_model_dir = "/runs/wordDetection/weights/best.pt" +cropped_img_folder_name = "user_output" +cropped_img_path = "/runs/detect/" + cropped_img_folder_name +folder_path = cropped_img_path + "/crops/word" +DBSCAN_dir = "/HCR/TextRearrange" +recog_dir = "/HCR/TextRecognition" +recog_model_dir='/HCR/TextRecognition/best_accuracy_s/best_accuracy_s.pth' +txt_file_path = "/HCR/TextRecognition/log_demo_result.txt" \ No newline at end of file diff --git a/process.py b/process.py new file mode 100644 index 0000000000000000000000000000000000000000..736e209185a4caf722e101968f476f65e586f614 --- /dev/null +++ b/process.py @@ -0,0 +1,76 @@ +import os +import shutil +import subprocess +from PIL import Image +import directories as Dir + +def clearDir(): + + #/text_detection/cookie/user_input + #shutil.rmtree('/cookie') + #os.remove("/cookie/user_input.jpg") + + #cropped_img_path = "/runs/detect/" + cropped_img_folder_name + shutil.rmtree(Dir.cropped_img_path) #'/runs/detect/user_output' + + #txt_file_path = "/HCR/TextRecognition/log_demo_result.txt" + os.remove(Dir.txt_file_path) + +def textDetection(im): + + #change dir to yolo folder + #yolo_dir = "/HCR/TextDetection/" + subprocess.call('cd'+ Dir.yolo_dir, shell=True) + + #transfrom ndarray type to PIL type + im = Image.fromarray(im) + + # save input image to cookie folder + subprocess.call('cd cookie', shell=True) + im.save("user_input.jpg", 'JPEG') + + #yolo_dir = "/HCR/TextDetection/" + subprocess.call('cd'+ Dir.yolo_dir, shell=True) + + # (Shell) run detect.py to get cropped word images + subprocess.call(['python','detect.py', + #User Input Data : /text_detection/cookie + '--source','/cookie', + #Text Detection Model : /runs/wordDetection/weights/best.pt + '--weights', Dir.detect_model_dir, + '--conf','0.25', + #Output Images Save Directory /runs/detect/user_output + '--name', Dir.cropped_img_folder_name, + '--save-crop', + '--save-conf']) + + #g = (size / max(im.size)) # gain + #im = im.resize((int(x * g) for x in im.size), Image.ANTIALIAS) # resize + + #results = model(im) # inference + #results.render() # updates results.imgs with boxes and labels + #return Image.fromarray(results.imgs[0]) + +def textRearrange(): + subprocess.call('cd' + Dir.DBSCAN_dir, shell=True) + subprocess.call(['python','DBSCAN.py']) + +def textRecognition(): + #%cd /content/drive/MyDrive/KITA/Text/lmdb/deep-text-recognition-benchmark + subprocess.call('cd '+Dir.recog_dir, shell=True) + #!CUDA_VISIBLE_DEVICES=0 python3 demo.py --Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn --image_folder /content/drive/MyDrive/KITA/Text/YOLO/runs/detect/youtube_data2/crops/word --saved_model /content/drive/MyDrive/KITA/Text/best_accuracy_s/best_accuracy_s.pth + subprocess.call('CUDA_VISIBLE_DEVICES="" python3 demo.py --Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn --image_folder ' + Dir.home_dir + Dir.cropped_img_path + '/crops/word --saved_model '+ Dir.recog_model_dir, shell=True) + +def getHcrResult(file_path):#*# + texts = "" + with open(file_path, 'r') as file: + lines = file.readlines() + for line in lines[3:]: + line = line.replace("\t","*",1) + line = line.replace(" ","*",1) + parts = line.replace(" ","") + parts2 = parts.split("*",2) + #print(len(parts2)) + texts = texts +" "+ str(parts2[1:2])[2:-2] + + return texts \ No newline at end of file diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..98025810654130199001b2fffa89bc276a1108c0 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,502 @@ +absl-py==1.4.0 +aiofiles==23.2.1 +aiohttp==3.9.1 +aiosignal==1.3.1 +alabaster==0.7.13 +albumentations==1.3.1 +altair==4.2.2 +annotated-types==0.6.0 +anyio==3.7.1 +appdirs==1.4.4 +argon2-cffi==23.1.0 +argon2-cffi-bindings==21.2.0 +array-record==0.5.0 +arviz==0.15.1 +astropy==5.3.4 +astunparse==1.6.3 +async-timeout==4.0.3 +atpublic==4.0 +attrs==23.1.0 +audioread==3.0.1 +autograd==1.6.2 +Babel==2.14.0 +backcall==0.2.0 +beautifulsoup4==4.11.2 +bidict==0.22.1 +bigframes==0.17.0 +bleach==6.1.0 +blinker==1.4 +blis==0.7.11 +blosc2==2.0.0 +bokeh==3.3.2 +bqplot==0.12.42 +branca==0.7.0 +build==1.0.3 +CacheControl==0.13.1 +cachetools==5.3.2 +catalogue==2.0.10 +certifi==2023.11.17 +cffi==1.16.0 +chardet==5.2.0 +charset-normalizer==3.3.2 +chex==0.1.7 +click==8.1.7 +click-plugins==1.1.1 +cligj==0.7.2 +cloudpickle==2.2.1 +cmake==3.27.9 +cmdstanpy==1.2.0 +colorama==0.4.6 +colorcet==3.0.1 +colorlover==0.3.0 +colour==0.1.5 +community==1.0.0b1 +confection==0.1.4 +cons==0.4.6 +contextlib2==21.6.0 +contourpy==1.2.0 +cryptography==41.0.7 +cufflinks==0.17.3 +cupy-cuda12x==12.2.0 +cvxopt==1.3.2 +cvxpy==1.3.2 +cycler==0.12.1 +cymem==2.0.8 +Cython==3.0.7 +dask==2023.8.1 +datascience==0.17.6 +db-dtypes==1.2.0 +dbus-python==1.2.18 +debugpy==1.6.6 +decorator==4.4.2 +defusedxml==0.7.1 +diskcache==5.6.3 +distributed==2023.8.1 +distro==1.7.0 +dlib==19.24.2 +dm-tree==0.1.8 +docutils==0.18.1 +dopamine-rl==4.0.6 +duckdb==0.9.2 +earthengine-api==0.1.384 +easydict==1.11 +ecos==2.0.12 +editdistance==0.6.2 +eerepr==0.0.4 +en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.6.0/en_core_web_sm-3.6.0-py3-none-any.whl#sha256=83276fc78a70045627144786b52e1f2728ad5e29e5e43916ec37ea9c26a11212 +entrypoints==0.4 +et-xmlfile==1.1.0 +etils==1.6.0 +etuples==0.3.9 +exceptiongroup==1.2.0 +fastai==2.7.13 +fastapi==0.108.0 +fastcore==1.5.29 +fastdownload==0.0.7 +fastjsonschema==2.19.0 +fastprogress==1.0.3 +fastrlock==0.8.2 +ffmpy==0.3.1 +filelock==3.13.1 +fiona==1.9.5 +firebase-admin==5.3.0 +Flask==2.2.5 +flatbuffers==23.5.26 +flax==0.7.5 +folium==0.14.0 +fonttools==4.47.0 +frozendict==2.3.10 +frozenlist==1.4.1 +fsspec==2023.6.0 +future==0.18.3 +gast==0.5.4 +gcsfs==2023.6.0 +GDAL==3.4.3 +gdown==4.6.6 +geemap==0.29.6 +gensim==4.3.2 +geocoder==1.38.1 +geographiclib==2.0 +geopandas==0.13.2 +geopy==2.3.0 +gin-config==0.5.0 +gitdb==4.0.11 +GitPython==3.1.40 +glob2==0.7 +google==2.0.3 +google-ai-generativelanguage==0.4.0 +google-api-core==2.11.1 +google-api-python-client==2.84.0 +google-auth==2.17.3 +google-auth-httplib2==0.1.1 +google-auth-oauthlib==1.2.0 +google-cloud-aiplatform==1.38.1 +google-cloud-bigquery==3.12.0 +google-cloud-bigquery-connection==1.12.1 +google-cloud-bigquery-storage==2.24.0 +google-cloud-core==2.3.3 +google-cloud-datastore==2.15.2 +google-cloud-firestore==2.11.1 +google-cloud-functions==1.13.3 +google-cloud-iam==2.13.0 +google-cloud-language==2.9.1 +google-cloud-resource-manager==1.11.0 +google-cloud-storage==2.8.0 +google-cloud-translate==3.11.3 +google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz#sha256=eb6190db7e94f83570d0663e5324b48dbc8d7ffbf066a99973922bc15318ecda +google-crc32c==1.5.0 +google-generativeai==0.3.2 +google-pasta==0.2.0 +google-resumable-media==2.7.0 +googleapis-common-protos==1.62.0 +googledrivedownloader==0.4 +gradio==4.13.0 +gradio_client==0.8.0 +graphviz==0.20.1 +greenlet==3.0.2 +grpc-google-iam-v1==0.13.0 +grpcio==1.60.0 +grpcio-status==1.48.2 +gspread==3.4.2 +gspread-dataframe==3.3.1 +gym==0.25.2 +gym-notices==0.0.8 +h11==0.14.0 +h5netcdf==1.3.0 +h5py==3.9.0 +holidays==0.39 +holoviews==1.17.1 +html5lib==1.1 +httpcore==1.0.2 +httpimport==1.3.1 +httplib2==0.22.0 +httpx==0.26.0 +huggingface-hub==0.20.1 +humanize==4.7.0 +hyperopt==0.2.7 +ibis-framework==6.2.0 +idna==3.6 +imageio==2.31.6 +imageio-ffmpeg==0.4.9 +imagesize==1.4.1 +imbalanced-learn==0.10.1 +imgaug==0.4.0 +importlib-metadata==7.0.0 +importlib-resources==6.1.1 +imutils==0.5.4 +inflect==7.0.0 +iniconfig==2.0.0 +install==1.3.5 +intel-openmp==2023.2.3 +ipyevents==2.0.2 +ipyfilechooser==0.6.0 +ipykernel==5.5.6 +ipyleaflet==0.18.1 +ipython==7.34.0 +ipython-genutils==0.2.0 +ipython-sql==0.5.0 +ipytree==0.2.2 +ipywidgets==7.7.1 +itsdangerous==2.1.2 +jax==0.4.23 +jaxlib @ https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.23+cuda12.cudnn89-cp310-cp310-manylinux2014_x86_64.whl#sha256=8e42000672599e7ec0ea7f551acfcc95dcdd0e22b05a1d1f12f97b56a9fce4a8 +jeepney==0.7.1 +jieba==0.42.1 +Jinja2==3.1.2 +joblib==1.3.2 +jsonpickle==3.0.2 +jsonschema==4.19.2 +jsonschema-specifications==2023.11.2 +jupyter-client==6.1.12 +jupyter-console==6.1.0 +jupyter-server==1.24.0 +jupyter_core==5.5.1 +jupyterlab-widgets==3.0.9 +jupyterlab_pygments==0.3.0 +kaggle==1.5.16 +kagglehub==0.1.4 +kaleido==0.2.1 +keras==2.15.0 +keyring==23.5.0 +kiwisolver==1.4.5 +langcodes==3.3.0 +launchpadlib==1.10.16 +lazr.restfulclient==0.14.4 +lazr.uri==1.0.6 +lazy_loader==0.3 +libclang==16.0.6 +librosa==0.10.1 +lida==0.0.10 +lightgbm==4.1.0 +linkify-it-py==2.0.2 +llmx==0.0.15a0 +llvmlite==0.41.1 +lmdb==1.4.1 +locket==1.0.0 +logical-unification==0.4.6 +lxml==4.9.4 +malloy==2023.1067 +Markdown==3.5.1 +markdown-it-py==3.0.0 +MarkupSafe==2.1.3 +matplotlib==3.7.1 +matplotlib-inline==0.1.6 +matplotlib-venn==0.11.9 +mdit-py-plugins==0.4.0 +mdurl==0.1.2 +miniKanren==1.0.3 +missingno==0.5.2 +mistune==0.8.4 +mizani==0.9.3 +mkl==2023.2.0 +ml-dtypes==0.2.0 +mlxtend==0.22.0 +more-itertools==10.1.0 +moviepy==1.0.3 +mpmath==1.3.0 +msgpack==1.0.7 +multidict==6.0.4 +multipledispatch==1.0.0 +multitasking==0.0.11 +murmurhash==1.0.10 +music21==9.1.0 +natsort==8.4.0 +nbclassic==1.0.0 +nbclient==0.9.0 +nbconvert==6.5.4 +nbformat==5.9.2 +nest-asyncio==1.5.8 +networkx==3.2.1 +nibabel==4.0.2 +nltk==3.8.1 +notebook==6.5.5 +notebook_shim==0.2.3 +numba==0.58.1 +numexpr==2.8.8 +numpy==1.23.5 +oauth2client==4.1.3 +oauthlib==3.2.2 +opencv-contrib-python==4.8.0.76 +opencv-python==4.8.0.76 +opencv-python-headless==4.8.1.78 +openpyxl==3.1.2 +opt-einsum==3.3.0 +optax==0.1.7 +orbax-checkpoint==0.4.4 +orjson==3.9.10 +osqp==0.6.2.post8 +packaging==23.2 +pandas==1.5.3 +pandas-datareader==0.10.0 +pandas-gbq==0.19.2 +pandas-stubs==1.5.3.230304 +pandocfilters==1.5.0 +panel==1.3.6 +param==2.0.1 +parso==0.8.3 +parsy==2.1 +partd==1.4.1 +pathlib==1.0.1 +pathy==0.10.3 +patsy==0.5.4 +peewee==3.17.0 +pexpect==4.9.0 +pickleshare==0.7.5 +Pillow==9.4.0 +pip-tools==6.13.0 +platformdirs==4.1.0 +plotly==5.15.0 +plotnine==0.12.4 +pluggy==1.3.0 +polars==0.17.3 +pooch==1.8.0 +portpicker==1.5.2 +prefetch-generator==1.0.3 +preshed==3.0.9 +prettytable==3.9.0 +proglog==0.1.10 +progressbar2==4.2.0 +prometheus-client==0.19.0 +promise==2.3 +prompt-toolkit==3.0.43 +prophet==1.1.5 +proto-plus==1.23.0 +protobuf==3.20.3 +psutil==5.9.5 +psycopg2==2.9.9 +ptyprocess==0.7.0 +py-cpuinfo==9.0.0 +py4j==0.10.9.7 +pyarrow==10.0.1 +pyasn1==0.5.1 +pyasn1-modules==0.3.0 +pycocotools==2.0.7 +pycparser==2.21 +pyct==0.5.0 +pydantic==2.5.3 +pydantic_core==2.14.6 +pydata-google-auth==1.8.2 +pydot==1.4.2 +pydot-ng==2.0.0 +pydotplus==2.0.2 +PyDrive==1.3.1 +PyDrive2==1.6.3 +pydub==0.25.1 +pyerfa==2.0.1.1 +pygame==2.5.2 +Pygments==2.16.1 +PyGObject==3.42.1 +PyJWT==2.3.0 +pymc==5.7.2 +pymystem3==0.2.0 +PyOpenGL==3.1.7 +pyOpenSSL==23.3.0 +pyparsing==3.1.1 +pyperclip==1.8.2 +pyproj==3.6.1 +pyproject_hooks==1.0.0 +pyshp==2.3.1 +PySocks==1.7.1 +pytensor==2.14.2 +pytest==7.4.3 +python-apt==0.0.0 +python-box==7.1.1 +python-dateutil==2.8.2 +python-louvain==0.16 +python-multipart==0.0.6 +python-slugify==8.0.1 +python-utils==3.8.1 +pytz==2023.3.post1 +pyviz_comms==3.0.0 +PyWavelets==1.5.0 +PyYAML==6.0.1 +pyzmq==23.2.1 +qdldl==0.1.7.post0 +qudida==0.0.4 +ratelim==0.1.6 +referencing==0.32.0 +regex==2023.6.3 +requests==2.31.0 +requests-oauthlib==1.3.1 +requirements-parser==0.5.0 +rich==13.7.0 +rpds-py==0.15.2 +rpy2==3.4.2 +rsa==4.9 +safetensors==0.4.1 +scikit-image==0.19.3 +scikit-learn==1.2.2 +scipy==1.11.4 +scooby==0.9.2 +scs==3.2.4.post1 +seaborn==0.12.2 +SecretStorage==3.3.1 +semantic-version==2.10.0 +Send2Trash==1.8.2 +shapely==2.0.2 +shellingham==1.5.4 +six==1.16.0 +sklearn-pandas==2.2.0 +smart-open==6.4.0 +smmap==5.0.1 +sniffio==1.3.0 +snowballstemmer==2.2.0 +sortedcontainers==2.4.0 +soundfile==0.12.1 +soupsieve==2.5 +soxr==0.3.7 +spacy==3.6.1 +spacy-legacy==3.0.12 +spacy-loggers==1.0.5 +Sphinx==5.0.2 +sphinxcontrib-applehelp==1.0.7 +sphinxcontrib-devhelp==1.0.5 +sphinxcontrib-htmlhelp==2.0.4 +sphinxcontrib-jsmath==1.0.1 +sphinxcontrib-qthelp==1.0.6 +sphinxcontrib-serializinghtml==1.1.9 +SQLAlchemy==2.0.23 +sqlglot==17.16.2 +sqlparse==0.4.4 +srsly==2.4.8 +stanio==0.3.0 +starlette==0.32.0.post1 +statsmodels==0.14.1 +sympy==1.12 +tables==3.8.0 +tabulate==0.9.0 +tbb==2021.11.0 +tblib==3.0.0 +tenacity==8.2.3 +tensorboard==2.15.1 +tensorboard-data-server==0.7.2 +tensorflow==2.15.0 +tensorflow-datasets==4.9.4 +tensorflow-estimator==2.15.0 +tensorflow-gcs-config==2.15.0 +tensorflow-hub==0.15.0 +tensorflow-io-gcs-filesystem==0.35.0 +tensorflow-metadata==1.14.0 +tensorflow-probability==0.22.0 +tensorstore==0.1.45 +termcolor==2.4.0 +terminado==0.18.0 +text-unidecode==1.3 +textblob==0.17.1 +tf-slim==1.1.0 +thinc==8.1.12 +thop==0.1.1.post2209072238 +threadpoolctl==3.2.0 +tifffile==2023.12.9 +tinycss2==1.2.1 +tokenizers==0.15.0 +toml==0.10.2 +tomli==2.0.1 +tomlkit==0.12.0 +toolz==0.12.0 +torch @ https://download.pytorch.org/whl/cu121/torch-2.1.0%2Bcu121-cp310-cp310-linux_x86_64.whl#sha256=0d4e8c52a1fcf5ed6cfc256d9a370fcf4360958fc79d0b08a51d55e70914df46 +torchaudio @ https://download.pytorch.org/whl/cu121/torchaudio-2.1.0%2Bcu121-cp310-cp310-linux_x86_64.whl#sha256=676bda4042734eda99bc59b2d7f761f345d3cde0cad492ad34e3aefde688c6d8 +torchdata==0.7.0 +torchsummary==1.5.1 +torchtext==0.16.0 +torchvision @ https://download.pytorch.org/whl/cu121/torchvision-0.16.0%2Bcu121-cp310-cp310-linux_x86_64.whl#sha256=e76e78d0ad43636c9884b3084ffaea8a8b61f21129fbfa456a5fe734f0affea9 +tornado==6.3.2 +tqdm==4.66.1 +traitlets==5.7.1 +traittypes==0.2.1 +transformers==4.35.2 +triton==2.1.0 +tweepy==4.14.0 +typer==0.9.0 +types-pytz==2023.3.1.1 +types-setuptools==69.0.0.0 +typing_extensions==4.9.0 +tzlocal==5.2 +uc-micro-py==1.0.2 +ultralytics==8.0.236 +uritemplate==4.1.1 +urllib3==2.0.7 +uvicorn==0.25.0 +vega-datasets==0.9.0 +wadllib==1.3.6 +wasabi==1.1.2 +wcwidth==0.2.12 +webcolors==1.13 +webencodings==0.5.1 +websocket-client==1.7.0 +websockets==11.0.3 +Werkzeug==3.0.1 +widgetsnbextension==3.6.6 +wordcloud==1.9.3 +wrapt==1.14.1 +xarray==2023.7.0 +xarray-einstats==0.6.0 +xgboost==2.0.3 +xlrd==2.0.1 +xxhash==3.4.1 +xyzservices==2023.10.1 +yarl==1.9.4 +yellowbrick==1.5 +yfinance==0.2.33 +zict==3.0.0 +zipp==3.17.0