LABEL_0 LABEL_1 LABEL_10 LABEL_100 LABEL_101 LABEL_102 LABEL_103 LABEL_104 LABEL_105 LABEL_106 LABEL_107 LABEL_108 LABEL_109 LABEL_11 LABEL_110 LABEL_111 LABEL_112 LABEL_113 LABEL_114 LABEL_115 LABEL_116 LABEL_117 LABEL_118 LABEL_119 LABEL_12 LABEL_120 LABEL_121 LABEL_122 LABEL_123 LABEL_124 LABEL_125 LABEL_126 LABEL_127 LABEL_128 LABEL_129 LABEL_13 LABEL_130 LABEL_131 LABEL_132 LABEL_133 LABEL_134 LABEL_135 LABEL_136 LABEL_137 LABEL_138 LABEL_139 LABEL_14 LABEL_140 LABEL_141 LABEL_142 LABEL_143 LABEL_144 LABEL_145 LABEL_146 LABEL_147 LABEL_148 LABEL_149 LABEL_15 LABEL_150 LABEL_151 LABEL_152 LABEL_153 LABEL_154 LABEL_155 LABEL_156 LABEL_157 LABEL_158 LABEL_159 LABEL_16 LABEL_160 LABEL_161 LABEL_162 LABEL_163 LABEL_164 LABEL_165 LABEL_166 LABEL_167 LABEL_168 LABEL_169 LABEL_17 LABEL_170 LABEL_171 LABEL_172 LABEL_173 LABEL_174 LABEL_175 LABEL_176 LABEL_177 LABEL_178 LABEL_179 LABEL_18 LABEL_180 LABEL_181 LABEL_182 LABEL_183 LABEL_184 LABEL_185 LABEL_186 LABEL_187 LABEL_188 LABEL_189 LABEL_19 LABEL_190 LABEL_191 LABEL_192 LABEL_193 LABEL_194 LABEL_195 LABEL_196 LABEL_197 LABEL_198 LABEL_199 LABEL_2 LABEL_20 LABEL_200 LABEL_201 LABEL_202 LABEL_203 LABEL_204 LABEL_205 LABEL_206 LABEL_207 LABEL_208 LABEL_209 LABEL_21 LABEL_210 LABEL_211 LABEL_212 LABEL_213 LABEL_214 LABEL_215 LABEL_216 LABEL_217 LABEL_218 LABEL_219 LABEL_22 LABEL_220 LABEL_221 LABEL_222 LABEL_223 LABEL_224 LABEL_225 LABEL_226 LABEL_227 LABEL_228 LABEL_229 LABEL_23 LABEL_230 LABEL_231 LABEL_232 LABEL_233 LABEL_234 LABEL_235 LABEL_236 LABEL_237 LABEL_238 LABEL_239 LABEL_24 LABEL_240 LABEL_241 LABEL_242 LABEL_243 LABEL_244 LABEL_245 LABEL_246 LABEL_247 LABEL_248 LABEL_249 LABEL_25 LABEL_250 LABEL_251 LABEL_252 LABEL_253 LABEL_254 LABEL_255 LABEL_256 LABEL_257 LABEL_258 LABEL_259 LABEL_26 LABEL_260 LABEL_261 LABEL_262 LABEL_263 LABEL_264 LABEL_265 LABEL_266 LABEL_267 LABEL_268 LABEL_269 LABEL_27 LABEL_270 LABEL_271 LABEL_272 LABEL_273 LABEL_274 LABEL_275 LABEL_276 LABEL_277 LABEL_278 LABEL_279 LABEL_28 LABEL_280 LABEL_281 LABEL_282 LABEL_283 LABEL_284 LABEL_285 LABEL_286 LABEL_287 LABEL_288 LABEL_289 LABEL_29 LABEL_290 LABEL_291 LABEL_292 LABEL_293 LABEL_294 LABEL_295 LABEL_296 LABEL_297 LABEL_298 LABEL_299 LABEL_3 LABEL_30 LABEL_300 LABEL_301 LABEL_302 LABEL_303 LABEL_304 LABEL_305 LABEL_306 LABEL_307 LABEL_308 LABEL_309 LABEL_31 LABEL_310 LABEL_311 LABEL_312 LABEL_313 LABEL_314 LABEL_315 LABEL_316 LABEL_317 LABEL_318 LABEL_319 LABEL_32 LABEL_320 LABEL_321 LABEL_322 LABEL_323 LABEL_324 LABEL_325 LABEL_326 LABEL_327 LABEL_328 LABEL_329 LABEL_33 LABEL_330 LABEL_331 LABEL_332 LABEL_333 LABEL_334 LABEL_335 LABEL_336 LABEL_337 LABEL_338 LABEL_339 LABEL_34 LABEL_340 LABEL_341 LABEL_342 LABEL_343 LABEL_344 LABEL_345 LABEL_346 LABEL_347 LABEL_348 LABEL_349 LABEL_35 LABEL_350 LABEL_351 LABEL_352 LABEL_353 LABEL_354 LABEL_355 LABEL_356 LABEL_357 LABEL_358 LABEL_359 LABEL_36 LABEL_360 LABEL_361 LABEL_362 LABEL_363 LABEL_364 LABEL_365 LABEL_366 LABEL_367 LABEL_368 LABEL_369 LABEL_37 LABEL_370 LABEL_371 LABEL_372 LABEL_373 LABEL_374 LABEL_375 LABEL_376 LABEL_377 LABEL_378 LABEL_379 LABEL_38 LABEL_380 LABEL_381 LABEL_382 LABEL_383 LABEL_384 LABEL_385 LABEL_386 LABEL_387 LABEL_388 LABEL_389 LABEL_39 LABEL_390 LABEL_391 LABEL_392 LABEL_393 LABEL_394 LABEL_395 LABEL_396 LABEL_397 LABEL_398 LABEL_399 LABEL_4 LABEL_40 LABEL_400 LABEL_401 LABEL_402 LABEL_403 LABEL_404 LABEL_405 LABEL_406 LABEL_407 LABEL_408 LABEL_409 LABEL_41 LABEL_410 LABEL_411 LABEL_412 LABEL_413 LABEL_414 LABEL_415 LABEL_416 LABEL_417 LABEL_418 LABEL_419 LABEL_42 LABEL_420 LABEL_421 LABEL_422 LABEL_423 LABEL_424 LABEL_425 LABEL_426 LABEL_427 LABEL_428 LABEL_429 LABEL_43 LABEL_430 LABEL_431 LABEL_432 LABEL_433 LABEL_434 LABEL_435 LABEL_436 LABEL_437 LABEL_438 LABEL_439 LABEL_44 LABEL_440 LABEL_441 LABEL_442 LABEL_443 LABEL_444 LABEL_445 LABEL_446 LABEL_447 LABEL_448 LABEL_449 LABEL_45 LABEL_450 LABEL_451 LABEL_452 LABEL_453 LABEL_454 LABEL_455 LABEL_456 LABEL_457 LABEL_458 LABEL_459 LABEL_46 LABEL_460 LABEL_461 LABEL_462 LABEL_463 LABEL_464 LABEL_465 LABEL_466 LABEL_467 LABEL_468 LABEL_469 LABEL_47 LABEL_470 LABEL_471 LABEL_472 LABEL_473 LABEL_474 LABEL_475 LABEL_476 LABEL_477 LABEL_478 LABEL_479 LABEL_48 LABEL_480 LABEL_481 LABEL_482 LABEL_483 LABEL_484 LABEL_485 LABEL_486 LABEL_487 LABEL_488 LABEL_489 LABEL_49 LABEL_490 LABEL_491 LABEL_492 LABEL_493 LABEL_494 LABEL_495 LABEL_496 LABEL_497 LABEL_498 LABEL_499 LABEL_5 LABEL_50 LABEL_500 LABEL_501 LABEL_502 LABEL_503 LABEL_504 LABEL_505 LABEL_506 LABEL_507 LABEL_508 LABEL_509 LABEL_51 LABEL_510 LABEL_511 LABEL_512 LABEL_513 LABEL_514 LABEL_515 LABEL_516 LABEL_517 LABEL_518 LABEL_519 LABEL_52 LABEL_520 LABEL_521 LABEL_522 LABEL_523 LABEL_524 LABEL_525 LABEL_526 LABEL_527 LABEL_528 LABEL_529 LABEL_53 LABEL_530 LABEL_531 LABEL_532 LABEL_533 LABEL_534 LABEL_535 LABEL_536 LABEL_537 LABEL_538 LABEL_539 LABEL_54 LABEL_540 LABEL_541 LABEL_542 LABEL_543 LABEL_544 LABEL_545 LABEL_546 LABEL_547 LABEL_548 LABEL_549 LABEL_55 LABEL_550 LABEL_551 LABEL_552 LABEL_553 LABEL_554 LABEL_555 LABEL_556 LABEL_557 LABEL_558 LABEL_559 LABEL_56 LABEL_560 LABEL_561 LABEL_562 LABEL_563 LABEL_564 LABEL_565 LABEL_566 LABEL_567 LABEL_568 LABEL_569 LABEL_57 LABEL_570 LABEL_571 LABEL_572 LABEL_573 LABEL_574 LABEL_575 LABEL_576 LABEL_577 LABEL_578 LABEL_579 LABEL_58 LABEL_580 LABEL_581 LABEL_582 LABEL_583 LABEL_584 LABEL_585 LABEL_586 LABEL_587 LABEL_588 LABEL_589 LABEL_59 LABEL_590 LABEL_591 LABEL_592 LABEL_593 LABEL_594 LABEL_595 LABEL_596 LABEL_597 LABEL_598 LABEL_599 LABEL_6 LABEL_60 LABEL_600 LABEL_601 LABEL_602 LABEL_603 LABEL_604 LABEL_605 LABEL_606 LABEL_607 LABEL_608 LABEL_609 LABEL_61 LABEL_610 LABEL_611 LABEL_612 LABEL_613 LABEL_614 LABEL_615 LABEL_616 LABEL_617 LABEL_618 LABEL_619 LABEL_62 LABEL_620 LABEL_621 LABEL_622 LABEL_623 LABEL_624 LABEL_625 LABEL_626 LABEL_627 LABEL_628 LABEL_629 LABEL_63 LABEL_630 LABEL_631 LABEL_632 LABEL_633 LABEL_634 LABEL_635 LABEL_636 LABEL_637 LABEL_638 LABEL_639 LABEL_64 LABEL_640 LABEL_641 LABEL_642 LABEL_643 LABEL_644 LABEL_645 LABEL_646 LABEL_647 LABEL_648 LABEL_649 LABEL_65 LABEL_650 LABEL_651 LABEL_652 LABEL_653 LABEL_654 LABEL_655 LABEL_656 LABEL_657 LABEL_658 LABEL_659 LABEL_66 LABEL_660 LABEL_661 LABEL_662 LABEL_663 LABEL_664 LABEL_665 LABEL_666 LABEL_667 LABEL_668 LABEL_669 LABEL_67 LABEL_670 LABEL_671 LABEL_672 LABEL_673 LABEL_674 LABEL_675 LABEL_676 LABEL_677 LABEL_678 LABEL_679 LABEL_68 LABEL_680 LABEL_681 LABEL_682 LABEL_683 LABEL_684 LABEL_685 LABEL_686 LABEL_687 LABEL_688 LABEL_689 LABEL_69 LABEL_690 LABEL_691 LABEL_692 LABEL_693 LABEL_694 LABEL_695 LABEL_696 LABEL_697 LABEL_698 LABEL_699 LABEL_7 LABEL_70 LABEL_700 LABEL_701 LABEL_702 LABEL_703 LABEL_704 LABEL_705 LABEL_706 LABEL_707 LABEL_708 LABEL_709 LABEL_71 LABEL_710 LABEL_711 LABEL_712 LABEL_713 LABEL_714 LABEL_715 LABEL_716 LABEL_717 LABEL_718 LABEL_719 LABEL_72 LABEL_720 LABEL_721 LABEL_722 LABEL_723 LABEL_724 LABEL_725 LABEL_726 LABEL_727 LABEL_728 LABEL_729 LABEL_73 LABEL_730 LABEL_731 LABEL_732 LABEL_733 LABEL_734 LABEL_735 LABEL_736 LABEL_737 LABEL_738 LABEL_739 LABEL_74 LABEL_740 LABEL_741 LABEL_742 LABEL_743 LABEL_744 LABEL_745 LABEL_746 LABEL_747 LABEL_748 LABEL_749 LABEL_75 LABEL_750 LABEL_751 LABEL_752 LABEL_753 LABEL_754 LABEL_755 LABEL_756 LABEL_757 LABEL_758 LABEL_759 LABEL_76 LABEL_760 LABEL_761 LABEL_762 LABEL_763 LABEL_764 LABEL_765 LABEL_766 LABEL_767 LABEL_77 LABEL_78 LABEL_79 LABEL_8 LABEL_80 LABEL_81 LABEL_82 LABEL_83 LABEL_84 LABEL_85 LABEL_86 LABEL_87 LABEL_88 LABEL_89 LABEL_9 LABEL_90 LABEL_91 LABEL_92 LABEL_93 LABEL_94 LABEL_95 LABEL_96 LABEL_97 LABEL_98 LABEL_99
---
license: apache-2.0
# inference: false
# pipeline_tag: zero-shot-image-classification
pipeline_tag: feature-extraction

# inference:
#   parameters:
tags:
- clip
- zh
- image-text
- feature-extraction
---

# Model Details

This model is a Chinese CLIP model trained on [Noah-Wukong Dataset](https://wukong-dataset.github.io/wukong-dataset/), which contains about 100M Chinese image-text pairs. We use ViT-L-14 from [openAI](https://github.com/openai/CLIP) as image encoder and Chinese pre-trained language model  [chinese-roberta-wwm-large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) as text encoder. We freeze the image encoder and only finetune the text encoder. The model was trained for 10 epochs and it takes about 5 days with 16 A100 GPUs. **This is a beta version, We will continueously update this model**

# Taiyi (太乙)
Taiyi models are a branch of the Fengshenbang (封神榜) series of models. The models in Taiyi are pre-trained with multimodal pre-training strategies. We will release more image-text model trained on Chinese dataset and benefit the Chinese community.


# Usage

```python3
from PIL import Image
import requests
import clip
import torch
from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
from transformers import CLIPProcessor, CLIPModel
import numpy as np

query_texts = ["一只猫", "一只狗",'两只猫', '两只老虎','一只老虎']  # 这里是输入文本的，可以随意替换。
# 加载Taiyi 中文 text encoder
text_tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-large-326M-Chinese")
text_encoder = BertForSequenceClassification.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-large-326M-Chinese").eval()
text = text_tokenizer(query_texts, return_tensors='pt', padding=True)['input_ids']

url = "http://images.cocodataset.org/val2017/000000039769.jpg"  # 这里可以换成任意图片的url
# 加载CLIP的image encoder
clip_model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")  
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
image = processor(images=Image.open(requests.get(url, stream=True).raw), return_tensors="pt")

with torch.no_grad():
    image_features = clip_model.get_image_features(**image)
    text_features = text_encoder(text).logits
    # 归一化
    image_features = image_features / image_features.norm(dim=1, keepdim=True)
    text_features = text_features / text_features.norm(dim=1, keepdim=True)
    # 计算余弦相似度 logit_scale是尺度系数
    logit_scale = clip_model.logit_scale.exp()
    logits_per_image = logit_scale * image_features @ text_features.t()
    logits_per_text = logits_per_image.t()
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()
    print(np.around(probs, 3))

```

# Evaluation

### Zero-Shot Classification

|  model   | dataset  | Top1 | Top5 |
|  ----  | ----  | ---- | ---- |
| Taiyi-CLIP-Roberta-326M-Chinese  | ImageNet1k-CN | 51.72% | 78.46% |

### Zero-Shot Text-to-Image Retrieval

|  model   | dataset  | Top1 | Top5 | Top10 |
|  ----  | ----  | ---- | ---- | ---- |
| Taiyi-CLIP-Roberta-326M-Chinese  | Flickr30k-CNA-test | 51.08 % | 78.20 %  | 85.94 % |
| Taiyi-CLIP-Roberta-326M-Chinese  | COCO-CN-test | 52.40 % | 80.50 %  | 89.60 % |
| Taiyi-CLIP-Roberta-326M-Chinese  | wukong50k | 60.16 % | 90.36% | 95.61% |


# Citation

If you find the resource is useful, please cite the following website in your paper.

```
@misc{Fengshenbang-LM,
  title={Fengshenbang-LM},
  author={IDEA-CCNL},
  year={2022},
  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}
```