Edit model card

Please Note!

This model is NOT the 19.2M images Characters Model on TrinArt, but an improved version of the original Trin-sama Twitter bot model. This model is intended to retain the original SD's aesthetics as much as possible while nudging the model to anime/manga style.

Other TrinArt models can be found at:

https://huggingface.co/naclbit/trinart_derrida_characters_v2_stable_diffusion

https://huggingface.co/naclbit/trinart_characters_19.2m_stable_diffusion_v1

Diffusers

The model has been ported to diffusers by ayan4m1 and can easily be run from one of the branches:

  • revision="diffusers-60k" for the checkpoint trained on 60,000 steps,
  • revision="diffusers-95k" for the checkpoint trained on 95,000 steps,
  • revision="diffusers-115k" for the checkpoint trained on 115,000 steps.

For more information, please have a look at the "Three flavors" section.

Gradio

We also support a Gradio web ui with diffusers to run inside a colab notebook: Open In Colab

Example Text2Image

# !pip install diffusers==0.3.0
from diffusers import StableDiffusionPipeline

# using the 60,000 steps checkpoint
pipe = StableDiffusionPipeline.from_pretrained("naclbit/trinart_stable_diffusion_v2", revision="diffusers-60k")
pipe.to("cuda")

image = pipe("A magical dragon flying in front of the Himalaya in manga style").images[0]
image

dragon

If you want to run the pipeline faster or on a different hardware, please have a look at the optimization docs.

Example Image2Image

# !pip install diffusers==0.3.0
from diffusers import StableDiffusionImg2ImgPipeline
import requests
from PIL import Image
from io import BytesIO

url = "https://scitechdaily.com/images/Dog-Park.jpg"

response = requests.get(url)
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((768, 512))

# using the 115,000 steps checkpoint
pipe = StableDiffusionImg2ImgPipeline.from_pretrained("naclbit/trinart_stable_diffusion_v2", revision="diffusers-115k")
pipe.to("cuda")

images = pipe(prompt="Manga drawing of Brad Pitt", init_image=init_image, strength=0.75, guidance_scale=7.5).images
image

If you want to run the pipeline faster or on a different hardware, please have a look at the optimization docs.

Stable Diffusion TrinArt/Trin-sama AI finetune v2

trinart_stable_diffusion is a SD model finetuned by about 40,000 assorted high resolution manga/anime-style pictures for 8 epochs. This is the same model running on Twitter bot @trinsama (https://twitter.com/trinsama)

Twitterใƒœใƒƒใƒˆใ€Œใจใ‚Šใ‚“ใ•ใพAIใ€@trinsama (https://twitter.com/trinsama) ใงไฝฟ็”จใ—ใฆใ„ใ‚‹SDใฎใƒ•ใ‚กใ‚คใƒณใƒใƒฅใƒผใƒณๆธˆใƒขใƒ‡ใƒซใงใ™ใ€‚ไธ€ๅฎšใฎใƒซใƒผใƒซใง้ธๅˆฅใ•ใ‚ŒใŸ็ด„4ไธ‡ๆžšใฎใ‚ขใƒ‹ใƒกใƒปใƒžใƒณใ‚ฌใ‚นใ‚ฟใ‚คใƒซใฎ้ซ˜่งฃๅƒๅบฆ็”ปๅƒใ‚’็”จใ„ใฆ็ด„8ใ‚จใƒใƒƒใ‚ฏใฎ่จ“็ทดใ‚’่กŒใ„ใพใ—ใŸใ€‚

Version 2

V2 checkpoint uses dropouts, 10,000 more images and a new tagging strategy and trained longer to improve results while retaining the original aesthetics.

ใƒใƒผใ‚ธใƒงใƒณ2ใฏ็”ปๅƒใ‚’1ไธ‡ๆžš่ฟฝๅŠ ใ—ใŸใปใ‹ใ€ใƒ‰ใƒญใƒƒใƒ—ใ‚ขใ‚ฆใƒˆใฎ้ฉ็”จใ€ใ‚ฟใ‚ฐไป˜ใ‘ใฎๆ”นๅ–„ใจใ‚ˆใ‚Š้•ทใ„ใƒˆใƒฌใƒผใƒ‹ใƒณใ‚ฐๆ™‚้–“ใซใ‚ˆใ‚Šใ€SDใฎใ‚นใ‚ฟใ‚คใƒซใ‚’ไฟใฃใŸใพใพๅ‡บๅŠ›ๅ†…ๅฎนใฎๆ”นๅ–„ใ‚’็›ฎๆŒ‡ใ—ใฆใ„ใพใ™ใ€‚

Three flavors

Step 115000/95000 checkpoints were trained further, but you may use step 60000 checkpoint instead if style nudging is too much.

ใ‚นใƒ†ใƒƒใƒ—115000/95000ใฎใƒใ‚งใƒƒใ‚ฏใƒใ‚คใƒณใƒˆใงใ‚นใ‚ฟใ‚คใƒซใŒๅค‰ใ‚ใ‚Šใ™ใŽใ‚‹ใจๆ„Ÿใ˜ใ‚‹ๅ ดๅˆใฏใ€ใ‚นใƒ†ใƒƒใƒ—60000ใฎใƒใ‚งใƒƒใ‚ฏใƒใ‚คใƒณใƒˆใ‚’ไฝฟ็”จใ—ใฆใฟใฆใใ ใ•ใ„ใ€‚

img2img

If you want to run latent-diffusion's stock ddim img2img script with this model, use_ema must be set to False.

latent-diffusion ใฎscriptsใƒ•ใ‚ฉใƒซใƒ€ใซๅ…ฅใฃใฆใ„ใ‚‹ddim img2imgใ‚’ใ“ใฎใƒขใƒ‡ใƒซใงๅ‹•ใ‹ใ™ๅ ดๅˆใ€use_emaใฏFalseใซใ™ใ‚‹ๅฟ…่ฆใŒใ‚ใ‚Šใพใ™ใ€‚

Hardware

  • 8xNVIDIA A100 40GB

Training Info

  • Custom dataset loader with augmentations: XFlip, center crop and aspect-ratio locked scaling
  • LR: 1.0e-5
  • 10% dropouts

Examples

Each images were diffused using K. Crowson's k-lms (from k-diffusion repo) method for 50 steps.

examples examples examples

Credits

  • Sta, AI Novelist Dev (https://ai-novel.com/) @ Bit192, Inc.
  • Stable Diffusion - Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bjorn

License

CreativeML OpenRAIL-M

Downloads last month
1,611
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Spaces using naclbit/trinart_stable_diffusion_v2 100