codingutf8 2023 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license image processor class for fuyu import math from typing import dict list optional union import numpy as np from imageprocessingutils import baseimageprocessor batchfeature from imagetransforms import pad resize tochanneldimensionformat from imageutils import channeldimension imageinput pilimageresampling getimagesize inferchanneldimensionformat isscaledimage isvalidimage makelistofimages tonumpyarray from utils import tensortype istorchavailable istorchdevice istorchdtype logging requiresbackends if istorchavailable import torch logger logging getloggername def makelistoflistofimages images unionlistlistimageinput listimageinput imageinput listlistimageinput if isvalidimageimages return images if isinstanceimages list and allisinstanceimage list for image in images return images if isinstanceimages list return makelistofimagesimage for image in images raise valueerrorimages must be a list of list of images or a list of images or an image class fuyubatchfeaturebatchfeature def converttotensorsself tensortype optionalunionstr tensortype none if tensortype is none return self istensor astensor self getisastensorfnstensortypetensortype def converttensorelem if istensorelem return elem return astensorelem def safeconverttensorelem try return converttensorelem except noqa e722 if key overflowingvalues raise valueerrorunable to create tensor returning overflowing values of different lengths raise valueerror unable to create tensor you should probably activate padding with paddingtrue to have batched tensors with the same length do the tensor conversion in batch for key value in self items if isinstancevalue list and isinstancevalue0 list listlistany listlisttensor selfkey safeconverttensorelem for elem in elems for elems in value elif isinstancevalue list listany listtensor selfkey safeconverttensorelem for elem in value else any tensor selfkey safeconverttensorvalue return self def toself args kwargs batchfeature requiresbackendsself torch import torch noqa newdata device kwargs getdevice check if the args are a device or a dtype if device is none and lenargs 0 device should be always the first argument arg args0 if istorchdtypearg the first argument is a dtype pass elif isinstancearg str or istorchdevicearg or isinstancearg int device arg else it s something else raise valueerrorfattempting to cast a batchfeature to type strarg this is not supported def toelem check if v is a floating point if torch isfloatingpointelem cast and send to device return elem toargs kwargs if device is not none return elem todevicedevice return elem we cast only floating point tensors to avoid issues with tokenizers casting longtensor to floattensor for k v in self items if isinstancev list and isinstancev0 list data structure is a list of lists newv for elems in v newv appendtoelem for elem in elems newdatak newv elif isinstancev list data structure is a list newdatak toelem for elem in v else newdatak tov self data newdata return self class fuyuimageprocessorbaseimageprocessor modelinputnames images imageinputids imagepatches imagepatchindicesperbatch imagepatchindicespersubsequence def init self doresize bool true size optionaldictstr int none resample pilimageresampling pilimageresampling bilinear dopad bool true paddingvalue float 1 0 paddingmode str constant donormalize bool true imagemean unionfloat listfloat 0 5 imagestd unionfloat listfloat 0 5 dorescale bool true rescalefactor float 1 255 patchsize optionaldictstr int none kwargs super initkwargs self doresize doresize self size size if size is not none else height 1080 width 1920 self resample resample self dopad dopad self paddingvalue paddingvalue self paddingmode paddingmode self donormalize donormalize self imagemean imagemean self imagestd imagestd self dorescale dorescale self rescalefactor rescalefactor self patchsize patchsize if patchsize is not none else height 30 width 30 def resize self image np ndarray size dictstr int resample pilimageresampling pilimageresampling bilinear dataformat optionalunionstr channeldimension none inputdataformat optionalunionstr channeldimension none kwargs np ndarray imageheight imagewidth getimagesizeimage inputdataformat targetheight targetwidth sizeheight sizewidth if imagewidth targetwidth and imageheight targetheight return image heightscalefactor targetheight imageheight widthscalefactor targetwidth imagewidth optimalscalefactor minheightscalefactor widthscalefactor newheight intimageheight optimalscalefactor newwidth intimagewidth optimalscalefactor scaledimage resize imageimage sizenewheight newwidth resampleresample dataformatdataformat inputdataformatinputdataformat kwargs return scaledimage def padimage self image np ndarray size dictstr int mode str constant constantvalues float 1 0 dataformat optionalunionstr channeldimension none inputdataformat optionalunionstr channeldimension none np ndarray imageheight imagewidth getimagesizeimage inputdataformat targetheight targetwidth sizeheight sizewidth paddingtop 0 paddingleft 0 paddingbottom targetheight imageheight paddingright targetwidth imagewidth paddedimage pad image paddingpaddingtop paddingbottom paddingleft paddingright modemode constantvaluesconstantvalues dataformatdataformat inputdataformatinputdataformat return paddedimage def preprocess self images doresize optionalbool none size optionaldictstr int none resample optionalpilimageresampling none dopad optionalbool none paddingvalue optionalfloat none paddingmode optionalstr none donormalize optionalbool none imagemean optionalfloat none imagestd optionalfloat none dorescale optionalbool none rescalefactor optionalfloat none patchsize optionaldictstr int none dataformat optionalunionstr channeldimension channeldimension first inputdataformat optionalunionstr channeldimension none returntensors optionaltensortype none doresize doresize if doresize is not none else self doresize size size if size is not none else self size resample resample if resample is not none else self resample dopad dopad if dopad is not none else self dopad dorescale dorescale if dorescale is not none else self dorescale rescalefactor rescalefactor if rescalefactor is not none else self rescalefactor donormalize donormalize if donormalize is not none else self donormalize imagemean imagemean if imagemean is not none else self imagemean imagestd imagestd if imagestd is not none else self imagestd paddingvalue paddingvalue if paddingvalue is not none else self paddingvalue paddingmode paddingmode if paddingmode is not none else self paddingmode dorescale dorescale if dorescale is not none else self dorescale rescalefactor rescalefactor if rescalefactor is not none else self rescalefactor patchsize patchsize if patchsize is not none else self patchsize if isinstanceimages list and anyisinstanceelem list and lenelem 2 for elem in images raise valueerrormultiple images for a single sample are not yet supported batchimages makelistoflistofimagesimages if doresize and size is none raise valueerrorsize must be specified if doresize is true if dorescale and rescalefactor is none raise valueerrorrescale factor must be specified if dorescale is true if donormalize and imagemean is none or imagestd is none raise valueerrorimagemean and imagestd must be specified if donormalize is true all transformations expect numpy arrays batchimages tonumpyarrayimage for image in images for images in batchimages if isscaledimagebatchimages00 and dorescale logger warningonce it looks like you are trying to rescale already rescaled images if the input images have pixel values between 0 and 1 set dorescalefalse to avoid rescaling them again if inputdataformat is none we assume that all images have the same channel dimension format inputdataformat inferchanneldimensionformatbatchimages00 originalimagesizes getimagesizeimages0 channeldiminputdataformat for images in batchimages if doresize batchimages self resizeimage sizesize inputdataformatinputdataformat for image in images for images in batchimages imagesizes getimagesizeimages0 channeldiminputdataformat for images in batchimages imageunpaddedheights imagesize0 for imagesize in imagesizes imageunpaddedwidths imagesize1 for imagesize in imagesizes scaleh is the same as scalew imagescalefactors resizedsize0 originalsize0 for originalsize resizedsize in ziporiginalimagesizes imagesizes if dopad batchimages self padimage image sizesize modepaddingmode constantvaluespaddingvalue inputdataformatinputdataformat for image in images for images in batchimages if dorescale batchimages self rescaleimage scalerescalefactor inputdataformatinputdataformat for image in images for images in batchimages if donormalize batchimages self normalizeimage meanimagemean stdimagestd inputdataformatinputdataformat for image in images for images in batchimages if dataformat is not none batchimages tochanneldimensionformatimage dataformat inputdataformat for image in images for images in batchimages data images batchimages imageunpaddedheights imageunpaddedheights imageunpaddedwidths imageunpaddedwidths imagescalefactors imagescalefactors return fuyubatchfeaturedatadata tensortypereturntensors def getnumpatchesself imageheight int imagewidth int patchsize dictstr int none int patchsize patchsize if patchsize is not none else self patchsize patchheight patchwidth self patchsizeheight self patchsizewidth if imageheight patchheight 0 raise valueerrorfimageheight must be divisible by patchheight if imagewidth patchwidth 0 raise valueerrorfimagewidth must be divisible by patchwidth numpatchesperdimh imageheight patchheight numpatchesperdimw imagewidth patchwidth numpatches numpatchesperdimh numpatchesperdimw return numpatches def patchifyimageself image torch tensor patchsize optionaldictstr int none torch tensor requiresbackendsself torch patchsize patchsize if patchsize is not none else self patchsize patchheight patchwidth patchsizeheight patchsizewidth todo refer to https github comarthurzuckertransformersblob0f0a3fe5ca5697ee58faeb5b53f049af720b5e98srctransformersmodelsvitmaemodelingvitmae pyl871 torch implementation is faster but does not handle nonsquares batchsize channels image shape unfoldedalongheight image unfold2 patchheight patchheight patches unfoldedalongheight unfold3 patchwidth patchwidth patches patches contiguous patches patches viewbatchsize channels 1 patchheight patchwidth patches patches permute0 2 3 4 1 patches patches reshapebatchsize 1 channels patchheight patchwidth return patches def preprocesswithtokenizerinfo self imageinput torch tensor imagepresent torch tensor imageunpaddedh torch tensor imageunpaddedw torch tensor imageplaceholderid int imagenewlineid int variablesized bool patchsize optionaldictstr int none fuyubatchfeature requiresbackendsself torch patchsize patchsize if patchsize is not none else self patchsize patchheight patchwidth patchsizeheight patchsizewidth only images that are present images listlisttorch tensor batchimagepatches listlisttorch tensor image input ids for every subsequence including ones with no image present batchimageinputids listlisttorch tensor for batchindex in rangeimageinput shape0 imageinputids imagepatches for subseqindex in rangeimageinput shape1 if imagepresentbatchindex subseqindex image imageinputbatchindex subseqindex imageheight imagewidth image shape1 image shape2 if variablesized the min is required here due to floating point issues math ceiltorch tensor300 cuda 30 11 newh min imageheight math ceilimageunpaddedhbatchindex subseqindex patchheight patchheight neww min imagewidth math ceilimageunpaddedwbatchindex subseqindex patchwidth patchwidth image image newh neww imageheight imagewidth newh neww numpatches self getnumpatchesimageheightimageheight imagewidthimagewidth tensorofimageids torch full numpatches imageplaceholderid dtypetorch int32 deviceimageinput device patches self patchifyimageimageimage unsqueeze0 squeeze0 assert numpatches patches shape0 if variablesized now terminate each line with newline tensorofimageids tensorofimageids reshape1 imagewidth patchwidth newlineids torch full tensorofimageids shape0 1 imagenewlineid dtypetorch int32 deviceimageinput device tensorofimageids torch cattensorofimageids newlineids dim1 tensorofimageids tensorofimageids reshape1 images appendimage imageinputids appendtensorofimageids imagepatches appendpatches else imageinputids appendtorch tensor dtypetorch int32 deviceimageinput device batchimageinputids appendimageinputids batchimagepatches appendimagepatches create imagepatchinputindices where nonnegative values correspond to image patches to be inserted in the stream imagepatchindicesperbatch listlisttorch tensor imagepatchindicespersubsequence listlisttorch tensor for sampleimageinputids in batchimageinputids indexoffset 0 perbatchindices persubsequenceindices for subseqimageinputids in sampleimageinputids indices of image patches patchesmask subseqimageinputids imageplaceholderid numpatches torch countnonzeropatchesmask indices torch arange numpatches dtypesubseqimageinputids dtype devicesubseqimageinputids device place those indices in the image input ids token stream with 1 representing nonindex tokens indicesinstreamperbatch torch fulllikesubseqimageinputids 1 indicesinstreampersubsequence torch fulllikesubseqimageinputids 1 patchesinds torch nonzeropatchesmask astupletrue0 indicesinstreamperbatchpatchesinds indices indexoffset indicesinstreampersubsequencepatchesinds indices perbatchindices appendindicesinstreamperbatch persubsequenceindices appendindicesinstreampersubsequence indexoffset numpatches imagepatchindicesperbatch appendperbatchindices imagepatchindicespersubsequence appendpersubsequenceindices return fuyubatchfeature data images images imageinputids batchimageinputids imagepatches batchimagepatches imagepatchindicesperbatch imagepatchindicesperbatch imagepatchindicespersubsequence imagepatchindicespersubsequence coding utf 8 2023 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license image processor class for fuyu batchfeature class for fuyu image processor and processor the outputs dictionary from the processors contains a mix of tensors and lists of tensors convert the inner content to tensors args tensor_type str or utils tensortype optional the type of tensors to use if str should be one of the values of the enum utils tensortype if none no modification is done noqa e722 do the tensor conversion in batch list list any list list tensor list any list tensor any tensor send all values to device by calling v to args kwargs pytorch only this should support casting in different dtypes and sending the batchfeature to a different device args args tuple will be passed to the to function of the tensors kwargs dict optional will be passed to the to function of the tensors returns batchfeature the same instance after modification noqa check if the args are a device or a dtype device should be always the first argument the first argument is a dtype it s something else check if v is a floating point cast and send to device we cast only floating point tensors to avoid issues with tokenizers casting longtensor to floattensor data structure is a list of lists data structure is a list this class should handle the image processing part before the main fuyuforcausallm in particular it should handle processing images taking a batch of images as input if the images are variable sized it resizes them based on the desired patch dimensions the image output is always img_h img_w of 1080 1920 then it patches up these images using the patchify_image function creating image input ids for each patch a placeholder id is given to identify where these patches belong in a token sequence for variable sized images each line of patches is terminated with a newline id image patch indices for each image patch the code maintains an index where these patches should be inserted in a token stream args do_resize bool optional defaults to true whether to resize the image to size size dict str int optional defaults to height 1080 width 1920 dictionary in the format height int width int specifying the size of the output image resample pilimageresampling optional defaults to resampling bilinear pilimageresampling filter to use when resizing the image e g pilimageresampling bilinear do_pad bool optional defaults to true whether to pad the image to size padding_value float optional defaults to 1 0 the value to pad the image with padding_mode str optional defaults to constant the padding mode to use when padding the image do_normalize bool optional defaults to true whether to normalize the image image_mean float optional defaults to 0 5 the mean to use when normalizing the image image_std float optional defaults to 0 5 the standard deviation to use when normalizing the image do_rescale bool optional defaults to true whether to rescale the image rescale_factor float optional defaults to 1 255 the factor to use when rescaling the image patch_size dict str int optional defaults to height 30 width 30 dictionary in the format height int width int specifying the size of the patches resize an image to size height size width args image np ndarray image to resize size dict str int dictionary in the format height int width int specifying the size of the output image resample pilimageresampling optional defaults to pilimageresampling bilinear pilimageresampling filter to use when resizing the image e g pilimageresampling bilinear data_format channeldimension or str optional the channel dimension format for the output image if unset the channel dimension format of the input image is used can be one of channels_first or channeldimension first image in num_channels height width format channels_last or channeldimension last image in height width num_channels format none or channeldimension none image in height width format input_data_format channeldimension or str optional the channel dimension format for the input image if unset the channel dimension format is inferred from the input image can be one of channels_first or channeldimension first image in num_channels height width format channels_last or channeldimension last image in height width num_channels format none or channeldimension none image in height width format returns np ndarray the resized image pad an image to size height size width args image np ndarray image to pad size dict str int dictionary in the format height int width int specifying the size of the output image data_format channeldimension or str optional the data format of the output image if unset the same format as the input image is used input_data_format channeldimension or str optional the channel dimension format of the input image if not provided it will be inferred utility function to preprocess the images and extract necessary information about original formats args images imageinput images to preprocess expects a single image a list or images or a list of lists of images pixel values range from 0 to 255 or between 0 and 1 if do_rescale is false do_resize bool optional defaults to self do_resize whether to resize the image to size size dict str int optional defaults to self size dictionary in the format height int width int specifying the size of the output image resample pilimageresampling optional defaults to self resample pilimageresampling filter to use when resizing the image e g pilimageresampling bilinear do_pad bool optional defaults to self do_pad whether to pad the image to size padding_value float optional defaults to self padding_value the value to pad the image with padding_mode str optional defaults to self padding_mode the padding mode to use when padding the image do_normalize bool optional defaults to self do_normalize whether to normalize the image image_mean float optional defaults to self image_mean the mean to use when normalizing the image image_std float optional defaults to self image_std the standard deviation to use when normalizing the image do_rescale bool optional defaults to self do_rescale whether to rescale the image rescale_factor float optional defaults to self rescale_factor the factor to use when rescaling the image patch_size dict str int optional defaults to self patch_size dictionary in the format height int width int specifying the size of the patches return_tensors str or tensortype optional the type of tensors to return can be one of unset return a list of np ndarray tensortype tensorflow or tf return a batch of type tf tensor tensortype pytorch or pt return a batch of type torch tensor tensortype numpy or np return a batch of type np ndarray tensortype jax or jax return a batch of type jax numpy ndarray data_format channeldimension or str optional defaults to channeldimension first the channel dimension format of the output image can be one of channels_first or channeldimension first image in num_channels height width format channels_last or channeldimension last image in height width num_channels format input_data_format channeldimension or str optional the channel dimension format for the input image if unset the channel dimension format is inferred from the input image can be one of channels_first or channeldimension first image in num_channels height width format channels_last or channeldimension last image in height width num_channels format all transformations expect numpy arrays we assume that all images have the same channel dimension format scale_h is the same as scale_w calculate number of patches required to encode an image args image_height int height of the image image_width int width of the image patch_size dict str int optional defaults to self patch_size dictionary in the format height int width int specifying the size of the patches convert an image into a tensor of patches args image torch tensor image to convert shape batch channels height width patch_size dict str int optional defaults to self patch_size dictionary in the format height int width int specifying the size of the patches todo refer to https github com arthurzucker transformers blob 0f0a3fe5ca5697ee58faeb5b53f049af720b5e98 src transformers models vit_mae modeling_vit_mae py l871 torch implementation is faster but does not handle non squares process images for model input in particular variable sized images are handled here args image_input torch tensor of shape batch_size subsequence_size num_channels height width tensor of images padded to model input size image_present torch tensor of shape batch_size subsequence_size num_images tensor of 1s and 0s indicating whether an image is present image_unpadded_h torch tensor of shape batch_size subsequence_size tensor of unpadded image heights image_unpadded_w torch tensor of shape batch_size subsequence_size tensor of unpadded image widths image_placeholder_id int the id of the image placeholder token comes from an associated tokenizer image_newline_id int the id of the image newline token comes from an associated tokenizer variable_sized bool whether to process images as variable sized patch_size dict str int optional defaults to self patch_size size of the patches only images that are present image input ids for every subsequence including ones with no image present the min is required here due to floating point issues math ceil torch tensor 300 cuda 30 11 now terminate each line with newline create image_patch_input_indices where non negative values correspond to image patches to be inserted in the stream indices of image patches place those indices in the image input ids token stream with 1 representing non index tokens
import math
from typing import Dict, List, Optional, Union

import numpy as np

from ...image_processing_utils import BaseImageProcessor, BatchFeature
from ...image_transforms import (
    pad,
    resize,
    to_channel_dimension_format,
)
from ...image_utils import (
    ChannelDimension,
    ImageInput,
    PILImageResampling,
    get_image_size,
    infer_channel_dimension_format,
    is_scaled_image,
    is_valid_image,
    make_list_of_images,
    to_numpy_array,
)
from ...utils import (
    TensorType,
    is_torch_available,
    is_torch_device,
    is_torch_dtype,
    logging,
    requires_backends,
)


if is_torch_available():
    import torch


logger = logging.get_logger(__name__)


def make_list_of_list_of_images(
    images: Union[List[List[ImageInput]], List[ImageInput], ImageInput]
) -> List[List[ImageInput]]:
    if is_valid_image(images):
        return [[images]]

    if isinstance(images, list) and all(isinstance(image, list) for image in images):
        return images

    if isinstance(images, list):
        return [make_list_of_images(image) for image in images]

    raise ValueError("images must be a list of list of images or a list of images or an image.")


class FuyuBatchFeature(BatchFeature):
    

    def convert_to_tensors(self, tensor_type: Optional[Union[str, TensorType]] = None):
        
        if tensor_type is None:
            return self

        is_tensor, as_tensor = self._get_is_as_tensor_fns(tensor_type=tensor_type)

        def _convert_tensor(elem):
            if is_tensor(elem):
                return elem
            return as_tensor(elem)

        def _safe_convert_tensor(elem):
            try:
                return _convert_tensor(elem)
            except:  
                if key == "overflowing_values":
                    raise ValueError("Unable to create tensor returning overflowing values of different lengths. ")
                raise ValueError(
                    "Unable to create tensor, you should probably activate padding "
                    "with 'padding=True' to have batched tensors with the same length."
                )

        
        for key, value in self.items():
            if isinstance(value, list) and isinstance(value[0], list):
                
                self[key] = [[_safe_convert_tensor(elem) for elem in elems] for elems in value]
            elif isinstance(value, list):
                
                self[key] = [_safe_convert_tensor(elem) for elem in value]
            else:
                
                self[key] = _safe_convert_tensor(value)
        return self

    def to(self, *args, **kwargs) -> "BatchFeature":
        
        requires_backends(self, ["torch"])
        import torch  

        new_data = {}
        device = kwargs.get("device")
        
        if device is None and len(args) > 0:
            
            arg = args[0]
            if is_torch_dtype(arg):
                
                pass
            elif isinstance(arg, str) or is_torch_device(arg) or isinstance(arg, int):
                device = arg
            else:
                
                raise ValueError(f"Attempting to cast a BatchFeature to type {str(arg)}. This is not supported.")

        def _to(elem):
            
            if torch.is_floating_point(elem):
                
                return elem.to(*args, **kwargs)
            if device is not None:
                return elem.to(device=device)

            return elem

        
        for k, v in self.items():
            if isinstance(v, list) and isinstance(v[0], list):
                
                new_v = []
                for elems in v:
                    new_v.append([_to(elem) for elem in elems])
                new_data[k] = new_v
            elif isinstance(v, list):
                
                new_data[k] = [_to(elem) for elem in v]
            else:
                new_data[k] = _to(v)
        self.data = new_data
        return self


class FuyuImageProcessor(BaseImageProcessor):
    

    model_input_names = [
        "images",
        "image_input_ids",
        "image_patches",
        "image_patch_indices_per_batch",
        "image_patch_indices_per_subsequence",
    ]

    def __init__(
        self,
        do_resize: bool = True,
        size: Optional[Dict[str, int]] = None,
        resample: PILImageResampling = PILImageResampling.BILINEAR,
        do_pad: bool = True,
        padding_value: float = 1.0,
        padding_mode: str = "constant",
        do_normalize: bool = True,
        image_mean: Union[float, List[float]] = 0.5,
        image_std: Union[float, List[float]] = 0.5,
        do_rescale: bool = True,
        rescale_factor: float = 1 / 255,
        patch_size: Optional[Dict[str, int]] = None,
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.do_resize = do_resize
        self.size = size if size is not None else {"height": 1080, "width": 1920}
        self.resample = resample
        self.do_pad = do_pad
        self.padding_value = padding_value
        self.padding_mode = padding_mode
        self.do_normalize = do_normalize
        self.image_mean = image_mean
        self.image_std = image_std
        self.do_rescale = do_rescale
        self.rescale_factor = rescale_factor
        self.patch_size = patch_size if patch_size is not None else {"height": 30, "width": 30}

    def resize(
        self,
        image: np.ndarray,
        size: Dict[str, int],
        resample: PILImageResampling = PILImageResampling.BILINEAR,
        data_format: Optional[Union[str, ChannelDimension]] = None,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
        **kwargs,
    ) -> np.ndarray:
        
        image_height, image_width = get_image_size(image, input_data_format)
        target_height, target_width = size["height"], size["width"]

        if image_width <= target_width and image_height <= target_height:
            return image

        height_scale_factor = target_height / image_height
        width_scale_factor = target_width / image_width
        optimal_scale_factor = min(height_scale_factor, width_scale_factor)

        new_height = int(image_height * optimal_scale_factor)
        new_width = int(image_width * optimal_scale_factor)

        scaled_image = resize(
            image=image,
            size=(new_height, new_width),
            resample=resample,
            data_format=data_format,
            input_data_format=input_data_format,
            **kwargs,
        )
        return scaled_image

    def pad_image(
        self,
        image: np.ndarray,
        size: Dict[str, int],
        mode: str = "constant",
        constant_values: float = 1.0,
        data_format: Optional[Union[str, ChannelDimension]] = None,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
    ) -> np.ndarray:
        
        image_height, image_width = get_image_size(image, input_data_format)
        target_height, target_width = size["height"], size["width"]
        padding_top = 0
        padding_left = 0
        padding_bottom = target_height - image_height
        padding_right = target_width - image_width
        padded_image = pad(
            image,
            padding=((padding_top, padding_bottom), (padding_left, padding_right)),
            mode=mode,
            constant_values=constant_values,
            data_format=data_format,
            input_data_format=input_data_format,
        )
        return padded_image

    def preprocess(
        self,
        images,
        do_resize: Optional[bool] = None,
        size: Optional[Dict[str, int]] = None,
        resample: Optional[PILImageResampling] = None,
        do_pad: Optional[bool] = None,
        padding_value: Optional[float] = None,
        padding_mode: Optional[str] = None,
        do_normalize: Optional[bool] = None,
        image_mean: Optional[float] = None,
        image_std: Optional[float] = None,
        do_rescale: Optional[bool] = None,
        rescale_factor: Optional[float] = None,
        patch_size: Optional[Dict[str, int]] = None,
        data_format: Optional[Union[str, ChannelDimension]] = ChannelDimension.FIRST,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
        return_tensors: Optional[TensorType] = None,
    ):
        

        do_resize = do_resize if do_resize is not None else self.do_resize
        size = size if size is not None else self.size
        resample = resample if resample is not None else self.resample
        do_pad = do_pad if do_pad is not None else self.do_pad
        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        padding_value = padding_value if padding_value is not None else self.padding_value
        padding_mode = padding_mode if padding_mode is not None else self.padding_mode
        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
        patch_size = patch_size if patch_size is not None else self.patch_size

        if isinstance(images, list) and any(isinstance(elem, list) and len(elem) >= 2 for elem in images):
            raise ValueError("Multiple images for a single sample are not yet supported.")

        batch_images = make_list_of_list_of_images(images)

        if do_resize and size is None:
            raise ValueError("Size must be specified if do_resize is True.")

        if do_rescale and rescale_factor is None:
            raise ValueError("Rescale factor must be specified if do_rescale is True.")

        if do_normalize and image_mean is None or image_std is None:
            raise ValueError("image_mean and image_std must be specified if do_normalize is True.")

        
        batch_images = [[to_numpy_array(image) for image in images] for images in batch_images]

        if is_scaled_image(batch_images[0][0]) and do_rescale:
            logger.warning_once(
                "It looks like you are trying to rescale already rescaled images. If the input"
                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
            )

        if input_data_format is None:
            
            input_data_format = infer_channel_dimension_format(batch_images[0][0])

        original_image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images]

        if do_resize:
            batch_images = [
                [self.resize(image, size=size, input_data_format=input_data_format) for image in images]
                for images in batch_images
            ]

        image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images]
        image_unpadded_heights = [[image_size[0]] for image_size in image_sizes]
        image_unpadded_widths = [[image_size[1]] for image_size in image_sizes]

        
        image_scale_factors = [
            [resized_size[0] / original_size[0]]
            for original_size, resized_size in zip(original_image_sizes, image_sizes)
        ]

        if do_pad:
            batch_images = [
                [
                    self.pad_image(
                        image,
                        size=size,
                        mode=padding_mode,
                        constant_values=padding_value,
                        input_data_format=input_data_format,
                    )
                    for image in images
                ]
                for images in batch_images
            ]

        if do_rescale:
            batch_images = [
                [self.rescale(image, scale=rescale_factor, input_data_format=input_data_format) for image in images]
                for images in batch_images
            ]

        if do_normalize:
            batch_images = [
                [
                    self.normalize(image, mean=image_mean, std=image_std, input_data_format=input_data_format)
                    for image in images
                ]
                for images in batch_images
            ]

        if data_format is not None:
            batch_images = [
                [to_channel_dimension_format(image, data_format, input_data_format) for image in images]
                for images in batch_images
            ]

        data = {
            "images": batch_images,
            "image_unpadded_heights": image_unpadded_heights,
            "image_unpadded_widths": image_unpadded_widths,
            "image_scale_factors": image_scale_factors,
        }
        return FuyuBatchFeature(data=data, tensor_type=return_tensors)

    def get_num_patches(self, image_height: int, image_width: int, patch_size: Dict[str, int] = None) -> int:
        
        patch_size = patch_size if patch_size is not None else self.patch_size
        patch_height, patch_width = self.patch_size["height"], self.patch_size["width"]

        if image_height % patch_height != 0:
            raise ValueError(f"{image_height=} must be divisible by {patch_height}")
        if image_width % patch_width != 0:
            raise ValueError(f"{image_width=} must be divisible by {patch_width}")

        num_patches_per_dim_h = image_height // patch_height
        num_patches_per_dim_w = image_width // patch_width
        num_patches = num_patches_per_dim_h * num_patches_per_dim_w
        return num_patches

    def patchify_image(self, image: "torch.Tensor", patch_size: Optional[Dict[str, int]] = None) -> "torch.Tensor":
        
        requires_backends(self, ["torch"])
        patch_size = patch_size if patch_size is not None else self.patch_size
        patch_height, patch_width = patch_size["height"], patch_size["width"]

        
        

        batch_size, channels, _, _ = image.shape
        unfolded_along_height = image.unfold(2, patch_height, patch_height)
        patches = unfolded_along_height.unfold(3, patch_width, patch_width)
        patches = patches.contiguous()
        patches = patches.view(batch_size, channels, -1, patch_height, patch_width)
        patches = patches.permute(0, 2, 3, 4, 1)
        patches = patches.reshape(batch_size, -1, channels * patch_height * patch_width)
        return patches

    def preprocess_with_tokenizer_info(
        self,
        image_input: "torch.Tensor",
        image_present: "torch.Tensor",
        image_unpadded_h: "torch.Tensor",
        image_unpadded_w: "torch.Tensor",
        image_placeholder_id: int,
        image_newline_id: int,
        variable_sized: bool,
        patch_size: Optional[Dict[str, int]] = None,
    ) -> FuyuBatchFeature:
        
        requires_backends(self, ["torch"])

        patch_size = patch_size if patch_size is not None else self.patch_size
        patch_height, patch_width = patch_size["height"], patch_size["width"]

        
        images: List[List[torch.Tensor]] = []
        batch_image_patches: List[List[torch.Tensor]] = []
        
        batch_image_input_ids: List[List[torch.Tensor]] = []
        for batch_index in range(image_input.shape[0]):
            image_input_ids = []
            image_patches = []
            for subseq_index in range(image_input.shape[1]):
                if image_present[batch_index, subseq_index]:
                    image = image_input[batch_index, subseq_index]
                    image_height, image_width = image.shape[1], image.shape[2]
                    if variable_sized:
                        
                        
                        new_h = min(
                            image_height,
                            math.ceil(image_unpadded_h[batch_index, subseq_index] / patch_height) * patch_height,
                        )
                        new_w = min(
                            image_width,
                            math.ceil(image_unpadded_w[batch_index, subseq_index] / patch_width) * patch_width,
                        )
                        image = image[:, :new_h, :new_w]
                        image_height, image_width = new_h, new_w

                    num_patches = self.get_num_patches(image_height=image_height, image_width=image_width)
                    tensor_of_image_ids = torch.full(
                        [num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device
                    )
                    patches = self.patchify_image(image=image.unsqueeze(0)).squeeze(0)
                    assert num_patches == patches.shape[0]

                    if variable_sized:
                        
                        tensor_of_image_ids = tensor_of_image_ids.reshape(-1, image_width // patch_width)
                        newline_ids = torch.full(
                            [tensor_of_image_ids.shape[0], 1],
                            image_newline_id,
                            dtype=torch.int32,
                            device=image_input.device,
                        )
                        tensor_of_image_ids = torch.cat([tensor_of_image_ids, newline_ids], dim=1)
                        tensor_of_image_ids = tensor_of_image_ids.reshape(-1)

                    images.append([image])
                    image_input_ids.append(tensor_of_image_ids)
                    image_patches.append(patches)
                else:
                    image_input_ids.append(torch.tensor([], dtype=torch.int32, device=image_input.device))

            batch_image_input_ids.append(image_input_ids)
            batch_image_patches.append(image_patches)

        
        
        image_patch_indices_per_batch: List[List[torch.Tensor]] = []
        image_patch_indices_per_subsequence: List[List[torch.Tensor]] = []

        for sample_image_input_ids in batch_image_input_ids:
            index_offset = 0
            per_batch_indices = []
            per_subsequence_indices = []
            for subseq_image_input_ids in sample_image_input_ids:
                
                patches_mask = subseq_image_input_ids == image_placeholder_id
                num_patches = torch.count_nonzero(patches_mask)
                indices = torch.arange(
                    num_patches, dtype=subseq_image_input_ids.dtype, device=subseq_image_input_ids.device
                )

                
                indices_in_stream_per_batch = torch.full_like(subseq_image_input_ids, -1)
                indices_in_stream_per_subsequence = torch.full_like(subseq_image_input_ids, -1)
                patches_inds = torch.nonzero(patches_mask, as_tuple=True)[0]

                indices_in_stream_per_batch[patches_inds] = indices + index_offset
                indices_in_stream_per_subsequence[patches_inds] = indices

                per_batch_indices.append(indices_in_stream_per_batch)
                per_subsequence_indices.append(indices_in_stream_per_subsequence)
                index_offset += num_patches

            image_patch_indices_per_batch.append(per_batch_indices)
            image_patch_indices_per_subsequence.append(per_subsequence_indices)

        return FuyuBatchFeature(
            data={
                "images": images,
                "image_input_ids": batch_image_input_ids,
                "image_patches": batch_image_patches,
                "image_patch_indices_per_batch": image_patch_indices_per_batch,
                "image_patch_indices_per_subsequence": image_patch_indices_per_subsequence,
            }
        )