## Summary
There is a segmentation fault in CABAC_encoder_bitstream::append_byte and CABAC_encoder_bitstream::write_startcode if the call to realloc fails in check_size_and_resize

## Tested with:
* Commit: 6fc550faeca2bbb1c82de5f29d970f6861146862 (master)
* PoC: any YUV input file.
* cmd: `./enc265/enc265 -i <yuv file>`

## Analysis

When executing `CABAC_encoder_bitstream::check_size_and_resize`, there is no check to ensure the call to `realloc` is successful. 

In case it is not, it will return NULL that will be set as the value of `data_mem`. Later on, the variable is accessed in `CABAC_encoder_bitstream::append_byte`:

```c++
  if (byte<=3) {
    /**/ if (state< 2 && byte==0) { state++; }
    else if (state==2 && byte<=3) {
      data_mem[ data_size++ ] = 3;

      if (byte==0) state=1;
      else         state=0;
    }
    else { state=0; }
  }
  else { state=0; }


  // write actual data byte

  data_mem[ data_size++ ] = byte;
  ```

and `CABAC_encoder_bitstream::write_startcode`:

```c++
  data_mem[ data_size+0 ] = 0;
  data_mem[ data_size+1 ] = 0;
  data_mem[ data_size+2 ] = 1;
  data_size+=3;
  ```

resulting in a segmentation fault due to trying to access a NULL pointer.

## Impact

* Crash

## Patch

In order to prevent a crash, a check for `data_mem` not being NULL should be added that can handle the case appropriately.

Potential segmentation fault due to incorrect realloc in CABAC_encoder_bitstream::check_size_and_resize
This is a GitHub Issue
repo:libde265
owner:strukturag
Title : Potential segmentation fault due to incorrect realloc in CABAC_encoder_bitstream::check_size_and_resize
Issue date:

--- start body ---
## Summary
There is a segmentation fault in CABAC_encoder_bitstream::append_byte and CABAC_encoder_bitstream::write_startcode if the call to realloc fails in check_size_and_resize

## Tested with:
* Commit: 6fc550faeca2bbb1c82de5f29d970f6861146862 (master)
* PoC: any YUV input file.
* cmd: `./enc265/enc265 -i <yuv file>`

## Analysis

When executing `CABAC_encoder_bitstream::check_size_and_resize`, there is no check to ensure the call to `realloc` is successful. 

In case it is not, it will return NULL that will be set as the value of `data_mem`. Later on, the variable is accessed in `CABAC_encoder_bitstream::append_byte`:

```c++
  if (byte<=3) {
    /**/ if (state< 2 && byte==0) { state++; }
    else if (state==2 && byte<=3) {
      data_mem[ data_size++ ] = 3;

      if (byte==0) state=1;
      else         state=0;
    }
    else { state=0; }
  }
  else { state=0; }


  // write actual data byte

  data_mem[ data_size++ ] = byte;
  ```

and `CABAC_encoder_bitstream::write_startcode`:

```c++
  data_mem[ data_size+0 ] = 0;
  data_mem[ data_size+1 ] = 0;
  data_mem[ data_size+2 ] = 1;
  data_size+=3;
  ```

resulting in a segmentation fault due to trying to access a NULL pointer.

## Impact

* Crash

## Patch

In order to prevent a crash, a check for `data_mem` not being NULL should be added that can handle the case appropriately.

--- end body ---

A potential segmentation fault in the CABAC_encoder_bitstream component of libde265 occurs when realloc fails in the check_size_and_resize function. If realloc returns NULL, subsequent accesses to data_mem in append_byte and write_startcode lead to a crash due to dereferencing a NULL pointer. This vulnerability can be exploited by providing any YUV input file, resulting in a Denial of Service (DoS).