Q:

I2c communication between ATMega328P and ESP8266ex can only send 8 bytes, bug?

I am developing an I2C bridge that enables me to virtual extend the pins available on the ESP8266ex in an 'intelligent' manner. The ESP8266ex is nice however has less usable pins. 
It looks like the extended pins are integrated into the ESP8266ex itself. Actually it is not because the bridge communicates via i2c in background to access the pins on the ATMega328P, however I can use the standard methods and functions such as pinMode, digitalRead/write and analogRead/Write. Because of this, the ESP8266ex is full in control to control the pins on ATMega328P.

 I2C MASTER/SLAVE CONFIG            GND
                                     o 
 ------------------------            |                       ------------------ 
 | MASTER         GND o |------------|---------------------- | o GND    SLAVE |
 |                   D2 |           <- i2c bus ->            | A4             |
 | ESP8266(ex)    SDA o |--------------------|-------------- | o SDA      A E |
 |                SCL o |------------|-----/ | /------------ | o SCL      T G |
 |                   D1 |            |       |               | A5         M A |
 |                +5V o |-|         [ ]     [ ]          --- | o +5V          |
 ------------------------ |    4.7K [ ]     [ ] 4.7K     |   -----------------
                          |         [ ]     [ ]          |
                          |          |       |           |
                          |----------|-------|-----------|  
 D2 = GPIO 4 (SDA)                   |
 D1 = GPIO 5 (SCL)                   o +5V

For example, to modify a pin on the ATMega328P, I can do the following (pins are remapped ofcourse):

  pinMode( D22, OUTPUT );     // Sets pin D13 = ONBOARD_LED on ATMega328P 
  digitalWrite( D22, HIGH );  // Turns the onboard LED ON on ATMega328P
  delay(2000);                // Wait two seconds
  digitalWrite( D22, LOW );   // Turns the onboard LED OFF on ATMega328P

This is working excellent, very straight forward functions with straight forward results, however, I also extend/chained the internal EEPROM with the external EEPROM to 'double' the size. The first 1K belongs to the ESP8266ex and the next 1K belongs to the ATMega328P.         

I developed a chain of functions for this and results in a bunch of easy to use functions:
bool setStorage( uint16_t iAddress, uint8_t* buffer, <parameters> );
bool setStorage( uint16_t iAddress, char* buffer, <parameters> );
bool getStorage( uint16_t iAddress, char* buffer, <parameters> );
.....
char* getStorage( uint16_t iAddress, char* psReturnDefault ); // Returns pointer buffered char array
....

For example, I can do:
setStorage( 2000UL, "Hello world" ); // Set 11 chars in EEPROM on ATMega328P on pos 977
delay(1000);
 // Read it back
Serial.println( "String is: " );
Serial.println( (char*)getStorage( 2000UL, "" ) );

Question
I verified the data written correctly and readed correctly however when the slave (ATMega328P) sending more than 8 bytes (with Wire.write()), the master (ESP8266ex) reads only a bunch of 0xFF's (with use of Wire.read()). So there is something wrong in between the I2C communication.
Checked/verified everything, buffersizes (32 bytes, that's enough for this example), check the contents of buffers, it is all perfectly fine. tried to send it buffered at once, nothing seems to help.
Is this a bug in the Wire library? Is there a workaround available? 

Part of my MASTER library (cannot post it all, too big for StackOverflow), to give you some idea what I am doing:
......

#define IPMB_HDR_DATASIZE         0x03
#define IPMB_MAX_DATACOUNT        (BUFFER_LENGTH-IPMB_HDR_DATASIZE)

......

typedef struct rIpmbRequestDataStruc
{
   uint8_t cmd;                          // Request command, take a look at IPMB_CMD_* above       
   uint8_t version;                      // Software version of request, must match
   uint8_t dataType;
   uint8_t data[ IPMB_MAX_DATACOUNT ];   // Data/parameters to be send
}; 

.........

bool i2cBridgeRequest( uint8_t iCmd,             // Request command
                       uint8_t*  puResult,       // Pointer to result var
                       uint16_t  iParam1,        // First parameter
                       uint16_t  iParam2 = 0,    // Second parameter, data or length  
                       uint8_t*  pParam3 = 0     // Byte data when stream or string
                     )
{
  bool     bSuccess   = i2cBridgeAvailable();

  uint8_t  iErrorCode = 0;
  uint8_t  iDataType  = 0;
  uint16_t iBytes     = 0;

  if( bSuccess )
  {

    rIpmbRequestDataStruc dataStruc;
    memset( (uint8_t*)&dataStruc, 0, sizeof( dataStruc ));

    dataStruc.cmd      = iCmd;
    dataStruc.version  = IPMB_DSI_VERSION;
    dataStruc.dataType = IPMB_DAT_TYPE_UINT16;

    uint16_t  i         = 0;
    uint16_t  iMax      = IPMB_MAX_DATACOUNT+IPMB_HDR_DATASIZE;
    uint8_t*  pParam    = 0;

    if( iCmd == IPMB_CMD_EEPROMREAD || iCmd == IPMB_CMD_DIGITALWRITE
      || iCmd == IPMB_CMD_ANALOGWRITE || iCmd == IPMB_CMD_EEPROMWRITE )
    {
      // First parameter must be 16 bits
      pParam = (uint8_t*)&iParam1;
      dataStruc.data[i++] = *pParam++;
      dataStruc.data[i++] = *pParam;
    }
    else { 
           dataStruc.dataType  = IPMB_DAT_TYPE_UINT8;
           dataStruc.data[i++] = iParam1; 
         } 

    if( iCmd == IPMB_CMD_DIGITALWRITE || iCmd == IPMB_CMD_ANALOGWRITE 
      || (iCmd == IPMB_CMD_CONFIG && iParam1 == IPMB_CFG_PWMCLOCK ) 
       || iCmd == IPMB_CMD_EEPROMREAD || pParam3 )
    {
      // Second parameter must be 16 bits
      pParam = (uint8_t*)&iParam2;
      dataStruc.data[i++] = *pParam++;
      dataStruc.data[i++] = *pParam;
     }
    else { dataStruc.data[i++] = iParam2; }

     // When pParam3 is specified, we expect iParam2 is the length
    if( pParam3 )
    {
      if( iParam2 > 1 ) 
       { dataStruc.dataType = IPMB_DAT_TYPE_STREAM; }

      iParam2+=IPMB_HDR_DATASIZE+1;
      while( i < iParam2 && i < iMax )
       { dataStruc.data[i++]=*pParam3++; }
    }
    else if( iCmd == IPMB_CMD_EEPROMREAD && iParam2 >= 1 )
          { dataStruc.dataType = IPMB_DAT_TYPE_STREAM; 
            Serial.println( "Data length = " );
            Serial.println( iParam2 );
          }      

    // Start transmission and send command and data
    Wire.beginTransmission( IPMB_I2C_ADDRESS );
    Wire.write( (uint8_t*)&dataStruc, IPMB_HDR_DATASIZE + i );
    bSuccess = ( Wire.endTransmission() == 0 );

    //Serial.println( bSuccess );

    // When data successfully send, perform command and data and ask result by request 
    if( bSuccess )
    { 
      //Wire.requestFrom( IPMB_I2C_ADDRESS, 3 + ( iCmd == IPMB_CMD_ANALOGREAD) ); 
      Wire.requestFrom( IPMB_I2C_ADDRESS, IPMB_HDR_DATASIZE+IPMB_MAX_DATACOUNT ); 
      //Serial.println( Wire.available() );
      if( Wire.available() > 2 )
       { 
          iErrorCode = Wire.read(); 
          if( !(iErrorCode >= IPMB_ECMD_MIN && iErrorCode <= IPMB_ECMD_MAX ))
           { iErrorCode = IPMB_ECMD_INVALID_RESPONSE; 

              // Debug read, reads only 0xFF's when received more than 8 bytes
             while( Wire.available() )
              { Serial.println( Wire.read(), HEX ); }

           }
       }
      else { iErrorCode = IPMB_ECMD_INVALID_RESPONSE; } 

      bSuccess = ( iErrorCode == IPMB_ECMD_OK );
    } 

    Serial.println( "ErrorCode:" );
    Serial.println( iErrorCode, HEX );

    if( bSuccess )
    { 
      iDataType = Wire.read();
      Serial.println( iDataType, HEX );

      if( iDataType != IPMB_DAT_TYPE_NONE )
      {
        uint8_t*  pFuncResult = puResult?puResult:(uint8_t*)&dataStruc.data[0];
        uint16_t  iMaxBytes   = i2cBridgeGetDataSize( iDataType );

        Serial.println( "Result is: " );
        Serial.println( (char*)pFuncResult );

        if( puResult )
         { memset( &pFuncResult[0], 0, sizeof( dataStruc.data )); }

        while( Wire.available() && iBytes < iMaxBytes )
         { pFuncResult[iBytes++] = Wire.read(); }

        if( iMaxBytes <= 4 )
         { bSuccess = ( iBytes == iMaxBytes ); }
        else { bSuccess = ( iBytes > 0 ); }  
      }
    }
    else {
           if( puResult ) 
            { *puResult = iErrorCode; } 
         }

     // Eat all left bytes if any
    while( Wire.available() )
     { Wire.read(); }

  }

  return bSuccess;
}

Part of my SLAVE library (cannot post it all, too big for StackOverflow), to give you some idea what I am doing:
.........

    typedef struct rIpmbResultDataStruc
    {
       uint8_t errorCode;
       uint8_t dataType;
       uint8_t data[ IPMB_MAX_DATACOUNT ];
    }; 

        ........

void eventHandleRequestReplyHandler()          // #2 Finish request, 

implement received data
    {
     /*
      Serial.print( "Bytes: " );
      Serial.println( __iIpmbDataByteCount );
      Serial.print( "Command: " );
      Serial.println( __rIpmbDataStruc.cmd );
      Serial.print( "Version: " );
      Serial.println( __rIpmbDataStruc.version, HEX );
      Serial.print( "DataType: " );
      Serial.println( __rIpmbDataStruc.dataType, HEX );
    */
      resetSendBuffer();

      uint16_t i         = 0;
      uint16_t iLength   = 0;
      uint8_t  iValue    = 0;
      uint16_t iAddress  = 0;

      // When reboot and sleep mode is previously requested, 
      // don't allow other commands 
      if( __bIpmbDoDeviceReset || __bIpmbDoDeviceSleep || isRebootSleepModeRequested() )
      {
        Wire.write( IPMB_ECMD_BUSY );
        Wire.write( IPMB_DAT_TYPE_NONE );
        Wire.write(0);
        return;
      }

      if( isValidCommand( __rIpmbDataStruc.cmd ) // Valid command received?
         && isValidVersion( __rIpmbDataStruc.version ) // Version the same?
           && isValidDataType( __rIpmbDataStruc.dataType ) ) // Valid dataType specified?
      {
        if( __rIpmbDataStruc.cmd == IPMB_CMD_DIGITALWRITE )
        {
           digitalWrite( getBuffDataUint16(0), getBuffDataUint16(1) );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_NONE );
           Wire.write(0);
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_ANALOGWRITE )
        {
           analogWrite( getBuffDataUint16(0), getBuffDataUint16(1) );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_NONE );
           Wire.write(0);
           return;
        }

        if( __rIpmbDataStruc.cmd == IPMB_CMD_DIGITALREAD )
        {
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_UINT8 );
           Wire.write( digitalRead( getBuffDataUint8(0) ));
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_ANALOGREAD )
        {
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_UINT16 );
           uint16_t iResult = analogRead( getBuffDataUint8(0) );
           uint8_t* pResult = (uint8_t*)&iResult; 
           Wire.write( *pResult++ );
           Wire.write( *pResult );
           return;
        }

        if( __rIpmbDataStruc.cmd == IPMB_CMD_PINMODE )
        {
           pinMode( getBuffDataUint8(0), getBuffDataUint8(1) );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_NONE );
           Wire.write(0);
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_EEPROMREAD )
        {
           Serial.println( "EEPROM READ");
           //Serial.println( getBuffDataUint16(0) );

           iAddress = IPMB_ADR_CUSTOM_DATA + getBuffDataUint16(0);
           iLength  = getBuffDataUint16(1);

           if( iLength > IPMB_MAX_DATACOUNT ) 
            { iLength = IPMB_MAX_DATACOUNT; }

           if( __rIpmbDataStruc.dataType == IPMB_DAT_TYPE_STREAM && iLength > 0 )
           {
             //Wire.write( IPMB_DAT_TYPE_STREAM );
             __rIpmbResultStruc.errorCode = IPMB_ECMD_OK; 
             __rIpmbResultStruc.dataType  = IPMB_DAT_TYPE_STREAM;

             while( i < iLength )
             {
                __rIpmbResultStruc.data[i++] = readStorage( iAddress++ );
             }

             //Serial.println( (char*)&__pIpmbResultByteBuff[0] );
             Wire.write( (uint8_t*)&__pIpmbResultByteBuff[0], 2+i ); 
           }
           else {
                  Wire.write( IPMB_DAT_TYPE_UINT8 );
                  Wire.write( readStorage( iAddress,
                                           getBuffDataUint8(1)
                                         )
                            );
                }            
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_EEPROMWRITE )
        {
           Serial.println( "EEPROM WRITE");
           Serial.println( getBuffDataUint16(0) );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_UINT8 );

           iAddress = IPMB_ADR_CUSTOM_DATA + getBuffDataUint16(0);

           if( __rIpmbDataStruc.dataType == IPMB_DAT_TYPE_STREAM )
            {
              iLength = getBuffDataUint16(1);
              Serial.println( iLength ); delay(100);

              while( i < iLength )
              {
                iValue = getBuffDataUint8(4+i);
                Serial.println( (char)iValue ); delay(100);
                if( writeStorage( iAddress++, iValue ) != iValue )
                {
                  Wire.write(0);
                  return;
                }

                ++i;
              }

              Wire.write( IPMB_ECMD_OK );
              Serial.println( "Done" ); delay(100);
            }
           else { 
                   Wire.write( writeStorage( iAddress,
                                             getBuffDataUint8(2),
                                             getBuffDataUint8(3),
                                             getBuffDataUint8(4)
                                            )
                             );
                }              
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_RESET )
        {
           //Serial.println( "SoftReset!" );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_NONE );
           Wire.write(0);
           __bIpmbDoDeviceReset = true;
           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_CONFIG )
        {
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_UINT8 );

           uint8_t iCfg = getBuffDataUint8(0);

           if( iCfg == IPMB_CFG_WIPE || iCfg == IPMB_CFG_WIPE_EEPROM )
            { 
              wipeStorage( iCfg == IPMB_CFG_WIPE_EEPROM );
              Wire.write( IPMB_ECMD_OK );

              // Always reset
              __bIpmbDoDeviceReset = true; 
            }
           else
           if( iCfg == IPMB_CFG_MCUCLOCK )
            { 
              Wire.write( 
                          setMcuClock( getBuffDataUint8(1), 
                                       __iIpmbConfigAutoSave
                                     ) 
                        ); 
            }
           else
           if( iCfg == IPMB_CFG_PWMCLOCK ) 
            { 
              Wire.write( 
                          setPwmClock( getBuffDataUint8(1),
                                       getBuffDataUint8(2),
                                       __iIpmbConfigAutoSave
                                     )
                          ); 
            }
           else { Wire.write(0); }

            // Set reboot flag if required
           if( __iIpmbRebootAtConfig && __iIpmbConfigAutoSave )
            { __bIpmbDoDeviceReset = true; }

           return;
        }

       if( __rIpmbDataStruc.cmd == IPMB_CMD_SLEEP )
        {
           //Serial.println( "Sleep" );
           Wire.write( IPMB_ECMD_OK );
           Wire.write( IPMB_DAT_TYPE_UINT8 );
           if( getBuffDataUint8(0) )
            { __bIpmbDoDeviceSleep = true; }
           Wire.write( (uint8_t)__bIpmbDoDeviceSleep );
           return;
        }
      }

      if( isValidCommand( __rIpmbDataStruc.cmd ) && !isValidVersion( __rIpmbDataStruc.version ))
       { Wire.write( IPMB_ECMD_INVALID_VERSION ); }
      else { Wire.write( IPMB_ECMD_INVALID_REQUEST ); }

      Wire.write( IPMB_DAT_TYPE_NONE );
      Wire.write(0);

      resetDataBuffer();
    }

A:

Finally found the bug and solution, after playing for hours with library code of the ESP, it is a problem of the ESP twi library. I post this as answer, maybe it can help somebody else. Cause: Timeout range too small and because of this, calls timeouts to early and functions relying on this will fail. That's the reason reading 0xFF's instead of real received data (it is actually there).
I know from previous projects that Esp8266 is very picky on delays, processes that takes too long for some reason and can cause a crash or device starts to malfunction, however, really, this timeout is too small especially when you are using more phiperials like a display for example or want to send more bytes over the BUS.
It is a bug in the ESP8266 twi library, inside the twi_init function. It has something to do with I2C BUS reading timeouts which cannot be changed via a class function (you can change BUS speed but not this), value is hardcoded inside this function.
function is in packages/esp8266/2.4.0/cores/esp8266/core_esp8266_si2c.c:
void twi_init(unsigned char sda, unsigned char scl){
  twi_sda = sda;
  twi_scl = scl;
  pinMode(twi_sda, INPUT_PULLUP);
  pinMode(twi_scl, INPUT_PULLUP);
  twi_setClock(100000);
  twi_setClockStretchLimit(230); // default value is 230 uS
}  

It is at the twi_setClockStretchLimit() instruction, the "ClockStretch" (whatever that means) is set to 230 uS, which is a way too low or narrow. 
To fix it, you need to cranck up this value to 600 or more and you must do this after initalization of the Wire library, for example:
Wire.begin();
 // Give it some time
delay( 500 );
 // default value is set to 230 uS, we change it here
twi_setClockStretchLimit(600); 
.....
.....

And now I can receive the full 32 bytes (default buffer limit). So when I ask the ATMega slave for a string (stream) inside the EEPROM, I will receive 28 bytes of data and 4 bytes data info (errorCode (byte), dataType (byte), length (2 bytes)).
Have fun ;-)

