parameters: (self, samples)
return_statement: return local_iter
docstring: This function needs to be overridden by the user to process the
          generated samples from generate_sample(self, str) function
          It is usually used as batch processing when a user wants to
          do preprocessing on a batch of samples, e.g. padding according to
          the max length of a sample in the batch
  
          Args:
              samples(list tuple): generated sample from generate_sample
  
          Returns:
              a python generator, the same format as return value of generate_sample
  
          Example:
  
              .. code-block:: python
  
                  import paddle.distributed.fleet.data_generator as dg
                  class MyData(dg.DataGenerator):
  
                      def generate_sample(self, line):
                          def local_iter():
                              int_words = [int(x) for x in line.split()]
                              yield ("words", int_words)
                          return local_iter
  
                      def generate_batch(self, samples):
                          def local_iter():
                              for s in samples:
                                  yield ("words", s[1].extend([s[1][0]]))
                  mydata = MyData()
                  mydata.set_batch(128)
docstring_summary: This function needs to be overridden by the user to process the
          generated samples from generate_sample(self, str) function
          It is usually used as batch processing when a user wants to
          do preprocessing on a batch of samples, e.g. padding according to
          the max length of a sample in the batch
docstring_tokens: This function needs to be overridden by the user to process the generated samples from generate_sample ( self str ) function It is usually used as batch processing when a user wants to do preprocessing on a batch of samples e . g . padding according to the max length of a sample in the batch
function: def generate_batch(self, samples):
          '''
          This function needs to be overridden by the user to process the
          generated samples from generate_sample(self, str) function
          It is usually used as batch processing when a user wants to
          do preprocessing on a batch of samples, e.g. padding according to
          the max length of a sample in the batch
  
          Args:
              samples(list tuple): generated sample from generate_sample
  
          Returns:
              a python generator, the same format as return value of generate_sample
  
          Example:
  
              .. code-block:: python
  
                  import paddle.distributed.fleet.data_generator as dg
                  class MyData(dg.DataGenerator):
  
                      def generate_sample(self, line):
                          def local_iter():
                              int_words = [int(x) for x in line.split()]
                              yield ("words", int_words)
                          return local_iter
  
                      def generate_batch(self, samples):
                          def local_iter():
                              for s in samples:
                                  yield ("words", s[1].extend([s[1][0]]))
                  mydata = MyData()
                  mydata.set_batch(128)
          '''
  
          def local_iter():
              for sample in samples:
                  yield sample
  
          return local_iter
function_tokens: def generate_batch ( self , samples ) : def local_iter ( ) : for sample in samples : yield sample return local_iter