parameters: (self, fwd_op_idx)
return_statement: return additional_sum_ops, grad_map
docstring: For each input name in the forward op, check if we will need to
          add gradient accumulation. If so, do gradient accumulation and return
          the list of gradient operators.
  
          The criteria for doing gradient accumulation is:
          (1) the specific input version has been used by multiple operators.
          (2) the current fwd_op_idx is the first to use that input, i.e. in the
              backward pass, is the last to optionally generate the gradient for
              the op.
          (3) For the operators that used the input, their gradient operators
              have generated more than 1 gradient.
  
          When accumulating operators, our current solution is to rename all the
          created gradients with an internal intermediate name, and then add a
          Sum() operator that adds up all the gradients. This may use more memory
          due to intermediate storage, but is usually the fastest approach as one
          can do one single sum for multiple intermediate gradients.
docstring_summary: For each input name in the forward op, check if we will need to
          add gradient accumulation. If so, do gradient accumulation and return
          the list of gradient operators.
docstring_tokens: For each input name in the forward op check if we will need to add gradient accumulation . If so do gradient accumulation and return the list of gradient operators .
function: def DoGradientAccumulation(self, fwd_op_idx):
          """For each input name in the forward op, check if we will need to
          add gradient accumulation. If so, do gradient accumulation and return
          the list of gradient operators.
  
          The criteria for doing gradient accumulation is:
          (1) the specific input version has been used by multiple operators.
          (2) the current fwd_op_idx is the first to use that input, i.e. in the
              backward pass, is the last to optionally generate the gradient for
              the op.
          (3) For the operators that used the input, their gradient operators
              have generated more than 1 gradient.
  
          When accumulating operators, our current solution is to rename all the
          created gradients with an internal intermediate name, and then add a
          Sum() operator that adds up all the gradients. This may use more memory
          due to intermediate storage, but is usually the fastest approach as one
          can do one single sum for multiple intermediate gradients.
          """
          forward_op, in_versions, out_versions = self.ssa[fwd_op_idx]
          additional_sum_ops = []
          grad_map = {}
          for _i, input_name in enumerate(set(forward_op.input)):
              input_version = in_versions[input_name]
              input_usage = self.input_usages[input_name][input_version]
              if (len(input_usage) <= 1 or fwd_op_idx != input_usage[0]):
                  # We do not need to do gradient accumulation yet.
                  continue
              generator = self.gradient_generators[input_name][input_version]
              try:
                  if not self._VerifyGradientGenerators(generator):
                      continue
              except RuntimeError as err:
                  raise RuntimeError(
                      "Gradients for param ''{}'' failed to verify: {}".format(
                          input_name,
                          err
                      )
                  )
  
              # Finally, let's create the sum operator.
              sum_ops, g = self._MakeSumOps(input_name, input_version)
              additional_sum_ops.extend(sum_ops)
              grad_map[input_name] = g
          return additional_sum_ops, grad_map
function_tokens: def DoGradientAccumulation ( self , fwd_op_idx ) : forward_op , in_versions , out_versions = self . ssa [ fwd_op_idx ] additional_sum_ops = [ ] grad_map = { } for _i , input_name in enumerate ( set ( forward_op . input ) ) : input_version = in_versions [ input_name ] input_usage = self . input_usages [ input_name ] [ input_version ] if ( len ( input_usage ) <= 1 or fwd_op_idx != input_usage [ 0 ] ) : # We do not need to do gradient accumulation yet. continue generator = self . gradient_generators [ input_name ] [ input_version ] try : if not self . _VerifyGradientGenerators ( generator ) : continue except RuntimeError as err : raise RuntimeError ( "Gradients for param ''{}'' failed to verify: {}" . format ( input_name , err ) ) # Finally, let's create the sum operator. sum_ops , g = self . _MakeSumOps ( input_name , input_version ) additional_sum_ops . extend ( sum_ops ) grad_map [ input_name ] = g return additional_sum_ops , grad_map