fastspeech2 reduction_fatcor=3 training problem
**Describe the bug**
With custom japanese dataset, I trained tacotron2 with reduction_factor = 3 using espnet2 recipe.
Using that model, i was trying to train fastspeech2 default config with reduction_factor = 3.

I extracted durations using inference stage and with that, I ran the script run.sh.
However, the error occurred at the stage 5 while extracting pitch and energy stats.
For more information, I attached the error log below.

Seems like while the durations got reduced as a consequence of applying reduction_factor = 3, 
pitches were not synchronized though.

**Basic environments:**
 - OS information:  Linux 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64
 - python version: 3.7.6 (default, Jan  8 2020, 19:59:22)  [GCC 7.3.0]
 - espnet version: espnet 0.9.4
 - pytorch version pytorch 1.4.0

**Environments from `torch.utils.collect_env`:**
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 440.100
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] numpydoc==0.9.2
[pip] pytorch-ranger==0.1.1
[pip] pytorch-wpe==0.0.0
[pip] torch==1.4.0
[pip] torch-complex==0.0.3
[pip] torch-optimizer==0.0.1a15
[pip] torchaudio==0.4.0
[pip] warpctc-pytorch==0.2.1
[pip] warprnnt-pytorch==0.1
[conda] _pytorch_select           0.2                       gpu_0  
[conda] blas                      1.0                         mkl  
[conda] mkl                       2020.0                      166  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] pytorch-wpe               0.0.0                    pypi_0    pypi
[conda] torch                     1.4.0                    pypi_0    pypi
[conda] torch-complex             0.0.3                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a15                 pypi_0    pypi
[conda] torchaudio                0.4.0                    pypi_0    pypi
[conda] warpctc-pytorch           0.2.1                    pypi_0    pypi
[conda] warprnnt-pytorch          0.1                      pypi_0    pypi

**Task information:**
 - Task: TTS
 - Recipe: custom dataset
 - ESPnet2

**To Reproduce**
Steps to reproduce the behavior:
- train fastspeech2 with reduction factor=3 config and teacher model as tacotron2 reduction factor =3 

**Error logs**
Paste the error logs. If applicable, add screenshots to help explain your problem.

File "/espnet2/tts/feats_extract/dio.py", line 176, in _average_by_duration
    assert d.sum() == len(x), f"{d.sum()} == {len(x)} error"
AssertionError: 225 == 677 error
x shape : (202800,)
x shape : (369300,)
x shape : (362100,)
x shape : (347400,)
x shape : (332400,)
x shape : (380700,)
x shape : (430200,)
x shape : (149400,)
x shape : (302100,)
x shape : (240000,)
x shape : (290100,)
x shape : (282000,)
x shape : (354900,)
x shape : (346500,)
x shape : (338100,)
x shape : (384000,)
x shape : (338400,)
x shape : (314400,)
x shape : (319200,)
x shape : (213900,)
diff --git a/espnet2/tasks/tts.py b/espnet2/tasks/tts.py
index 3232d7254..9a031b4ef 100644
--- a/espnet2/tasks/tts.py
+++ b/espnet2/tasks/tts.py
@@ -301,8 +301,24 @@ class TTSTask(AbsTask):
         energy_normalize = None
         if getattr(args, "pitch_extract", None) is not None:
             pitch_extract_class = pitch_extractor_choices.get_class(args.pitch_extract)
+            if args.pitch_extract_conf.get("reduction_factor", None) is not None:
+                assert args.pitch_extract_conf.get(
+                    "reduction_factor", None
+                ) == args.tts_conf.get("reduction_factor", 1)
+            else:
+                args.pitch_extract_conf["reduction_factor"] = args.tts_conf.get(
+                    "reduction_factor", 1
+                )
             pitch_extract = pitch_extract_class(**args.pitch_extract_conf)
         if getattr(args, "energy_extract", None) is not None:
+            if args.energy_extract_conf.get("reduction_factor", None) is not None:
+                assert args.energy_extract_conf.get(
+                    "reduction_factor", None
+                ) == args.tts_conf.get("reduction_factor", 1)
+            else:
+                args.energy_extract_conf["reduction_factor"] = args.tts_conf.get(
+                    "reduction_factor", 1
+                )
             energy_extract_class = energy_extractor_choices.get_class(
                 args.energy_extract
             )
diff --git a/espnet2/tts/feats_extract/dio.py b/espnet2/tts/feats_extract/dio.py
index ba4f82597..48f7249aa 100644
--- a/espnet2/tts/feats_extract/dio.py
+++ b/espnet2/tts/feats_extract/dio.py
@@ -7,7 +7,6 @@ import logging
 
 from typing import Any
 from typing import Dict
-from typing import Optional
 from typing import Tuple
 from typing import Union
 
@@ -47,11 +46,12 @@ class Dio(AbsFeatsExtract):
         fs: Union[int, str] = 22050,
         n_fft: int = 1024,
         hop_length: int = 256,
-        f0min: Optional[int] = 80,
-        f0max: Optional[int] = 400,
+        f0min: int = 80,
+        f0max: int = 400,
         use_token_averaged_f0: bool = True,
         use_continuous_f0: bool = True,
         use_log_f0: bool = True,
+        reduction_factor: int = None,
     ):
         assert check_argument_types()
         super().__init__()
@@ -66,6 +66,9 @@ class Dio(AbsFeatsExtract):
         self.use_token_averaged_f0 = use_token_averaged_f0
         self.use_continuous_f0 = use_continuous_f0
         self.use_log_f0 = use_log_f0
+        if use_token_averaged_f0:
+            assert reduction_factor >= 1
+        self.reduction_factor = reduction_factor
 
     def output_size(self) -> int:
         return 1
@@ -80,6 +83,7 @@ class Dio(AbsFeatsExtract):
             use_token_averaged_f0=self.use_token_averaged_f0,
             use_continuous_f0=self.use_continuous_f0,
             use_log_f0=self.use_log_f0,
+            reduction_factor=self.reduction_factor,
         )
 
     def forward(
@@ -108,6 +112,7 @@ class Dio(AbsFeatsExtract):
 
         # (Optional): Average by duration to calculate token-wise f0
         if self.use_token_averaged_f0:
+            durations = durations * self.reduction_factor
             pitch = [
                 self._average_by_duration(p, d).view(-1)
                 for p, d in zip(pitch, durations)
@@ -170,9 +175,8 @@ class Dio(AbsFeatsExtract):
 
         return f0
 
-    @staticmethod
-    def _average_by_duration(x: torch.Tensor, d: torch.Tensor) -> torch.Tensor:
-        assert d.sum() == len(x)
+    def _average_by_duration(self, x: torch.Tensor, d: torch.Tensor) -> torch.Tensor:
+        assert 0 <= len(x) - d.sum() < self.reduction_factor
         d_cumsum = F.pad(d.cumsum(dim=0), (1, 0))
         x_avg = [
             x[start:end].masked_select(x[start:end].gt(0.0)).mean(dim=0)
diff --git a/espnet2/tts/feats_extract/energy.py b/espnet2/tts/feats_extract/energy.py
index 33ef4fb9c..d80f3af53 100644
--- a/espnet2/tts/feats_extract/energy.py
+++ b/espnet2/tts/feats_extract/energy.py
@@ -5,7 +5,6 @@
 
 from typing import Any
 from typing import Dict
-from typing import Optional
 from typing import Tuple
 from typing import Union
 
@@ -29,11 +28,12 @@ class Energy(AbsFeatsExtract):
         n_fft: int = 1024,
         win_length: int = None,
         hop_length: int = 256,
-        window: Optional[str] = "hann",
+        window: str = "hann",
         center: bool = True,
         normalized: bool = False,
         onesided: bool = True,
         use_token_averaged_energy: bool = True,
+        reduction_factor: int = None,
     ):
         assert check_argument_types()
         super().__init__()
@@ -46,6 +46,9 @@ class Energy(AbsFeatsExtract):
         self.win_length = win_length
         self.window = window
         self.use_token_averaged_energy = use_token_averaged_energy
+        if use_token_averaged_energy:
+            assert reduction_factor >= 1
+        self.reduction_factor = reduction_factor
 
         self.stft = Stft(
             n_fft=n_fft,
@@ -70,6 +73,7 @@ class Energy(AbsFeatsExtract):
             center=self.stft.center,
             normalized=self.stft.normalized,
             use_token_averaged_energy=self.use_token_averaged_energy,
+            reduction_factor=self.reduction_factor,
         )
 
     def forward(
@@ -107,6 +111,7 @@ class Energy(AbsFeatsExtract):
 
         # (Optional): Average by duration to calculate token-wise energy
         if self.use_token_averaged_energy:
+            durations = durations * self.reduction_factor
             energy = [
                 self._average_by_duration(e[:el].view(-1), d)
                 for e, el, d in zip(energy, energy_lengths, durations)
@@ -120,9 +125,8 @@ class Energy(AbsFeatsExtract):
         # Return with the shape (B, T, 1)
         return energy.unsqueeze(-1), energy_lengths
 
-    @staticmethod
-    def _average_by_duration(x: torch.Tensor, d: torch.Tensor) -> torch.Tensor:
-        assert d.sum() == len(x)
+    def _average_by_duration(self, x: torch.Tensor, d: torch.Tensor) -> torch.Tensor:
+        assert 0 <= len(x) - d.sum() < self.reduction_factor
         d_cumsum = F.pad(d.cumsum(dim=0), (1, 0))
         x_avg = [
             x[start:end].mean() if len(x[start:end]) != 0 else x.new_tensor(0.0)
diff --git a/test/espnet2/tts/feats_extract/test_dio.py b/test/espnet2/tts/feats_extract/test_dio.py
index c599d71b1..7d6269057 100644
--- a/test/espnet2/tts/feats_extract/test_dio.py
+++ b/test/espnet2/tts/feats_extract/test_dio.py
@@ -6,8 +6,12 @@ from espnet2.tts.feats_extract.dio import Dio
 
 @pytest.mark.parametrize("use_continuous_f0", [False, True])
 @pytest.mark.parametrize("use_log_f0", [False, True])
-@pytest.mark.parametrize("use_token_averaged_f0", [False, True])
-def test_forward(use_continuous_f0, use_log_f0, use_token_averaged_f0):
+@pytest.mark.parametrize(
+    "use_token_averaged_f0, reduction_factor", [(False, 1), (True, 1), (True, 3)]
+)
+def test_forward(
+    use_continuous_f0, use_log_f0, use_token_averaged_f0, reduction_factor
+):
     layer = Dio(
         n_fft=128,
         hop_length=64,
@@ -17,24 +21,47 @@ def test_forward(use_continuous_f0, use_log_f0, use_token_averaged_f0):
         use_continuous_f0=use_continuous_f0,
         use_log_f0=use_log_f0,
         use_token_averaged_f0=use_token_averaged_f0,
+        reduction_factor=reduction_factor,
     )
-    xs = torch.randn(2, 256)
+    xs = torch.randn(2, 384)
     if not use_token_averaged_f0:
-        layer(xs, torch.LongTensor([256, 128]))
+        layer(xs, torch.LongTensor([384, 128]))
     else:
-        ds = torch.LongTensor([[3, 0, 2], [3, 0, 0]])
+        ds = torch.LongTensor([[3, 3, 1], [3, 0, 0]]) // reduction_factor
         dlens = torch.LongTensor([3, 1])
         ps, _ = layer(
-            xs, torch.LongTensor([256, 128]), durations=ds, durations_lengths=dlens
+            xs, torch.LongTensor([384, 128]), durations=ds, durations_lengths=dlens
         )
         assert torch.isnan(ps).sum() == 0
 
 
-def test_output_size():
-    layer = Dio(n_fft=4, hop_length=1, f0min=40, f0max=800, fs="16k")
+@pytest.mark.parametrize(
+    "use_token_averaged_f0, reduction_factor", [(False, 1), (True, 1), (True, 3)]
+)
+def test_output_size(use_token_averaged_f0, reduction_factor):
+    layer = Dio(
+        n_fft=4,
+        hop_length=1,
+        f0min=40,
+        f0max=800,
+        fs="16k",
+        use_token_averaged_f0=use_token_averaged_f0,
+        reduction_factor=reduction_factor,
+    )
     print(layer.output_size())
 
 
-def test_get_parameters():
-    layer = Dio(n_fft=4, hop_length=1, f0min=40, f0max=800, fs="16k")
+@pytest.mark.parametrize(
+    "use_token_averaged_f0, reduction_factor", [(False, 1), (True, 1), (True, 3)]
+)
+def test_get_parameters(use_token_averaged_f0, reduction_factor):
+    layer = Dio(
+        n_fft=4,
+        hop_length=1,
+        f0min=40,
+        f0max=800,
+        fs="16k",
+        use_token_averaged_f0=use_token_averaged_f0,
+        reduction_factor=reduction_factor,
+    )
     print(layer.get_parameters())
diff --git a/test/espnet2/tts/feats_extract/test_energy.py b/test/espnet2/tts/feats_extract/test_energy.py
index de6c22c81..7bb9a05e4 100644
--- a/test/espnet2/tts/feats_extract/test_energy.py
+++ b/test/espnet2/tts/feats_extract/test_energy.py
@@ -4,32 +4,53 @@ import torch
 from espnet2.tts.feats_extract.energy import Energy
 
 
-@pytest.mark.parametrize("use_token_averaged_energy", [False, True])
-def test_forward(use_token_averaged_energy):
+@pytest.mark.parametrize(
+    "use_token_averaged_energy, reduction_factor", [(False, None), (True, 1), (True, 3)]
+)
+def test_forward(use_token_averaged_energy, reduction_factor):
     layer = Energy(
         n_fft=128,
         hop_length=64,
         fs="16k",
         use_token_averaged_energy=use_token_averaged_energy,
+        reduction_factor=reduction_factor,
     )
-    xs = torch.randn(2, 256)
+    xs = torch.randn(2, 384)
     if not use_token_averaged_energy:
-        es, elens = layer(xs, torch.LongTensor([256, 128]))
+        es, elens = layer(xs, torch.LongTensor([384, 128]))
         assert es.shape[1] == max(elens)
     else:
-        ds = torch.LongTensor([[3, 0, 2], [3, 0, 0]])
+        ds = torch.LongTensor([[3, 3, 1], [3, 0, 0]]) // reduction_factor
         dlens = torch.LongTensor([3, 1])
         es, _ = layer(
-            xs, torch.LongTensor([256, 128]), durations=ds, durations_lengths=dlens
+            xs, torch.LongTensor([384, 128]), durations=ds, durations_lengths=dlens
         )
         assert torch.isnan(es).sum() == 0
 
 
-def test_output_size():
-    layer = Energy(n_fft=4, hop_length=1, fs="16k")
+@pytest.mark.parametrize(
+    "use_token_averaged_energy, reduction_factor", [(False, None), (True, 1), (True, 3)]
+)
+def test_output_size(use_token_averaged_energy, reduction_factor):
+    layer = Energy(
+        n_fft=4,
+        hop_length=1,
+        fs="16k",
+        use_token_averaged_energy=use_token_averaged_energy,
+        reduction_factor=reduction_factor,
+    )
     print(layer.output_size())
 
 
-def test_get_parameters():
-    layer = Energy(n_fft=4, hop_length=1, fs="16k")
+@pytest.mark.parametrize(
+    "use_token_averaged_energy, reduction_factor", [(False, None), (True, 1), (True, 3)]
+)
+def test_get_parameters(use_token_averaged_energy, reduction_factor):
+    layer = Energy(
+        n_fft=4,
+        hop_length=1,
+        fs="16k",
+        use_token_averaged_energy=use_token_averaged_energy,
+        reduction_factor=reduction_factor,
+    )
     print(layer.get_parameters())
['espnet2/tasks/tts.py', 'espnet2/tts/feats_extract/energy.py', 'espnet2/tts/feats_extract/dio.py', 'test/espnet2/tts/feats_extract/test_dio.py', 'test/espnet2/tts/feats_extract/test_energy.py']