<story><title>Show HN: AI dub tool I made to watch foreign language videos with my 7-year-old</title><url>https://speakz.ai</url><text>Hey HN!&lt;p&gt;I love watching YouTube with my 7-year-old daughter. Unfortunately, the best stuff is often in English (we&amp;#x27;re German). So I made an AI tool that translates videos directly, using the original voices. All other sounds, as well as background music, are preserved, too.&lt;p&gt;Turns out that it works for many other language pairs, too. So far, it can create dubs in English, Mandarin Chinese, Spanish, Arabic, French, Russian, German, Italian, Korean, Polish and Dutch.&lt;p&gt;The main challenge in building this was to get the balance right between translating the original meaning and getting the timing right. Especially for language pairs like English -&amp;gt; German, where the target ist often longer than the source (&amp;quot;bat&amp;quot; -&amp;gt; &amp;quot;Fle-der-maus&amp;quot;, &amp;quot;speed&amp;quot; -&amp;gt; &amp;quot;Ge-schwin-dig-keit&amp;quot;).&lt;p&gt;Let me know what you think! :)</text></story><parent_chain><item><author>sorenjan</author><text>I know Germany dub most video, but wouldn&amp;#x27;t a seven year old be able to read subtitles? It&amp;#x27;s a great way for her to learn English, it&amp;#x27;s how most Swedes learn it before starting school. I think there&amp;#x27;s a pretty strong correlation between countries&amp;#x27; average English proficiency and how common dubbing is.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;haonowshaokao.com&amp;#x2F;2013&amp;#x2F;05&amp;#x2F;18&amp;#x2F;does-dubbing-tv-harm-language-learning&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;haonowshaokao.com&amp;#x2F;2013&amp;#x2F;05&amp;#x2F;18&amp;#x2F;does-dubbing-tv-harm-la...&lt;/a&gt;&lt;p&gt;Edit: I forgot to mention that the samples on the website is impressive and well made. How do you do the speaker diarization and voice cloning?</text></item></parent_chain><comment><author>leobg</author><text>Yeah, this isn&amp;#x27;t really helpful for her to learn English. This is more when we watch The Anatomy Lab, or BBC&amp;#x27;s &amp;quot;The Incredible Human Journey&amp;quot;. She&amp;#x27;ll already be asking me a lot of questions about the content. So if I had to translate on top of that, it would be tedious.&lt;p&gt;Subtitles - those are actually being generated as well. I&amp;#x27;ve generated SRT files during development. Color coded by speaker, and on a per-word basis, for me to get the timing right.&lt;p&gt;Basically, if you have a YouTube channel, you can take any video from your channel, run it through Speakz.ai, and you&amp;#x27;ll get 15+ additional audio tracks in different languages, plus 15+ subtitle files (SRT).&lt;p&gt;Voice cloning and speaker diarization was a bit of a challenge. On the one hand, I want to duplicate the voice that is being spoken right now. On the other hand, sometimes &amp;quot;right now&amp;quot; is just a short &amp;quot;Yeah&amp;quot; (like in the Elon interview) which doesn&amp;#x27;t give you a lot of &amp;quot;meat&amp;quot; to work with in terms of embedding the voice&amp;#x27;s characteristics.&lt;p&gt;Right now, I&amp;#x27;m using a mix of signals:&lt;p&gt;- Is the utterance padded by pauses before&amp;#x2F;after?
- Is the utterance a complete sentence?
- Does the voice of the utterance sound significantly different from the voice of the previous utterance?&lt;p&gt;It&amp;#x27;s a deep, deep rabbit hole. I was tempted to go much deeper. But I thought I better check if anybody besides myself actually cares before I do that... :)</text></comment>
<story><title>Show HN: AI dub tool I made to watch foreign language videos with my 7-year-old</title><url>https://speakz.ai</url><text>Hey HN!&lt;p&gt;I love watching YouTube with my 7-year-old daughter. Unfortunately, the best stuff is often in English (we&amp;#x27;re German). So I made an AI tool that translates videos directly, using the original voices. All other sounds, as well as background music, are preserved, too.&lt;p&gt;Turns out that it works for many other language pairs, too. So far, it can create dubs in English, Mandarin Chinese, Spanish, Arabic, French, Russian, German, Italian, Korean, Polish and Dutch.&lt;p&gt;The main challenge in building this was to get the balance right between translating the original meaning and getting the timing right. Especially for language pairs like English -&amp;gt; German, where the target ist often longer than the source (&amp;quot;bat&amp;quot; -&amp;gt; &amp;quot;Fle-der-maus&amp;quot;, &amp;quot;speed&amp;quot; -&amp;gt; &amp;quot;Ge-schwin-dig-keit&amp;quot;).&lt;p&gt;Let me know what you think! :)</text></story><parent_chain><item><author>sorenjan</author><text>I know Germany dub most video, but wouldn&amp;#x27;t a seven year old be able to read subtitles? It&amp;#x27;s a great way for her to learn English, it&amp;#x27;s how most Swedes learn it before starting school. I think there&amp;#x27;s a pretty strong correlation between countries&amp;#x27; average English proficiency and how common dubbing is.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;haonowshaokao.com&amp;#x2F;2013&amp;#x2F;05&amp;#x2F;18&amp;#x2F;does-dubbing-tv-harm-language-learning&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;haonowshaokao.com&amp;#x2F;2013&amp;#x2F;05&amp;#x2F;18&amp;#x2F;does-dubbing-tv-harm-la...&lt;/a&gt;&lt;p&gt;Edit: I forgot to mention that the samples on the website is impressive and well made. How do you do the speaker diarization and voice cloning?</text></item></parent_chain><comment><author>supafastcoder</author><text>I think it’s a cultural difference. I’m also from a non-dubbing country (Netherlands) and I can’t stand dubbed content either. On the other hand people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it.</text></comment>