<story><title>Native Matrix VoIP with Element Call</title><url>https://element.io/blog/introducing-native-matrix-voip-with-element-call/</url></story><parent_chain><item><author>Arathorn</author><text>The competitive gap with Discord in terms of media quality is probably something like:&lt;p&gt;* Need a low-latency SFU.  This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we&amp;#x27;ve built MCUs and media servers of all flavours in the past.  MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party.&lt;p&gt;* Needs a SFU with good rate control (and&amp;#x2F;or FEC).  This is probably the single most important thing to get right in terms of quality.  Signal wrote up a good overview of why: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;signal.org&amp;#x2F;blog&amp;#x2F;how-to-build-encrypted-group-calls&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;signal.org&amp;#x2F;blog&amp;#x2F;how-to-build-encrypted-group-calls&amp;#x2F;&lt;/a&gt;&lt;p&gt;* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc).  Ideally you need something like &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;krisp.ai&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;krisp.ai&amp;#x2F;&lt;/a&gt; or &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;workspaceupdates.googleblog.com&amp;#x2F;2021&amp;#x2F;06&amp;#x2F;background-noise-reduction-cancellation-update.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;workspaceupdates.googleblog.com&amp;#x2F;2021&amp;#x2F;06&amp;#x2F;background-n...&lt;/a&gt; in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard.  However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;seshat&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;seshat&lt;/a&gt;), we&amp;#x27;ll have a go at doing something similar for this problem too.&lt;p&gt;* Excellent automatic gain control.  The importance of normalising&amp;#x2F;compressing everyone&amp;#x27;s audio so they&amp;#x27;re equivalent loudness is really important.&lt;p&gt;We&amp;#x27;re also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.&lt;p&gt;We can probably also be more bullish about supporting new audio codecs like Lyra.&lt;p&gt;EDIT: oh, the other obvious thing is echo cancellation - but we&amp;#x27;re currently at the mercy of the browser&amp;#x27;s WebRTC stack for that.  However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.</text></item><item><author>jfkimmes</author><text>Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Discord threw a lot of time and money at optimizing voice in their product. Can you just jump in and realistically compete?&lt;p&gt;There are a lot of performance&amp;#x2F;latency&amp;#x2F;sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Zoom, etc. I feel like this is a problem-space that can be optimized to an arbitrarily deep extent. Just two examples that come to mind are SFU performance&amp;#x2F;efficiency and noise reduction, two things where e.g Jitsi notoriously lags behind Zoom.</text></item><item><author>Arathorn</author><text>From my perspective, the really exciting thing about this that it works equally well in mobile web browsers as well as desktop web - clicking on a link on Mobile Safari should Do The Right Thing without having to install anything.&lt;p&gt;Moreover, because it&amp;#x27;s built on Matrix, MSC3401 (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;matrix-doc&amp;#x2F;blob&amp;#x2F;matthew&amp;#x2F;group-voip&amp;#x2F;proposals&amp;#x2F;3401-group-voip.md&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;matrix-doc&amp;#x2F;blob&amp;#x2F;matthew&amp;#x2F;group-...&lt;/a&gt;) means that we&amp;#x27;ll finally have decentralised cascading video&amp;#x2F;voice conferences once the SFU (selective forwarding unit) component is added into the mix.  So, for instance, users on the same homeserver will get their video feeds relayed locally with minimal latency... and then users on another remote homeserver will also get mixed locally with minimal latency, trunking the two together.  If the link dies or one homeserver dies, the conference will keep going - i.e. precisely the same semantics as normal Matrix.</text></item></parent_chain><comment><author>pthatcherg</author><text>Hey, I&amp;#x27;m the author of the Signal blog post about SFUs.  I have a few questions&amp;#x2F;comments:&lt;p&gt;1.  I don&amp;#x27;t think there are many good open source SFUs to choose from.  I know of 2, maybe 3 (including our new one).  There may be many, but few have good rate control.   But maybe I just don&amp;#x27;t know about them?  I&amp;#x27;d be happy to learn of more good ones.&lt;p&gt;2.  Echo cancellation is certainly a hard problem, but it doesn&amp;#x27;t conflict with E2EE unless you do it on the server, which isn&amp;#x27;t necessary.  So perhaps it may be a somewhat harder problem because you close off one possible approach (doing it server-side), but many (most?) echo cancellation solutions are done client-side.&lt;p&gt;3.  You may not be completely dependent on WebRTC&amp;#x27;s echo cancellation any more because of 
the new MediaStreamTrackProcessor  and MediaStreamTrackGenerator APIs.  I don&amp;#x27;t know if it will work for echo cancellation, but it might.</text></comment>
<story><title>Native Matrix VoIP with Element Call</title><url>https://element.io/blog/introducing-native-matrix-voip-with-element-call/</url></story><parent_chain><item><author>Arathorn</author><text>The competitive gap with Discord in terms of media quality is probably something like:&lt;p&gt;* Need a low-latency SFU.  This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we&amp;#x27;ve built MCUs and media servers of all flavours in the past.  MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party.&lt;p&gt;* Needs a SFU with good rate control (and&amp;#x2F;or FEC).  This is probably the single most important thing to get right in terms of quality.  Signal wrote up a good overview of why: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;signal.org&amp;#x2F;blog&amp;#x2F;how-to-build-encrypted-group-calls&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;signal.org&amp;#x2F;blog&amp;#x2F;how-to-build-encrypted-group-calls&amp;#x2F;&lt;/a&gt;&lt;p&gt;* Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc).  Ideally you need something like &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;krisp.ai&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;krisp.ai&amp;#x2F;&lt;/a&gt; or &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;workspaceupdates.googleblog.com&amp;#x2F;2021&amp;#x2F;06&amp;#x2F;background-noise-reduction-cancellation-update.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;workspaceupdates.googleblog.com&amp;#x2F;2021&amp;#x2F;06&amp;#x2F;background-n...&lt;/a&gt; in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard.  However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;seshat&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;seshat&lt;/a&gt;), we&amp;#x27;ll have a go at doing something similar for this problem too.&lt;p&gt;* Excellent automatic gain control.  The importance of normalising&amp;#x2F;compressing everyone&amp;#x27;s audio so they&amp;#x27;re equivalent loudness is really important.&lt;p&gt;We&amp;#x27;re also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.&lt;p&gt;We can probably also be more bullish about supporting new audio codecs like Lyra.&lt;p&gt;EDIT: oh, the other obvious thing is echo cancellation - but we&amp;#x27;re currently at the mercy of the browser&amp;#x27;s WebRTC stack for that.  However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.</text></item><item><author>jfkimmes</author><text>Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Discord threw a lot of time and money at optimizing voice in their product. Can you just jump in and realistically compete?&lt;p&gt;There are a lot of performance&amp;#x2F;latency&amp;#x2F;sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Zoom, etc. I feel like this is a problem-space that can be optimized to an arbitrarily deep extent. Just two examples that come to mind are SFU performance&amp;#x2F;efficiency and noise reduction, two things where e.g Jitsi notoriously lags behind Zoom.</text></item><item><author>Arathorn</author><text>From my perspective, the really exciting thing about this that it works equally well in mobile web browsers as well as desktop web - clicking on a link on Mobile Safari should Do The Right Thing without having to install anything.&lt;p&gt;Moreover, because it&amp;#x27;s built on Matrix, MSC3401 (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;matrix-doc&amp;#x2F;blob&amp;#x2F;matthew&amp;#x2F;group-voip&amp;#x2F;proposals&amp;#x2F;3401-group-voip.md&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;matrix-org&amp;#x2F;matrix-doc&amp;#x2F;blob&amp;#x2F;matthew&amp;#x2F;group-...&lt;/a&gt;) means that we&amp;#x27;ll finally have decentralised cascading video&amp;#x2F;voice conferences once the SFU (selective forwarding unit) component is added into the mix.  So, for instance, users on the same homeserver will get their video feeds relayed locally with minimal latency... and then users on another remote homeserver will also get mixed locally with minimal latency, trunking the two together.  If the link dies or one homeserver dies, the conference will keep going - i.e. precisely the same semantics as normal Matrix.</text></item></parent_chain><comment><author>ryukafalz</author><text>&amp;gt; We&amp;#x27;re also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds.&lt;p&gt;Thank you! This is something I&amp;#x27;ve wanted in a videoconferencing app since I discovered it was a thing, and I&amp;#x27;m glad it&amp;#x27;s Element doing it.</text></comment>