TECHNICAL FIELD

The present disclosure relates to media container formats such as ISOBMFF for encapsulating encoded audio and/or video content.

BACKGROUND

“Media container” is a term of art in the field of transporting and storage of media data. It refers to a data structure for wrapping up content such as audio and/or video (“AV”) content for transport over a transmission medium or storage on a data storage medium. For instance the transmission medium may comprise a network such as the Internet or a cellular network; or the storage medium may comprise a magnetic storage device such as a hard disk drive, an electronic medium such as a solid state drive or memory stick, or an optical medium such as a CD. A storage device such as a memory stick or removable drive can also be used as a means of transporting data from one device to another.

Audio and video media data is typically formatted in multiple layers for transmission or storage. These may be referred to as protocol layers or layers of formatting. At a first such layer the audio or video content is encoded according to an audio or video encoding scheme, the encoding comprising compression of the content. The encoding scheme may also be referred to as an encoding format. Typically the encoding scheme is a standardized scheme. For instance for audio the encoding scheme may comprise MP3, AC-4, MPEG-D USAC or an MPEG-H audio standard; or for video the encoding may comprise H.264, HEVC, etc. A device, software or specification for encoding and decoding audio or video media according to a given scheme is sometimes referred to as a codec (coder-decoder).

At a second, higher layer, the encoded media content is then encapsulated in a media container according to a media container format (also sometimes called a wrapper format). The media container format specifies metadata for inclusion in the container to wrap up the encapsulated content (the payload of the container). The metadata describes how one or more items of media and/or data are arranged within the container. For instance the container metadata may specify that audio track of encoding format I (e.g. AC-4) starts at byte X for Y bytes, and/or video track of encoding format II (e.g. HEVC) starts at byte V for W bytes, etc. In addition, the metadata typically lists more details on the coded format, such as (in audio) number of channels and sampling rate, or (in video) aspect ratio and number of frames per second, etc. An example of a media container format is ISOBMFF (ISO Base Media File Format), which is standardized by ISO/IEC 14496-12 and which also encompasses a number of extensions thereof. For the present purposes, a reference to ISOBMFF is taken to refer to ISO/IEC 14496-12 or any of the extensions derived therefrom, such as a MP4, 3GP, 3G2, Motion JPEG 2000, etc. Another example of a container format is the MPEG-2 Transport Stream (TS) format.

In the case of transport over a network, one form of transport is streaming. This means the bitstream is played out at the receiving device as-and-when it is received, i.e. so one received part of the stream is being played out while another, later part of the stream is still being downloaded to the receiving device (as opposed to waiting for all the content to download before playing out). For streaming the bitstream is broken down into smaller pieces, sometimes called segments or file chunks. Each such segment or chunk is encapsulated in a media container according to a media container format such as ISOBMFF or MPEG-2 TS (ISO/IEC 13818-1). At a third layer above the second layer (above the container format), there is provided streaming protocol to support the transport of the stream in pieces (e.g. segments or file chunks). The streaming protocol specifies a format for a media presentation description file, wherein this description file comprises pointers to the individual pieces of the stream (e.g. individual segments or file chunks). The description file of the streaming protocol is downloaded to the receiving device upon initiating the streaming, and then used to successively fetch the different chunks of the stream from the transmitting device (e.g. server) in a streamed fashion. The description file may also be downloaded again later during streaming to refresh it upon a certain event, such as a time running out. Examples of streaming protocols include MPEG-DASH (Dynamic Adaptive Streaming over HTTP) and HLS (HTTP Live Streaming).

As well as audio and video content, nowadays an item of media such as streamed TV show or tune can also comprise an embedded data track. This is conventionally used to provide additional content. For instance this could comprise timed-text such as subtitles, additional media such as actor profiles, and/or interactive content. Such data may be embedded in the container of a media container format such as ISOBMFF.

SUMMARY

An issue with media data structures is that there exist different possible variants of the formatting, e.g. different standards of container format or different audio or video encoding schemes. This means that a given media engine at the receive side, depending on its compatibility, may or may not be able to fully process the received data structure in order to play out the audio and/or video content contained therein. In some cases only partial play-out may be possible or in other case no play out may be possible at all. At present, to deal with this, content providers have to create differently formatted versions of the same content in order to anticipate the possibility of different capability of the receive side. This is wasteful in terms of storage or processing resources: either the provider has to pre-generate and store two or more different versions of a given item of content (e.g. a given show or tune), which is wasteful of storage space; or else the provider has to dynamically convert the formatting on the fly as-and-when the resource is requested by the receiver, which incurs a lot of processing resource at the transmit side.

As mentioned, media container files support the ability to include supplementary data as well as AV content. Conventionally this is used to provide additional content such as subtitles, etc. However, the data track could be used to embed other kinds of data, even runnable code such as a script for running in a browser at the receive side (e.g. JavaScript). It is recognized herein that this ability could be used to include code to be run at the receive side for adapting the formatting of the data structure. This can be used to enable a provider to provide a given version of a media data structure to a receiving apparatus that only supports a different formatting. E.g. this can be used to provide one version of a data structure for receipt by different receiving apparatuses with support for different formats.

According to one aspect disclosed herein, there is provided a method of playing out media from a media engine run on a receiving apparatus. The method comprises: at the receiving apparatus, receiving a media data structure comprising audio or video content formatted in a plurality of layers, including at least a first layer comprising the audio or video content encoded according to an audio or video encoding scheme respectively, and a second layer encapsulating the encoded content in one or more media containers according to a media container format. The method further comprises: determining that at least one of said media containers further encapsulates runnable code for processing at least some of the formatting of the media data structure in order to support playout of the audio or video content by the media engine, and running the code on a code engine of the receiving apparatus in order to perform said processing of the media data structure for input to the media engine. A processed version of the data structure, following the processing by at least said code, is input to the media engine in order to play out the media content.

In embodiments, the method may comprise identifying a capability the media engine is lacking for processing the formatting of the media data structure, and said determination may comprise determining that the runnable code encapsulated in the received container file supports said lacking capability. In embodiments, said running of the code may be performed on condition of said identification and determination.

In embodiments the delivered code may replace a corresponding component of the media engine (e.g. a decoder or decapsulating component), or may supplement such a component (e.g. to transcode the encoding scheme or convert the container format).

In embodiments the one or more media containers may comprise a sequence of media containers (e.g. segments or file chunks) each encapsulating a respective portion of the audio or video data, each according to said media container format.

In embodiments the receipt of the data structure may comprise receiving the data structure in a streamed manner.

In embodiments, one or both of the code engine and media engine may be incorporated in a web browser run on the receiving apparatus, or a plug-in to the web browser.

In embodiments, said content may be audio content, in which case said encoding scheme is an audio encoding scheme. In some such embodiments the media data structure may additionally comprise video encoded according to a video coding scheme.

In other embodiments said content may be video content and said encoding scheme may be a video encoding scheme. In some such embodiments the media data structure may additionally comprise audio encoded according to an audio coding scheme.

In embodiments, the media encoding scheme used in the received data structure is an ETSI standard or an MPEG standard.

In embodiments the audio encoding scheme used in the received media data structure may comprise one of:AC-4 (ETSI TS 103 190)AC-3 or EAC-3 (ETSI TS 102 366)MPEG-4 AAC (ISO/IEC 14496-3)MPEG-D USAC (ISO/IEC 23003-3)MPEG-H Audio (ISO/IEC 23008-3)

In embodiments, the container format may be an ISO Base Media File Format (ISOBMFF).

Note that an ISOBMFF file as referred to herein may refer to ISO/IEC 14496-12 or any extension or derivative thereof, such as MP4, 3GP, 3G2 or CMAF.

In embodiments the data structure may be streamed, said one or more media containers comprising a plurality of media containers each encapsulating a portion of the audio or video content according to said media container format. In such embodiments, the layers of formatting in the data structure further may comprise a third layer comprising a media presentation description file pointing to each of the plurality of media containers in the second layer, the media presentation description file being formatted according to a streaming format. For instance the streaming format may comprise MPEG-DASH or HLS.

In embodiments the code may take the form of a script and the code engine may be a script engine, which may be any form of virtual machine or sandboxed execution environment.

In embodiments the code may take the form of JavaScript. Alternatively other languages such as Webassembly could be used.

In embodiments said processing may comprise processing at least part of the formatting of the first layer.

In embodiments, said processing may comprises processing at least part of the formatting of the second layer.

In embodiments, the media engine may comprise no decoder capable of decoding the encoding scheme used in the received media resource; and said processing may comprise either: decoding the content for playout by the media engine, or transcoding the media content to a second encoding scheme which the media engine is capable of decoding.

For example, the second encoding scheme may comprise one of: an AAC standard, MP3, Opus, or a PCM scheme.

In embodiments, the media data structure may comprise multiple audio streams or multiple video streams, and said processing may comprise multiplexing the multiple audio streams into a same audio stream for playout via a same audio buffer of the media engine, or multiplexing the multiple video streams into a same video stream for playout via a same video buffer of the media engine.

In some such embodiments, the streams may take the form of different streams at the first layer (e.g. different substreams or channels), wherein the one or more media containers comprise a sequence of media containers each comprising a portion of each of said streams. In this case said multiplexing comprises multiplexing the portions form the multiple first-layer streams into a single stream for playout via a same audio buffer of the media engine, or similarly for video.

Alternatively the streams may take the form of streams from different sequences of said containers at the second layer. In this case said multiplexing may comprise multiplexing the different sequences into a single stream for playout via a same audio buffer of the media engine, or similarly for video.

In embodiments, for audio or video, the different streams may comprise a base layer stream and at least one enhancement layer stream, and the multiplexing may comprise combining the base layer stream and the at least one enhancement layer stream into one.

In embodiments, the media engine may not be capable of accessing the container format used in the received resource, and said processing may comprise converting to a second container format which the media engine is capable of accessing.

For example, the second container format may be MPEG-2 TS.

In embodiments the processed data structure may be input to the media engine via a web socket.

According to another aspect disclosed herein, there is provided a computer program comprising software embodied on computer-readable storage and configured so as when run on one or more processors of the receiving apparatus to perform a method in accordance with any of the embodiments disclosed herein.

According to another aspect disclosed herein, there is provided a receiving apparatus (e.g. user equipment) for performing the method of any embodiment disclosed herein, the receiving apparatus comprising: an interface for the receipt of the media data structure; a controller configured to perform said determination; the code engine, being arranged to perform said running of the code; and the media engine, being arranged to perform said playout of the media content following said processing.

According to another aspect disclosed herein, there is provided a method of supplying media, the method comprising: outputting a media data structure for receipt by a receiving apparatus, the media data structure comprising audio or video content formatted in a plurality of layers, including at least a first layer encoding the audio or video content according to an audio or video encoding scheme respectively, and a second layer encapsulating the encoded content in one or more media containers according to a media container format; wherein at least one of the media containers further encapsulates runnable code for processing at least some of the formatting of the media data structure in order to support playout of the audio or video content by a media engine of the receiving apparatus.

The method of supplying may be performed by a generator of the media resource or a forwarding provider of the media resource.

According to another aspect disclosed herein, there is provided apparatus (e.g. a server) for generating a media data structure, the apparatus comprising one or more processors and storage storing code arranged to run on at least one of the one or more processors; the code being configured so as when thus run to perform operations of: generating a media data structure comprising audio or video content formatted in a plurality of layers, including at least a first layer encoding the audio or video content according to an audio or video encoding scheme respectively, and a second layer encapsulating the encoded content in one or more media containers according to a media container format; and further encapsulating, in at least one of the media containers, runnable code for processing at least part of the formatting of the media data structure in order to support playout of the audio or video content by a media engine.

In embodiments the outputting or generating of the media data structure may comprise outputting or generating a media resource having any of the properties disclosed from the perspective of the receive-side method, program or apparatus.

According to another aspect disclosed herein, there is provided a media data structure embodied on computer-readable storage, the media data structure comprising: audio or video content formatted in a plurality of layers, including at least a first layer encoding the audio or video content according to an audio or video encoding scheme respectively, and a second layer encapsulating the encoded content in one or more media containers according to a media container format; and encapsulated in at least one of the media containers, runnable code for processing at least part of the formatting of the media data structure in order to support playout of the audio or video content by a media engine

In embodiments the media data structure may have any of the properties disclosed in relation to the method, apparatus or program.

