diff --git a/media-source-respec.html b/media-source-respec.html index c04fcb7..f792442 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -118,7 +118,7 @@ - +
This specification extends {{HTMLMediaElement}} [[HTML]] to allow JavaScript to generate media streams for playback. @@ -127,7 +127,14 @@
-

On top of editorial updates, substantives changes since publication as a W3C Recommendation in November 2016 are the addition of a {{SourceBuffer/changeType()}} method to switch codecs, the possibility to create and use {{MediaSource}} objects off the main thread in dedicated workers, and the removal of the createObjectURL() extension to the {{URL}} object following its integration in the File API [[FILEAPI]]. For a full list of changes done since the previous version, see the commits.

+

On top of editorial updates, substantive changes since publication as a W3C Recommendation in November 2016 are +

+ For a full list of changes done since the previous version, see the commits.

The working group maintains a list of all bug reports that the editors have not yet tried to address.

Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways. Vendors interested in implementing this specification before it eventually reaches the Candidate Recommendation stage should track the GitHub repository and take part in the discussions.

@@ -243,12 +250,22 @@

Definitions

Random Access Point

A position in a [=media segment=] where decoding and continuous playback can begin without relying on any previous data in the segment. For video this tends to be the location of I-frames. In the case of audio, most audio frames can be treated as a random access point. Since video tracks tend to have a more sparse distribution of random access points, the location of these points are usually considered the random access points for multiplexed streams.

-
SourceBuffer byte stream format specification
-

The specific [=byte stream format specification=] that describes the format of the byte stream accepted by a SourceBuffer instance. The - [=byte stream format specification=], for a SourceBuffer object, is initially selected based on the |type:DOMString| passed to the - {{MediaSource/addSourceBuffer()}} call that created the object, and can be updated by {{SourceBuffer/changeType()}} calls on the object.

- -
SourceBuffer configuration
+
SourceBuffer byte stream format specification or WebCodecs chunks buffering expectations
+

+ A {{SourceBuffer}} object can dynamically be configured to either expect to be sent a byte stream via + {{SourceBuffer/appendBuffer()}} that conforms to a specific [=byte stream format specification=], or to be + sent sequences of WebCodecs [[WEBCODECS]] chunks via {{SourceBuffer/appendEncodedChunks()}}. + When configured to expect a byte stream, the SourceBuffer byte + stream format specification is the specific [=byte stream format specification=] that describes the + format of the byte stream expected and accepted by a {{SourceBuffer}} object is initially selected based on + the |type:DOMString| passed to the {{MediaSource/addSourceBuffer()}} call that created the object, and can be + updated by {{SourceBuffer/changeType()}} calls on the object. + When the {{SourceBuffer}} is instead using a WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}} + [[WEBCODECS]] in the most recent of either the {{MediaSource/addSourceBuffer()}} call that created the + {{SourceBuffer}} or a {{SourceBuffer/changeType()}} call, the {{SourceBuffer}} is instead expecting WebCodecs + chunks to be appended via {{SourceBuffer/appendEncodedChunks()}}.

+ +
SourceBuffer track configuration

A specific set of tracks distributed across one or more SourceBuffer objects owned by a single MediaSource instance.

Implementations MUST support at least 1 MediaSource object with the following @@ -323,13 +340,31 @@

MediaSource Object

attribute EventHandler onsourceended; attribute EventHandler onsourceclose; static readonly attribute boolean canConstructInDedicatedWorker; - SourceBuffer addSourceBuffer (DOMString type); - undefined removeSourceBuffer (SourceBuffer sourceBuffer); - undefined endOfStream (optional EndOfStreamError error); - undefined setLiveSeekableRange (double start, double end); - undefined clearLiveSeekableRange (); - static boolean isTypeSupported (DOMString type); -}; + + // "optional" with default empty SourceBufferConfig dictionary is used here + // to pass WebIDL verification. Behavior of the method enforces that either + // a valid mime-type string or a valid SourceBufferConfig are provided. + SourceBuffer addSourceBuffer (optional TypeOrConfig typeOrConfig = {}); + + undefined removeSourceBuffer (SourceBuffer sourceBuffer); + undefined endOfStream (optional EndOfStreamError error); + undefined setLiveSeekableRange (double start, double end); + undefined clearLiveSeekableRange (); + static boolean isTypeSupported (DOMString type); +}; + +dictionary SourceBufferConfig { + // Precisely one of these WebCodecs config objects must be populated to + // signal via addSourceBuffer or changeType the intent to buffer the + // corresponding WebCodecs media. + AudioDecoderConfig audioConfig; // For appending EncodedAudioChunks + VideoDecoderConfig videoConfig; // For appending EncodedVideoChunks +}; + +// This typedef simplifies the syntax for overloading addSourceBuffer and +// changeType to receive either a mime-type string or a WebCodecs config. +typedef (DOMString or SourceBufferConfig) TypeOrConfig; +

Attributes

sourceBuffers of type SourceBufferList, readonly
@@ -362,7 +397,7 @@

Attributes

  • If the {{SourceBuffer/updating}} attribute equals true on any SourceBuffer in {{MediaSource/sourceBuffers}}, then throw an {{InvalidStateError}} exception and abort these steps.
  • Run the [=duration change=] algorithm with |new duration:unrestricted double| set to the value being assigned to this attribute.

    The [=duration change=] algorithm will adjust |new duration| higher if there is any currently buffered coded frame with a higher end time.

    -

    {{SourceBuffer/appendBuffer()}} and {{MediaSource/endOfStream()}} can update the duration under certain circumstances.

    +

    {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/appendEncodedChunks()}} and {{MediaSource/endOfStream()}} can update the duration under certain circumstances.

    • onsourceopen of type {{EventHandler}}
    @@ -378,25 +413,66 @@

    Attributes

    creating and using a {{MediaSource}} object in a dedicated worker, and mitigates the need for higher latency detection polyfills like attempting creation of a {{MediaSource}} object from a dedicated worker, especially if the feature is not supported.

    -

    Methods

    addSourceBuffer
    +
    +

    Methods

    +
    addSourceBuffer

    Adds a new SourceBuffer to {{MediaSource/sourceBuffers}}.

      -
    1. If |type:DOMString| is an empty string then throw a {{TypeError}} exception and abort these steps.
      2. -
    2. If |type| contains a MIME type that is not supported or contains a MIME type that is not supported with the types specified for the other SourceBuffer objects in {{MediaSource/sourceBuffers}}, then throw a {{NotSupportedError}} exception and abort these steps.
      3. -
    3. If the user agent can't handle any more SourceBuffer objects or if creating a SourceBuffer - based on |type| would result in an unsupported [=SourceBuffer configuration=], - then throw a {{QuotaExceededError}} exception and abort these steps. +
    4. If |typeOrConfig:TypeOrConfig| is a DOMString, then run the following steps: +
        +
      1. If |typeOrConfig| is empty then throw a {{TypeError}} exception and abort these steps and + this method.
        2. +
      2. If |typeOrConfig| contains a MIME type that is not supported or contains a MIME type that is not + supported with the types specified for the other {{SourceBuffer}} objects in + {{MediaSource/sourceBuffers}}, then throw a {{NotSupportedError}} exception and abort these + steps and this method.
        3. +
      +
      5. +
    5. If |typeOrConfig| is not a DOMString, then run the following steps: +
        +
      1. If |typeOrConfig| has both an {{SourceBufferConfig/audioConfig}} and a + {{SourceBufferConfig/videoConfig}} or has neither, then throw a {{TypeError}} exception and abort + these steps and this method.
        2. +
      2. If |typeOrConfig| has neither a neither a [=valid AudioDecoderConfig=] in + {{SourceBufferConfig/audioConfig}} nor a [=valid VideoDecoderConfig=] in + {{SourceBufferConfig/videoConfig}}, then throw a {{TypeError}} exception and abort these steps and + this method.
        3. +
      3. If |typeOrConfig| contains a decoder configuration that is not supported or is not supported with + the types specified for the other {{SourceBuffer}} objects in {{MediaSource/sourceBuffers}}, then + throw a {{NotSupportedError}} exception and abort these steps and this method.
        4. +
      +
      6. +
    6. If the user agent can't handle any more {{SourceBuffer}} objects or if creating a {{SourceBuffer}} based + on |typeOrConfig| would result in an unsupported [=SourceBuffer track configuration=], then throw a + {{QuotaExceededError}} exception and abort these steps.

      For example, a user agent MAY throw a {{QuotaExceededError}} exception if the media element has reached the {{HTMLMediaElement/HAVE_METADATA}} readyState. This can occur if the user agent's media engine does not support adding more tracks during playback.

    7. If the {{MediaSource/readyState}} attribute is not in the {{ReadyState/""open""}} state then throw an {{InvalidStateError}} exception and abort these steps.
      8. -
    8. Create a new SourceBuffer object and associated resources.
      9. -
    9. Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the "Generate - Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is associated with - |type|. -
    10. +
    11. Create a new {{SourceBuffer}} object and associated resources.
      12. +
    12. +
      +
      If |typeOrConfig| is a DOMString:
      +
      Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the + "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is + associated with |typeOrConfig|. +
      +
      Otherwise:
      +
        +
      1. Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to false. +
        WebCodecs encoded chunks are required to have timestamps, so there is no need to + generate them.
        +
        2. +
      2. Copy the {{SourceBufferConfig}} from |typeOrConfig| into the new object's {{SourceBuffer/[[input + webcodecs configs and chunks]]}} for potential handling later during that object's [=chunks append=] + algorithm.
        3. +
      +
      +
      +
      13. +
    13. If the {{SourceBuffer/[[generate timestamps flag]]}} equals true:
      @@ -413,7 +489,9 @@

      Attributes

    14. Add the new object to {{MediaSource/sourceBuffers}} and [=queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/sourceBuffers}}.
    15. Return the new object.
    -
    ParameterTypeNullableOptionalDescription
    |type|{{DOMString}}✘✘
    Return type: SourceBuffer
    + + +
    ParameterTypeNullableOptionalDescription
    |typeOrConfig|{{TypeOrConfig}}✘✘
    Return type: SourceBuffer
    removeSourceBuffer

    Removes a {{SourceBuffer}} from {{MediaSource/sourceBuffers}}.

    @@ -423,9 +501,20 @@

    Attributes

  • If the |sourceBuffer|.{{SourceBuffer/updating}} attribute equals true, then run the following steps:
    1. Abort the [=buffer append=] algorithm if it is running.
      2. +
    2. Abort the [=chunks append=] algorithm if it is running.
    3. Set the |sourceBuffer|.{{SourceBuffer/updating}} attribute to false.
      4. -
    4. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
      5. -
    5. [=Queue a task=] to [=fire an event=] named {{updateend}} at |sourceBuffer|.
      6. +
    6. +
      If the |sourceBuffer| has a {{Promise}} in {{SourceBuffer/[[pending append chunks + promise]]}}:
      +
      Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset + {{SourceBuffer/[[pending append chunks promise]]}}.
      +
      Otherwise:
      +
        +
      1. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
        2. +
      2. [=Queue a task=] to [=fire an event=] named {{updateend}} at |sourceBuffer|.
        3. +
      +
      +
    • @@ -680,6 +769,11 @@

    Attributes

    This method returning true implies that HTMLMediaElement.canPlayType() will return "maybe" or "probably" since it does not make sense for a MediaSource to support a type the HTMLMediaElement knows it cannot play.

    +

    + For proactively checking support for a WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}}, use the + respective {{AudioDecoder}} {{AudioDecoder/isConfigSupported()}} or {{VideoDecoder}} + {{VideoDecoder/isConfigSupported()}} method. +

    ParameterTypeNullableOptionalDescription
    |type|{{DOMString}}✘✘
    Return type: {{boolean}}
    @@ -802,7 +896,7 @@

    Attaching to a media element

  • Continue the by running the remaining steps, with these clarifications:
      -
    1. Text in the or the that refers to "the download", "bytes received", or "whenever new data for the current media resource becomes available" refers to data passed in via {{SourceBuffer/appendBuffer()}}.
      2. +
    2. Text in the or the that refers to "the download", "bytes received", or "whenever new data for the current media resource becomes available" refers to data passed in via {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/appendEncodedChunks()}}.
    3. References to HTTP in the and the do not apply because the HTMLMediaElement does not fetch media data via HTTP when a MediaSource is attached.
    • @@ -862,7 +956,7 @@

    Seeking

    to {{HTMLMediaElement/HAVE_METADATA}}.

    Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

    -
  • The media element waits until an {{SourceBuffer/appendBuffer()}} call causes the [=coded frame processing=] algorithm to set +
  • The media element waits until an {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/appendEncodedChunks()}} call causes the [=coded frame processing=] algorithm to set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to a value greater than {{HTMLMediaElement/HAVE_METADATA}}.

    The web application can use {{SourceBuffer/buffered}} and {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to determine what the media element needs to resume playback.

    • @@ -1128,17 +1222,28 @@

    SourceBuffer Object

    enum AppendMode {
     "segments",
     "sequence"
-};
    - - + + @@ -1408,9 +1637,9 @@

    SourceBuffer Object

    Track Buffers

    -

    A track buffer stores the [=track descriptions=] and [=coded frames=] for an individual - track. The track buffer is updated as [=initialization segments=] and [=media segments=] are appended to the - SourceBuffer.

    +

    A track buffer stores the [=track descriptions=] and [=coded frames=] for an + individual track. The track buffer is updated as [=initialization segments=], [=media segments=], decoder + configurations and {{EncodedChunks}} are given to the {{SourceBuffer}}.

    Each [=track buffer=] has a last decode timestamp variable that stores the decode timestamp of the last [=coded frame=] appended in the current [=coded frame group=]. The variable is initially @@ -1516,8 +1745,9 @@

    Segment Parser Loop

    empty when the {{SourceBuffer}} object is created.

    Each {{SourceBuffer}} object has a [[\buffer full flag]] internal - slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes. It is set - to false when the {{SourceBuffer}} object is created and gets updated as data is appended and removed.

    + slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes or + {{SourceBuffer/appendEncodedChunks()}} is allowed to accept more encoded chunks. It is set to false when the + {{SourceBuffer}} object is created and gets updated as data is appended and removed.

    Each {{SourceBuffer}} object has a [[\group start timestamp]] internal slot that keeps track of the starting timestamp for a new [=coded frame group=] in the @@ -1544,6 +1774,7 @@

    Segment Parser Loop

    This flag is set by {{MediaSource/addSourceBuffer()}} when the {{SourceBuffer}} object is created and is updated by {{SourceBuffer/changeType()}}.

    +

    When the segment parser loop algorithm is invoked, run the following steps:

      @@ -1619,6 +1850,15 @@

      Reset Parser State

    1. If the {{SourceBuffer/mode}} attribute equals {{AppendMode/""sequence""}}, then set the {{SourceBuffer/[[group start timestamp]]}} to the {{SourceBuffer/[[group end timestamp]]}}
    2. Remove all bytes from the {{SourceBuffer/[[input buffer]]}}.
      3. +
    3. If {{SourceBuffer/[[input webcodecs configs and chunks]]}} contains any {{EncodedChunks}}, remove them + from {{SourceBuffer/[[input webcodecs configs and chunks]]}}, but retain any {{AudioDecoderConfig}} or + {{VideoDecoderConfig}} that may be in that internal slot. +
      Keeping an unprocessed WebCodecs config in this internal slot lets future + {{SourceBuffer/appendEncodedChunks()}} reuse the config in the [=chunks append=] algorithm because + there is no other way to retain such an unprocessed config, except perhaps by the app calling + {{SourceBuffer/changeType()}} with the config again. +
      +
    4. Set {{SourceBuffer/[[append state]]}} to [=WAITING_FOR_SEGMENT=].
    @@ -1629,8 +1869,18 @@

    Append Error

    1. Run the [=reset parser state=] algorithm.
    2. Set the {{SourceBuffer/updating}} attribute to false.
      3. -
    3. [=Queue a task=] to [=fire an event=] named {{error}} at this SourceBuffer object.
      4. -
    4. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
      5. +
    5. +
      If this {{SourceBuffer}} has a {{Promise}} in {{SourceBuffer/[[pending append chunks + promise]]}}:
      +
      Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[pending + append chunks promise]]}}.
      +
      Otherwise:
      +
        +
      1. [=Queue a task=] to [=fire an event=] named {{error}} at this {{SourceBuffer}} object.
        2. +
      2. [=Queue a task=] to [=fire an event=] named {{updateend}} at this {{SourceBuffer}} object.
        3. +
      +
      +
    6. Run the [=end of stream=] algorithm with the |error:EndOfStreamError| parameter set to {{EndOfStreamError/""decode""}}.
    @@ -1678,6 +1928,13 @@

    Prepare Append

    Buffer Append

    When {{SourceBuffer/appendBuffer()}} is called, the following steps are run to process the appended data.

      +
    1. If this {{SourceBuffer}} is currently configured to expect processing of WebCodecs {{EncodedChunks}}, + then run the [=append error=] algorithm and abort this algorithm. +
      The expectation of processing an appended bytestream versus appended {{EncodedChunks}} + is based on the most recently successful execution of the initial {{MediaSource/addSourceBuffer()}} that + created this {{SourceBuffer}} or potentially a more recent {{SourceBuffer/changeType()}} call that may + have changed the expectation.
      +
    2. Run the [=segment parser loop=] algorithm.
    3. If the [=segment parser loop=] algorithm in the previous step was aborted, then abort this algorithm.
    4. Set the {{SourceBuffer/updating}} attribute to false.
      5. @@ -1686,6 +1943,82 @@

      Buffer Append

    +
    +

    Chunks Append

    +

    Each {{SourceBuffer}} object has a [[\pending append chunks + promise]] internal slot that stores the promise necessary to communicate the completion of + asynchronous steps begun by a call to {{SourceBuffer/appendEncodedChunks()}}. If there is no asynchronous + {{SourceBuffer/appendEncodedChunks()}} operation in progress, then this slot is unset.

    + +

    Each {{SourceBuffer}} object has an [[\input webcodecs configs and + chunks]] internal slot that stores a queue of unprocessed {{EncodedChunks}} and at most one + {{SourceBufferConfig}}. The {{SourceBufferConfig}} is added if the initial call to + {{MediaSource/addSourceBuffer()}} that created this object was provided a {{SourceBufferConfig}}, or if + there is a more recent successful call to {{SourceBuffer/changeType()}} on this {{SourceBuffer}} that + provided a {{SourceBufferConfig}}. The contents of this slot are processed by the [=chunks append=] + algorithm to buffer the necessary decoder configuration and coded frames into the underlying [=track + buffer=].

    + +

    When the chunks append algorithm is invoked, run the following steps to process the appended + {{EncodedChunks}} relative to the current {{SourceBufferConfig}}:

    +
      +
    1. If this {{SourceBuffer}} is currently configured to expect processing of an appended bytestream, + then run the [=append error=] algorithm and abort this algorithm. +
      The expectation of processing an appended bytestream versus appended {{EncodedChunks}} + is based on the most recently successful execution of the initial {{MediaSource/addSourceBuffer()}} that + created this {{SourceBuffer}} or potentially a more recent {{SourceBuffer/changeType()}} call that may + have changed the expectation.
      +
      2. +
    2. If there is a {{SourceBufferConfig}} in this {{SourceBuffer}}'s {{SourceBuffer/[[input webcodecs configs + and chunks]]}}, then run the following steps: +
        +
      1. Let |config:SourceBufferConfig| be the {{SourceBufferConfig}} from {{SourceBuffer/[[input webcodecs + configs and chunks]]}}. +
        There must be precisely one {{AudioDecoderConfig}} or {{VideoDecoderConfig}} in + |config|; this condition was already enforced by the {{MediaSource/addSourceBuffer()}} or + {{SourceBuffer/changeType()}} call that placed |config| into {{SourceBuffer/[[input webcodecs + configs and chunks]]}}.
        +
        2. +
      2. Remove the {{SourceBufferConfig}} from {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
        3. +
      3. Run the [=initialization segment received=] algorithm with |config| denoting a single track + configuration. If that algorithm aborts, then abort this algorithm.
        4. +
      +
      3. +
    3. In the order in which they were added to {{SourceBuffer/[[input webcodecs configs and chunks]]}}, + prepare the {{EncodedChunks}} for coded frame processing by running the following steps for each chunk: +
        +
      1. Let |chunk:EncodedAudioChunk or EncodedVideoChunk| be the next encoded chunk being prepared, + dequeued (removed) from the front of {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
        2. +
      2. If |chunk| is an {{EncodedAudioChunk}} but the most recently processed |config| by this object was a + {{VideoDecoderConfig}}, or if |chunk| is an {{EncodedVideoChunk}} but the most recently processed + |config| by this object was an {{AudioDecoderConfig}}, then run the [=append error=] algorithm and + abort this algorithm. +
        3. +
      3. Create a coded frame from |chunk|. Treat the frame as a random access point iff |chunk|'s type + is "key". Set the frame's decode timestamp to be 0. +
        Using a constant decode timestamp disables the [=coded frame processing=] + algorithm's discontinuity detection. Note that other mechanisms for signalling discontinuous + sequences of frames survive this simplification, with the most reliable yet lightweight being calling + {{SourceBuffer/abort()}} when {{SourceBuffer/updating}} is false. A more heavyweight option would be + to call {{SourceBuffer/changeType()}}. Also note that, though WebCodecs does not define a decode + timestamp attribute for encoded chunks, if reliable buffering of chunks into MSE needs real decode + timestamps, this spec may be refined to improve that support. +
        +
        4. +
      +
      4. +
    4. Run the [=coded frame processing=] algorithm with the frames in order from the previous step. +
      This algorithm, unlike the [=segment parser loop=] algorithm, deterministically + consumes and clears all of its input state and processes all corresponding coded frames. Even the steps + which might abort this algorithm early do so after the [=append error=] algorithm clears any remaining + input state and rejects the promise. +
      +
      5. +
    5. Set the {{SourceBuffer/updating}} attribute to false. +
    6. Resolve the {{SourceBuffer/[[pending append chunks promise]]}} and unset that internal slot.
      7. +
    +
    +

    Range Removal

    Follow these steps when a caller needs to initiate a JavaScript visible range removal @@ -1705,7 +2038,8 @@

    Range Removal

    Initialization Segment Received

    -

    The following steps are run when the [=segment parser loop=] successfully parses a complete [=initialization segment=]:

    +

    The following steps are run when the [=segment parser loop=] successfully parses a complete + [=initialization segment=] or the [=chunks append=] algorithm handles a {{SourceBufferConfig}}:

    Each SourceBuffer object has a [[\first initialization segment received flag]] internal slot that tracks whether the first [=initialization segment=] has been appended and received by this algorithm. This @@ -2066,10 +2400,11 @@

    Initialization Segment Received

    Coded Frame Processing

    -

    When complete [=coded frames=] have been parsed by the [=segment parser loop=] then the following steps are run:

    +

    When complete [=coded frames=] have been parsed by the [=segment parser loop=] or emitted by the [=chunks + append=] algorithm, then the following steps are run:

    1. -

      For each [=coded frame=] in the [=media segment=] run the following steps:

      +

      For each [=coded frame=] run the following steps:

      1. Loop Top:
        If {{SourceBuffer/[[generate timestamps flag]]}} equals true:
        @@ -2256,7 +2591,7 @@

        Coded Frame Processing

        If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_FUTURE_DATA}} and the new [=coded frames=] cause {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to have a {{TimeRanges}} that includes the current playback position and [=enough data to ensure uninterrupted playback=], then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_ENOUGH_DATA}}.

        Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

        2. -
      2. If the [=media segment=] contains data beyond the current {{MediaSource/duration}}, then run the +
      3. If the result of coded frame processing buffers data beyond the current {{MediaSource/duration}}, then run the [=duration change=] algorithm with |new duration:unrestricted double| set to the maximum of the current duration and the {{SourceBuffer/[[group end timestamp]]}}.
      @@ -2319,9 +2654,11 @@

      Coded Frame Removal

      Coded Frame Eviction

      -

      This algorithm is run to free up space in this {{SourceBuffer}} when new data is appended.

      +

      This algorithm is run to free up space in this {{SourceBuffer}} when new data is appended. New data is + either the bytes being appended via {{SourceBuffer/appendBuffer()}} or the WebCodecs {{EncodedChunks}} being + appended via {{SourceBuffer/appendEncodedChunks()}}.

        -
      1. Let |new data:BufferSource| equal the data that is about to be appended to this SourceBuffer. +
      2. Let |new data:BufferSource or EncodedChunks| equal the data that is about to be appended to this SourceBuffer.

        Need to recognize step here that implementations MAY decide to set {{SourceBuffer/[[buffer full flag]]}} true here if it predicts that processing |new data| in addition to any existing bytes in @@ -2809,6 +3146,10 @@

        Byte Stream Formats

        The byte stream format specifications in the registry are not intended to define new storage formats. They simply outline the subset of existing storage format structures that implementations of this specification will accept.

        Byte stream format parsing and validation is implemented in the [=segment parser loop=] algorithm.

        +

        When currently configured by {{MediaSource/addSourceBuffer()}} or {{SourceBuffer/changeType()}} to + expect appends of WebCodecs encoded chunks via {{SourceBuffer/appendEncodedChunks()}} instead of a bytestream + via {{SourceBuffer/appendBuffer}}, the {{SourceBuffer}} does not have a specific bytestream format + associated.

        This section provides general requirements for all byte stream format specifications:

          @@ -2882,9 +3223,12 @@

          Byte Stream Formats

          Examples

          Example use of the Media Source Extensions

          -
          -
          - 
          <script>
+      
          
+        
          Buffering bytes from a bytestream format
          
+        
          
+          
          
+            
          +<script>
   function onSourceOpen(videoTag, e) {
     var mediaSource = e.target;
 
@@ -2973,9 +3317,98 @@ 
          Examples
          
   mediaSource.addEventListener('sourceopen', onSourceOpen.bind(this, video));
   video.src = window.URL.createObjectURL(mediaSource);
 </script>
-          
          
+
          +
          -
        +
      +
      +

      Buffering WebCodecs encoded chunks

      +
      +
      + 
      +<body>
+<script>
+  // Very simple demuxer bound to a specific media file.
+  // Replace with other source of WebCodecs configs and encoded chunks.
+  let next_chunk_index = 0;
+  let buffer;
+  let chunk_duration = 100 * 1000;  // 100 milliseconds for this example.
+  let metadata = [
+    {
+      offset: /* chunk offset, e.g. 42 */,
+      size: /* chunk size, e.g. 4242 */,
+      type: /* chunk type, e.g. "key" or "delta" */
+    }, // Plus additional chunks' metadata listed here.
+  ];
+
+  function getConfig() {
+    return( /* e.g. { videoConfig: { codec: "vp09.00.10.08" } } */ );
+  }
+
+  async function getNextEncodedChunk() {
+    if (next_chunk_index >= metadata.length)
+      return null;
+    if (next_chunk_index == 0)
+      buffer = await (await fetch(/* e.g. "vp9.chunks" */)).arrayBuffer();
+    let chunk_metadata = metadata[next_chunk_index];
+    let chunk_timestamp = chunk_duration * next_chunk_index;
+    next_chunk_index++;
+    // EncodedAudioChunks could also be buffered into MSE, but only one form of
+    // media (audio versus video) can be buffered into a specific SourceBuffer,
+    // so this simple example is video-only.
+    return new EncodedVideoChunk( {
+      type: chunk_metadata.type,
+      timestamp: chunk_timestamp,
+      duration: chunk_duration,
+      data: new Uint8Array(buffer, chunk_metadata.offset, chunk_metadata.size)
+    } );
+  }
+
+  // MSE player using the WebCodecs content generated, above.
+  async function getOpenMediaSource() {
+    return new Promise(async (resolve, reject) => {
+      const v = document.createElement("video");
+      document.body.appendChild(v);
+      const mediaSource = new MediaSource();
+      const url = URL.createObjectURL(mediaSource);
+      mediaSource.addEventListener("sourceopen", () => {
+        URL.revokeObjectURL(url);
+        if (mediaSource.readyState != "open")
+          reject();
+        else
+          resolve([ v, mediaSource ]);
+      }, { once: true });
+      v.src = url;
+    });
+  }
+
+  async function bufferMedia() {
+    let [ videoElement, mediaSource ] = await getOpenMediaSource();
+    let sourceBuffer = mediaSource.addSourceBuffer(await getConfig());
+
+    // This simple player attempts to buffer everything immediately. Full
+    // players will condition buffering, for example, on playback progress
+    // and availability of chunks.
+    while (null != (chunk = await(getNextEncodedChunk()))) {
+      // Note that appending a sequence instead of a single chunk here
+      // could be more performant.
+      await sourceBuffer.appendEncodedChunks(chunk);
+    }
+
+    mediaSource.endOfStream();
+    videoElement.controls = true;  // Show default controls.
+  }
+
+  if (!SourceBuffer.prototype.hasOwnProperty("appendEncodedChunks"))
+    console.log("MSE-for-WebCodecs support is missing.")
+  else
+    bufferMedia();
+</script>
+</body>
+
      +
      +
      +
    Enumeration description
    segments -

    The timestamps in the media segment determine where the [=coded frames=] are placed in the presentation. Media segments can be appended in any order.

    -
    sequence -

    Media segments will be treated as adjacent in time independent of the timestamps in the media segment. Coded frames in a new media segment will be placed immediately after the coded - frames in the previous media segment. The {{SourceBuffer/timestampOffset}} attribute will be updated if a new offset is needed to make the new media segments adjacent to the previous media segment. - Setting the {{SourceBuffer/timestampOffset}} attribute in {{AppendMode/""sequence""}} mode allows a media segment to be placed at a specific position in the timeline without any knowledge - of the timestamps in the media segment. -

    +}; + +
    Enumeration description
    segments +

    The timestamps in the media segment or {{EncodedChunks}} determine where the [=coded frames=] are placed in + the presentation. Media segments can be appended in any order. Discontinuous {{EncodedChunks}} should not be + appended in the same {{SourceBuffer/appendEncodedChunks()}} call: the application should instead wait for the + {{SourceBuffer}} to no longer be {{SourceBuffer/updating}}, and then call {{SourceBuffer/abort()}} or + {{SourceBuffer/changeType()}} before appending {{EncodedChunks}} that are discontinuous with those in the + previous {{SourceBuffer/appendEncodedChunks()}} call.

    +
    sequence +

    Media segments will be treated as adjacent in time independent of the timestamps in the media segment. Coded + frames in a new media segment will be placed immediately after the coded frames in the previous media segment. + As with {{AppendMode/""segments""}} mode, discontinuous {{EncodedChunks}} should not be appended in the same + {{SourceBuffer/appendEncodedChunks()}} call. + The {{SourceBuffer/timestampOffset}} attribute will be updated if a new offset is needed to make the new media + segments adjacent to the previous media segment. Setting the {{SourceBuffer/timestampOffset}} attribute in + {{AppendMode/""sequence""}} mode allows a media segment to be placed at a specific position in the timeline + without any knowledge of the timestamps in the media segment.

    -
    [Exposed=(Window,DedicatedWorker)]
+
    +[Exposed=(Window,DedicatedWorker)]
 interface SourceBuffer : EventTarget {
                     attribute AppendMode          mode;
     readonly        attribute boolean             updating;
@@ -1155,13 +1260,30 @@ 
    SourceBuffer Object
    
                     attribute EventHandler        onerror;
                     attribute EventHandler        onabort;
     undefined appendBuffer (BufferSource data);
+    Promise < undefined > appendEncodedChunks(EncodedChunks chunks);
     undefined abort ();
-    undefined changeType (DOMString type);
+
+    // "optional" with default empty SourceBufferConfig dictionary is used here
+    // to pass WebIDL verification. Behavior of the method enforces that either
+    // a valid mime-type string or a valid SourceBufferConfig are provided.
+    undefined changeType (optional TypeOrConfig typeOrConfig = {});
+
     undefined remove (double start, unrestricted double end);
-};
    
+};
+
+// This typedef simplifies the syntax for describing either a single audio or
+// video chunk, or a sequence of chunks. While this typedef does not provide the
+// ability to enforce that all members of a sequence of chunks are the same kind
+// (audio versus video), the behavior of the appendEncodedChunks method enforces
+// that constraint: the implementation MUST reject a sequence containing both
+// audio and video chunks.
+typedef (sequence< (EncodedAudioChunk or EncodedVideoChunk) >
+            or EncodedAudioChunk or EncodedVideoChunk) EncodedChunks;
+
    [[HTML]] {{AudioTrackList}}, {{VideoTrackList}} and {{TextTrackList}} need Window+DedicatedWorker exposure.
    -

    Attributes

    mode of type AppendMode
    +

    Attributes

    +
    mode of type AppendMode

    Controls how a sequence of [=media segments=] are handled. This attribute is initially set by {{MediaSource/addSourceBuffer()}} after the object is created, and can be updated by {{SourceBuffer/changeType()}} or setting this attribute.

    On getting, Return the initial value or the last value that was successfully set.

    On setting, run the following steps:

    @@ -1185,7 +1307,7 @@

    SourceBuffer Object

  • Update the attribute to |new mode|.
    • updating of type {{boolean}}, readonly
    -

    Indicates whether the asynchronous continuation of an {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/remove()}} +

    Indicates whether the asynchronous continuation of an {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/appendEncodedChunks()}} or {{SourceBuffer/remove()}} operation is still being processed. This attribute is initially set to false when the object is created.

    buffered of type {{TimeRanges}}, readonly

    Indicates what {{TimeRanges}} are buffered in the SourceBuffer. This @@ -1211,7 +1333,7 @@

    SourceBuffer Object

  • Return the current value of this attribute.
    • timestampOffset of type {{double}}
    -

    Controls the offset applied to timestamps inside subsequent [=media segments=] that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.

    +

    Controls the offset applied to timestamps inside subsequent [=media segments=] or {{EncodedChunks}} that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.

    On getting, Return the initial value or the last value that was successfully set.

    On setting, run the following steps:

      @@ -1272,17 +1394,68 @@

      SourceBuffer Object

      The event handler for the {{error}} event.

    onabort of type {{EventHandler}}

    The event handler for the {{abort}} event.

    -

    Methods

    appendBuffer
    +
    +

    Methods

    +
    appendBuffer
    +

    Appends the segment data in an BufferSource[[!WEBIDL]] to the {{SourceBuffer}}.

      -
    1. Run the [=prepare append=] algorithm.
      2. +
    2. Run the [=prepare append=] algorithm. If it throws exception and aborts, then these steps are also + aborted.
    3. Add |data:BufferSource| to the end of the {{SourceBuffer/[[input buffer]]}}.
    4. Set the {{SourceBuffer/updating}} attribute to true.
    5. [=Queue a task=] to [=fire an event=] named {{updatestart}} at this SourceBuffer object.
      6. -
    6. Asynchronously run the [=buffer append=] algorithm.
      7. +
    7. Asynchronously run the [=buffer append=] algorithm. +
      If this {{SourceBuffer}} is currently configured by the most recent + of the {{MediaSource/addSourceBuffer()}} call that created this object or a more recent + {{SourceBuffer/changeType()}} call to expect to buffer WebCodecs {{EncodedChunks}} via + {{SourceBuffer/appendEncodedChunks()}}, the asynchronous [=buffer append=] algorithm will detect this and + trigger the [=append error=] algorithm.
      +
    -
    ParameterTypeNullableOptionalDescription
    |data|{{BufferSource}}✘✘
    Return type: {{undefined}}
    abort
    +
    ParameterTypeNullableOptionalDescription
    |data|{{BufferSource}}✘✘
    Return type: {{undefined}}
    + +
    appendEncodedChunks
    +
    +

    Appends WebCodecs [[WEBCODECS]] {{EncodedChunks}} to the {{SourceBuffer}}.

    + +
      +
    1. Run the [=prepare append=] algorithm. If it throws exception and aborts, then these steps are also + aborted.
      2. +
    2. If any of the chunks in |chunks:EncodedChunks| have a null value for their duration attribute, throw a + {{TypeError}} exception and abort these steps. +
      This requirement is necessary to understand all chunks' presentation intervals in + the coded frame processing algorithm, enabling correct buffering when frames overlap other frames, are later + overlapped by other frames, or are in a disjoint portion of the presentation timeline. +
      +
      3. +
    3. Add |chunks| to the end of {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
      4. +
    4. Set the {{SourceBuffer/updating}} attribute to true. +
      This promise-based method differs from the asynchronous event-signalling + {{SourceBuffer/appendBuffer()}} method: promise resolution or rejection is the way to detect the + analogues of "abort", "error", "updatestart", "update", and "updateend" events that are not enqueued + by {{SourceBuffer/appendEncodedChunks()}}.
      +
      5. +
    5. Let |promise:Promise| be a new {{Promise}}.
      6. +
    6. Set {{SourceBuffer/[[pending append chunks promise]]}} to be |promise|.
      7. +
    7. Return |promise| to the caller and run the rest of these steps asynchronously.
      8. +
    8. Run the [=chunks append=] algorithm. +
      If this {{SourceBuffer}} is currently configured by the most recent + of the {{MediaSource/addSourceBuffer()}} call that created this object or a more recent + {{SourceBuffer/changeType()}} call to expect to buffer bytes parsed from a {{BufferSource}} via + {{SourceBuffer/appendBuffer()}}, the asynchronous [=chunks append=] algorithm will detect this and + trigger the [=append error=] algorithm.
      +
      9. +
    + + +
    ParameterTypeNullableOptionalDescription
    |chunks|{{EncodedChunks}}✘✘
    +
    Return type: {{Promise}} < {{undefined}} >
    +
    + + +
    abort

    Aborts the current segment and resets the segment parser.

      @@ -1292,9 +1465,20 @@

      SourceBuffer Object

    1. If the {{SourceBuffer/updating}} attribute equals true, then run the following steps:
      1. Abort the [=buffer append=] algorithm if it is running.
        2. +
      2. Abort the [=chunks append=] algorithm if it is running.
      3. Set the {{SourceBuffer/updating}} attribute to false.
        4. -
      4. [=Queue a task=] to [=fire an event=] named {{abort}} at this SourceBuffer object.
        5. -
      5. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
        6. +
      6. +
        If this object has a {{Promise}} in {{SourceBuffer/[[pending append chunks promise]]}}:
        +
        Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset + {{SourceBuffer/[[pending append chunks promise]]}}.
        +
        Otherwise:
        +
          +
        1. [=Queue a task=] to [=fire an event=] named {{abort}} at this {{SourceBuffer}} object.
          2. +
        2. [=Queue a task=] to [=fire an event=] named {{updateend}} at this {{SourceBuffer}} + object.
          3. +
        +
        +
    2. Run the [=reset parser state=] algorithm.
      3. @@ -1303,12 +1487,38 @@

      SourceBuffer Object

    No parameters.
    Return type: {{undefined}}
    changeType
    -

    Changes the MIME type associated with this object. Subsequent {{SourceBuffer/appendBuffer()}} calls will expect the newly appended bytes to conform to the new type. +

    Changes the [=SourceBuffer byte stream format specification or WebCodecs chunks buffering expectations=] for + this object. Enables switching among bytestreams, codecs, or even between bytestream parsing versus + buffering of WebCodecs encoded chunks in the same {{SourceBuffer}}.

      -
    1. If |type:DOMString| is an empty string then throw a {{TypeError}} exception and abort these steps.
      2. -
    2. If this object has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw an {{InvalidStateError}} exception and abort these steps.
      3. -
    3. If the {{SourceBuffer/updating}} attribute equals true, then throw an {{InvalidStateError}} exception and abort these steps.
      4. -
    4. If |type| contains a MIME type that is not supported or contains a MIME type that is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects in the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a {{NotSupportedError}} exception and abort these steps.
      5. +
    5. If |typeOrConfig:TypeOrConfig| is an empty DOMString, then throw a {{TypeError}} exception and abort + these steps.
      6. +
    6. If |typeOrConfig| is not a DOMString, then run the following steps: +
        +
      1. If |typeOrConfig| has both an {{SourceBufferConfig/audioConfig}} and a + {{SourceBufferConfig/videoConfig}} or has neither, then throw a {{TypeError}} exception and abort + these steps and this method.
        2. +
      2. If |typeOrConfig| has neither a neither a [=valid AudioDecoderConfig=] in + {{SourceBufferConfig/audioConfig}} nor a [=valid VideoDecoderConfig=] in + {{SourceBufferConfig/videoConfig}}, then throw a {{TypeError}} exception and abort these steps and + this method.
        3. +
      +
      7. + +
    7. If this object has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media + source=], then throw an {{InvalidStateError}} exception and abort these steps.
      8. + +
    8. If the {{SourceBuffer/updating}} attribute equals true, then throw an {{InvalidStateError}} exception + and abort these steps.
      9. + +
    9. If |typeOrConfig| is a DOMString that contains a MIME type that is not supported or that contains a MIME + type that is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects + in the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a + {{NotSupportedError}} exception and abort these steps.
      10. +
    10. If |typeOrConfig| is not a DOMString, and if it contains a decoder configuration that is not supported + or is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects in the + {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a {{NotSupportedError}} + exception and abort these steps.

    11. If the {{MediaSource/readyState}} attribute of the [=parent media source=] is in the {{ReadyState/""ended""}} state then run the following steps:

        @@ -1317,9 +1527,28 @@

        SourceBuffer Object

    12. Run the [=reset parser state=] algorithm.
      13. -
    13. Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to the value in - the "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is - associated with |type|.
      14. + +
    14. +
      +
      If |typeOrConfig| is a DOMString:
      +
      Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to the value in the + "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is + associated with |typeOrConfig|. +
      +
      Otherwise:
      +
        +
      1. Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to false. +
        WebCodecs encoded chunks are required to have timestamps, so there is no need to + generate them.
        +
        2. +
      2. Remove any {{SourceBufferConfig}} that may be in this {{SourceBuffer}} object's + {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
        3. +
      3. Copy the {{SourceBufferConfig}} from |typeOrConfig| into this {{SourceBuffer}} object's {{SourceBuffer/[[input + webcodecs configs and chunks]]}} for potential handling later during [=chunks append=] algorithm.
        4. +
      +
      +
      +
    15. If the {{SourceBuffer/[[generate timestamps flag]]}} equals true:
      @@ -1345,8 +1574,8 @@

      SourceBuffer Object

    		Description
    |type|{{DOMString}}|typeOrConfig|{{TypeOrConfig}} ✘ ✘