Draft for Autoplay Detection spec

[alastor0325]

This draft is written based on the discussion from issue#12, and we can still make any modification on this if we reach to another new consensus later. In addition, the website version preview can be viewed in here.

[tjenkinson]

Looks great to me!

[tjenkinson]

Curious if we think this enum may be expanded in the future or if this will be fixed?

[chrisn]

Do you have a use case in mind that may need a different option?

[tjenkinson]

Nope I don’t at the moment. Just curious if that’s something that would be ok to happen, and it’s related to my other comment on whether media element and web audio will always have values that satisfy both apis.

[tjenkinson]

Do we need to mention the conditions for web audio to run too?

[alastor0325]

Like "Media and web audio are allowed to autoplay"? Also see this comment for a potential problem that media element and web audio has a different result.

[tjenkinson]

Will the document api always be able to satisfy media element and web audio (or gif??)? I’m wondering is it’s safe to couple them together?

E.g could a browser block web audio autoplay but allow a muted video autoplay?

[alastor0325]

Yep I just think of a similar question as well, see this comment. Probably we should let this document.autoplayPolicy focus on only media element? and have another API for exposing the sticky activation?

[dalecurtis]

Do you want to say more about has a user gesture might affect this?

[gkatsev]

Does the note or example below meet this? Lines 156 and 166.

[alastor0325]

Yes, I used note and example below to demonstrate the possible situaion of using user gesture activation to detect blocking autoplay for user-agents. @dalecurtis let me know if you think that's not enough or I should add more details in the note and example. In addition, because user-agents could use different strategy for blocking autoplay, eg. Safari doesn't use the user activation model. That makes me figure that avoiding to mention the user gesture in the normative form might be better?

[alastor0325]

In addition, I just think of one potential issue. If the user-agent is using a non-user-activation way for blocking autoplay, then it seems possible that we would have different result for media element and web audio in term of allowing to play?

Eg.
If a user-agent implements a click-to-play strategy (only allow an media element to start after user clicks on it), so event if a user clicks on the page (not element), the document's autoplaypolicy would still be disallowed. But web audio uses sticky activation, it should be allowed to start after the page granted the sticky activation. In this situation, if document.autoplayPolicy returns disallowed then it doesn't reflect the web audio's state. But if it returns allowed then it violates the media element's state.

I wonder how y'all think about this case? Thank you.

[tjenkinson]

To me it sounds like we shouldn't try and represent both web audio and the media element with a single property.

Maybe some static ones on the actual apis?

Like HTMLMediaElement.autoplayPolicy and AudioContext.autoplayPolicy?

[alastor0325]

That sounds reasonable to me, let me post a new comment on the thread and ask other people's opinion.

[dalecurtis]

Ditto. Anything to be said about user gestures? I.e., "... allowed to autoplay within the current execution context."

[alastor0325]

I wonder if we should add "...within the current execution context" for allowed-muted, not allowed? Because that somehow indicates that allowed-muted is a temporary state.

[chrisn]

@alastor0325 Thanks for drafting this! I made a number of minor comments. What I suggest is to review all the feedback, and any comments that need follow up discussion can be converted to separate issues. I have a couple more substantial questions I'll leave for after this PR is merged.

[gkatsev]

Does the note or example below meet this? Lines 156 and 166.

[alastor0325]

So far, I think most of issues have been solved, but there is still one problem which isn't clear to me. @tjenkinson also proposed that to have another property on WebAudio API to separate the use of media element and web audio.

@chrisn @gkatsev @dalecurtis (who have left review suggestions on this PR before), would you mind to provide your thought about this issue? Also, @padenot, what would you feel to have that (a new attribute for web audio) based on the issue I mentioned above?

Thank you all.

[dalecurtis]

Sorry, I'm heading out on paternity leave, so won't have a chance to review this soon. @chcunningham to review or route.

[chcunningham]

Routing to @liberato-at-chromium for review

[liberato-at-chromium]

Hi all, nice to meet you. I'm relatively new to this process, so please point out (gently :) ) any breaches of decorum on my part.

@tjenkinson also proposed that to have another property on WebAudio API to separate the use of media element and web audio.

I'm still catching up. But at first glance, I think that having a single property on Document would be confusing. Web developers will have a specific API in mind, so letting them ask that API directly seems like it's more likely to produce actionable information. Else I'm not sure that we can meaningfully let the policies between APIs differ.

To put it another way, what would a developer infer in response to allowed-muted at the document level, if they are interested in WebAudio?

[tjenkinson]

👋 hi!

I'm relatively new to this process, so please point out (gently :) ) any breaches of decorum on my part

snap

I think that having a single property on Document would be confusing. Web developers will have a specific API in mind, so letting them ask that API directly seems like it's more likely to produce actionable information. Else I'm not sure that we can meaningfully let the policies between APIs differ.

Completely agree. What do others think?

[alastor0325]

Sorry for my late reply.

I'm still catching up. But at first glance, I think that having a single property on Document would be confusing. Web developers will have a specific API in mind, so letting them ask that API directly seems like it's more likely to produce actionable information. Else I'm not sure that we can meaningfully let the policies between APIs differ.

Per this comment, document API seems to be designed for helping you decide which source to use, and I think that's something video.autoplayPolicy can do as well. And some people started thinking about they can use that to determine whether web audio can be allowed to autoplay or not. At the time when document API was introduced, not sure if allowed-muted already appeared on the discussion or not (probably not?)

In this draft, we tend to avoid define the exact implementation of blocking strategy for the user agents, so we didn't discuss the user gesture activation too much on the spec, which allows the user agents to have a choice to implement different strategies to block autoplay. But web audio has a clear defintion for its allow-to-start, so the problem appeared in this discussion

To put it another way, what would a developer infer in response to allowed-muted at the document level, if they are interested in WebAudio?

Web audio requires sticky user gesture activation, but our API doesn't tell anything about that. So I will say allowed-muted probably meaningless for web audio developers. I will prefere to have a dedicated API for web audio, not put it on the document.

[alastor0325]

I talked to @padenot (the web audio expert in Mozilla) about this issue yesterday, and he said that they thought that allowing web audio API to start when document.autoplayPolicy == allowed is enough at the time the document API was discussed. But as the situation I mentioned, it doesn't work if the blocking policy is not based on the sticky activation.

In this draft, we tend to avoid define the exact implementation of blocking strategy for the user agents, so we didn't discuss the user gesture activation too much on the spec, which allows the user agents to have a choice to implement different strategies to block autoplay. But web audio has a clear defintion for its allow-to-start, so the problem appeared in this discussion

As above reason, the APIs for autoplay detection won't represent the status of user activation. Web audio can't use API to detect whether the page gets sticky activation or not.

In addition, as we don't define the exact implementation of blocking strategy for the user agents, that provides the flexibilty of the spec. Otherwise, we have to define ONE policy that can only be used for blocking autoplay, and all user agents have to agree with that. As far as I know, Safari doesn't use user activation for blocking autoplay, and Firefox also provides options to allow users to switch to non-sticky activation for blocking autoplay. Therefore, I think provide the flexibility of the spec is important.

Therefore, we think the solution would be either adding an attirbute on AudioContext (eg. AudioContext.canAllowToStart, or adding an attribute on document to expose whether the document gets sticky activation, such as document.hasStickyActivation. Doing the latter one might also help other APIs requiring the stick activation. But I don't know if that is a good idea, and that might be out of the scope of this spec?

@annevk @jan-ivar would you mind to help me leave your thought about adding document.hasStickyActivation? Thank you so much.

[annevk]

There is an outstanding PR for that feature (with a slightly different signature) by @dtapuska and @mustaqahmed at whatwg/html#4009. However, it's quite old and there's no reason stated why it was abandoned. Perhaps they can comment here (or there). I do think it would be best if it were defined in HTML as that also defines the overall user activation model.

Aside: I also think that media element extensions would be best defined in HTML or at least directly integrated with HTML. Having elements defined in many different places usually leads to subtle issues.

[alastor0325]

Sorry for my late reply. After checking the PR for the user activation, I think that is enough for the web audio case and we could make this spec only for media element. If this proposal works for everyone, then I will start modifying my draft again.
@chrisn @eric-carlson @liberato-at-chromium @gkatsev @tjenkinson How would y'all think? Thank you so much.

[gkatsev]

Taking whatwg/html#4009 into account, what would be the end points for the autoplay policy property?

[chrisn]

@annevk said:

Aside: I also think that media element extensions would be best defined in HTML or at least directly integrated with HTML. Having elements defined in many different places usually leads to subtle issues.

Thanks, we'll discuss this point in the Media WG.

[chrisn]

We discussed integrating into HTML in yesterday's Media WG meeting. We want to continue development in Media WG for now, and review HTML integration when the design has progressed a bit further.

[alastor0325]

In the last Media WG Meeting, people suggested me to merge this first draft version, and then open other small issues to track other open questions, eg. should we create new API on AudioContext, or just use a new API design. Instead of keeping discuss everything on this issue, which would make it super long and hard to focus on the thing we really want to discuss. Therefore, I will go to merge this PR first, and start other discussion on other new issues.