Going cross eyed with happy eyeballs

Remember my "happy eyeballs" post? Well, it looks like it's causing issues with YouTube clips both on MacOS X (Mountain Lion) as IOS (6). The symptoms? Clips that refuse to launch and result in a black screen filled with pseudo static and a "An error has occurred. Please try again later." Even worse, some clips actually start to play and stop after a while. BTW, the error message is very helpful Google, thanks. The issue exists on Safari and Firefox; Chrome seems to immune.

Pseudo static and the frustrating "An error has occurred" message

I was quite sure it had to do something with the dual stacked IPv4/IPv6 set-up we use at home as disabling IPv6 on my MBP made the issue disappear. But what is causing the problem? And is there a fix?

Googling quickly pointed to "happy eyeballs", which confirmed my findings after some observations and a good deal of TCPdumping. But why would this happen as "happy eyeballs" was created to prevent bad user experience on IPv6 in the first place.

To understand what happens while viewing a YouTube clip, I used HTTPfox on Firefox. HTTPfox is basically a local proxy which shows all the requests generated by the page you're visiting. The following screenshot shows a clip I was viewing and that stopped halfway...

"Battlestar Galactica" getting stuck between IPv4 and IPv6 transitions.

Digging into the output of HTTPfox one can easily retrieve the URLs that deliver the actual video stream. Google has its own Content Delivery Network (CDN), which presumably tries to deliver data as close as possible to the end user.

In the above example the content cache was:

• o-o---preferred---sn-cxab5jvh-cg0z---v18---lscache7.c.youtube.com
  

And now it becomes interesting as this host resolves to:

• CNAME o-o.preferred.sn-cxab5jvh-cg0z.v18.lscache7.c.youtube.com
• A 194.78.99.141
• AAAA 2a00:1450:400e:d::11

How and where the actual caching node is defined in the HTTP exchange is not really clear to me, but typically this is "somewhere" defined through geoIP lookups of the source IP of the requester. The obvious issue here is that my geo-location is completely different over IPv4 and IPv6 as I use a Hurricane Electric tunnel for my IPv6 connectivity. And even if my provider would deliver native IPv6 connectivity, it will still show a very different routing path.

While one is looking at a YouTube clip, multiple HTTP GETs are send out to the content cache to retrieve gradually the content. And here lives the culprit! As "happy eyeballs" kicks in and tries to deliver connectivity over the "best path", it might actually change its decision during the session. In other words, one HTTP GET can go out over IPv4 but seconds later, the next HTTP GET would go over IPv6. In theory this shouldn't be a problem if all caches are equal, but it depends heavily on how the caches actually operate and the application works. If sessions are involved i.e., are we sure that the IPv4 and IPv6 are actually the same machine(s); if not how is the system supposed to stitch the IPv4 and IPv6 session together? I believe this is exactly what happens with Safari and Firefox on YouTube. The fact that my IPv4 and IPv6 routing is completely different and the likelihood that the IPv4 and IPv6 infrastructure is not the same on the CDN side, might very well result in lost, unknown and zombie sessions.

For a pseudo streaming application like HTTP-based QuickTime streams or YouTube clips, "happy eyeballs" is very dangerous. Pseudo streaming applications send chunks of media data and rely on the fact that the network is faster than the realtime stream. For this specific clip it sends packets of 1,6 MB according to the HTTPfox output.

Conclusion: the browser should not only rely on the OS implementation of "happy eyeballs" as it could change IP stacks during a session or HTTP conversation with potential undesired side effects. It should stick to one IP stack after the initial decision handler to preserve session integrity.

As to view YouTube clips, I think I'll stick with Chrome until Apple and Mozilla engineers have figured out a way to fix this issue.