Tackling the IP Address Crisis: Alibaba’s Youku Leads IPv6 Adoption
As the world faces a shortage of IPv4 addresses, Youku is pioneering the transition to IPv6
As the worldwide system for sending and receiving data packets across the internet, the Internet Protocol is the foundation of all digital communications. However, the current Internet Protocol version 4 (IPv4) is rapidly running out of available IP addresses for new devices, limiting growth capacity for emerging technologies like the Internet of Things.
Ahead of the 2018 11.11 Global Shopping Festival, video streaming platform Youku spearheaded adoption of the updated Internet Protocol version 6 (IPv6) by becoming the first video industry giant to embrace the protocol and drive the participation necessary to support it going forward. More recently, Youku has doubled down in pursuit of its canary release of mobile clients following successful release of PC clients for its VIP users in September 2018.
For Youku, gaining the new system’s first user took months; reaching 10 users took weeks; reaching 1000 users took days; but reaching 500,000 users took just a few hours. The next step for Youku is to deploy the full release, at which its user count will rise even higher. In preparation, this article examines how Youku rolled out the IPv6 release, the challenges it faced, and the solutions it adopted.
Youku’s Rollout
Youku’s plans for IPv6 started early in 2018. During the World Cup, Youku investigated the technical feasibility of IPv6 and prepared and verified its transition plan. Then, during the 11.11 Global Shopping Festival, IPv6 was rolled out to a selection of Youku PC and app users in the Chinese domestic education network and in key front-line cities. With the IPv6 technology, these users were able to enjoy live, on-demand HD services. This user experience could be likened to driving on a newly paved highway without other traffic to slow down for. Users had access to exclusive user identities, exclusive client network detection functions, exclusive membership cards, and exclusive “red packet” bonuses.
Why upgrade to IPv6?
IPv6 offers a number of advantages over IPv4 for Youku. Crucially, IPv6 has a larger-capacity address space that does not have the same address shortage as IPv4 and allows true stateless address autoconfiguration. This overcomes the current lack of IPv4 address availability and enables unlimited expansion of services. In addition, the header format used for IPv6 is greatly simplified, reducing the processing overhead for the header. This capacity increase and reduced overhead are crucial for the effective deployment of an “Internet of Everything” approach to the Internet of Things.
Just as importantly, IPv6 increases security and allows personalized services. It mandates the use of the IPSec protocol to ensure the integrity and confidentiality of end-to-end communication at the network layer. The use of flow tags allows Youku to provide personalized network services for different data packet types to effectively guarantee service quality for specific services.
The transition plan
Youku’s IPv6 transition plan spans three phases:
1. External network transition
There are three main goals for this phase: achieve rapid external services of apps; meet the needs of IPv6 ecological development by focusing on the web/App request service as the core; and provide the different requirements for apps through the external network.
2. Intranet transition
In this phase Youku gradually expands direct interactions between apps and IPv6 such as the crawler, mailbox, DB, and storage. This requires the use of IPv6 in intranet servers, and dual stack delivery for the entire network.
3. Full IPv6
Once more than 50% of apps have migrated to IPv6, any new apps adopt IPv6 development by default. IPv4 services continue to be used for users of older apps that are left behind, and 4over6 is adopted for the intranet packaging. Compared to dual stacks, IPv6 costs less and has faster table lookups, requiring only one set of protocol stacks to be maintained.
Scope of the IPv6 transition
The overall scope of IPv6 transition is large, with the following factors comprising the entire link:
1. The client’s network, including mobile network and local area network. The mobile network’s transition is dependent on the carrier, and the user’s LAN must support IPv6.
2. User devices, including mobile phones, PCs, TVs, refrigerators, and washing machines. Manufacturers must update these, and unbranded devices may not be updated.
3. OS/browser. These are Apple, Google, and Microsoft’s responsibility.
4. Client app/PC webpage. This is the first element falling within the scope of Youku’s service transition.
5. HttpDNS. This sends IPv6 addresses to the client. It supports the dual-stack network based on a set policy and requires support from a transformed httpdns.
6. localDNS. DNS resolution requires DNS to support IPv6 resolution and AAAA records to be added to the domain name resolution record.
7. Network link. Carriers need to support IPv6, including data center exit, network routing, and so on.
8. LVS. The exits for all services need to support IPv6, and forward requests to realsrv.
9. Access layer. This forwards requests to the service server with the client IPv6 address.
10. Business services. This falls within the scope of Youku’s service transition, which will be discussed in further detail later in this article.
Everything that involves devices, networks, apps, servers, and services fall into the scope of transition and requires the cooperation of many partners across the industry. However, only the client app/PC webpage and business services fall within the scope of Youku’s transition work.
IPv6 Transformation for Developers
For Youku developers, transition of the client app/PC webpage and the server were the sole areas of concern.
Client
IPv6 requires the update of second-party libraries such as the basic network package NetworkSDK. For clients that use httpdns to parse, they needed to be updated so that the client network determined and received the AAAA record sent by the httpdns server. For clients that use localdns to parse, they needed to be updated so that the identification of the AAAA record resolution was added to the DNS service request parameter. For clients that use third-party libraries, IPv6 must be supported, which involves updating the libraries to newer versions that support IPv6 where possible. However, in some cases this was not possible and developers replaced the libraries instead.
PC/server
The PC/server area involved more potential issues and need for transition work.
Transition was required if any of the service processes contained one or more of the following:
1. The use of IP address libraries
Developers checked whether an address library was used by examining the geographical basis of the user IP. If a library was used, the address library was upgraded to the IPv6 version along with the calling method.
2. Judgment of IP address format
If the user IP needed to be verified, regular expression judgment of the IPv6 address format was added.
3. IP address storage
If there was IP address storage such as in tables, the length of the corresponding field was changed for IPv6. IPv6 addresses are longer than IPv4 ones. For example, with MySQL the recommended field type is VARBINARY (16).
4. Dependence on modifications on the link
If the IP used as an interface parameter was passed to a downstream dependent service, the downstream dependent service also needed to be transitioned.
5. Obtaining the client’s IP address
If the client’s IP address was obtained from the header of the client request, then a single request can only get either the IPv4 or the IPv6 address in a dual-stack environment. It is impossible to get both. If the client address is passed through a field in the request body, developers had to consider whether to get all the v4 and v6 addresses of the client. If doing so, the request field was extended and IPv4 and IPv6 was divided into two fields. This also required the server to be transitioned for receiving the request.
6. Transition of data products such as logs and collection of data
If any collection tools used do not support IPv6, the collected data will not align with the server’s request log, resulting in gaps. Similar problems arise in service scenarios such as advertising, monitoring, and so on. From a service perspective, it was necessary to distinguish between IPv4 user requests and IPv6 user requests, and perform data analysis on them separately. Therefore, any data products, data storage, and so on had to be updated to support collection of the user’s IPv6 data.
7. Security products
Content security requires text security filtering, seven-layer traffic cleaning, and so on. Security products needed to be transitioned, and some second-party packages/clients required updates.
8. Monitoring
Any monitoring configurations that use the user IP as judgment criteria/statistical criteria needed modification.
9. Big data statistics
If the user IP was used as judgment criteria/statistical criteria, then these services also needed to be transitioned.
10. Dependent services
In Youku’s case, all original Alibaba Cloud products in use needed to be transitioned to support IPv6 as well. This includes VPC, ECS, OSS, and CDN.
Challenges Encountered During the Transition
As a pioneer in IPv6 transition, Youku encountered a number of problems during the transition, and was prepared to face various unexpected challenges while innovating in this space.
Lack of available IPv6 environments
Initially, the office network did not have an IPv6 access environment, hindering service development.
The intranet had not yet been transitioned, limiting access to the daily (test) environment. At the beginning, when the basic environment was not available, Youku used an IPv6 over IPv4 link VPN to connect to the test environment. To implement that, certificates needed to be added to the PC/client and hosts needed to be changed. However, hosts could not be changed on mobile terminals without getting administrator privileges illegally. This initially caused issues, but it was still possible to start the service tests.
This situation improved over time. The basic network team and IT team cooperated to deploy multiple IPv6 access environments, open up IPv6 exports, and open up IPv6 links between the office network and data centers. This allowed Youku to gradually enable IPv6 on the external network, daily environment, pre-release, and official release. Through DNS hijacking and other means, Youku steadily improved the service testing experience from merely being able to be tested to the same level possible for IPv4 testing, achieving standard test efficiency.
OS network modules
Since containers must support obtaining IPv6 addresses from request headers, the user IP must be passed through level by level. As a result, the network modules at various levels of the server needed to be upgraded and the message header needed to be expanded as well. For example, the toa module that is used to allow the backend realserver to see the real clientip rather than the lvs dip had to be upgraded.
At the same time, applications such as tengine/nginx also need to be upgraded to versions that support IPv6 (supporting new toa modules, and so on). However, due to reasons beyond Youku’s control, various older versions could not be upgraded. Youku avoided the risk that self-upgrading these network modules could transform the application as a whole; through the upgrade of old-version applications and removal of nginx, Youku successfully upgraded the server as a whole with the installation of a tengine-proxy (which can be installed on the ecs test machine or host machine). In addition, it reduced the workload of service transition by modifying the access layer architecture.
Special address library requirements
Address libraries are not uniform across Youku. Youku has its own regional code; the Youku advertising service uses regional code provided by the China Advertising Association, and other services use Alibaba Group’s address library.
The first part of the solution was to standardize the address libraries used by Youku. All services migrated using Alibaba Group’s own address library across the board. This involved coordination with the developers of the address library to ensure it met the needs of the use scenarios presented. Coordination was key to reducing the workload during standardization.
Secondly, Youku developed a number of solutions to solve scenarios such as advertising that must use a unified industry address library, which may not yet be updated to IPv6. For example, with the China Advertising Association’s address library, Youku added the regional code from the library to Alibaba Group’s address library. This allowed the advertising service to temporarily use the Group’s address library for transition and testing so that the service would not be compromised.
Youku then followed up with this by contacting industry associations such as the China Advertising Association to speed up production of their IPv6 address libraries, and provided Alibaba Group’s address database to them for free. This not only fulfilled Alibaba’s corporate responsibility, but also accelerated the transition progress of the entire industry. In the end, from setting up the project to finally producing the address library, the industry association took less than a month. However, to collect the data, the Group had spent a year and a half beforehand doing ground work.
The MTU problem
In the IPv4 era, devices on the three layers of the intermediate network would be sharded. As a result, the MTU was generally set to a maximum value of 1500 to reduce network overhead. However, to reduce complexity and the costs of devices for intermediate network layers, these devices are no longer sharded in the IPv6 protocol and are specified during negotiation between the two ends.
In the case of the default MTU (1500), a large amount of packet loss occurs in the intermediate devices due to the the sum of NAT conversion, TCPOption, and other additional consumption exceeding 1500. The short-term solution was to enable SYNProxy and negotiate with the ends through MSS and adjust the MTU to a minimum of 1280. When the intermediate layer MTU was less than 1280, a network error was reported.
Verifying whether a client is using IPv6
This challenge presented a serious concern. When a network has begun using IPv6 and the service is running normally, it must be confirmed that the network is running on IPv6 and has not been downgraded. Capturing client logs is one approach to this, and the most accurate. However, Youku was already working to transform the client, so instead integrated a way to verify the network’s use of IPv6 directly into the client.
Problem Description:
During the canary release, related BUs and ordinary users participate in testing. It became necessary to adopt convenient and easy-to-use means of identifying whether the network is accessed through IPv6.
Implementation: Utilize Youku’s Wanxiang platform features
Add rules to match the client IPv6 address type (regular); for client configuration, add an ICON, which is displayed when the IPv6 is being used;
In the right-hand figure, the Expert Tester icon is labelled by the red box. If users see this icon, it means that they are using IPv6 for communication. This icon represents a high-speed private line between the user’s device and Youku. IPv6 is a new data highway, and the symbol indicates that the user is one of the few people who has a license to use that highway. They get an unparalleled service; for example, exclusive membership, exclusive ultra-high definition film sources, exclusive custom resources, and higher discounts.
Protocol rollback
- In a dual-stack environment, the node can theoretically communicate with any optional protocol.
- Most operating system protocol stacks and application implementations will default to IPv6.
- When the IPv6 path is “unavailable”, it can automatically roll back to IPv4.
- But the older rollback algorithm affects user experience to a large extent.
- Adopt a “more aggressive” rollback algorithm
The algorithm was first defined in RFC6555 (“Happy Eyeballs Algorithm”) and has been deployed in:
Browsers and operating systems such as Chrome, Firefox, Safari, OS X, and iOS (Internet Explorer relies on operating system implementation).
- For the mobile client, Youku have greater control
In the IPv4 era, similar functions have been implemented in response to scenarios such as WIFI/cellular roaming.
- For the PC client, the situation is complicated and needs to be treated carefully and fully tested.
Security issues
The carrier’s export function is currently available. However, black hole capability is still in development and currently unavailable.
There are basic security products available to use with IPv6. After binding the domain name, the user can access any services directly. The canary release is still growing and expanding, and more will be added over time.
CDN canary release
The CDN domain name is controlled by Alibaba Cloud and cannot be in the same scope as the canary release of the service. Youku added IPv6-exclusive CDN domain names to sidestep this issue. By adding business logic on the service side, they could issue different domain names to match the pace of the canary release.
Confirming the Service is Running Normally
There are a number of different parts of the platform and service that Youku had to confirm were running normally during testing and canary release — namely the service layer, access layer, data platform, and basic network.
· For the service layer, Youku used an IPv6 monitoring platform for service configuration and compared the monitoring curves of IPv4 and IPv6.
· For the access layer, it measured the total volume of IPv6 traffic, sub-domain name, sub-interface display success, and success rate and RT.
· For the data platform, it analyzed big data and reports of service indicators.
· For the basic network, it measured the link success rate of provincial carriers, IPv6 user ratio, link quality and link latency, IPv6 downgrade, IPv4 ratio, and so on.
Using these, Youku was able to ensure that services were running properly and that they were able to quickly identify where any issues occurred.
Looking to the Future
As Youku continues to roll out its IPv6 transition, its development team is looking to what it allows them to do in the future. With IPv6, Youku removes the bottleneck problem of IPv4 so that the number of devices people use is no longer an issue. Every device can be interconnected to form a networked Internet of Everything, making it easier than ever before for Youku to work with partners across the industry and improve the experience for end-users and content owners.
For video creation, the life cycle runs from filming and processing to production and finally presentation, and at each step a dedicated team takes the basic film and polishes it into something that reflects the director’s vision. In the past, people watched movies in cinemas where lighting could be controlled, and creative teams made movies according to that experience. Now, people watch movies on mobile phones, iPads, PCs, TVs, and projectors. The pictures people see on such devices today are not what directors originally hoped to present.
However, with the Internet of Everything and the possibility of working with partners across the industry, smart post-processing becomes possible for end user devices.
To quickly batch process the Youku content library into a version suitable for sharing across the internet, Youku uses an algorithm that adaptively adjusts the mapping curve according to the shade of the content. It enhances the contrast of dark areas and bright areas, enabling viewers to watch videos closely resembling their directors’ original intent on their mobile devices.
Similarly, IPv6 and the Internet of Everything makes it easier to tackle piracy and the challenges facing copyright protection. Using blockchain technology, IPv6, and 5G, each device records playback information to the blockchain. To change any content, an attacker must modify more than 51% of the devices, making piracy infinitely more expensive. By reducing the cost of anti-piracy protection, copyright becomes more affordable for content owners.
The value of technology lies in helping human beings, as opposed to replacing human beings. By safeguarding the success of a project using technology, the success of the project in turn promotes the implementation of a technology. Youku embraces this philosophy of “Technology drives products”.
As the canary release expands, the next phase of Youku’s IPv6 transition is coming. Without ordinary users even knowing what IPv6 is, Youku will demonstrate its benefits through better services and products. As video quality improves, loading times become shorter, videos become accessible from anywhere, and membership periods are extended free of charge, people will inevitably embrace the new technology that is helping Youku build a better future.
(The Original article is written by Wu Lingxiao吴灵晓)
Alibaba Tech
First hand and in-depth information about Alibaba’s latest technology → Facebook: “Alibaba Tech”. Twitter: “AlibabaTech”.
