Alibaba Tech

Jan 16, 2018

14 min read

12 Alibaba Techs made Open-source in 2017

Part I: Distributed systems that are high performance and available to all

Since the launch of the first batch of open-source projects in 2011, the developers at Alibaba have been actively involved in the development of open-source communities. As the number of open-source projects grows every year, Alibaba now boasts 150+ open-source projects, many of which have over ten thousand stars. Alibaba was also the only Chinese company included in Github’s list of top contributors in 2017, winning five out of the top 20 awards at the 2017 Open-source Chinese Software Conference. In Part I of this article, we take a look at the first five of Alibaba’s latest new open-source projects rated by Open Source China last year.

Even though these open-source projects are not included in Alibaba’s KPI assessments, relevant teams are fully committed to sharing lessons learnt to help develop the whole community. This is the true spirit of the open-source community, where both sharing and learning happens side by side.

Building High-performance Distributed Systems

Though mainframe systems require fewer node configurations and are easier to deploy, they are expensive and over-reliant on single points. This has led to the rise of distributed systems, which offer the advantages of improved concurrency, scalability, and reliability, all while reducing overall costs. Alibaba continues to help develop open-source distributed system technologies by releasing projects and updates.

1. Dubbo

Dubbo is a highly acclaimed Java-based Remote Procedure Call (RPC) framework that was first released in 2012. Extremely popular with open-source enthusiasts, Dubbo saw a major upgrade in 2017 to better serve its large user base and stay relevant to RPC technology trends. The design principles for this overhaul focused on stability, scalability, performance, easy communication, and flexibility in use with open-source protocols and peripheral systems. The upgrade also addresses system defects and gaps across versions 2.5.4 to 2.5.8, and now supports Netty 4 and Java 8.

The official website and files also saw changes, notably a new English version. Future versions of Dubbo will incorporate a native cloud feature, multilingual versions, and better service support to build a well-integrated ecosystem.

Find out more:

2. RocketMQ

RocketMQ is a third-generation distributed message-oriented middleware launched in 2012, with an enterprise version used by over a hundred companies and institutes. Host to all of Alibaba’s circulation messages for the Singles’ Day sales on November 11, RocketMQ delivered 1.2 trillion messages for the 2017 event, peaking at 170,000 per second.

In September 2017, the Apache Software Foundation announced that Alibaba’s RocketMQ had achieved the status of a top-level project (TLP). Already the very first domestic non-Hadoop ecosystem and Internet middleware TLP in the Apache community, RocketMQ went on to win the CJKOSS excellence award at the 12th China-Korea-Japan Open-source Software Conference in 2015.

RocketMQ’s features include:

· Latency of over 99.6% responses within 1 millisecond

· Matches tracking and audit requirements of financial companies

· Suitable for industrial-grade applications due to high-volume messaging capacity

· Neutrally supports various messaging protocols, including JMS and OpenMessaging

· Reliably provides sufficient storage without sacrificing performance

Find out more:

3. Druid and Fastjson

The only database connection pool in use by Alibaba, Druid easily supports and monitors extreme scenarios such as the Singles’ Day sales. Built and evolved over the course of six years, Druid is a key part of the solutions offered by many technical companies.

Fastjson is a versatile, high-performance JSON library used in every Java-based Alibaba software. Often employed in scenarios involving cache serialization, protocol interaction, Web output and Android development, Fastjson’s easy-to-use interface maximizes performance based on an “assumed ordered fast matching ” algorithm.

Since they’re fairly advanced, both Druid and Fastjson saw no major changes this year, only smaller fixes in response to issues highlighted by user communities. In the near future, Druid can be expected to be enhanced and integrated with Alibaba Cloud’s own monitoring products, with the parser module branching out as a separate major project.

Find out more:

4. ApsaraCache

A Redis branch of Apsara DB used by the Alibaba Cloud, Alibaba open-sourced ApsaraCache 2.8 with certain v3.0 functionalities to provide a stable, smooth performance to users. Even prior to being made open-source in October 2017, ApsaraCache was commonly used across various web categories, including live broadcast (Miaopai, Yizhibo, Inke and CNTV), gaming (Immomo Game, Longyuan Network, LongTu Game and Changyou), and lifestyle (news platform Jinri Toutiao, AutoNavi, Ant Financial, and DJI Innovations).

ApsaraCache has been open-source since Redis. However, since it was designed to be minimalist, the stability and performance was challenged by the growth of user groups and use cases in China. Since cloud users were unable to use it effectively, Alibaba invested in its reconstruction, and will enable future versions to support Redis module mechanisms. Open-sourcing ApsaraCache aims at facilitating long-term growth and improvements, and reflects Alibaba’s development capabilities. The upcoming version will remove host stability bottlenecks caused by frequent AOF rewriting and include full-volume synchronization to mitigate replication interruptions from weak network connections.

Find out more:

5. Pouch and Dragonfly

Pouch is a rich container technology developed in-house by Alibaba for various system services offering safety-isolation protection and isolation dimensions. Currently, over hundreds of thousands of Pouch containers support large-scale co-location of internal offline and online operations.

Traditional download methods often cause download failures, where large file sizes make retries inefficient, and mass downloads result in file collapses on the client side.

Dragonfly is a P2P file distribution system that reduces pressure on source files and resumes broken downloads regardless of cause. Integrated with Docker and Pouch technologies, Dragonfly’s various functions include image preheating, and meets the mass download requirements of Docker or Pouch users. Thanks to its built-in P2P technology, Dragon’s image download speed is in fact much faster than that of Docker, and also saves cross-border bandwidth for foreign nodes.

The Pouch and Dragonfly architecture

Dragonfly’s open-source version supports the Apache 2.0 protocol and makes P2P file distribution, container image distribution, and disk capacity pre-check available. The enterprise version, embedded in Alibaba Cloud and container services (public/private cloud), supports additional features like a RAM file system, intelligent network flow control, intelligent dynamic compression, and intelligent scheduling strategy.

Find out more:

The Client Side of the Internet Age

Along with the evolution of Web2.0 and Web3.0, performance-based improvement of wireless terminal hardware and advocacy for user experience concepts, websites and apps are becoming more important and more complex. This is reflected in the evolution of JavaScript, from its modest beginnings in 1995 to its current, staggering, levels of popularity. There are around 230,000 open-source JavaScript projects in GitHub’s repositories, 2.3 times as many as open-source Python projects, and much higher than the number of open-source Java and PHP projects put together. JavaScript’s popularity can be attributed to the extraordinary performance of AJAX in front-end development, and the compelling potential Node.js presents for server-side programming with JavaScript.

Two enterprise-class open-source projects that Alibaba has contributed to the community are Ant Design, a UI design language implemented on React, and Egg.js, a Web framework.

6. Ant Design

Ant Design is the result of middle and back-end design experiences from the development of Ant Financial. Its product interaction and visual style aims to be “inconspicuous, definite and cheerful”, and improve user and designer experiences. Abundant, flexible and valuable base components are accumulated to provide solutions for frequently asked questions. TypeScript supports complete type definition, and the high-quality out-of-the-box React components are based on npm + webpack + dva enterprise-class development frameworks.

Major Ant Design developments of 2017:

1. Ant Design 2.x continued its evolution from version 2.6.0 to version 2.13.11.

2. The front-end scaffolding for Ant Design Pro was introduced in October, providing best practices for typical middle and back-end application scenarios.

3. Ant Design 3.0 was released in early December, providing a brand-new design system and many new features. The release log can be found here.

In the two years since its open-sourcing, Ant Design has continued improving by collecting various design and development requirements. The design team responsible for Ant Design is constantly refining design specifications and adjusting overall visual effects, something which is well reflected in version 3.0.

Currently, the main focus remains on publicizing the long-term project development plan, collecting new requirements and issuing reports regularly through DingTalk and GitHub, and interacting closely with the community. The project has fixed cycles for the development of minor versions, where versions with bug fixes are released on a weekly basis and bigger updates are released on a monthly basis.

Ant Design is one of the most popular UI component libraries in the React community, and is used by technology companies including Alibaba, Tencent, Baidu, Meituan, Didi, and Ant Design has garnered over 21,485 stars in GitHub, and was nominated in Open Source China’s 2017 Top 20 list of open-source Chinese software.

Ant Design has reached a relatively advanced stage after the release of v3.0, and will not be undergoing major changes anytime soon. Instead, the team will be working to enhance the visual layout and theme configurations, and promote internationalization and integration.

Find out more:

7. Egg

Egg is an enterprise-class, basic web framework developed to overcome deficiencies found with Node.js. When micro-service architecture is adopted, specific tasks such as service granularity, API interface and development maintenance are involved, and the front and back-end face the conflict of flexible user experience vs. back-end service general utility. Node.js has managed to overcome this widely-criticized weakness, and npm package is now the most popular peer, with vast improvement in single-threaded problems, avoidance of burdensome callbacks with the help of ES norm, a type system based on mature TypeScript, and better overall performance.

Unfortunately, the infrastructure maturity and popularity of Node.js are not a match made in heaven, and so it is essentially still a newcomer. In the industry, there are very few Web framework practices, and the community repeats a lot of work from scratch and lacks unified referential norms, which brings many repeated and unnecessary issues. Compared to the application, Node.js fundamental technology research is still rare, so ease of availability is a stumbling block.

The application of Node.js in Alibaba started in 2011. After experiencing difficulties, including starting from scratch, Alibaba began to seriously consider not only unifying the ecology build, but also customization support. Egg was born at the right time and is anticipated to be a fundamental framework for Web development to help all teams in building upper-layer framework. Egg was cultivated from engineering practices of large-scale enterprise applications, including robustly supporting Singles’ Day sales on November 11, and was polished in four versions of microkernels throughout 3 years.

Egg is based on Koa and is totally different from the upper-layer frameworks of Sails/LoopBack. It lies in the middle-layer and provides “microkernel + plug-in component mechanism + framework customization capability”, delivering a whole set of unified agreements for the service of upper-layer frameworks. It also adopts a gradual development method with extreme flexibility, so it is suitable for the fast development of small project and development by enterprise-level teams.

Within just a year of open-sourcing, Egg has received over 500 Pull Requests (272 in the host library), handled over 1,200 issues, and published over 40 official documents of over 100,000 words. It has also received over 6,000+ stars in GitHub and nearly 10,000 PVs on its official website.

Within Alibaba, it has served over 500 front end developers and over 1,000 apps, while outside Alibaba, Egg has received feedback for improvement from over 90 developers in various companies, including Baidu, Tencent, Quanmin and Dxy, and even Rod Vagg, Director of Node TSC, who got involved retweeting about Egg!

In the future, Egg will focus on the optimization of developer experience and the continued sharing of best practices.

Besides the 1,000+ apps within Alibaba, the technology team also eliminated some bottom-layer bugs in Node.js, and explored with Intel the further improvement in Node.js functions. In addition, Alibaba has just released a Node.js performance platform, and provides Node.js running + monitoring and diagnosis services for free.

Find out more:

8. Atlas

Atlas, a flexible Android development framework, was created to assist with internal issues at Alibaba. With the continuous development of mobile Taobao, the R&D team became larger and functions more complicated, which required the fast iteration of end-side businesses. On the other hand, the problems encountered during development, operation and maintenance continually increased, which also required fixing online bugs. As such, the mobile Taobao team began to think about how to simplify the complicated problems and conduct dynamic deployment to improve mobile development efficiency.

Atlas dockerizes to solve large-scale team cooperation issues, which is suitable for the development of large or small-sized apps on system versions higher than Android 4.0 and supports dynamic publishing. Generally speaking, the industry usually uses plug-in component frameworks. However, Atlas adopts modularization, which divides businesses and reuses common parts in consideration of reusability. The project adopts the approach of OSGI (Open Grid Services Infrastructure (OGSI)) and separates business to independently form a bundle for isolation and decoupling operations, so that concurrent development, fast iteration and dynamic deployment can be realized.

Atlas is a result of five years of development by Alibaba’s mobile team, and makes technical preparations for static and dynamic modularization. After the official open-sourcing this year, Atlas has maintained the publishing of a major version every two months, and uses the same coding both in Alibaba and the community. Altas’ improvements lie in stability, tool chain upgrade, dexpatch, etc. With regard to dynamic deployment, new functions include component support, dexpatch function, and bundle view reuse.

As an early open-source project, initially Altas was not perfect and received criticism for its high barriers to entry, incomplete documentation and primitive demos. The Altas team immediately decided to form a DingTalk group to help developers understand and have access to Atlas, and make better documentation and demos. Many Chinese Android apps are using Atlas (such as Tmall, Taobao, Xiami, Youku and Alibaba Cloud), as well as at least three non-Chinese apps.

Atlas’ vision is to “bring developers back to the golden era of app development”. In the future, Altas will continue to improve documentation, upgrade the toolchain, lower access costs, improve stability and compatibility, introduce a complementary Emas for Atlas business services, and increase convenience for server-side publishing.

Find out more:

Best Practice Reference Specifications

The following two technologies made open-source by Alibaba last year concern specific best practice reference specifications.

9. Alibaba Java Development Specifications

Alibaba’s development of Java specifications and corresponding automatic testing tools (DE test plug-in components [IDEA, Eclipse]), is the result of a wide-spread collaboration between a number of senior Alibaba technical staff, has enabled developers to implement specifications more conveniently and quickly.

The plug-in component scans the codes and shows ones which are not compliant with the specified blocker/critical/major levels in the lower part or even the IDEA. Real-time testing functions are also provided based on the inspection mechanism, and quickly spot issues when coding. For the former codes, one-key repairing functions are employed for certain rules.

These Java code specifications have been made public in a plug-in component form, and are leading the way ahead in Java language standardization. Currently, the plug-in component is found integrated in the Yunxiao Public Cloud Platform.

Find out more:

10. OpenMessaging

The OpenMessaging project was initiated by Alibaba in cooperation with Yahoo, Didi Chuxing and Streamlio. Its main aim is to create an app development standard for distributed messaging and stream processing that is company- and platform-neutral. The longer-term standpoint is that, with the evolution of standards, more internet and cloud computing companies will participate into the project and ecosystem.

In the cloud computing era, messaging has become a key segment of modern data-driven architecture where the middleware can link the data source, data handling engine and apps all together. However, messaging encounters two major problems worldwide. First is the lack of industry standards that are supplier-neutral, thus resulting in vast complications and incompatibilities in various message middleware. Second is the inability of current solutions to be compatible with cloud architecture, i.e. non-cloud native architecture, thereby impeding technical support for emerging business requirements including big data, stream computing and IoT.

Besides linking cross-platform, cross-language, cross-product and cross-cloud standards, OpenMessaging will provide cloud native capabilities for cloud computing and hybrid cloud architecture to realize seamless transfers across multiple cloud platforms, and provide one-stop solutions for finance, ecommerce, IoT and big data enterprises.

Find out more:

The era of IoT

Though the earliest conceptual origins of IoT lie in 1982 when Carnegie Mellon University began exploring the interconnections between intelligent devices, it wasn’t until 2005 that it began to gain traction with the International Telecom Union’s (ITU) report on the internet. Indeed, ITU should be commended for its foresight- a little over a decade later in 2016, Gartner put the number of IoT devices in use at 6.4 billion, expecting the figure to balloon to 20.41 billion by 2020.

11. AliOS Things

Today, there is more attention on IoT than ever before. However, the operation systems market in highly fragmented, and the systems themselves feature several drawbacks, such as high development and maintenance costs, difficult deployment of cloud-integrated apps, systemic and security risks, and inadequate hardware ecosystems, development tools, and functional components.

AliOS Things, the integrated platform used by all Alibaba Cloud IoT business devices, targets such weak points by providing a secure, reliable, and highly optimized basic OS, differentiated functional features and developing tools, and cloud-end integrated application frameworks.

Little wonder then that it is the most widely used lightweight IoT operation system for MCUs of across consumption, connection, and controlling classes. Its many high-level applications, including security device management, voice recognition, cloud-end integrated development and full-chain optimization, are based on all major IoT open platforms suitable for use in everyday life, urban management and manufacturing. Currently, AliOS Things has been transplanted to 21 types of chips from 17 domestic and foreign chip makers, including STMicroelectronics, Espressif, and Beken.

After being made open-source this October, AliOS Things has released 3 major versions and regular updates, rich documents, and supports an active developer community.

Find out more:

12. AliOS Lite

Alibaba will soon be open-sourcing AliOS Lite, a lightweight version of aliOS based on JavaScript that supports all intelligent processes for IoT devices with lower CPU performance capacities, such as smart watches and cameras.

In a period when the industry is struggling to find IoT-based operating systems that are highly compatible and pose lower barriers to development, AliOS Lite provides processing solutions for devices with memory as little as 256 MB, and guarantees uniform action for various APIs, all while integrating Alibaba’s proprietary machine intelligence capabilities in face recognition and image classification. In addition to this, AliOS Lite also supports mainstream machine learning frameworks like Tensorflow and Caffe. It also improves system efficiency through streamlining, fast rendering and unified background program hosting.

It innovatively supports lightweight multi-thread programming, enables multi-kernel hardware, helps increase app operation speeds by 35%, and optimizes memory consumption levels by 20%.

Alibaba Tech

First hand and in-depth information about Alibaba’s latest technology → Search “Alibaba Tech” on Facebook