Implementing Distributed Tracing in Dubbo: An Alibaba Guide to Zipkin
Learn how to implement Zipkin in Dubbo for complex distributed systems, from configuring dependencies to performing tracing analysis
As applications expand in scale, traditional application architecture can struggle to support development demands. This has led to the transformation of service architecture, led by distributed service frameworks such as Dubbo.
Along with this service architecture transformation, applications have also been further split to finer granularity, with a vast number of different applications being developed by different teams. These factors result in increased complexity throughout the distributed system, making it difficult to locate the causes of issues when they arise.
Major technology companies have generated various responses to this issue. Google published a paper on Dapper, its distributed systems tracing infrastructure; Alibaba launched its own distributed tracing systems, among which EagleEye is the most well-known; Twitter developed Zipkin, an open source distributed tracing system.
This article looks at some of the technical fundamentals of Zipkin, and shows developers how to use Zipkin in Dubbo to implement distributed tracing.
Understanding Zipkin
Zipkin is an open source distributed tracing system from Twitter that collects information on distributed service execution time. Zipkin traces service call links and analyzes service execution delays.
Zipkin architecture
The Zipkin server includes the collector, storage, API, and UI user interface, and corresponds to the openzipkin/zipkin project on GitHub. The components that report the consumed-time information called in the collecting application are symbiotic with the application.
Versions are available in different languages including Go, PHP, Javascript, Net, and Ruby. The Java client implementation is at openzipkin/brave.
Zipkin working processes
When the user initiates a call, the Zipkin client generates a globally unique trace ID at the ingress for the entire invocation link. The Zipkin client also generates a span ID for each distributed call in the link.
There can be a parent-child relationship between span and span, which represents the upstream and downstream relationships in distributed calls. There can also be a sibling relationship, which represents two sub-calls under the current call. A set of spans together forms a trace.
Each span is separated by call boundaries. In Zipkin, the call boundary is represented by the following four annotations:
· CS — Client Sent: Client sent the request
· SR — Server Receive: Server received the request
· SS — Server Send: Server completed the process and sent a response to the client
· CR — Client Receive: Client received the result
You can use timestamps on these four annotations to find the time spent on different stages of a complete call. For example:
· SR — CS shows the time spent on the request in the network
· SS — SR shows the time taken by the server to process the request
· CR — SS shows the time the network took to respond
· CR — CS shows the overall time spent on a call
Zipkin passes trace-related information on the calling link and asynchronously reports the consumed-time of the current call to the Zipkin server at the end of each call boundary. When the Zipkin server receives the trace information, it stores it. Zipkin supports several storage types such as in-memory, MySQL, Cassandra, and ElasticSearch. Zipkin’s web UI then extracts the trace information from storage for analysis via API access and displays it, as shown in the following figure:
Using Zipkin in Dubbo
As Dubbo is supported by Brave, integrating Zipkin-based link tracing in Dubbo is simple. The following content explains how to use Zipkin in Dubbo according to the guidelines supported by Dubbo RPC in Brave.
Installing the Zipkin Server
You can install Zipkin by following the quick start guide in the Zipkin official documentation as follows:
$ curl -sSL https://zipkin.io/quickstart.sh | bash -s
$ java -jar zipkin.jarWhen the Zipkin server is installed in this way, the storage type used is in-memory. When the server is shutdown, all trace information collected will be lost, and will not be suitable for use in production systems. If the trace information is used in a production system, other storage types also need to be configured. Zipkin supports MySQL, Cassandra, and ElasticSearch. Cassandra and ElasticSearch are recommended. Refer to the official documentation for relevant configuration.
For convenience, the storage type used in this article is in-memory. The following prompts show on the terminal after successful startup:
$ java -jar zipkin.jar
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
********
** **
* *
** **
** **
** **
** **
********
****
****
**** ****
****** **** ***
****************************************************************************
******* **** ***
**** ****
**
**
***** ** ***** ** ** ** ** **
** ** ** * *** ** **** **
** ** ***** **** ** ** ***
****** ** ** ** ** ** ** **
:: Powered by Spring Boot :: (v2.0.5.RELEASE)
...
o.s.b.w.e.u.UndertowServletWebServer : Undertow started on port(s) 9411 (http) with context path ''
2018-10-10 18:40:31.605 INFO 21072 --- [ main] z.s.ZipkinServer : Started ZipkinServer in 6.835 seconds (JVM running for 8.35)Once the above prompts have displayed, visit the web verification page on http://localhost:9411 in the browser.
Configuring Maven Dependency
Introducing Brave dependencies
Create a new Java project and introduce Brave-related dependencies in pom.xml, as follows:
<properties>
<brave.version>5.4.2</brave.version>
<zipkin-reporter.version>2.7.9</zipkin-reporter.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- Introducing the BOM file of zipkin brave -->
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-bom</artifactId>
<version>${brave.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<!-- Introducing the BOM file of zipkin reporter -->
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-bom</artifactId>
<version>${zipkin-reporter.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<!-- 1. Via integration with Brave to support dubbo -->
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-instrumentation-dubbo-rpc</artifactId>
</dependency>
<!-- 2. brave's spring bean support -->
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-spring-beans</artifactId>
</dependency>
<!-- 3. Supporting traceId and spanId in the MDC(Mapped Diagnostic Context)of SLF4J -->
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-context-slf4j</artifactId>
</dependency>
<!-- 4. Using okhttp3 as reporter -->
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-sender-okhttp3</artifactId>
</dependency>
</dependencies>The Brave-related dependencies to introduce here include:
1. Brave’s support for Dubbo, brave-instrumentation-dubbo-rpc: https://github.com/openzipkin/brave/blob/master/instrumentation/dubbo-rpc/README.md
2. Brave’s support for Spring beans, brave-spring-beans: https://github.com/openzipkin/brave/blob/master/spring-beans/README.md
3. Brave’s support for SLF4J, so traceId and spanId can be used in MDC, brave-context-slf4j:https://github.com/openzipkin/brave/blob/master/context/slf4j/README.md
4. Zipkin-sender-okhttp3 and use okhttp3 to report data: https://github.com/openzipkin/zipkin-reporter-java
Introducing Dubbo-related dependencies
The Dubbo-related dependencies are Dubbo itself and the Zookeeper client. The following examples use a standalone Zookeeper server as a service discovery.
<dependencies>
<!-- 1. Zookeeper client dependency -->
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>2.12.0</version>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- 2. Dubbo dependency -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>dubbo</artifactId>
<version>2.6.2</version>
</dependency>
</dependencies>Additional information:
1. Dubbo relies on a separate Zookeeper server for service discovery. The client used here is Curator.
2. When introducing Dubbo framework’s dependencies, in principle any 2.6 version will work. Version 2.6.2 is used here.
Implementation
The scenario this process builds is a service dependency chain with two nodes.
This means that when a Dubbo client invokes service A, service A will continue to call service B. In this example, Service A is the greeting service, and the downstream Service B, on which it depends, is the hello service.
Defining a service interface
To do this, you need to firstly define two service interfaces, the GreetingService and the HelloService.
- com.alibaba.dubbo.samples.api.GreetingService
package com.alibaba.dubbo.samples.api;
public interface GreetingService {
String greeting(String message);
}2. com.alibaba.dubbo.samples.api.HelloService
package com.alibaba.dubbo.samples.api;
public interface HelloService {
String hello(String message);
}Implementing the service interface
In order to separate the two types of code, all implementation codes related to the HelloService are placed under the hello sub-package, and all implementation codes related to the GreetingService are placed under the greeting sub-package.
- Implementing com.alibaba.dubbo.samples.api.HelloService
package com.alibaba.dubbo.samples.service.hello;
import com.alibaba.dubbo.samples.api.HelloService;
import java.util.Random;
public class HelloServiceImpl implements HelloService {
@Override
public String hello(String message) {
try {
// Simulate business logic processing time via sleep
Thread.sleep(new Random(System.currentTimeMillis()).nextInt(1000));
} catch (InterruptedException e) {
// no op
}
return "hello, " + message;
}
}2. Implementing com.alibaba.dubbo.samples.api.GreetingService
package com.alibaba.dubbo.samples.service.greeting;
import com.alibaba.dubbo.samples.api.GreetingService;
import com.alibaba.dubbo.samples.api.HelloService;
import java.util.Random;
public class GreetingServiceImpl implements GreetingService {
// downstream dependency service,injecting into the service agent by using spring container while running
private HelloService;
public void setHelloService(HelloService helloService) {
this.helloService = helloService;
}
@Override
public String greeting(String message) {
try {
// Simulate business logic processing time via sleep
Thread.sleep(new Random(System.currentTimeMillis()).nextInt(1000));
} catch (InterruptedException e) {
// no op
}
return "greeting, " + helloService.hello(message);
}
}It should be noted here that while GreetingServiceImpl is being implemented, it declares a HelloService type member variable. Also, once the greeting method has executed its own logic, it calls the hello method on HelloService.
The HelloService implementation will be injected externally in the running state. What is injected is not the HelloServiceImpl implementation, but the HelloService remote call agent.
Using this kind of method, the goal of completing one Dubbo service while calling another remote Dubbo service is accomplished.
From a link tracing perspective, the client calling a GreetingService is a span. The GreetingService calling the HelloService is another span. The two spans have a parent-child relationship, and belong to the same trace; that is, they both belong to the same call link.
Also, during the implementation of GreetingServiceImpl and HelloServiceImpl, consumed processing time is simulated via Thread.sleep for better display on the Zipkin UI.
Configuration
In order to simply explain how to use Zipkin, this article uses the most traditional Spring XML configuration for the configuration and programming model instead of using more advanced techniques. There are more advanced methods which use annotation or even Spring Boot; relevant Dubbo and Zipkin documents can be consulted for more information.
1. Expose the HelloService service
Add the following configuration in resources/spring/hello-service.xml to expose HelloServiceImpl as a Dubbo service:
· The locally started Zookeeper server is used as the registry center, and the address is set as default value. zookeeper://127.0.0.1:2181
· Expose services on port 20880 with Dubbo native service
· Register HelloServiceImpl as a Spring bean whose ID is helloService so that it can be referenced to this implementation class in subsequent <dubbo:service>.
· Expose HelloServiceImpl via <dubbo:service> as Dubbo service
<!-- Defining the application name of HelloService -->
<dubbo:application name="hello-service-provider"/>
<!-- Designating registry center address -->
<dubbo:registry address="zookeeper://127.0.0.1:2181"/>
<!-- Using Dubbo native protocol to expose service on Port 20880 -->
<dubbo:protocol name="dubbo" port="20880"/>
<!-- Declaring the implementation of HelloServiceImpl as a spring bean-->
<bean id="helloService" class="com.alibaba.dubbo.samples.service.hello.HelloServiceImpl"/>
<!-- Declaring the HelloServiceImpl as a Dubbo service -->
<dubbo:service interface="com.alibaba.dubbo.samples.api.HelloService" ref="helloService"/>2. Add Zipkin related configuration
Add Zipkin related configuration in resources/spring/hello-service.xml:
· Modify the configuration of Dubbo service exposure and adding the Zipkin tracing filter to the filter chain in Dubbo.
· Configure the Zipkin sender and the tracing Spring bean according to https://github.com/openzipkin/brave/blob/master/spring-beans/README.md
<!-- 1. Modifying Dubbo's services exposure configuration and adding zipkin's tracing filter in filter chain -->
<dubbo:service interface="com.alibaba.dubbo.samples.api.HelloService" ref="helloService" filter="tracing"/>
<!-- 2. zipkin related configuration -->
<!-- Using OKHttp to send trace information to Zipkin Server。 Here the Zipkin Server startups locally -->
<bean id="sender" class="zipkin2.reporter.beans.OkHttpSenderFactoryBean">
<property name="endpoint" value="http://localhost:9411/api/v2/spans"/>
</bean>
<bean id="tracing" class="brave.spring.beans.TracingFactoryBean">
<property name="localServiceName" value="hello-service"/>
<property name="spanReporter">
<bean class="zipkin2.reporter.beans.AsyncReporterFactoryBean">
<property name="sender" ref="sender"/>
<!-- wait up to half a second for any in-flight spans on close -->
<property name="closeTimeout" value="500"/>
</bean>
</property>
<property name="currentTraceContext">
<bean class="brave.spring.beans.CurrentTraceContextFactoryBean">
<property name="scopeDecorators">
<bean class="brave.context.slf4j.MDCScopeDecorator" factory-method="create"/>
</property>
</bean>
</property>
</bean>3. Add HelloService startup class
Read the configured spring/hello-service.xml in com.alibaba.dubbo.samples.service.hello.Application via ClassPathXmlApplicationContext to initialize as well as start a Spring context.
package com.alibaba.dubbo.samples.service.hello;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import java.io.IOException;
public class Application {
public static void main(String[] args) throws IOException {
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("spring/hello-service.xml");
context.start();
System.out.println("Hello service started");
// press any key to exit
System.in.read();
}
}4. Expose the GreetingService service and use Zipkin
Configure GreetingService in resources/spring/greeting-service.xml. The steps involved are similar to HelloService. This article focuses on how to configure the dependencies of downstream services in the GreetingService. The integral XML configuration is as follows:
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dubbo="http://dubbo.apache.org/schema/dubbo"
xmlns="http://www.springframework.org/schema/beans"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://dubbo.apache.org/schema/dubbo http://dubbo.apache.org/schema/dubbo/dubbo.xsd">
<!-- 1. Defining the application name of GreetingService -->
<dubbo:application name="greeting-service-provider"/>
<!-- 2. Designating registry center address -->
<dubbo:registry address="zookeeper://127.0.0.1:2181"/>
<!-- 3. Using Dubbo native protocol on Port 20881 to exposure service -->
<dubbo:protocol name="dubbo" port="20881"/>
<!-- 4. Declaring the remote agent of HelloService and adding tracing filter in the filter chain in Dubbo-->
<dubbo:reference id="helloService" check="false" interface="com.alibaba.dubbo.samples.api.HelloService" filter="tracing"/>
<!-- 5. Declaring the implementation of GreetingServiceImpl as a spring bean and assemble the remote agent of HelloService into it-->
<bean id="greetingService" class="com.alibaba.dubbo.samples.service.greeting.GreetingServiceImpl">
<property name="helloService" ref="helloService"/>
</bean>
<!-- 6. Declaring the GreetingServiceImpl as a Dubbo service and adding tracing filter in filter chain in Dubbo -->
<dubbo:service interface="com.alibaba.dubbo.samples.api.GreetingService" ref="greetingService" filter="tracing"/>
<!-- 7. zipkin related configuration -->
<bean id="sender" class="zipkin2.reporter.beans.OkHttpSenderFactoryBean">
<property name="endpoint" value="http://localhost:9411/api/v2/spans"/>
</bean>
<bean id="tracing" class="brave.spring.beans.TracingFactoryBean">
<property name="localServiceName" value="greeting-service"/>
<property name="spanReporter">
<bean class="zipkin2.reporter.beans.AsyncReporterFactoryBean">
<property name="sender" ref="sender"/>
<!-- wait up to half a second for any in-flight spans on close -->
<property name="closeTimeout" value="500"/>
</bean>
</property>
<property name="currentTraceContext">
<bean class="brave.spring.beans.CurrentTraceContextFactoryBean">
<property name="scopeDecorators">
<bean class="brave.context.slf4j.MDCScopeDecorator" factory-method="create"/>
</property>
</bean>
</property>
</bean>
</beans>The configuration is similar to HelloService as above. Note the following two points:
· In step 3 in the preceding code, note that the service needs to be exposed on a different port, otherwise it will conflict with HelloService. In this case, port 20881 is selected.
· Declare the remote agent of HelloService via step 4 in the preceding code, and then assemble it into GreetingService via step 5 in the preceding code to finish the declaration of the dependency of upstream and downstream service.
Adding the startup class of GreetingService is similar to HelloService: initialize a new Spring context by configuring spring/greeting-service.xml.
package com.alibaba.dubbo.samples.service.greeting;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import java.io.IOException;
public class Application {
public static void main(String[] args) throws IOException {
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("spring/greeting-service.xml");
context.start();
System.out.println("Greeting service started");
// press any key to exit
System.in.read();
}
}1. Implement client
Initialize a Spring context via resources/spring/client.xml and obtain the remote agent from GreetingService to initiate a remote call.
package com.alibaba.dubbo.samples.client;
import com.alibaba.dubbo.samples.api.GreetingService;
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class Application {
public static void main(String[] args) {
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("spring/client.xml");
context.start();
// Obtaining remote agent and initiating the call
GreetingService = (GreetingService) context.getBean("greetingService");
System.out.println(greetingService.greeting("world"));
}
}The configuration in resource/spring/client.xml is similar to the Dubbo service configuration, which is mainly about configuring the remote agent and Zipkin.
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dubbo="http://dubbo.apache.org/schema/dubbo"
xmlns="http://www.springframework.org/schema/beans"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://dubbo.apache.org/schema/dubbo http://dubbo.apache.org/schema/dubbo/dubbo.xsd">
<!-- 1. Defining the application name of dubbo's client end -->
<dubbo:application name="dubbo-client"/>
<!-- 2. Designating register center address-->
<dubbo:registry address="zookeeper://127.0.0.1:2181"/>
<!-- 3. Declaring the remote agent of GreetingService and adding tracing filter in Dubbo's filter chain.-->
<dubbo:reference id="greetingService" check="false" interface="com.alibaba.dubbo.samples.api.GreetingService" filter="tracing"/>
<!-- 4. zipkin related settings -->
<bean id="sender" class="zipkin2.reporter.beans.OkHttpSenderFactoryBean">
<property name="endpoint" value="http://localhost:9411/api/v2/spans"/>
</bean>
<bean id="tracing" class="brave.spring.beans.TracingFactoryBean">
<property name="localServiceName" value="client"/>
<property name="spanReporter">
<bean class="zipkin2.reporter.beans.AsyncReporterFactoryBean">
<property name="sender" ref="sender"/>
<!-- wait up to half a second for any in-flight spans on close -->
<property name="closeTimeout" value="500"/>
</bean>
</property>
<property name="currentTraceContext">
<bean class="brave.spring.beans.CurrentTraceContextFactoryBean">
<property name="scopeDecorators">
<bean class="brave.context.slf4j.MDCScopeDecorator" factory-method="create"/>
</property>
</bean>
</property>
</bean>
</beans>When the project is completed, the contents structure is as follows:
Running the Project
Run the entire link to check the Zipkin link tracing performance.
Starting the Zookeeper server
Execute the following commands to start a local Zookeeper server. If you have not installed it, download it from the ZooKeeper official website:
$ zkServer start
Starting the Zipkin server
Execute the following commands and start a Zipkin Server locally:
$ curl -sSL https://zipkin.io/quickstart.sh | bash -s
$ java -jar zipkin.jar
Starting HelloService
Start HelloService with the following commands, or start it directly on IDE:
$ mvn exec:java -Dexec.mainClass=com.alibaba.dubbo.samples.service.hello.Application
If startup is successful, “Hello service started” will display on the client end.
Starting GreetingService
Start GreetingService with the following commands, or start it directly on IDE:
$ mvn exec:java -Dexec.mainClass=com.alibaba.dubbo.samples.service.greeting.Application
If startup is successful, “Greeting service started” will display on the terminal.
Running the Dubbo client
Initiate remote call to GreetingService by running the client end of Dubbo with the following commands or start it directly on IDE:
$ mvn exec:java -Dexec.mainClass=com.alibaba.dubbo.samples.client.Application
When the execution is successful, the client end will display “greeting, hello, world” on the terminal.
Performing tracing analysis
Open the browser and visit http://localhost:9411. Press “Find Traces” to find the tracing analysis you just called as shown below:
You can also select each span to learn the details within the current-called boundary. For instance, the details of the span of hello-service is as shown below:
Further Reading
The aim of this article is to introduce basic link tracing concepts, and to outline the basics of using Zipkin. This article has also shown how to use Dubbo to build a simple call link, and how to use Zipkin for full link tracing. Zipkin supports Dubbo well, so the entire process is quite straightforward.
Zipkin’s support for Dubbo is based on the filter extension mechanism of Dubbo. This page on GitHub has more information.
The instances mentioned in this article can be obtained from the “dubbo-samples-zipkin” sub-module here. Additionally, Spring Cloud Sleuth 2.0 now supports Dubbo.
Alibaba Tech
First hand and in-depth information about Alibaba’s latest technology → Facebook: “Alibaba Tech”. Twitter: “AlibabaTech”.
