Cloud Trace

Overview

Cloud Trace is a feature of the Google Cloud Platform that allows you to view the RPCs (remote procedure calls) invoked by your App Engine application and to view and analyze the time taken to complete each RPC.

An App Engine RPC is a round trip network call between your application and an App Engine service. For example, each of these calls made by a Python application result in RPCs:

• Datastore calls such as db.get(), db.put(), or db.query().
• Memcache calls such as memcache.get(), or memcache.get_multi().
• URL Fetch calls such as urlfetch.fetch().

Cloud Trace selects a representative subset of requests to your application, and traces all RPCs made by your application when handling those requests. The traces can help you find performance bottlenecks in your application. These can include:

• Unnecessary service calls that are not needed to process the request
• Repeated service calls to get the same data
• Serial service calls that can be batched or executed in parallel

Cloud Trace also allows you to view the latency distribution for a set of requests and extract trace samples for different latency percentiles by analyzing the traces. You can also compare performance of two sets of requests. For example, you can compare the performance of your application before and after a release by comparing the traces for requests received.

Accessing Cloud Trace

To access Cloud Trace, open your project in the Google Developers Console and select Monitoring > Traces in the left-hand navigation panel. The Trace List view opens by default. Click on the Settings tab, and make sure that Traces is set to "On". You must have Owner or Edit permissions to change this setting.

Viewing traces

The Trace List displays a sortable table that shows some or all of the recent requests to your application (Cloud Trace does not guarantee that all requests will be traced).

For each request, the table shows the following:

• Latency: The time it took the application to process the request
• URI: The relative URI that handled the request
• Time: The time at which the request was made

You can search for traces corresponding to a set of requests by doing any of the following:

• Enter the relative URI or relative URI prefix that handled the requests in the Search field. The URI prefix must begin with "/".
• Filter for a particular type of HTTP request (e.g. "POST") with the **HTTP METHOD" drop-down list.
• Filter for requests returning a particular HTTP status code (e.g. 500 for "Internal Server Error") with the HTTP Status drop-down list.
• Select an application module from the Module drop-down list.
• Select a version of the module in the Version drop-down list.
• Select a time range for the requests in the Time Range drop-down list.

Once you have entered or specified the desired values, you can click Search to see the traces for the set of requests that match all fields. For example, if you want to view the traces for requests sent to the default version of your application at /tasks/add_comment, /tasks/delete_comments, and /tasks/view_comments URIs in the last one hour, you can:

• Enter /tasks in the search box.
• Select the default version of your app in the Version drop-down list.
• Select Last hour in the Time Range drop-down list.

Viewing trace details

You can click any URI in the URI column in the Trace List view to view details about the request.

The Trace Details display shows the following:

• URI: Fully-qualified URI that handled the request, including query parameters
• Response time: The time it took the application to process the request
• Timestamp: The time at which the request was made
• HTTP Method: The HTTP method specified in the request
• Status code: The status code returned for the request. 200 is "OK".
• Module: the name of the module that handled the request.
• Version: The version of the module that handled this request.

You can also click on tabs in the panel below to navigate to the following additional views:

• Timeline: Displays a chart that shows the total time it took the application to process the request, as well as the time it took to process each RPC call initiated by the request. Click on any of the bars in the graph to see details for that request or RPC call. (Displayed by default.)
• Summary: Displays a list of the request and RPC URIs along with the number of invocations and the total time it took the application to process them.

Viewing logs

Under "Log", click View to view the App Engine logs for the request.

Note that when viewing App Engine logs, if the request has been traced by Cloud Trace, you can click the duration field displayed on the first row of the log entry to view its trace details.

Analyzing traces

You can use the Cloud Trace feature to create and view analysis reports that show the latency distribution of requests to your application. Analysis reports also display a table with the latency values for different percentile ranges.

Creating analysis reports

To create an analysis report, click the Analysis Reports tab that appears at the top of Cloud Trace interface. This allows you to view a list of reports that you have previously created. Click New Report to create the new report.

The following form is displayed:

Enter the following in the form:

• URI: The relative URI of requests whose traces you want to analyze. If you leave this field blank, it defaults to / (all requests).
• Http Method: You can limit your report to only include requests that specify a particular HTTP method.
• Http Status: You can limit your report to only include requests that return a particular HTTP status code.
• Report Name: An optional name for the report. If you do not enter a name, a default name in the format trace_name@start_datetime is used.
• Module: You can select a particular module from your application.
• Version: If you want to analyze requests for a particular application version, select it here.
• Time Range: Specify a time range for the requests.

You can also select the Create comparison checkbox to compare the traces for requests received by the same set of URIs but at different time periods, or requests received by the same set of URIs in different versions of your module at same or different time periods. When you select the Create comparison checkbox, additional fields are displayed that let you specify a module, version, or time period.

Click Submit to create the report. It may take several minutes before the report is complete, particularly if you are analyzing a large number of requests (greater than 10,000). The Analysis Reports tab appears after you have created the report and displays a list of reports that are in-progress and completed.

Viewing analysis reports

In the Analysis Reports tab, click the name of the report you want to view.

The Overall latency for requests that make remote procedure calls panel shows the latency distribution of requests in the report using a logarithmic time scale on the X axis. The Latency panel shows the latency for values for different percentile ranges. You can click the links to sample traces provided to view the Trace Details for requests whose latency values fall within a given percentile.

If the analysis has detected any bottlenecks, there will also be a Bottlenecks panel that highlights the RPCs that are significant latency contributors at different latency percentiles.

All rights reserved.