The Go Blog

Generating code

22 December 2014

A property of universal computation—Turing completeness—is that a computer program can write a computer program. This is a powerful idea that is not appreciated as often as it might be, even though it happens frequently. It's a big part of the definition of a compiler, for instance. It's also how the go test command works: it scans the packages to be tested, writes out a Go program containing a test harness customized for the package, and then compiles and runs it. Modern computers are so fast this expensive-sounding sequence can complete in a fraction of a second.

There are lots of other examples of programs that write programs. Yacc, for instance, reads in a description of a grammar and writes out a program to parse that grammar. The protocol buffer "compiler" reads an interface description and emits structure definitions, methods, and other support code. Configuration tools of all sorts work like this too, examining metadata or the environment and emitting scaffolding customized to the local state.

Programs that write programs are therefore important elements in software engineering, but programs like Yacc that produce source code need to be integrated into the build process so their output can be compiled. When an external build tool like Make is being used, this is usually easy to do. But in Go, whose go tool gets all necessary build information from the Go source, there is a problem. There is simply no mechanism to run Yacc from the go tool alone.

Until now, that is.

The latest Go release, 1.4, includes a new command that makes it easier to run such tools. It's called go generate, and it works by scanning for special comments in Go source code that identify general commands to run. It's important to understand that go generate is not part of go build. It contains no dependency analysis and must be run explicitly before running go build. It is intended to be used by the author of the Go package, not its clients.

The go generate command is easy to use. As a warmup, here's how to use it to generate a Yacc grammar. Say you have a Yacc input file called gopher.y that defines a grammar for your new language. To produce the Go source file implementing the grammar, you would normally invoke the standard Go version of Yacc like this: 

go tool yacc -o gopher.go -p parser gopher.y

The -o option names the output file while -p specifies the package name.

To have go generate drive the process, in any one of the regular (non-generated) .go files in the same directory, add this comment anywhere in the file: 

//go:generate go tool yacc -o gopher.go -p parser gopher.y

This text is just the command above prefixed by a special comment recognized by go generate. The comment must start at the beginning of the line and have no spaces between the // and the go:generate. After that marker, the rest of the line specifies a command for go generate to run.

Now run it. Change to the source directory and run go generate, then go build and so on: 

$ cd $GOPATH/myrepo/gopher
$ go generate
$ go build
$ go test

That's it. Assuming there are no errors, the go generate command will invoke yacc to create gopher.go, at which point the directory holds the full set of Go source files, so we can build, test, and work normally. Every time gopher.y is modified, just rerun go generate to regenerate the parser.

For more details about how go generate works, including options, environment variables, and so on, see the design document.

Go generate does nothing that couldn't be done with Make or some other build mechanism, but it comes with the go tool—no extra installation required—and fits nicely into the Go ecosystem. Just keep in mind that it is for package authors, not clients, if only for the reason that the program it invokes might not be available on the target machine. Also, if the containing package is intended for import by go get, once the file is generated (and tested!) it must be checked into the source code repository to be available to clients.

Now that we have it, let's use it for something new. As a very different example of how go generate can help, there is a new program available in the golang.org/x/tools repository called stringer. It automatically writes string methods for sets of integer constants. It's not part of the released distribution, but it's easy to install: 

$ go get golang.org/x/tools/cmd/stringer

Here's an example from the documentation for stringer. Imagine we have some code that contains a set of integer constants defining different types of pills: 

package painkiller

type Pill int

const (
    Placebo Pill = iota
    Aspirin
    Ibuprofen
    Paracetamol
    Acetaminophen = Paracetamol
)

For debugging, we'd like these constants to pretty-print themselves, which means we want a method with signature, 

func (p Pill) String() string

It's easy to write one by hand, perhaps like this: 

func (p Pill) String() string {
    switch p {
    case Placebo:
        return "Placebo"
    case Aspirin:
        return "Aspirin"
    case Ibuprofen:
        return "Ibuprofen"
    case Paracetamol: // == Acetaminophen
        return "Paracetamol"
    }
    return fmt.Sprintf("Pill(%d)", p)
}

There are other ways to write this function, of course. We could use a slice of strings indexed by Pill, or a map, or some other technique. Whatever we do, we need to maintain it if we change the set of pills, and we need to make sure it's correct. (The two names for paracetamol make this trickier than it might otherwise be.) Plus the very question of which approach to take depends on the types and values: signed or unsigned, dense or sparse, zero-based or not, and so on.

The stringer program takes care of all these details. Although it can be run in isolation, it is intended to be driven by go generate. To use it, add a generate comment to the source, perhaps near the type definition: 

//go:generate stringer -type=Pill

This rule specifies that go generate should run the stringer tool to generate a String method for type Pill. The output is automatically written to pill_string.go (a default we could override with the -output flag).

Let's run it: 

$ go generate
$ cat pill_string.go
// generated by stringer -type Pill pill.go; DO NOT EDIT

package pill

import "fmt"

const _Pill_name = "PlaceboAspirinIbuprofenParacetamol"

var _Pill_index = [...]uint8{0, 7, 14, 23, 34}

func (i Pill) String() string {
    if i < 0 || i+1 >= Pill(len(_Pill_index)) {
        return fmt.Sprintf("Pill(%d)", i)
    }
    return _Pill_name[_Pill_index[i]:_Pill_index[i+1]]
}
$

Every time we change the definition of Pill or the constants, all we need to do is run 

$ go generate

to update the String method. And of course if we've got multiple types set up this way in the same package, that single command will update all their String methods with a single command.

There's no question the generated method is ugly. That's OK, though, because humans don't need to work on it; machine-generated code is often ugly. It's working hard to be efficient. All the names are smashed together into a single string, which saves memory (only one string header for all the names, even if there are zillions of them). Then an array, _Pill_index, maps from value to name by a simple, efficient technique. Note too that _Pill_index is an array (not a slice; one more header eliminated) of uint8, the smallest integer sufficient to span the space of values. If there were more values, or there were negatives ones, the generated type of _Pill_index might change to uint16 or int8: whatever works best.

The approach used by the methods printed by stringer varies according to the properties of the constant set. For instance, if the constants are sparse, it might use a map. Here's a trivial example based on a constant set representing powers of two: 

const _Power_name = "p0p1p2p3p4p5..."

var _Power_map = map[Power]string{
    1:    _Power_name[0:2],
    2:    _Power_name[2:4],
    4:    _Power_name[4:6],
    8:    _Power_name[6:8],
    16:   _Power_name[8:10],
    32:   _Power_name[10:12],
    ...,
}

func (i Power) String() string {
    if str, ok := _Power_map[i]; ok {
        return str
    }
    return fmt.Sprintf("Power(%d)", i)
}

In short, generating the method automatically allows us to do a better job than we would expect a human to do.

There are lots of other uses of go generate already installed in the Go tree. Examples include generating Unicode tables in the unicode package, creating efficient methods for encoding and decoding arrays in encoding/gob, producing time zone data in the time package, and so on.

Please use go generate creatively. It's there to encourage experimentation.

And even if you don't, use the new stringer tool to write your String methods for your integer constants. Let the machine do the work.

By Rob Pike

Go 1.4 is released

10 December 2014

Today we announce Go 1.4, the fifth major stable release of Go, arriving six months after our previous major release Go 1.3. It contains a small language change, support for more operating systems and processor architectures, and improvements to the tool chain and libraries. As always, Go 1.4 keeps the promise of compatibility, and almost everything will continue to compile and run without change when moved to 1.4. For the full details, see the Go 1.4 release notes.

The most notable new feature in this release is official support for Android. Using the support in the core and the libraries in the golang.org/x/mobile repository, it is now possible to write simple Android apps using only Go code. At this stage, the support libraries are still nascent and under heavy development. Early adopters should expect a bumpy ride, but we welcome the community to get involved.

The language change is a tweak to the syntax of for-range loops. You may now write "for range s {" to loop over each item from s, without having to assign the value, loop index, or map key. See the release notes for details.

The go command has a new subcommand, go generate, to automate the running of tools to generate source code before compilation. For example, it can be used to automate the generation of String methods for typed constants using the new stringer tool. For more information, see the design document.

Most programs will run about the same speed or slightly faster in 1.4 than in 1.3; some will be slightly slower. There are many changes, making it hard to be precise about what to expect. See the release notes for more discussion.

And, of course, there are many more improvements and bug fixes.

In case you missed it, a few weeks ago the sub-repositories were moved to new locations. For example, the go.tools packages are now imported from "golang.org/x/tools". See the announcement post for details.

This release also coincides with the project's move from Mercurial to Git (for source control), Rietveld to Gerrit (for code review), and Google Code to Github (for issue tracking and wiki). The move affects the core Go repository and its sub-repositories. You can find the canonical Git repositories at go.googlesource.com, and the issue tracker and wiki at the golang/go GitHub repo.

While development has already moved over to the new infrastructure, for the 1.4 release we still recommend that users who install from source use the Mercurial repositories.

For App Engine users, Go 1.4 is now available for beta testing. See the announcement for details.

From all of us on the Go team, please enjoy Go 1.4, and have a happy holiday season.

By Andrew Gerrand

Half a decade with Go

10 November 2014

Five years ago we launched the Go project. It seems like only yesterday that we were preparing the initial public release: our website was a lovely shade of yellow, we were calling Go a "systems language", and you had to terminate statements with a semicolon and write Makefiles to build your code. We had no idea how Go would be received. Would people share our vision and goals? Would people find Go useful?

At launch, there was a flurry of attention. Google had produced a new programming language, and everyone was eager to check it out. Some programmers were turned off by Go's conservative feature set—at first glance they saw "nothing to see here"—but a smaller group saw the beginnings of an ecosystem tailored to their needs as working software engineers. These few would form the kernel of the Go community.

Gopher illustration by Renee French

After the initial release, it took us a while to properly communicate the goals and design ethos behind Go. Rob Pike did so eloquently in his 2012 essay Go at Google: Language Design in the Service of Software Engineering and more personally in his blog post Less is exponentially more. Andrew Gerrand's Code that grows with grace (slides) and Go for Gophers (slides) give a more in-depth, technical take on Go's design philosophy.

Over time, the few became many. The turning point for the project was the release of Go 1 in March 2012, which provided a stable language and standard library that developers could trust. By 2014, the project had hundreds of core contributors, the ecosystem had countless libraries and tools maintained by thousands of developers, and the greater community had many passionate members (or, as we call them, "gophers"). Today, by our current metrics, the Go community is growing faster than we believed possible.

Where can those gophers be found? They are at the many Go events that are popping up around the world. This year we saw several dedicated Go conferences: the inaugural GopherCon and dotGo conferences in Denver and Paris, the Go DevRoom at FOSDEM and two more instances of the biannual GoCon conference in Tokyo. At each event, gophers from around the globe eagerly presented their Go projects. For the Go team, it is very satisfying to meet so many programmers that share our vision and excitement.

More than 1,200 gophers attended GopherCon in Denver and dotGo in Paris.

There are also dozens of community-run Go User Groups spread across cities worldwide. If you haven't visited your local group, consider going along. And if there isn't a group in your area, maybe you should start one?

Today, Go has found a home in the cloud. Go arrived as the industry underwent a tectonic shift toward cloud computing, and we were thrilled to see it quickly become an important part of that movement. Its simplicity, efficiency, built-in concurrency primitives, and modern standard library make it a great fit for cloud software development (after all, that's what it was designed for). Significant open source cloud projects like Docker and Kubernetes have been written in Go, and infrastructure companies like Google, CloudFlare, Canonical, Digital Ocean, GitHub, Heroku, and Microsoft are now using Go to do some heavy lifting.

So, what does the future hold? We think that 2015 will be Go's biggest year yet.

Go 1.4—in addition to its new features and fixes—lays the groundwork for a new low-latency garbage collector and support for running Go on mobile devices. It is due to be released on December 1st 2014. We expect the new GC to be available in Go 1.5, due June 1st 2015, which will make Go appealing for a broader range of applications. We can't wait to see where people take it.

And there will be more great events, with GothamGo in New York (15 Nov), another Go DevRoom at FOSDEM in Brussels (Jan 31 and Feb 1; get involved!), GopherCon India in Bengaluru (19-21 Feb), the original GopherCon back at Denver in July, and dotGo on again at Paris in November.

The Go team would like to extend its thanks to all the gophers out there. Here's to the next five years.

To celebrate 5 years of Go, over the coming month the Gopher Academy will publish a series of articles by prominent Go users. Be sure to check out their blog for more Go action.

By Andrew Gerrand

Go at Google I/O and Gopher SummerFest

6 October 2014

Introduction

The week of June 23rd was a good week for gophers in San Francisco. Go was a big part of Google I/O on Wednesday and Thursday, and on Monday we took advantage of the large gopher population to run the Go SummerFest, a special instance of the GoSF meetup. This blog post is a recap of both events.

Gopher SummerFest

On the Monday, more than 200 gophers gathered at the Google office in San Francisco to hear a series of talks:

  • The State of Go, (slides and video) by Andrew Gerrand.
  • I was wrong, again! (slides and video), by Derek Collison.
  • Go at Splice (slides), by Matt Aimonetti
  • Quick testing with quick (slides), by Evan Shaw
  • Something about Go (no slides), by Blake Mizerany.

More comments and pictures from the event are available on the meetup event page.

Go at Google I/O

On the Wednesday and Thursday, Go was at Google I/O in two different formats: the Go booth in the sandbox area and the Go code labs available in the code lab area and all around the world through I/O Extended.

The Go booth

The Go booth was part of the Developer Sandbox area.

For the two days of the conference, some gophers from Google and other companies gave a series of talks and demonstrations. The talks were not recorded, but the slides and some screencasts and blog posts will be shared soon.

  • Organizing Go Code, by David Crawshaw. (slides)
  • Testing Techniques, by Andrew Gerrand. (video and slides)
  • Go for Java Developers, by Francesc Campoy. (slides)
  • Camlistore: Android, ARM, App Engine, Everywhere, by Brad Fitzpatrick. (slides)
  • Go Compilation Complexities, by Ian Lance Taylor. (slides)
  • SourceGraph: a Code Search Engine in Go, by Quinn Slack. (video and slides)

We also organized Q&A sessions and lightning talks by members of the Go community:

The Go code lab

This year attendees of Google I/O had a code lab area with self-service computers where they could sit and learn Go. The code labs were also available to anyone through the Google I/O extended brand. You can try it yourself at io2014codelabs.appspot.com.

Conclusion

Thanks to the organizers, speakers, and attendees who helped make these events a great success. See you next year. (Or at dotGo this week!)

By Francesc Campoy

Deploying Go servers with Docker

26 September 2014

Introduction

This week Docker announced official base images for Go and other major languages, giving programmers a trusted and easy way to build containers for their Go programs.

In this article we'll walk through a recipe for creating a Docker container for a simple Go web application and deploying that container to Google Compute Engine. If you're not familiar with Docker, you should read Understanding Docker before reading on.

The demo app

For our demonstration we will use the outyet program from the Go examples repository, a simple web server that reports whether the next version of Go has been released (designed to power sites like isgo1point4.outyet.org). It has no dependencies outside the standard library and requires no additional data files at run time; for a web server, it's about as simple as it gets.

Use "go get" to fetch and install outyet in your workspace: 

$ go get github.com/golang/example/outyet

Write a Dockerfile

Create a file named Dockerfile in the outyet directory with the following contents: 

# Start from a Debian image with the latest version of Go installed
# and a workspace (GOPATH) configured at /go.
FROM golang

# Copy the local package files to the container's workspace.
ADD . /go/src/github.com/golang/example/outyet

# Build the outyet command inside the container.
# (You may fetch or manage dependencies here,
# either manually or with a tool like "godep".)
RUN go install github.com/golang/example/outyet

# Run the outyet command by default when the container starts.
ENTRYPOINT /go/bin/outyet

# Document that the service listens on port 8080.
EXPOSE 8080

This Dockerfile specifies how to construct a container that runs outyet, starting with the basic dependencies (a Debian system with Go installed; the official golang docker image), adding the outyet package source, building it, and then finally running it.

The ADD, RUN, and ENTRYPOINT steps are common tasks for any Go project. To simplify this, there is an onbuild variant of the golang image that automatically copies the package source, fetches the application dependencies, builds the program, and configures it to run on startup.

With the onbuild variant, the Dockerfile is much simpler: 

FROM golang:onbuild
EXPOSE 8080

Build and run the image

Invoke Docker from the package directory to build an image using the Dockerfile: 

$ docker build -t outyet .

This will fetch the golang base image from Docker Hub, copy the package source to it, build the package inside it, and tag the resulting image as outyet.

To run a container from the resulting image: 

$ docker run --publish 6060:8080 --name test --rm outyet

The --publish flag tells docker to publish the container's port 8080 on the external port 6060.

The --name flag gives our container a predictable name to make it easier to work with.

The --rm flag tells docker to remove the container image when the outyet server exits.

With the container running, open http://localhost:6060/ in a web browser and you should see something like this:

(If your docker daemon is running on another machine (or in a virtual machine), you should replace localhost with the address of that machine. If you're using boot2docker on OS X or Windows you can find that address with boot2docker ip.)

Now that we've verified that the image works, shut down the running container from another terminal window: 

$ docker stop test

Create a repository on Docker Hub

Docker Hub, the container registry from which we pulled the golang image earlier, offers a feature called Automated Builds that builds images from a GitHub or BitBucket repository.

By committing the Dockerfile to the repository and creating an automated build for it, anyone with Docker installed can download and run our image with a single command. (We will see the utility of this in the next section.)

To set up an Automated Build, commit the Dockerfile to your repo on GitHub or BitBucket, create an account on Docker Hub, and follow the instructions for creating an Automated Build.

When you're done, you can run your container using the name of the automated build: 

$ docker run goexample/outyet

(Replace goexample/outyet with the name of the automated build you created.)

Deploy the container to Google Compute Engine

Google provides container-optimized Google Compute Engine images that make it easy to spin up a virtual machine running an arbitrary Docker container. On startup, a program running on the instance reads a configuration file that specifies which container to run, fetches the container image, and runs it.

Create a containers.yaml file that specifies the docker image to run and the ports to expose: 

version: v1beta2
containers:
- name: outyet
  image: goexample/outyet
  ports:
  - name: http
    hostPort: 80
    containerPort: 8080

(Note that we're publishing the container's port 8080 as external port 80, the default port for serving HTTP traffic. And, again, you should replace goexample/outyet with the name of your Automated Build.)

Use the gcloud tool to create a VM instance running the container: 

$ gcloud compute instances create outyet \
    --image container-vm-v20140925 \
    --image-project google-containers \
    --metadata-from-file google-container-manifest=containers.yaml \
    --tags http-server \
    --zone us-central1-a \
    --machine-type f1-micro

The first argument (outyet) specifies the instance name, a convenient label for administrative purposes.

The --image and --image-project flags specify the special container-optimized system image to use (copy these flags verbatim).

The --metadata-from-file flag supplies your containers.yaml file to the VM.

The --tags flag tags your VM instance as an HTTP server, adjusting the firewall to expose port 80 on the public network interface.

The --zone and --machine-type flags specify the zone in which to run the VM and the type of machine to run. (To see a list of machine types and the zones, run gcloud compute machine-types list.)

Once this has completed, the gcloud command should print some information about the instance. In the output, locate the networkInterfaces section to find the instance's external IP address. Within a couple of minutes you should be able to access that IP with your web browser and see the "Has Go 1.4 been released yet?" page.

(To see what's happening on the new VM instance you can ssh into it with gcloud compute ssh outyet. From there, try sudo docker ps to see which Docker containers are running.)

Learn more

This is just the tip of the iceberg—there's a lot more you can do with Go, Docker, and Google Compute Engine.

To learn more about Docker, see their extensive documentation.

To learn more about Docker and Go, see the official golang Docker Hub repository and Kelsey Hightower's Optimizing Docker Images for Static Go Binaries.

To learn more about Docker and Google Compute Engine, see the Container-optimized VMs page and the google/docker-registry Docker Hub repository.

By Andrew Gerrand

See the index for more articles.