Practical advice for Go library authors

Writing better libraries

14 July 2016

Jack Lindamood

Software Engineer, Twitch

Goal of this talk

Many ways to do common things

Which way is preferred and why

Common pitfalls of Go libraries

What I wish I knew

General naming advice

Generally imports are not renamed

Standard practice is PACKAGE.FUNCTION()

Think of package as part of the name

// In Java, the package name is silent
Java:  new BufferedReader()

// In Go, the package name is where it is used
Go:    bufio.NewReader()

Naming examples

Tips for structs

stuttering struct names ok, but avoid if you can

// bad.  Too generic
var c client.Client

// stuttering, but accepted
var ctx context.Context

// optimal example
var h http.Client

Tips for functions

package functions should never stutter

// Ok.  Reads as a background context
context.Background()

// Not ok.  Context redundant
context.NewContext()

Object construction

No rigid constructor like other languages

Zero value sometimes works

Constructor function (NewXYZ) has ramifications

Using the struct or constructor function

// Using the struct
x := http.Client{}

// Constructor function
y := bytes.NewBuffer(...)

Zero value considerations

// Examples of a zero value
var c http.Client
y := bytes.Buffer{}

Read operations work well on nil map/slice/chan

Great for making your zero value behave

Less useful if

struct needs background goroutines

non zero/empty defaults are the best

Zero struct support is viral!!!!

So is nil support

Working with zero values

Example for defaults

// net/http/server.go
func (srv *Server) ListenAndServe() error {
    addr := srv.Addr
    if addr == "" {
        addr = ":http"
    }
    // ...
}

Example for nil reads

// bytes/buffer.go
type Buffer struct {
    buf []byte
    off int
}

func (b *Buffer) Len() int { return len(b.buf) - b.off }

New() constructor function

Not as terse as making zero value work

Constructors can do anything. Struct initialization only does one thing.

Risk people use struct w/o getting it from New

Some libraries hide this with a private struct, public interface

// anti-pattern.go
type authImpl struct {
}

type Auth interface {
    // Tends to be very large
}

func NewAuth() Auth {
}

What do these do?

func directUse() {
    x := grpc.Client{
        Port:     123,
        Hostname: "elaine82.sjc",
    }
    // ...
}

func constructor() {
    x := grpc.NewClient(123, "elaine82.sjc")
    // ...
}

Does constructor() do the following

spawn goroutines

allocate extra memory

panic

Singleton options

Go stdlib makes heavy use of singletons
not a fan of this

Beware hidden singletons (expvar/http/rand/etc)

General way singletons are done

var std = New(os.Stderr, "", LstdFlags)

// Singleton functions in package.  Same as struct functions
func Print(v ...interface{}) {
    std.Print(v...)
}

func (l *Logger) Print(v ...interface{}) {
    // ...
}

Optional config w/ functions

Use NewXYZ(WithThing(), WithThing()) can be clunky

Has not gained adoption

type Server struct {
    speed int
}

func Speed(rate int) func(*Server) {
    return func(s *Server) {
        s.speed = rate
    }
}

func NewServer(options ...func(*Server)) {
}

x := NewServer(Speed(10))

Config struct

Gaining popularity

Easy to understand all available options

Does zero work for default config? Sometimes not

Important to document if config is usable after being passed in

Easier to work with config management

// Usually JSON in config management
type Config struct {
    Speed int
}

func NewServer(conf *Config) {
}

Logging counter-example

import "log"

type Server struct {}

func (s *Server) handleRequest(r *Request) {
    if r.user == 0 {
        log.Println("No user set")
        return
    }
    // ...
}

Direct print to std streams

There are many logging frameworks

Cannot turn it off

Logging advice

Is it needed?

Logging is a callback

Log to an interface

Don't assume log package

Respect stdout/stderr

No singleton!

Better logging example

type Log interface {
    Println(v ...interface{})
}
type Server struct {
    Logger Log
}
func (s *Server) log(v ...interface{}) {
    if s.Logger != nil {
        s.Logger.Println(v...)
    }
}
func (s *Server) handleRequest(r *Request) {
    if r.user == 0 {
        s.log("No user set")
        return
    }    
}

Can we do even better?

Structured logging

interfaces vs structs

Accept interfaces, return structs

Some libraries only expose interfaces. Keep all structs private.

Tend to be large interfaces

Usually no need to include interface from outside your package/stdlib

Ok to do, just not needed
Prevents import cycles
Implicit interfaces get around this
context.Context made this difficult

Means an API that uses standard objects is usually best

Easier to implement

Let's make a random package

type Rand interface {
    ExpFloat64() float64
    Float32() float32
    Float64() float64
    Int() int
    Int31() int32
    Int31n(n int32) int32
    Int63() int64
    Int63n(n int64) int64
    Intn(n int) int
    NormFloat64() float64
    Perm(n int) []int
    Read(p []byte) (n int, err error)
    Seed(seed int64)
    Uint32() uint32
}

There are lots of ways to generate random things

What is wrong with this?

Avoiding large interfaces (rand.Rand)

Lots of ways to get random things

rand.Rand{} would be a very large interface

rand.Source is an interface. Very small. The building block.

Lesson: Turn large interfaces into small interfaces with wrapper logic

// A Source represents a source of uniformly-distributed
// pseudo-random int64 values in the range [0, 1<<63).
type Source interface {
    Int63() int64
    Seed(seed int64)
}

// A Rand is a source of random numbers.
type Rand struct {
    src Source
}

When to panic

Never?

panic() in a spawned goroutine is the worst

Panic is usually ok if:

Function begins with MustXYZ()
Operations on nil

// MustXYZ calls XYZ, panics on error.
// Note: Users still have the option to call XYZ directly
func MustCompile(str string) *Regexp {
    regexp, error := Compile(str)
    if error != nil {
        panic(`...`)
    }
    return regexp
}

Checking errors

Check all errors on interface function calls

Especially the ones you don't expect to error!

If you can't return it, always do something.

Log it
Increment something

When are returning errors appropriate

When a promise could not be kept

When an answer could not be given

// Counterexample
HasPermission(userID) error

// Better
HasPermission(userID) (bool, error)

Enabling debuggability for your library

Especially complex libraries can have state that isn't clear to the user

Suggest exposing an expvar.Var{} as a function

Debug logging can help

Ideally expose Stat() function for atomic integers around tracking information

type Stats struct {
    // atomic.AddInt64(&s.stats.TotalRequests, 1)
    TotalRequests int64
}
type Server struct {
    DebugLog Log
    stats    Stats
}
func (s *Server) Var() expvar.Var {
    return expvar.Func(func() interface{} {
        return struct{...}
    }
}

Designing for testing

Complex libraries could benefit users with a simple test helper

httptest is an example

Abstract away time/IO/os calls with interfaces

Maintain control

// Package stub.  For trim structs
var Now = time.Now

// Inside struct for most control
type Scheduler struct {
    Now func() time.Time
}

func (s *Scheduler) now() time.Time {
    if s.Now != nil {
        return s.Now()
    }
    return time.Now()
}

Channels

Stdlib has very few channels in the public API

Push what you would use channels for up the stack

Channels are rarely parameters to functions

Callbacks vs channels

Mixing mutexes and channels can be dangerous

Honestly, channels rarely belong in a public API

When to spawn goroutines

Some libraries use New() to spawn their goroutines.

Not ideal: Stdlib uses .Serve() functions

Close() should end all daemon threads

Push goroutine creation up the stack

// counterexample
func NewServer(..) *Server {
    s := &Server{...}
    go s.Start()
    return s
}

// ideal
func userLand() {
    // ...
    s := http.Server{}
    go s.Serve(l)
    // ...
}

When to use context.Context and when not to

All blocking/long operations should be cancelable!

Generally an abuse to store context.Context

When to use context.Value()

What other languages would use thread local for

So easy to abuse

Try hard not to use it

Singletons and context.Value() obscure your program's state machine

Context.Value() Should inform not control

Seriously though, try not to use context.Value()

If something is hard to do, make someone else do it

Great advice in library design, system design, and Dilbert life

The less hard things you try to do, the less likely you'll screw up whatever you're doing.

Hard things (threading/Mutexes/Deadlock/channels/encryption)

Push problems up the stack

Logging

Goroutines

Locking

Corollary: Try not to do things

Designing for efficiency

Correctness still trumps efficiency

Minimizing memory allocs is usually first priority

Avoid creating an API that forces memory allocations

Easy to optimize internals

Hard to change an API

func (s *Object) Encode() []byte {
    // ....
}

func (s *Object) WriteTo(w io.Writer) {
    // ...
}

Using /vendor in libraries

Package management is not ideal for go libraries

Don't use /vendor for libraries.

Will have issues if the library has singletons

Try to use implicit interfaces and injection

Don't expose vendored deps as part of the library's external API

Their github.com/a isn't the same as your github.com/b/vendor/github.com/a

Honestly, just don't use libraries in your library

npm::left-pad

Build tags

Cross OS libraries

Writing libraries that are compatible with new versions of Go

Example: http.Request.Cancel channel

Use build tags like // +build go1.5

Integration tests

Integration testing for libraries (with // +build integration)

Staying clean

Numerous static analysis tools for go

Use almost all of them

Once experienced, you can lint/grep them away

Build options (travis/circle/etc)

Many free continuous testing integrations for open source libraries

Helps ensure code works for various Go versions

Thank you