Go Patterns: Pipelines

by about Go, Software Engineering in Technology

Sometimes, when reviewing code, you come across a huge spaghetti of sadness. You’d love to refactor it, but it is just a list of steps that need to be executed in order.

Think about a problem like installing a server: you need to partition the hard disk, create the filesystem, copy the files, install the bootloader, etc. If you are developing this for the first time, you might write everything down in one go.

Initially, this code may be good and work as intended, but as you need to change it over and over again it may become messy. How do you refactor it?

The first step will undoubtedly be to split it into separate function, and the resulting main function may look something like this:

func installHost() error {
    if err := performStep1(); err != nil {
        return fmt.Errorf("step 1 failed (%w)", err)
    }
    if err := performStep2(); err != nil {
        return fmt.Errorf("step 2 failed (%w)", err)
    }
    if err := performStep3(); err != nil {
        return fmt.Errorf("step 3 failed (%w)", err)
    }
    if err := performStep4(); err != nil {
        return fmt.Errorf("step 4 failed (%w)", err)
    }
}

All is good, but even this may become unmaintainable if you have 3-4 steps.

Using callbacks ▲ Back to top

As a first step of optimizing the code above, let’s use the fact that functions can be passed around in variables and shorten the installHost function. First, we create a slice that contains the functions to run as values.

func installHost() error {
    steps := []func() error{
        performStep1,
        performStep2,
        performStep3,
        performStep4,
    }

    for i, stepFunc := range steps {
        if err := stepFunc(); err != nil {
            return fmt.Errorf("step %d failed (%w)", i, err)
        }
    }
}

Passing state ▲ Back to top

What if you wanted to pass some information between steps? After all, there are not many use cases for running isolated steps.

We start by defining our data structure:

type pipelineData struct {
    field1 string
    field2 int
}

Now we can modify the code from above to create a pointer to such a data structure and pass it to the function. We also need to adapt our function signature.

func installHost() error {
    data := &pipelineData{}
    steps := []func(*pipelineData) error{
        performStep1,
        performStep2,
        performStep3,
        performStep4,
    }

    for i, stepFunc := range steps {
        if err := stepFunc(data); err != nil {
            return fmt.Errorf("step %d failed (%w)", i, err)
        }
    }
}

Making it reusable ▲ Back to top

Let’s think a little further: you would probably want to use this pattern in a couple of places. What if we made it reusable? We could create a function that accepts the functions to call and then iterates over these, calling them in order.

func pipeline(
    steps []func() error,
) error {
    for i, step := range steps {
        if err := step(); err != nil {
            return fmt.Errorf("step %d failed (%w)", i, err)
        }
    }
    return nil
}

There is one drawback to using this method: we can’t pass our data structure from before. Or can we? (Of course we can.)

We can use receivers to tie the functions to the data set. We can implement the steps with the *pipelineData receiver. We then initialize our data variable as before and pass the functions to the pipeline with the data attached.

func (p *pipelineData) performStep1() error {
    // Do what step 1 needs to do.
}

func main() {
    data := &pipelineData{}
    err := pipeline([]func() error {
        data.performStep1,
        data.performStep2,
        data.performStep3,
        data.performStep4,
    })
    //...
}

This is very similar to how OOP works in Go.

You can declare functions with receivers both with and without pointers for the variable. If you remove the pointer, the function will not be able to modify the data.

That’s it! You can now create a pipeline and don’t need to unnecessarily bloat your code. You can even extend this pattern to your liking.