Bart Fokker

Tutorial: Identicon generator in Go

Bart Fokker — Wed, 11 Jan 2017 00:00:00 GMT

Introduction

In this tutorial we are going to build a simple identicon generator in Go. The source code can be found here: Github repo. But what exactly is an Identicon? I think most of you are familiar with the standard logo you get when creating a new account on Github.

Example:

What you see here is a visual representation of a hash value. What this basically means is that a input is hashed with a hashing algorithm(md5, sha256) and the output is used to generate a image. This way an identicon will always be the same when we generate it with the same input.

Too much talking already, lets start coding!

First we start by defining an Identicon struct

type Identicon struct {
    name       string
    hash       [16]byte
    color      [3]byte
}

The struct holds 3 properties:

name: This will be the input name given to generate an identicon.
hash: 16 byte array containing the hashed value of the name, we are using [16]byte here because our hashing functions returns a byte array with 16 values.
color: 3 byte array containing Red,Green,Blue

Nice we have defined the basic container to save the values, now we can listen to user input and hash the incoming value. We use the Sum method from the "crypto/md5" package. This will return us a [16]byte checksum of the input.

func hashInput(input []byte) Identicon {
    // generate checksum from input
    checkSum := md5.Sum(input)
    // return the identicon
    return Identicon{
            name: string(input),
            hash: checkSum,
    }
}

We can use the method as follow:

// converting string to byte array
data := []byte("bart")
// calling hashinput method on this byte array
hashInput(data)
// outputs: [245 65 70 163 252 130 171 23 229 38 86 149 178 63 100 107]

Now we need to define a color for our Identicon, we want the identicon to always be the same when we generate it again with the same input. Our trick is to take the first 3 bytes of the checksum, this way we get the same color over and over again. The first value will be Red, the second value Green and the third value will be Blue. This way we generate a valid RGB value and we can use it together with the color package in go.

func pickColor(identicon Identicon) Identicon {
    // first we make a byte array with length 3
    rgb := [3]byte{}
    // next we copy the first 3 values from the hash to the rgb array
    copy(rgb[:], identicon.hash[:3])
    // we than assign it to the color value
    identicon.color = rgb
    // return the modified identicon
    return identicon
}

Right now we have an Identicon which holds a name, color and the hash. What's special about Identicons is that the left hand side is the mirror of the right hand side. We have to generate a 5x5 grid. If we take this byte array as example: [245 65 70 163 252 130 171 23 229 38 86 149 178 63 100 107]. The result we want from the buildgrid function is

[ 246, 65, 70, 65, 245,
    163, 252, 130, 252, 163,
    171, 23, 229, 23, 171,
    38, 86, 149, 86, 38,
    178, 63, 100, 63, 178 ]

As you can see, the left hand side is mirrored on the right hand side in the array. We could use a matrix but for this example a simple array is sufficient. First we have to add another field to the identicon struct which will hold the value of the grid

type Identicon struct {
    name       string
    hash       [16]byte
    color      [3]byte
    grid       []byte // New property to hold the grid
}

func buildGrid(identicon Identicon) Identicon {
    // Create empty grid
    grid := []byte{}

    // Process the hash in chunks of 3 bytes to build 5 mirrored rows.
    // Loop condition ensures we take exactly 5 chunks (3*5 = 15 bytes) from the 16-byte MD5 hash.
    for i := 0; i+3 <= len(identicon.hash); i += 3 {
        // Create a 5-byte chunk and copy the 3 bytes into it
        chunk := make([]byte, 5)
        copy(chunk, identicon.hash[i:i+3])

        // Mirror the first two values to create the 5-wide row
        chunk[3] = chunk[1]
        chunk[4] = chunk[0]

        // Append the chunk to the grid
        grid = append(grid, chunk...)
    }

    identicon.grid = grid
    return identicon
}

Now we almost have a grid we can use to draw a real image, but first we have to specify which points in the grid we are going to fill with the color. If you look at the grid you can see it would be a nice fit if we only fill in the odd values. For this to work we have to keep track of the index inside the grid, because otherwise we won't know which of the items to fill. We do this by specifying another struct, which will hold the value and the index of the item in the grid.

type GridPoint struct {
    value byte
    index int
}

We have to keep track of the GridPoints inside the Identicon structure.

type Identicon struct {
    name       string
    hash       [16]byte
    color      [3]byte
    grid       []byte
    gridPoints []GridPoint // Filtered points in the grid
}

The GridPoints will be the values we are going to fill in with the previous generated color. Lets have a look at the method to filter the odds grid.

func filterOddSquares(identicon Identicon) Identicon {
    grid := []GridPoint{} // create a placeholder to hold the values of the loop
    for i, code := range identicon.grid { // loop over the grid
        if code%2 == 0 { // check if the value is odd or not
            // create a new Gridpoint where we save the value and the index in the grid
            point := GridPoint{
                value: code,
                index: i,
            }
            // append the item to the new grid
            grid = append(grid, point)
        }
    }
    // set the property
    identicon.gridPoints = grid
    return identicon // return the modified identicon
}

So right now we have an Identicon which has the values we need to fill in, but we can't draw an image of only those values. We have to transform them to a pixelmap. Because we want to draw a rectangle for every odd point we need to calculate the topLeft x and y and the bottom right x and y. With those values we can make a rectangle.

First we specify a structure to hold the dimensions.

type Point struct {
    x, y int
}

type DrawingPoint struct {
    topLeft     Point
    bottomRight Point
}

But how can we relate the values from the grid to actual pixels on a pixelmap? We specify here first that our image will be of size 250 pixels by 250 pixels. As mentioned before our grid was 5x5 so 1 piece in the grid will be 50 pixels by 50 pixels, because 250/5 = 50. With this knowledge we can calculate the borders to draw. With two simple formula's:

horizontal: (x % 5) * 50
vertical: (x / 50) * 50 where x = index of the gridpoint

I won't zoom in any further on the formula's, because they are out of scope of this article. But remember for now that they allow us to calculate the positions in the pixelmap. Lets have a look at how we can use those formula's to generate dimensions for our rectangle. First we modify our Identicon struct to also hold a property of pixelMap. This pixelMap will contain the dimensions to draw in the png.

type Identicon struct {
    name       string
    hash       [16]byte
    color      [3]byte
    grid       []byte
    gridPoints []GridPoint
    pixelMap   []DrawingPoint // pixelMap for drawing
}

Now we can save the pixelmap in our Identicon struct

func buildPixelMap(identicon Identicon) Identicon {
    drawingPoints := []DrawingPoint{} // define placeholder for drawingpoints

    // Closure, this function returns a Drawingpoint
    pixelFunc := func(p GridPoint) DrawingPoint {
        // This is the formula, we use the index from the gridpoint to calculate the horizontal dimension
        horizontal := (p.index % 5) * 50
        // This is the formula, we use the index from the gridpoint to calculate the vertical dimension
        vertical := (p.index / 5) * 50
        // this is the topleft point with x and the y
        topLeft := Point{horizontal, vertical}
        // the bottom right point is just the topleft point +50 because 1 block in the grid is 50x50
        bottomRight := Point{horizontal + 50, vertical + 50}

        return DrawingPoint{ // We then return the drawingpoint
            topLeft,
            bottomRight,
        }
    }

    for _, gridPoint := range identicon.gridPoints {
        // for every gridPoint we calculate the drawingpoints and we add them to the array
        drawingPoints = append(drawingPoints, pixelFunc(gridPoint))
    }
    identicon.pixelMap = drawingPoints // set the drawingpoint value on the identicon
    return identicon // return the modified identicon
}

Sweet our datastructure now also holds a pixelmap we can use to draw rectangles. To actually draw rectangle we need to take the four dimensions from the DrawingPoint struct. And pass them to a draw function. for drawing rectangles I'll use a external library draw2dRepo. This library make's it very easy for us to draw a rectangle. We pass in a image, color and the dimensions to our rect function. This will create a filled rectangle for us.

func rect(img *image.RGBA, col color.Color, x1, y1, x2, y2 float64) {
    gc := draw2dimg.NewGraphicContext(img) // Prepare new image context
    gc.SetFillColor(col) // set the color
    gc.MoveTo(x1, y1) // move to the topleft in the image
    // Draw the lines for the dimensions
    gc.LineTo(x1, y1)
    gc.LineTo(x1, y2)
    gc.MoveTo(x2, y1) // move to the right in the image
    // Draw the lines for the dimensions
    gc.LineTo(x2, y1)
    gc.LineTo(x2, y2)
    // Set the linewidth to zero
    gc.SetLineWidth(0)
    // Fill the stroke so the rectangle will be filled
    gc.FillStroke()
}

Now we can draw filled rectangles. We just have to loop over the pixelMap and call the rectangle function.

func drawRectangle(identicon Identicon) error {
    // We create our default image containing a 250x250 rectangle
    var img = image.NewRGBA(image.Rect(0, 0, 250, 250))
    // We retrieve the color from the color property on the identicon
    col := color.RGBA{identicon.color[0], identicon.color[1], identicon.color[2], 255}

    // Loop over the pixelmap and call the rect function with the img, color and the dimensions
    for _, pixel := range identicon.pixelMap {
        rect(
            img,
            col,
            float64(pixel.topLeft.x),
            float64(pixel.topLeft.y),
            float64(pixel.bottomRight.x),
            float64(pixel.bottomRight.y),
        )
    }
    // Finally save the image to disk
    return draw2dimg.SaveToPngFile(identicon.name+".png", img)
}

Sweet!! we have made an Identicon generator, all the functions are there but there is only thing left. As you may have noticed most of the functions where modifying an Identicon and then returning an Identicon. This is a perfect example for a pipe function, we can make a placeholder for the function like this:

type Apply func(Identicon) Identicon

And we can define a function which takes x amount of Apply types and apply them to an Identicon

func pipe(identicon Identicon, funcs ...Apply) Identicon {
    for _, applyer := range funcs {
        identicon = applyer(identicon)
    }
    return identicon
}

Finally lets wrap everything in a main funtion and define a flag for the name input.

func main() {
    var (
        name = flag.String("name", "", "Set the name where you want to generate an Identicon for")
    )
    flag.Parse()

    if *name == "" {
        flag.Usage()
        os.Exit(0)
    }

    data := []byte(*name)
    identicon := hashInput(data)

    // Pass in the identicon, call the methods which you want to transform
    identicon = pipe(identicon, pickColor, buildGrid, filterOddSquares, buildPixelMap)

    // we can use the identicon to insert to our drawRectangle function
    if err := drawRectangle(identicon); err != nil {
        log.Fatalln(err)
    }
}

Running this with input bart will give us:

Congratulations, you just made your own identicon generator.

Conclusion

Thanks for reading my blog post, I hope you all enjoyed it. I was inpsired by a elixir course on udemy where this was a exercise. I thought it would be fun to rewrite it in go. In a few lines you can create your own identicon generator, currently you can't generate bigger or smaller identicons. I leave that one to implement for the reader.

Happy programming!

Build your own ASCII github issue table in Go!

Bart Fokker — Fri, 27 Jan 2017 00:00:00 GMT

Introduction

In this blogpost I'm going to show you how you can create your own github issue table for inside your terminal. We will be covering some cool features and the implementation.

Code can be found here: Github repo

The looks of the application

Programming

What we want to achieve with the application is that we can specify a github repository url somehow and display x amount of issues in a pretty printed table. Since our application is going to be a command line application we need some way to retrieve the input from the user. For this we specify 2 flags:

var (
    amount = flag.Int("amount", -1, "Set the maximum amount of issues to show")
    repo   = flag.String("repo", "", "Repository on http://www.github.com to list the issues from")
)

This is the only input our application need (for now). After the user have filled in our flags we need a way to parse that input.

u, err := url.Parse(githubRepoURL)
if err != nil {
	fmt.Printf("Invalid url! %s", err)
	os.Exit(1)
}
path := strings.Split(u.EscapedPath(), "/")
if len(path) < 3 {
	fmt.Println("Invalid github URL")
	os.Exit(1)
}

Example github url path: /barthr/issuestable Here we parse the url and we split it on the "/". If the path contains less than 3 items the url is invalid, this means there isn't a repository name and a repository owner. With this information we can retrieve the issues from the github API. For this we use the Go github client. This client library provides us with methods for retrieving issues from repository's.

Lets do this!

// New github client
client = github.NewClient(nil)

// Initiate empty options with 100 records to fetch each page (maximum)
opt := &github.IssueListByRepoOptions{
    ListOptions: github.ListOptions{PerPage: 100},
}

var issues []*github.Issue
// Check for amount less than 0, if that is the case (no amount specified) we fetch all the issues
// Amount flag is -1 if not set (this way we keep looping until there aren't any pages left)
for len(issues) <= *amount || *amount < 0 {
    // Retrieve first batch of items
    newIssues, resp, err := client.Issues.ListByRepo(path[1], path[2], opt)
    if err != nil {
    	log.Fatalf("error fetching issues for %v/%v: %v", path[1], path[2], err)
    }
    // add them to the issue array
    issues = append(issues, newIssues...)
    // if the response contains no next page stop looping, we fetched all the issues
    if resp.NextPage == 0 {
    	break
    }
    // Set the next page on the options passed to the ListByRepo call
    opt.ListOptions.Page = resp.NextPage
}

First we initiate a issue array. This will be the array which holds the issues we want to show. Because we can't get all the issues at once from the github api, we have to add pagination to fetch the remaining ones. Luckily the github api client supports this. The ListByRepo call also returns a page object which holds the next page. We keep looping until we have the amount specified by the flag. If no amount value is given we keep looping until we have all the issues

We can now check if we have any issues and if we don't we shouldn't be writing a table

if len(issues) == 0 {
    fmt.Printf("No issues found for this repository")
    os.Exit(0)
}

Finally we can draw the table for our issues. The things we want to show are the index, number of the issue, created_at and the title. We will be using a library for drawing the ASCII table Go tablewriter

// Create a new tablewriter to Stdout
table := tablewriter.NewWriter(os.Stdout)
// Set the headers we want
table.SetHeader([]string{"index", "number", "created_at", "title"})

// Loop over the issues
for index, issue := range issues {
    // if the index == the amount flag than we need to stop looping, the desired result is there
    if index == *amount {
    	break
    }
    // Create a row for the tablewriter (the tablewriter expects a string array)
    // We convert all of our variables to a string
    row := make([]string, 4)
    row[0] = strconv.Itoa(index)
    row[1] = strconv.Itoa(*issue.Number)
    row[2] = issue.CreatedAt.String()
    row[3] = *issue.Title
    table.Append(row)
}
// Set a row line
table.SetRowLine(true)
// Render it to the screen
table.Render()

We can use the application like this:

issuestable -repo github.com/golang/go -amount 100

Conclusion

With a few lines of code (and with the help of a few library's) we can easily create our custom ASCII table displaying github issues for the repo we want. This is cool, but a little bit boring at the moment. It would be nice if we can also specify custom time events and maybe even issue tags. This is all very easy with the github client library and we can easily add more items to our ascii table. You could even take this further and make a whole interactive issue browser inside your terminal. I will leave that one as a exercise for the reader :).

Thanks all for reading my blog post!

Clean http handlers in Go!

Bart Fokker — Fri, 07 Jul 2017 00:00:00 GMT

Introduction

For this blog post we are going to take a look at the http.HandlerFunc type and how we can improve it to make more elegant and clean handlers. Following the idioms of Go and staying compatible with the standard library.

Handlers

In Go A Handler is a type which responds to an HTTP request.

type Handler interface {
        ServeHTTP(http.ResponseWriter, *http.Request)
}

Any struct implementing the ServeHTTP method from the interface can be used to handle http requests. This is very powerful and flexible. It is easy to add http handling capabilities to any structure in your program.

Although most of the time this is not the way a lot of people use it. It is still limited to the fact that you need to implement it on a structure which is not always what you would like. A single function would be easier to use than implementing an interface. Luckily Go has an solution for this.

Say Hi to the http.HandlerFunc

type HandlerFunc func(http.ResponseWriter, *http.Request)

The HandlerFunc is basically an adapter for the Handler interface. Because the HandlerFunc is a type it can implement methods on that type. Note that the type is actually a function so any function containing the same signature as the HandlerFunc can be easily casted to this type. When passing it as a parameter this happens implicitly and you wont even know the difference.

Ok cool, so here we are we've seen how the Handler and HandlerFunc works, but how can we actually extend them? And why do we even need to extend them, they are already powerful aren't they?

If you have ever written more complex http request handlers in Go you probably know that they can grow really big because of the verbose error handling and early returns.

func IndexHandler(w http.ResponseWriter, r *http.Request) {
  // Do something
  v, err := ...
  // check for err
  if err != nil {
    w.WriteHeader(http.StatusInternalServerError)
    w.Write([]byte(err.Error()))
    return
  }
  // More code
  // ...
}

This is common code and because the functions returns void we have to terminate the function early if we want to stop the function from writing more to the given io.Writer. Of course an else would be possible here as well but that would just decrease the readability, because statements will be more and more nested and harder to follow.

But most of the functions in Go returns errors when something failed instead of notifying through an given pointer in the parameters (after all we aren't programming in C right)?

So how can we change this and still make use of the great integration with the standard library and the http.Handler interface. For this we are going to take the idea of http.HandlerFunc and create our own adapter for the Handler interface which can work with return types.

But first we need to find the perfect return types for our functions. Only an error would not be sufficient because we still have to set the status code on the ResponseWriter. We could make a generic struct which can contain most of the information we would like to send to the caller, this would look like this:

// Map of string to string where the key is the key for the header
// And the value is the value for the header
type Headers map[string]string

// Generic response object for our handlers
type Response struct {
	// StatusCode
	Status int
	// Content Type to writer
	ContentType string
	// Content to be written to the response writer
	Content io.Reader
	// Headers to be written to the response writer
	Headers Headers
}

The same as how the HandlerFunc works we create a type alias for our function definition. The type will return the new created response object by us.

type Action func(r *http.Request) *Response

We omitted the response writer as parameter because we don't need it in our functions. We won't be writing to the response writer from inside our function (this breaks the paradigm we want to accomplish). The response struct is the way for us to write content to the response writer. Now we need to make our Action type compatible with the http.Handler interface by implementing the ServeHTTP method on it.

func (a Action) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
	if response := a(r); response != nil {
		if response.ContentType != "" {
			rw.Header().Set("Content-Type", response.ContentType)
		}
		for k, v := range response.Headers {
			rw.Header().Set(k, v)
		}
		rw.WriteHeader(response.Status)
		_, err := io.Copy(rw, response.Content)

		if err != nil {
			rw.WriteHeader(http.StatusInternalServerError)
		}
	} else {
		rw.WriteHeader(http.StatusOK)
	}
}

A few things going on here. First we declare the method on the function and we call the type Action which is in essence just a plain function. So every function with the same type signature as the Action can be converted to this Action type. As we know our Action type gave back a pointer to a Response struct. For this to work correctly we need to check if the pointer is not nil. Otherwise the program would panic when calling the Status and Content property on them. By using a pointer it gives us also the extra benefit of that we can return nil in our functions and nothing will be done (the default value of 200 OK will be send to the caller). We still have the same flexibility of the regular http.HandlerFunc and we are in control of when something will be written to the response writer.

Ok cool, with this in place we have some nice functions to work with. We can now return in our handlers and everything will be fine. We can now write some wrapper functions so we don't have to manually create our Response struct.

Because our response struct works with an io.Reader interface we cannot just simply return the error in there so we need to first create a wrapper for this. We use a io.Reader here because this way we stay flexible, we can return any reader (even a stream reader) in our handlers and it will be streamed to the response writer.

func Error(status int, err error, headers Headers) *Response {
	return &Response{
		Status:  status,
		Content: bytes.NewBufferString(err.Error()),
		Headers: headers,
	}
}

This function pretty much explains itself. We pass in an error and we let the function convert it to an io.Reader with using an internal buffer. We can use it like this:

func Index(r *http.Request) *Response {
	return Error(404, errors.New("not found"), nil)
}

Sweet! that looks way more clear than before. Let's take it a step further, nowadays a lot of people are making rest api's which spit out JSON to the caller. We can easily create a function for this.

type errorResponse struct {
	Error string `json:"error"`
}

func ErrorJSON(status int, err error, headers Headers) *Response {
	errResp := errorResponse{
		Error: err.Error(),
	}

	b, err := json.Marshal(errResp)

	if err != nil {
		return Error(http.StatusInternalServerError, err, headers)
	}
	return &Response{
		Status:      status,
		ContentType: "application/json",
		Content:     bytes.NewBuffer(b),
		Headers:     headers,
	}
}

NOTE We can use the ErrorJSON functions again in our handlers. And it will do the conversion to JSON for us.

func Index(r *http.Request) *Response {
	return ErrorJSON(http.StatusNotFound, errors.New("not found"), nil)
}

and it will print:

{
  "error": "not found"
}

With those helper functions we can create responses for every content-type you would like; ErrorXML etc. We've seen error handling and how we can elegant create custom responses for our errors. How does this work for returning something else than en error?

We can create a generic functions (same as the error function) for regular data as well.

func Data(status int, content []byte, headers Headers) *Response {
	return &Response{
		Status:  status,
		Content: bytes.NewBuffer(content),
		Headers: headers,
	}
}

Example usage:

func Index(r *http.Request) *Response {
	return Data(http.StatusOK, []byte("test"), nil)
}

Same as the errors we could take this a step further and implement some helper functions who will do marshalling of data to JSON.

func DataJSON(status int, v interface{}, headers Headers) *Response {
	b, err := json.Marshal(v)

	if err != nil {
		return ErrorJSON(http.StatusInternalServerError, err, headers)
	}

	return &Response{
		Status:      status,
		ContentType: "application/json",
		Content:     bytes.NewBuffer(b),
		Headers:     headers,
	}
}

Here we accept an interface in our method and let the json package take care of the conversion between the incoming v and the byte array. If we encounter some error during marshalling we just return our ErrorJSON function and the caller will be notified with the error (note this should probably be logged instead of returning the actual error to the caller). We do the same trick as in our ErrorJSON method and set the right content type. Usage is the same as all the other methods.

type temp struct {
	Message string `json:"msg"`
}

func Index(r *http.Request) *Response {
	return DataJSON(http.StatusOK, temp{"test"}, nil)
}

We can also create our helper function for the standard io.Reader this way we can return any reader we would like. This could be a external http or anything implementing the io.Reader interface.

func DataWithReader(status int, r io.Reader, headers Headers) *Response {
	return &Response{
		Status:  status,
		Content: r,
		Headers: headers,
	}
}

With this in place we have all the flexibility we would like and can return anything we can even think off. We eliminated the verbose writing to the response writer and made our handlers look way cleaner and easier to follow. Without losing perfomance or flexibility.

Compatibility with the standard library

Because our handlers are still of type http.Handler we can use them anywhere where the http.Handler interface is used. Lets try it out! We are going to create middleware for logging the details about a request

func logger(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Printf("[%s] User agent => %s Remote addr => %s", r.Method, r.UserAgent(), r.RemoteAddr)
		next.ServeHTTP(w, r)
	})
}

We can create a route in main

func main() {
	http.Handle("/test", logger(Action(Index)))
	http.ListenAndServe(":8080", nil)
}

And it all works. We can chain the middleware and use existing middleware with our new handler types.

Conclusion

The http.Handler interfaces gives a lot of flexibility and by using type aliasing in Go we can easily convert our functions to actual methods which implement the Handler interface. It is even possible to extend our Response object with more options. This is all up to you and you can modify the wrappers to use the new options you define. (Extra headers for example).

the full code can be found here Github Gist. Let me know what you think about it and what could be improved.

Thanks for reading and happy coding! (y)

Clean code using decorators

Bart Fokker — Sat, 23 Jun 2018 00:00:00 GMT

Introduction

In this blogpost I'm going to explain how to keep your code clean using decorators. By applying decorators the open close principle is maintained, code becomes easier to extend and easier to adjust.

Background

According to wikipedia the decorator pattern is a design pattern that allows behaviour to be added to an individual object, without affecting the behavior of other objects from the same class. In go this means that we extend the functionality of a function or method(s) of a struct without changing the original functions or method(s).

Consider the following interface:

type Adder interface {
  Add(x, y int) int
}

We can implement this interface and use it somewhere in our code

package main

type adderImpl struct {}

func (adderImpl) Add(x, y int) int {
  return x + y
}

func main() {
  var a Adder = adderImpl{}

  fmt.Println(a.Add(1,2))
  // prints => 3
}

This is idiomatic go code and works fine for adding numbers. Because we have an interface with 1 function we can also create a function signature for it, which implements the Adder interface. This way we can use it even more easily, it's the same principle as the http.HandlerFunc from the standard library.

type AdderFunc func(x, y int) int

func (a AdderFunc) Add(x, y int) int {
  return a(x, y)
}

What happens in the code above is that we define a type which is a function. On this function type we implement the original Adder interface and propagate the Add call to the AdderFunc. This way we won't need to implement the interface everytime on a struct and can just use a function to implement the behaviour.

package main

func main() {
  a := AdderFunc(
    func(x, y int) int {
      return x + y
    },
  )
  fmt.Println(Do(a))
  // prints => 3
}

func Do(adder Adder) int {
  return adder.Add(1, 2)
}

Note the type conversion for the anonymous function to the AdderFunc is required because it's not able to implicitly convert an anonymous function to the Adder interface directly. If the Do function would accept an AdderFunc instead of an Adder it is possible to pass in the anonymous function directly and the compiler knows how to convert the function to the AdderFunc type implicitly.

Middleware

Let's say we wan't to add logging capabilities to the adder function. We can do a naive way and add logging to the original function but this will introduce a dependency on the log package in our business domain logic. It also violates the single responsiblity and open close principle because the code is not only adding numbers anymore but also logging. Below I illustrate an example of the wrong way.

Wrong way

package main

func main() {
  a := AdderFunc(
    func(x, y int) (result int) {
      defer func(t time.Time) {
        log.Printf("took=%v, x=%v, y=%v, result=%v", time.Since(t), x, y, result)
      }(time.Now())
      return x + y
    },
  )
  fmt.Println(Do(a))
  // prints => 2009/11/10 23:00:00 took=0s, x=1, y=2, result=3
  // prints => 3
}

func Do(adder Adder) int {
  return adder.Add(1, 2)
}

There is a lot happening in the code above, after the x + y is added there runs another function using defer which logs the input and output values, aswell as the time it took to calculate x + y. Currently our code is all in main, this is not really a violation but when we start seperating the code in packages the logging code starts to occur everywhere.

A better approach would be to apply the Decorator pattern often called Middleware in Go. This pattern at its simplest is running code before and/or after our original code. The Middleware pattern is the decorator pattern applied in Go. When using custom Middleware functions wrapping our interface we can add behaviour on top of the original behaviour. Our Middleware function takes in an Adder and returns a new decorated Adder.

// Middleware function, this function takes in a `Adder` and returns a new `Adder`.
type AdderMiddleware func(Adder) Adder

func Wraplogger(logger *log.Logger) AdderMiddleware {
  return func(a Adder) Adder {
    // Using `AdderFunc` to implement the `Adder` interface.
    fn := func(x, y int) (result int) {
      defer func(t time.Time) {
        logger.Printf("took=%v, x=%v, y=%v, result=%v", time.Since(t), x, y, result)
      }(time.Now())
      // Propogate call to original adder
      return a.Add(x, y)
    }
    // Return a new `Adder` wrapped with the loggin functionality
    return AdderFunc(fn)
  }
}

As you can see the AdderFunc type is used, this type makes sure that the fn function is converted to implement the Adder interface. The fn closure calls the logger after the original Adder is called. The beauty of this is that the WrapLogger function doesn't know any of the internals of the original Adder it only extends the behaviour of the Adder by loggin the input and output. This way we adhere to the open close principle. By combining the power of first class functions and the middleware pattern we are able to add additional behaviour on top of the original behaviour.

Middleware can be written for any type of functionality you want to add to an Adder. Consider the case that we also need to cache the results, because adding our numbers takes to much compute time, we can easily add a caching layer on top of our Adder by writing a Middleware function. I'll be using a sync.Map from the std library here to implement the caching.

NOTE that this is pure for illustration purposes and caching the results will likely be a performance hit.

func WrapCache(cache *sync.Map) AdderMiddleware {
  return func(a Adder) Adder {
    fn := func(x, y int) int {
      key := fmt.Sprintf("x=%dy=%d", x, y)
      val, ok := cache.Load(key)
      if ok {
        return val.(int)
      }
      result := a.Add(x, y)
      cache.Store(key, result)
      return result
    }
    return AdderFunc(fn)
  }
}

The sole purpose of the WrapCache function is to cache the response of the Adder it's wrapping. The original Adder has no clue about this caching functionality. If we get a new requirement to only cache even number we only have to change the WrapCache function to cache only on even numbers. Also if we need to use an external cache like redis or memcache we can write a new caching middleware function for this type of cache and use that to wrap the original adder instead of using the WrapCache method. This allows us to easily switch between implementations (also interfaces can be used to depend on only abstractions in the WrapCache method)

Testing

Testing using this pattern becomes easier aswell, we can test the business logic seperatly from the caching logic and logging logic. In our business logic we only have to test the original Adder functionality, in for example the WrapCache function we only have to test if the result is cached, we don't care about the result itselfs only if it's correctly cached or not.

Chaining

When you have many functions decorating the original behaviour combining them can become messy.

func main() {
  a = WrapCache(&sync.Map{})(WrapLogger(log.New(os.Stdout, "test", 1))(a))
  a.Add(10, 20)
}

For this we can have an elegant solution called chaining, this solution makes it more human readable to see how the wrapping works.

func Chain(outer AdderMiddleware, middleware ...AdderMiddleware) AdderMiddleware {
  return func(a Adder) Adder {
    topIndex := len(middleware) - 1
    for i := range middleware {
      a = middleware[topIndex-i](a)
    }
    return outer(a)
  }
}

The above code loops through the given middleware functions in reverse order and inserts the Adder in it. With every iteration the Adder is overwritten to the wrapped adder which is used as input for the next middleware. The reason why the loop is in reverse order is because this way the wrapped function is called in the order the middlewares are declared. That means that the first middleware function is the outermost middleware wrapper. Chain is also a middleware function, but instead of having to do all this wrapping manually, the Chain function can be used like this, making the main function way cleaner:

func main() {
  var a Adder = AdderFunc(
    func(x, y int) int {
      return x + y
    },
  )
  a = Chain(
    WrapLogger(log.New(os.Stdout, "test", 1)),
    WrapCache(&sync.Map{}),
  )(a)

  a.Add(10, 20)
}

When reading this codes it more obvious what is happening, adding new middleware functions is also a bit easier and doesn't make the code messy.

Further thoughts

This pattern can be really usefull to seperate the business logic from things which don't belong in the business logic, like caching, logging and metrics. It also allows you to think in terms of small composable interfaces and how to cleanly add additional features on top of them without changing the original code and behaviour. In the end your main function starts to look very clean and clear for a future developer. My main functions started to look like this:

gc := geocoder.Chain(
  geocoder.WrapLogger(svcLogger.WithField("geocoder", "google-maps")),
  geocoder.WrapMetrics(mockMetrics),
  geocoder.WrapLRUCache(mockMetrics, 8192),
)(geocoder.Googlemaps(gm, "NL"))

svc := geoparser.Chain(
  geoparser.WrapLogger(svcLogger.WithField("geoparser", "parser")),
  geoparser.WrapMetrics(mockMetrics),
)(geoparser.WrapService(mockMetrics, p, gc))

My functions became easier to test and were also easier to adjust, this pattern allowed for better maintainability overall. The pattern can also be applied to interfaces with multiple functions, but this is left as an exercise for the reader.

Thanks for reading and happy programming (y)!

Running code can be found here playground and gist

Mocking system time in tests

Bart Fokker — Sun, 23 Dec 2018 00:00:00 GMT

Introduction

Recently I was working on my personal project and I encountered a situation where I was calling time.Now() to get the current system time. I was working on a function to calculate someone's age from a birthday, I needed this function to display it in my app. I did not want another field in my database with someone's age, since in my opinion this calculation from the birthday is pretty trivial.

Coding

Since the function to calculate the age is a function with a clear in and output we could easily test it using table driven tests. If you don't know what table tests this is a good explanation. The benefits of using table driven tests are that adding and adjusting test cases is fairly easy, which is exactly what I needed to test my Age function.

My implementation of the Age function looks like this.

package account

import (
	"time"
)

type UserProfile struct {
    Birthday time.Time
    // other fields omitted
}

func (up UserProfile) Age() uint {
	now := time.Now()
	currentYear := now.Year()
	// if the current year is equal then or lower then the birthday it's always zero
	// because no years have passed
	if currentYear <= up.Birthday.Year() {
		return 0
	}
	difference := currentYear - up.Birthday.Year()
	// if the birthday of the user profile is higher then the current month
	// we know the user birthday hasn't been yet.
	if up.Birthday.Month() > now.Month() {
		return uint(difference - 1)
	}
	// check if the birthday is the same as the current month
	// and if the day is later in the month then the current day
	if monthDayIsBefore(up.Birthday, now) {
		return uint(difference - 1)
	}
	return uint(difference)
}

func monthDayIsBefore(t, s time.Time) bool {
	return t.Month() == s.Month() && t.Day() > s.Day()
}

When I started with the tests I soon found out that I needed a way to control the time I was comparing against. For example, with the current implementation there is no way to control the now variable. This variable always depends on the system time, I found this to be bad for reproducibility of the tests.

The only assertion I could made against the return value was that the value is bigger then 0. There was no actual way to test the age, without updating the tests all the time. I needed a way to mock the now variable. The part I needed to mock was the call to time.Now(), because this one was using the implementation from the standard library which uses the system time underneath.

There are multiple ways to abstract this behaviour but I choose to go for the interface way.

I defined the following interface

package clock

import (
	"time"
)

type Clock interface {
	Now() time.Time
}

Instead of calling time.Now() I passed this interface as a parameter to the age function and called the Now() function from the interface. Note that instead of passing it as a parameter it could also be part of the struct.

func (up UserProfile) Age(c clock.Clock) uint {
    now := c.Now()
    // rest omitted for clarity
}

Now the function itself is fully under our control, it doesnt depend on any external factors like the Now function from the time package. I made 2 implementations based on the Clock interface, one which uses the standard library time.Now() and one which uses a fixed time for in the tests.

package clock

import (
	"time"
)

type Mock time.Time

func (m Mock) Now() time.Time { return time.Time(m) }

type Real struct{}

func (Real) Now() time.Time { return time.Now() }

In the tests we pass the Mock type with the fixed time and in the business logic we use the Real implementation.

My tests started looked like the following.

package account

import (
	"testing"
)

func TestAge(t *testing.T) {
	testCases := []struct {
		desc        string
		profile     UserProfile
		clock       clock.Clock
		expectedAge uint
	}{
		{
			desc: "Test with birthday in the future",
			profile: UserProfile{
				Birthday: time.Date(2011, 1, 1, 1, 1, 1, 1, time.UTC),
			},
			clock:       clock.Mock(time.Date(2010, 1, 1, 1, 1, 1, 1, time.UTC)),
			expectedAge: 0,
		},
		{
			desc: "Test with birthday in the same year but in the past",
			profile: UserProfile{
				Birthday: time.Date(2010, 1, 1, 1, 1, 1, 1, time.UTC),
			},
			clock:       clock.Mock(time.Date(2010, 2, 1, 1, 1, 1, 1, time.UTC)),
			expectedAge: 0,
		},
		// cases omitted for clarity
	}
	for _, tC := range testCases {
		t.Run(tC.desc, func(t *testing.T) {
			if tC.profile.Age(tC.clock) != tC.expectedAge {
				t.Errorf("Expected profile age %v to be equal to expected age %v", tC.profile.Age(tC.clock), tC.expectedAge)
			}
		})
	}
}

Cleaning up the public API

Right now the dependencies are really clear and in the public exported API. For the caller of the Age function this might be a little bit to much, you might expect that just calling Age will just return the actual Age compared to the current time. Now something which is only useful for testing is leaked in the public API.

We can fix this with a private helper function.

package account

type UserProfile struct {
	// fields omitted for clarity
}

func (up UserProfile) Age() uint {
	return up.age(clock.Real{})
}

func (up UserProfile) age(c clock.Clock) uint {
	// implementation of age function
}

In the tests we test the private age function, which contains the actual age calculation. The exported Age function just propagates the call to the private function with the real system time implementation.

Conclusion

With this approach we are able to control the exact values used in the Age function, to assert the expected behaviour. On actual runtime we still have the flexibility to use the system clock but with the downside that we have to pass it as a parameter to the age function. The example shown in the blog post is a very simplistic example, but the concept can be used in many different places and not only for mocking out time.

Running a serverless Go web application

Bart Fokker — Sun, 27 Oct 2019 00:00:00 GMT

Introduction

Recently Google introduced the beta program for Google Cloud Run. This is a service to run stateless containers on a fully managed environment by Google. It is essentially serverless for any container, scaling your containers up and down as requests (and peaks) come and go. The benefit of using tools like Cloud Run is that it abstracts away all the infrastructure, thus you can focus more on what really matters, building your application.

The reason why I'm so excited is because it essentially allows me to run my (already) dockerized applications in a serverless manner, reducing costs and maintenance burdens. You can turn any dockerized application into a serverless application using Cloud Run and in this post we are going to explore this with Go.

Setting up

Requirements

You need an Google Cloud Platform account, you can create one here
You need to enable billing on your Google Cloud Project to use Cloud Run see this
You need to install gcloud sdk locally and make sure that you are logged in, see the guide

With everything set up in terms of account configuration we can enable Cloud Run in the GCP console. In the console head over to the menu on the left and press Cloud Run. This will move you to a page where you can enable Cloud Run for your current project (see picture below). After pressing the enable button Cloud Run will be available for your current project.

Since cloud run is still in beta we need to install the beta components in the gcloud utility and set the gcloud region to one of the available regions for Cloud run:

Enable cloud run in the gcloud utility

gcloud components install beta

Set the region to one of the available regions:

europe-west1
us-central1
us-east1
asia-northeast1

Executing the following command will set your default Cloud run region, this can be overridden with the --region flag

$ gcloud config set run/region <REGION>

Building the application

With that out of the way we can actually start building the application. We will be building an application which generates a random avatar based on the input you give it. We will be using the code from my previous post and turn it in a serverless API, to generate identicons (avatars) on the fly.

I've adjusted the code from my previous post to be used as a library, this way we can use it for a cli and for the web application we are building right now. You can find the code here

Let's start writing some code for the web version!

import (
	"github.com/barthr/identicon"
)

func main() {
  http.HandlerFunc("/identicon/generate", func(w http.ResponseWriter, r *http.Request) {
    name := r.URL.Query().Get("name")
    if name == "" {
      http.Error(w, "no name given", http.StatusPreconditionFailed)
      return
    }
    log.Printf("generating identicon for %s", name)

    w.Header().Set("Content-Type", "image/png")
    if err := identicon.Generate([]byte(name)).WriteImage(w); err != nil {
      http.Error(w, "failed generating identicon", http.StatusInternalServerError)
    }
  })
  http.ListenAndServe(":"+os.Getenv("PORT"), nil)
}

We can run the code locally by running the following command PORT=8080 go run main.go

This is everything needed to run a simple web application on cloud run. First we register a http handler which expects some input. After this input is considered "valid" we call our identicon library which will return a Identicon struct.

On the identicon struct there is a method WriteImage(w io.Writer) error, by using a io.Writer here as input we can easily change where the image is written to, in this case the http.ResponseWriter is used but in case of the cli version the image can be written to an os.File or anything which implements the io.Writer interface.

We are now ready to deploy this to cloud run. Cloud run requires that your code is bundled in a docker container and is able to serve http requests on a port configurable with the PORT environment variable.

Lets create a Dockerfile, I'll be using multistage builds so that the end container only contains the executable binary, I wont go in depth on how the Dockerfile and multistage builds work, if you want to learn more about this have a look at this link.

FROM golang:1.13 as builder

# Create and change to the app directory.
WORKDIR /app

# Retrieve application dependencies.
COPY go.* ./
RUN go mod download

# Copy local code to the container image.
COPY . ./

# Build the binary.
RUN CGO_ENABLED=0 GOOS=linux go build -mod=readonly -v -o server cmd/api/main.go


FROM alpine:3
RUN apk add --no-cache ca-certificates

# Copy the binary to the production image from the builder stage.
COPY --from=builder /app/server /server

# Run the web service on container startup.
CMD ["/server"]

With that out of the way we can submit a build to Google cloud build. Google cloud build will build our container and store it in cloud registry.

Execute the following command to publish a build to google cloud build

$ gcloud builds submit --tag gcr.io/<PROJECT_ID>/identicon-generator

If cloud build successfully builds the container it will be published to the Google container registry. We can use the image in the registry to submit a deploy to Cloud run. There are multiple ways to do it but I'll be using the terminal to deploy the container to cloud run.

Execute the following command to publish a deploy to cloud run

$ gcloud beta run deploy --image gcr.io/<PROJECT_ID>/identicon-generator --platform managed

There are a few things going on here. First we tell gcloud that we want to use the managed platform using the --platform flag, it is possible to install Cloud run on your own Kubernetes cluster but that is out of scope of this article. Next we define what image we want to deploy, we point this to the image just published by cloud build, by default it will fetch the latest version of this image but it is also possible to deploy a specific version by adding a tag.

The command will prompt us with a question if we want allow unauthenticated requests, we input yes and now our cloud run function will be publicly exposed.

You can view your cloud run functions in the Google cloud run dashboard

You can now test your cloud function by going to the url which cloud run provides to us. My identicon generator is running on https://identicon-generator-ziolee7x5a-ew.a.run.app/identicon/generate?name=super+cool. You can test it out with various inputs, the first boot can be a bit slow because the container is having a cold start but subsequent requests will be very quick because there is no need for a cold start.

Conclusion

Deploying web applications with cloud run is very easy. It is as simple as defining a Dockerfile and publishing the image to google cloud. It removes the burden to maintain any infrastructure and you only pay for the resources it uses. Scaling is done automatically and in combination with hosted databases like cloud sql you have a fully managed solution which is very easy to maintain and also very scalable.

The benefits of using a service like cloud run is that your local environment is not depending on any kind of serverless framework or library, it is just a regular go application which you can test and run in the same way as you're used to.

Cloud run is still in beta and will hopefully soon be covered by the SLA and available in more regions. In a later post I will explain how to automate the build process from a git push to a deployment in cloud run.

The full code can be found here

Thanks for reading and happy coding 🚀

How to organize your code base

Bart Fokker — Sun, 19 Jan 2020 00:00:00 GMT

Introduction

Besides scaling the application itself for user demand there is another very important aspect to scaling which is organizing the source code itself. We've all heard (or seen) the horror stories about very big unmaintainable code bases. In this post I will elaborate on what I think is a good practice on maintaining your source code as the project grows.

The why

Over the (short) period of my career I've encountered relatively simple systems and more complex systems. With these projects at some point the dicussion arises on how the folder structure should be laid out. Most of the time people start out with what they know, based on experience from previous systems. While Go embraces simplicity I feel like this subject is not in any way simple.

A sensible directory structure is important since it is something you will work with every day.

In the community there is no clear consensus on how to organize your repository. This is because there is no right or wrong way, it really depends on your situation. What I will share with you in this blog post is my perspective on organizing source code so you will (hopefully) be able to make these decisions a bit better. Don't take this as a silver bullet because as with a lot of things in software development, it depends.

Approach

There are several blog posts/videos about directory layout and ports & adapaters etc. While I think these way of doing things can be good I don't think it is the right way when you are just starting out on building your application.

The main goal should always be keep it simple

Making things complex by adding a lot of packages initially doesn't make sense, these packages should organically grow.

As an example. Let's say you are building an api. There are a lot of things necessary for building an api: middleware, authentication, logging, routers etc. You can split all these up in a separate package which would give you a bunch of packages.

This overcomplicates from the beginning and is a form of premature optimisation. A single package would be sufficient where every logical piece is in a separate file or even in 1 file (if the logic itself is small enough). This approach works perfectly well and it scales quite good.

At some point this one package model starts to grow too big when you are adding more features. This is the moment when you start moving code out into separate packages.

You might be thinking, isn't this a lot of work? Well it might be, but most of this can be accomplished by tooling making it essentially free and thus, saving a lot of other developers the hassle of understanding the directory structure beforehand.

How do you know when something is "too big"? This is based on your own experience, whenever you have trouble finding something or navigation the code base that is an indication that the packages are growing to big or not adhering to having a single responsbility.

Conclusion

The nice thing about this approach is that you are not trying to think beforehand what package names logically make sense, which most of the time is very hard to predict. The names for your packages will organically arise, when you feel like some things start to grow too big or things should be isolated in a separate package (like logging for example).

This approach especially works well in a microservice architecture, since these services are quite scoped and not too big. At my current employment we have services which have only one package and as a result are very easy to maintain. This minimalistic approach makes sure that you postpone these significant decisions to when the time is right.

Postpone these significant decisions to when the time is right.

As a result I always start with a main package and with the first abstraction which is to split everything logically into separate files. Whenever I feel the need for moving these to a package (for instance the single package starts to grow too big) I create a new package and move the files to there.

Thanks for reading and happy coding 🚀

Credits

Thanks to Edward Poot for proofreading my post.

Multi command fzf history

Bart Fokker — Mon, 22 Dec 2025 00:00:00 GMT

TL;DR I improved the default fzf history widget to allow multiple commands and stitch them together using &&

I am a very heavy terminal user who likes to optimise his workflow in the shell using various command line tools. With the restriction that I always try to stay as close to the original feeling of the shell as possible. I prefer the Unix way, combining multiple (single-focused) tools instead of using a single tool to do it all.

As much as I like the "bare" built-ins and tools my shell and OS provide, it has its flaws too and can be improved. Specifically when searching through long lists. For this, my go-to has been fzf for a very long time, mostly to fuzzy search through my shell history.

This allows me to efficiently search through my history and execute that specific command. It is a workflow I've been very accustomed to and use countless times every day.

However, sometimes I want to run multiple commands at once. But in the (standard) history fuzzy search that is not possible. I approached something related a couple of years ago with my own cli redo. It worked well enough for my use case but over time I tend to forget the shell aliases and what exactly they do. Also, it is yet another tool.

So this "annoyance" slowly crept back into my brain. And because I really don't want to add "yet another tool" to my workflow I started to investigate if I could alter the fzf-history-widget to provide multi-select. Lucky for me fzf already provides a --multi flag. So that gave me the confidence that it was possible.

Looking at the way fzf implements the history widget for zsh, I took the original code and added the --multi flag to it.

4c4
<       FZF_DEFAULT_OPTS=$(__fzf_defaults "" "-n2..,.. --scheme=history --bind=ctrl-r:toggle-sort --wrap-sign '\t↳ ' --highlight-line ${FZF_CTRL_R_OPTS-} --query=${(qqq)LBUFFER} +m --read0 --multi")
---
>       FZF_DEFAULT_OPTS=$(__fzf_defaults "" "-n2..,.. --scheme=history --bind=ctrl-r:toggle-sort --wrap-sign '\t↳ ' --highlight-line ${FZF_CTRL_R_OPTS-} --query=${(qqq)LBUFFER} +m --read0") \
8c8
<       FZF_DEFAULT_OPTS=$(__fzf_defaults "" "-n2..,.. --scheme=history --bind=ctrl-r:toggle-sort --wrap-sign '\t↳ ' --highlight-line ${FZF_CTRL_R_OPTS-} --query=${(qqq)LBUFFER} +m --multi")
---
>       FZF_DEFAULT_OPTS=$(__fzf_defaults "" "-n2..,.. --scheme=history --bind=ctrl-r:toggle-sort --wrap-sign '\t↳ ' --highlight-line ${FZF_CTRL_R_OPTS-} --query=${(qqq)LBUFFER} +m") \

The only thing to do now was to stitch the multi-line output together using &&. The original code was fetching the history item using vi-fetch-history by number and executing that history item directly. I changed the code to combine the commands into one single command using &&.

< local ret=$?
< if [ -n "$selected" ]; then
<     if [[ $(awk '{print $1; exit}' <<< "$selected") =~ ^[1-9][0-9]* ]]; then
<       zle vi-fetch-history -n $MATCH
<     else # selected is a custom query, not from history
<       LBUFFER="$selected"
<     fi
< fi
< zle reset-prompt
< return $ret
---
> # Remove the history number (for example: 800 echo 'test') will become echo 'test'
> cmds=("${(@f)$(printf '%s\n' "${selected[@]}" | awk -F ' ' '{$1=""; sub(/^ +/,""); print}')}")
> # Put them into the editor buffer combined with &&
> BUFFER="${(j: && :)cmds}"
# This ensure the buffer will be set on the input prompt
> CURSOR=${#BUFFER}
> zle reset-promp

And now with the default fzf key binding to select multiple entries <tab> I could finally execute multiple commands from my shell history.

I didn't bother to implement it for bash, so my solution is zsh only. Porting it wouldn't be too difficult, but that was out of scope for this.

The full code can be found in this commit 77510e8. Thank you for reading!