[Go as in Golang] Standard net/http config will break your production environment

or a less click baity title: An introduction to net/http timeouts

Source: https://commons.wikimedia.org/wiki/File:Gophercolor.jpg

First of all, as you may already recognized from the titles, this blogpost is standing on the shoulder of giants. The following two blog posts inspired me to revise the net/http timeouts, as the linked blog post are at some parts outdated:

Give them a visit after you read this post and see how things have changed in such a short time ;)

Why you should not use the standard net/http config?

The go core team decided to not set any timeouts at all on the standard net/http client or server config and that is a real sane decision. Why?

To not break things! Timeouts are a highly individual setting and in more of the cases a to short timeout will break your application with a unexplainable error, than a too long one (or in GOs case none) would.

Imaging following different use cases of the go net/http client:

1) Downloading a big file (10GB) from a webserver. With an average (german) internet connection this would take round about five minutes.

=> The timeout for the connection should be longer than five minutes, because anything less would break your application by canceling the download in the middle (or third, or whatever percentage) of the file.

2) Accessing a REST API with a lot of concurrent connections. This normally should take at most a few seconds per connection

=> The timeout should be not more than 10 seconds, as anything that takes longer would mean, that you are keeping that connection open for to long and starving your application as it only can have X (depending on system, configuration and coding) open connections. So if that REST API you access is broken in any way that it keeps the connections open without sending you the data you need, you want to prevent it from doing so.

So, for what scenario should the standard lib be optimized? Trust me, you do not want to decide that for millions of developers around the globe.

That is why we have to set the timeouts, so that they fit our use case!

So never use the standard go http client/server! It will break your production system! (Happened to me, as I forgot my own rule ones)

What type of timeouts occur in a HTTP connection?

I assume you have a basic understanding of the TCP and HTTP protocols. (If not, Wikipedia is a good starting point for that)

There are mainly three different categories of timeouts that can occur:

As you already might expect from our two examples in the introduction, the timeout that we have to care about the most is the one regarding the body. The other ones are most of the time shorter and similar in every setup. (E.g. there is only a certain amount of headers that will be send) We still have to think and care about timeouts in the header as there are certain DOS attacks that play with malformed headers, or never closing a header (SLOWLORIS DOS attack) but we will come to this in a later point of the post.

You should at least do this: The easy path

net/http gives you the possibility to set a timeout for the complete transfer of data (setup, headers, body). It is not as fine grained as with the later bespoken solutions, but it will help you to prevent the most obvious problems:

So you should at least use this timeouts on every go net/http client/server you use!

Client

The following example client, gives you a complete timeout of 5 seconds.

c := &http.Client{
	Timeout: 5 * time.Second,
}
c.Get("https://blog.simon-frey.eu/")

If the connection is still open, it will be canceled with net/http: request canceled (Client.Timeout exceeded while reading ...)

So this timeout would work for small files, but not for download of a large file. We will see how we can have a variable timeout for the body later in the post.

Server

For the server we have to set two timeouts in the easy setup. Read and write. So the ReadTimeout defines how long you allow a connection to be open during a client sends data. And with WriteTimeout it is in the other direction. (Yeah it could also be, that you send data somewhere and the packages never get accepted TCP-ACK and your server would starve again)

s := &http.Server{
	ReadTimeout: 1 * time.Second,
	WriteTimeout: 10 * time.Second,
	Addr:":8080",
}
s.ListenAndServe()

So this server would listen on port 8080 and have your desired timeouts.

For a lot of use cases, this easy path may be enough. But please read on and see what other things are possible :D

[Client] In-depth configuration of timeouts

One thing to note before we get started here is the following differentiation:

Connection setup

In the following setup are two parameters, we set with a timeout. They differ in their connection type:

In a 2019 setup, you should always try to talk to encrypted HTTPS endpoints, so there are very rare cases where it makes sense to only set one of the two parameters.

 c := &http.Client{
    Transport: &http.Transport{
		DialContext:(&net.Dialer{
			Timeout:   3 * time.Second,
		}).DialContext,
		TLSHandshakeTimeout:   10 * time.Second,
    }
}
c.Get("https://blog.simon-frey.eu/")

With setting these parameters you define how long the setup of a connection should last at longest. This helps you with 'detecting' (for actually detection you have to do more than this few lines) of hosts that are down in a faster manner. So you are not waiting in your project for a host, that is/was down in the first place.

Response headers

Now as we have an established (hopefully HTTPS) connection, we have to receive the meta information about the content we get. This meta information is stored in the headers. We can set timeouts, how long we want the server to be able to answer us.

Here again are two different timeouts to be defined:

So you want to have the complete header information ExpectContinueTimeout + ResponseHeaderTimeout after your did send you complete request

c := &http.Client{
	Transport: &http.Transport{
		ExpectContinueTimeout: 4 * time.Second,
		ResponseHeaderTimeout: 10 * time.Second,
    },
}
c.Get("https://blog.simon-frey.eu/")

With setting this parameters, we can define how long we accept the server to take for an answer and therefore also for internal operations.

Imagine following scenario: Your access an API, that will resize an image you send to it. So you upload the image and normally it takes ~1 second to resize the image and than start sending it back to your service. But maybe the API crashes of whatever reasons and takes 60 seconds to resize the image. As you now defined the timeouts, you can abort after a couple of seconds and tell your own customers that API xyz is down and that you are in contact with the supplier...better than having your fancy image editor loading for ages and not showing any status information, and that all because of a bug that is not even your fault!

Body

Per definition, the timeout for the body is the hardest, as this is the part of the response that will vary the most in size and thereby time it needs for transfer.

We will cover two approaches that help you to define a timeout on the body:

Static timeout

We are dropping all errors in the example code. You should not do that!

c := &http.Client{}
resp, _ := c.GET("https://blog.simon-frey.eu")
defer resp.Body.Close()

time.AfterFunc(5*time.Second, func() {
	resp.Body.Close()
})
bodyBytes,_ := ioutil.ReadAll(resp.Body)

In the code example we set a timer, that executes resp.Body.Close() after it finished. With this command we close the body and the ioutil.ReadAll will throw a read on closed response body error.

Variable timeout

We are dropping most of the errors in the example code. You should not do that!

c := &http.Client{}
resp, _ := c.GET(https://blog.simon-frey.eu")
defer resp.Body.Close()

timer := time.AfterFunc(5*time.Second, func() {
	resp.Body.Close()
})	
                 
bodyBytes := make([]byte, 0)
for {
	//We reset the timer, for the variable time
	timer.Reset(1 * time.Second)

	_, err = io.CopyN(bytes.NewBuffer(bodyBytes), resp.Body, 256)
	if err == io.EOF {
		// This is not an error in the common sense
        // io.EOF tells us, that we did read the complete body
			break
	} else if err != nil {
		//You should do error handling here
        break
	}
}

The difference here is, that we have a endless loop, that iterates over the body and copies data out of it. There are two options how this loop will be left:

This solutions works, because io.CopyN is blocking. So if there is not enough (in our case 256 bytes) to read from the body it will wait. If the timeout triggers during that time, we stop the execution.

My 'default' config

Again: This is my very own opinion on the timeouts and you should adapt them to the requirements of your project! I do not use this exact same setup in every project!

c := &http.Client{
	Transport: &http.Transport{
		DialContext:(&net.Dialer{
			Timeout:   10 * time.Second,
			KeepAlive: 10 * time.Second,
		}).DialContext,
		TLSHandshakeTimeout:   10 * time.Second,
           
		ExpectContinueTimeout: 4 * time.Second,
		ResponseHeaderTimeout: 3 * time.Second,
		
        // Prevent endless redirects
        Timeout: 10 * time.Minute,
	},
}

[Server] In-depth configuration of timeouts

As there are no certain dial up timeouts for http.Server we will directly start into the timeouts for the headers.

Headers

For the request headers we have a certain timeout: ReadeHeaderTimeout, which represents the time until the full request header (send by a client) should be read. So if a client takes longer to send the headers the connection will time out. This timeout is especially important against attacks like SLOWLORIS as here the header never gets closed and the connection thereby will be kept open all the time.

s := &http.Server{
	ReadHeaderTimeout:20 *time.Second,
}
s.ListenAndServe()

As you may already have recognized, there is only a ReadHeaderTimeout, because for the sending of data to the client go does not have a certain distinction between the headers and the body for the timeout

Body

Here we have to differentiate between request (that is send from the client to the server) and the response body.

Response body

For the response body there is only one static solution for a timeout:

s := &http.Server{
	WriteTimeout:20 *time.Second,
}
s.ListenAndServe()

As long as the connection is open, we can not differentiate if the data was send correctly or if the client is doing bogus here. But as we know our payload data, it is quite straight forward to set the timeout here on our past information we have about our server. So if you are a file server this timeout should be longer than for a API server. You can set no timeout for testing purpose and track how long a 'normal' request takes. Add a few percent of variance there and then you should be good to go!

Request body

**Attention: If you did set the WriteTimeout it will have an effect on the request timeout as well. This is because of the defintion if the WriteTimeout. It starts when the headers of the request where read. ** So if reading from the request body takes 5 seconds and your write timeout is 4 seconds it will also kill the reading of the request body!

For the request body there are again two possible solutions:

Static

For a static timeout we can use the ReadTimeout parameter we already used in the easy path:

s := &http.Server{
	ReadTimeout:20 * time.Second,
}
s.ListenAndServe()
Variable

For the variable timeout we need to work on the level of the handlers. Do not set a ReadTimeout, because the static timeout will interfere with the variable one. Also you must not set WriteTimeout as it is counted from the end of the request header and thereby also will interfere with the variable header

We have to define our own handler for the server, in our example we call it timeoutHandler. This handler does nothing than reading from the body with our loop and timeout if there is no data send anymore.

type timeoutHandler struct{}
func (h timeoutHandler) ServeHTTP(w http.ResponseWriter, r *http.Request){
	defer r.Body.Close()

	timer := time.AfterFunc(5*time.Second, func() {
		r.Body.Close()
	})

	bodyBytes := make([]byte, 0)
	for {
		//We reset the timer, for the variable time
		timer.Reset(1 * time.Second)
        
		_, err := io.CopyN(bytes.NewBuffer(bodyBytes), r.Body, 256)
		if err == io.EOF {
			// This is not an error in the common sense
			// io.EOF tells us, that we did read the complete body
			break
		} else if err != nil {
			//You should do error handling here
			break
		}
	}
}

func main() {
	h := timeoutHandler{}
	s := &http.Server{
		ReadHeaderTimeout:20 *time.Second,
		Handler:h,
		Addr:":8080",
	}
	s.ListenAndServe()
}

It is a similar approach to the one we did choose in the client. You have define this timeout loop in every handler you have separately. So you maybe should consider building a function for that, so that you don't have to rewrite the coder over and over again.

My 'default' config

Again: This is my very own opinion on the timeouts and you should adapt them to the requirements of your project! I do not use this exact same setup in every project!

s := &http.Server{
	ReadHeaderTimeout:20 *time.Second,
	ReadTimeout: 1 * time.Minute,

    WriteTimeout: 2 * time.Minute,
}

Conclusion

I hope you liked this blog post and it helped you to understand the different timeouts in go a little bit better. If you have any feedback, questions or just want to say 'Servus' (bavarian german for hello) do not hesitate to contact me!

Feedback: Email

Donate: Donate Button or Patreon

RSS Feed – This work is licensed under Creative Commons Attribution 4.0 International License

Sources

https://golang.org/pkg/net/http/

https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779

https://blog.cloudflare.com/exposing-go-on-the-internet/

Gopher Image (CC BY-SA 3.0): Wikimedia