Making HTTP requests beyond the 32MB urlfetch limit on Google App Engine

Since Google App Engine’s urlfetch service imposes a 32MB response size limit on each request, one has to roll their own http.Client implementation using appengine/socket to make requests exceeding that limit.

Sadly the appengine/socket implementation is not working with HTTP calls inside the local development server, such that one can not use the same solution and code in development and production.

The two issues made me create the fetchall package, a simple drop-in replacement for urlfetch that gives you a http.Client implementation that handles arbitrarily large HTTP requests (within the time limits of Google App Engine).

When running in the development server, the implementation will simply use Go’s net package instead of appengine/socket in order for request to work locally as well as in the cloud.

To use this package simply replace urlfetch.Client with fetchall.Client in your Go code.

GitHub: github.com/tp/fetchall
GoDoc: godoc.org/timm.io/fetchall