5
\$\begingroup\$

Goproxy is a Go package implementing an HTTP proxy. I am using it to store requests from a client (eg: a web browser) for further processing.

In an http.Request struct, the Body is an io.ReadCloser, it can only be read once.

The problem I faced is that I need to read the Body to store it, but still make it accessible to goproxy so that it can be sent to the server.

I solved it by reading the body with ioutil.ReadAll, and then creating a ReadCloser with the byte slice I got. I then assign this ReadCloser to the request Body.

package main

import (
    "bytes"
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "net/url"
    "regexp"
    "sync"

    "github.com/elazarl/goproxy"
)

var requests map[string][]Request
var mutex sync.Mutex

type Request struct {
    Method        string
    URL           url.URL
    Proto         string // "HTTP/1.0"
    ProtoMajor    int    // 1
    ProtoMinor    int    // 0
    Header        http.Header
    Body          []byte
    ContentLength int64
    Host          string
    PostForm      url.Values
    RequestURI    string
}

type RequestBody struct {
    *bytes.Reader
}

func (r RequestBody) Close() error {
    return nil
}

func storeRequest(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        // only print error for now
        log.Println(err)
        return r, nil
    }
    r.Body.Close()
    r.Body = RequestBody{bytes.NewReader(body)}
    fmt.Println(string(body))
    mutex.Lock()
    requests[r.URL.Host] = append(requests[r.URL.Host], Request{
        Method:        r.Method,
        URL:           *r.URL,
        Proto:         r.Proto,
        ProtoMajor:    r.ProtoMajor,
        ProtoMinor:    r.ProtoMinor,
        Header:        r.Header,
        Body:          body,
        ContentLength: r.ContentLength,
        Host:          r.Host,
        PostForm:      r.PostForm,
        RequestURI:    r.RequestURI,
    })
    mutex.Unlock()
    return r, nil
}

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*$"))).
        HandleConnect(goproxy.AlwaysMitm)
    proxy.OnRequest().DoFunc(storeRequest)

    requests = make(map[string][]Request)

    addr := flag.String("addr", ":8080", "proxy listen address")
    flag.Parse()
    log.Fatal(http.ListenAndServe(*addr, proxy))
}

I would like to know if this is a correct way of solving this problem.

\$\endgroup\$
2
  • 1
    \$\begingroup\$ It looks like you're caching solely based on URL. You should really be caching by method too. Actually, you probably should only be caching HEAD and GET requests. And you should probably be saving some of the request headers. Accept-Language, for example, can have an effect on the response. You should also be complying with Cache-Control response headers. I'm not sure how much of this goproxy does. It also isn't clear to me that your cache is ever used (it looks like you only store). \$\endgroup\$ Dec 25, 2018 at 0:16
  • \$\begingroup\$ @BaileyParker that does not matter for as long he is not doing for cache management. surely he is not in regard to current source code. \$\endgroup\$
    – mh-cbon
    Dec 20, 2019 at 9:34

2 Answers 2

1
\$\begingroup\$

It is fine to read the body and replace it, although see ioutil.NopCloser.

However,

  • it does not make sense to duplicate the http.Request type. Re use it instead. If you need finer control, embed it.
  • take care to make true copy when passing around slices, maps and pointers.
  • the code is lacking structured factorization, make use of type definition

This is a revised version i suggest,

package main

import (
    "bytes"
    "flag"
    "io"
    "io/ioutil"
    "log"
    "net/http"
    "net/url"
    "regexp"
    "sync"

    "github.com/elazarl/goproxy"
)

type store struct {
    requests map[string][]http.Request
    sync.Mutex
}

func (s *store) Add(r *http.Request) {
    var c http.Request
    c.Method = r.Method
    c.Proto = r.Proto
    c.ProtoMajor = r.ProtoMajor
    c.ProtoMinor = r.ProtoMinor
    c.Host = r.Host
    c.RequestURI = r.RequestURI
    c.ContentLength = r.ContentLength
    c.URL = r.URL

    c.Header = http.Header{}
    for k, vv := range r.Header {
        for _, v := range vv {
            c.Header.Set(k, v)
        }
    }

    c.PostForm = url.Values{}
    for k, vv := range r.PostForm {
        for _, v := range vv {
            c.PostForm.Set(k, v)
        }
    }

    c.Form = url.Values{}
    for k, vv := range r.Form {
        for _, v := range vv {
            c.Form.Set(k, v)
        }
    }

    var b bytes.Buffer
    io.Copy(&b, r.Body)
    r.Body.Close()
    buf := b.Bytes()

    var b1 bytes.Buffer
    b1.Write(buf)
    r.Body = ioutil.NopCloser(&b1)

    c.Body = ioutil.NopCloser(bytes.NewBuffer(buf))

    key := r.URL.Host
    s.Lock()
    s.requests[key] = append(s.requests[key], c)
    s.Unlock()
}

type storeHandler struct {
    requests *store
}

func (s storeHandler) Handle(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
    s.requests.Add(r)
    return r, nil
}

func main() {
    h := storeHandler{requests: &store{requests: map[string][]http.Request{}}}

    proxy := goproxy.NewProxyHttpServer()
    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*$"))).
        HandleConnect(goproxy.AlwaysMitm)
    proxy.OnRequest().DoFunc(h.Handle)

    addr := flag.String("addr", ":8080", "proxy listen address")
    flag.Parse()
    log.Fatal(http.ListenAndServe(*addr, proxy))
}
\$\endgroup\$
0
\$\begingroup\$

The hardest part of your question:

Is it allowed to change Body field value of http.Request struct?

Part of http.Request documentation:

// Body is the request's body.
//
// For client requests a nil body means the request has no
// body, such as a GET request. The HTTP Client's Transport
// is responsible for calling the Close method.
//
// For server requests the Request Body is always non-nil
// but will return EOF immediately when no body is present.
// The Server will close the request body. The ServeHTTP
// Handler does not need to.
Body io.ReadCloser

The documentation says nothing about it. So we need to find some top level policy on such things. As far as I know there is no one in Go documentation.

Body field type is io.ReadCloser interface, the actual struct type is hidden behind it.

  • the actual type may have non exported fields which work in conjuction with Close method,
  • the hidden struct may be part of some reusable pool and will be lost,
  • HTTP trailer headers may be somewhere behind EOF.

You may take a look at current net/http sources to find the answer, but it can change in next Go release.

So I guess your best bet is to ask the Go community or even propose a change to Body documentation that will clear things out.


Another approach would be to wrap original Body and read it together with request handler.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.