After working several projects about skimming data in GO. I have some experiences about it. These are :

Tool for crawler (Go vs PuerkitoBio/goquery)

Before Go, i tried with Python (Scrapy), it works, but i did not feel comfortable with it. I do not know why, but after trying Go i do not want to think about Python anymore. Maybe, it is enough for me now. The only third party i need to import is PuerkitoBio/goquery. It is used for extracting response body.

Chrome and debugger

Before sticking with Chrome, i used to try so many Chrome extension like Web Sniffer to catch requests. But after using Debugger in Chrome for a while . I recognize, it is all i need to see the requests. Only 1 thing maybe you need to pay attention is the setting Preserve log, it will not clear the requests. My helpful screenshot

Understanding the way to make request with the body in GO

Because we do not create that website’s back-end, so we do not know about it. We have to follow the request’s information.
Query String Parameters

req, err := http.NewRequest("GET", url, nil)
req.Header.Add("Cookie", oreillyCookie)
req.Header.Add("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36")
query := req.URL.Query()
query.Add("param1", "param1")
query.Add("param2", "param1")

req.URL.RawQuery = query.Encode()
resp, err := client.Do(req)
if err != nil {

defer resp.Body.Close()

Request Payload

str := `{"param1":"param1","param2":"param2"}`

res, _ := http.NewRequest("POST", url, bytes.NewBuffer([]byte(str)))
res.Header.Add("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36")
res.Header.Add("content-type", "application/json")
res.Header.Add("x-requested-with", "XMLHttpRequest")

resp, err := client.Do(res)
if err != nil {

defer resp.Body.Close()

Form Data

form := url.Values{}
form.Add("param1", "param1")

res, _ = http.NewRequest("POST", url, strings.NewReader(form.Encode()))
res.Header.Add("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36")
res.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err = client.Do(res)
if err != nil {

defer resp.Body.Close()


After a while doing this things, sometimes it makes me crazy, do not understand why it does not work although i things did extactly. But it looks like a game, when you figure out what the point is, you win the game. I am really excited about that moment. Try, read message error, try again.

It has a lot more things to achieve, to discover. But again, with Go i feel very comfortable about the simple and efficiency.