思元的開發筆記

關於 web service, unity, blogger 等軟體工程筆記

Golang HTTP Request 遇到的坑：An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

王思元 8:35 下午 程式語言 , Golang Edit icon 沒有留言
Golang

之前服務要上線前，開始跑內部測試發現的問題，從 Event log 留意到服務會自動關閉，從中找到錯誤訊息如下：

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

經過層層的線索尋找，透過一些工具 (i.e. netstat) 查看 socket 狀態，終於找到原因。由於該服務會使用 HTTP 向其他服務要求資料，但資料處理完後，並沒有立刻關閉連線，導致 TCP 連線資源佔用，此外也沒有限制 TCP 連線建立數量，所以最後就爆炸發生上述的錯誤訊息。

釋放 HTTP Request

一般使用 http.Request，透過 http.Client 建立連線並取得回應：

/*
import (
   "net/http"
   "io/ioutil"
)
*/

req, _ := http.NewRequest(http.MethodGet, url, nil)
rep, _ := http.DefaultClient.Do(req)
body, _ := ioutil.ReadAll(rep.Body)

之前以為這樣就可以愉快的處理 url 回應的 body 資料，而不用再管其他事情，但經由社團先進們指點，才留意到 Golang 文件說明，若讀取完 rep.Body 而不呼叫 rep.Body.Close()，則該 request 所使用到的 TCP 連線資源不會立即釋放，而是會佔用一段時間後，由 Golang 內部機制釋放其連線資源 (GC?)。

由於我們內部測試在短時間內送出大量的要求，該服務程式在短時間內建立大量的 HTTP 要求，但取得完回應後，卻沒有立刻將連線資源釋放，而是持續佔用，Golang 內部實作持續建立新的 TCP 連線，最後發生該錯誤，TCP/IP port exhaustion，把 TCP 資源用光了。

因此，應該加入釋放資源的程式碼：

defer rep.Body.Close()

另外從網路上的討論，若沒有要讀取 rep.Body，與應該將該 buffer 清空：

io.Copy(ioutil.Discard, rep.Body)

一個完整範例如下：

rep, err := http.DefaultClient.GET(url)
if err != nil {
   log.Fatal(err)
}
io.Copy(ioutil.Discard, rep.Body)
rep.Body.Close()

Reference

附錄 Socket 狀態

使用 netstat -na 查看目前系統的 socket 狀態：

Active Connections

  Proto Local Address Foreign Address State PID
  TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 832
  TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 2140
  TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING 380
  TCP 0.0.0.0:5985 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:47001 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:49664 0.0.0.0:0 LISTENING 612
  TCP 0.0.0.0:49665 0.0.0.0:0 LISTENING 532
  TCP 0.0.0.0:49666 0.0.0.0:0 LISTENING 1120
  TCP 0.0.0.0:49667 0.0.0.0:0 LISTENING 1120
  TCP 0.0.0.0:49668 0.0.0.0:0 LISTENING 1660
  TCP 0.0.0.0:49671 0.0.0.0:0 LISTENING 1688
  TCP 0.0.0.0:49681 0.0.0.0:0 LISTENING 712
  TCP 0.0.0.0:49771 0.0.0.0:0 LISTENING 720
  TCP 172.31.30.24:139 0.0.0.0:0 LISTENING 4
  TCP 172.31.30.24:443 60.251.127.174:62171 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62177 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62178 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62183 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62184 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62188 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62191 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62192 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62196 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62197 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62198 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62199 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62200 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62201 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62202 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62203 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62204 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62205 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62206 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62207 SYN_RECEIVED 2140
  TCP 172.31.30.24:3389 60.251.127.174:61449 ESTABLISHED 380
  TCP 172.31.30.24:49687 172.31.16.196:7008 ESTABLISHED 2140
  TCP 172.31.30.24:49689 172.31.16.196:7004 ESTABLISHED 2140
  TCP 172.31.30.24:50746 172.31.16.196:7008 ESTABLISHED 2140
  TCP 172.31.30.24:50747 23.50.91.27:80 TIME_WAIT 0
  TCP 172.31.30.24:50763 207.46.154.155:80 TIME_WAIT 0
  TCP 172.31.30.24:50784 111.221.29.100:443 ESTABLISHED 3104
  TCP 172.31.30.24:50847 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50851 172.31.16.196:7006 ESTABLISHED 2140
  TCP 172.31.30.24:50906 137.116.139.114:443 TIME_WAIT 0
  TCP 172.31.30.24:50949 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50950 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50956 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50959 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50960 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50963 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50964 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50965 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50966 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50967 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50968 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50969 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50970 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50971 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50972 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50973 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50974 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50975 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50976 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50977 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50978 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50979 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50980 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50981 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50982 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50983 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50984 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50985 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50986 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50987 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50988 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50989 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50990 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50991 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50992 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50993 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50994 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50995 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50996 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50997 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50998 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50999 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51000 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51001 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51002 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51003 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51004 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51005 172.31.23.164:80 TIME_WAIT 0
---- 以下省略 172.31.30.24:51006 - 172.31.30.24:52500 .. TIME_WAIT 狀態

沒有留言: