關於 web service, unity, blogger 等軟體工程筆記

Golang HTTP Request 遇到的坑:TCP/IP port exhaustion

Edit icon 沒有留言
Golang

之前服務要上線前,開始跑內部測試發現的問題,從 Event log 留意到服務會自動關閉,從中找到錯誤訊息如下:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

經過層層的線索尋找,透過一些工具 (i.e. netstat) 查看 socket 狀態,終於找到原因。由於該服務會使用 HTTP 向其他服務要求資料,但資料處理完後,並沒有立刻關閉連線,導致 TCP 連線資源佔用,此外也沒有限制 TCP 連線建立數量,所以最後就爆炸發生上述的錯誤訊息。

釋放 HTTP Request

一般使用 http.Request,透過 http.Client 建立連線並取得回應:

/*
import (
   "net/http"
   "io/ioutil"
)
*/

req, _ := http.NewRequest("GET", url, nil)
rep, _ := http.DefaultClient.Do(req)
body, _ := ioutil.ReadAll(rep.Body)

之前以為這樣就可以愉快的處理 url 回應的 body 資料,而不用再管其他事情,但經由社團先進們指點,才留意到 Golang 文件說明,若讀取完 rep.Body 而不呼叫 rep.Body.Close(),則該 request 所使用到的 TCP 連線資源不會立即釋放,而是會佔用一段時間後,由 Golang 內部機制釋放其連線資源 (GC?)。

由於我們內部測試在短時間內送出大量的要求,該服務程式在短時間內建立大量的 HTTP 要求,但取得完回應後,卻沒有立刻將連線資源釋放,而是持續佔用,Golang 內部實作持續建立新的 TCP 連線,最後發生開頭的錯誤,TCP/IP port exhaustion,把 TCP 資源用光了。

因此,應該加入釋放資源的程式碼:

defer rep.Body.Close()

另外從網路上的討論,若沒有要讀取 rep.Body,建議可以將該 buffer 清空:

io.Copy(ioutil.Discard, rep.Body)

Reference

附錄 Socket 狀態

使用 netstat -na 查看目前系統的 socket 狀態:

Active Connections

  Proto Local Address Foreign Address State PID
  TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 832
  TCP 0.0.0.0:443 0.0.0.0:0 LISTENING 2140
  TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING 380
  TCP 0.0.0.0:5985 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:47001 0.0.0.0:0 LISTENING 4
  TCP 0.0.0.0:49664 0.0.0.0:0 LISTENING 612
  TCP 0.0.0.0:49665 0.0.0.0:0 LISTENING 532
  TCP 0.0.0.0:49666 0.0.0.0:0 LISTENING 1120
  TCP 0.0.0.0:49667 0.0.0.0:0 LISTENING 1120
  TCP 0.0.0.0:49668 0.0.0.0:0 LISTENING 1660
  TCP 0.0.0.0:49671 0.0.0.0:0 LISTENING 1688
  TCP 0.0.0.0:49681 0.0.0.0:0 LISTENING 712
  TCP 0.0.0.0:49771 0.0.0.0:0 LISTENING 720
  TCP 172.31.30.24:139 0.0.0.0:0 LISTENING 4
  TCP 172.31.30.24:443 60.251.127.174:62171 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62177 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62178 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62183 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62184 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62188 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62191 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62192 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62196 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62197 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62198 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62199 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62200 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62201 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62202 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62203 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62204 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62205 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62206 ESTABLISHED 2140
  TCP 172.31.30.24:443 60.251.127.174:62207 SYN_RECEIVED 2140
  TCP 172.31.30.24:3389 60.251.127.174:61449 ESTABLISHED 380
  TCP 172.31.30.24:49687 172.31.16.196:7008 ESTABLISHED 2140
  TCP 172.31.30.24:49689 172.31.16.196:7004 ESTABLISHED 2140
  TCP 172.31.30.24:50746 172.31.16.196:7008 ESTABLISHED 2140
  TCP 172.31.30.24:50747 23.50.91.27:80 TIME_WAIT 0
  TCP 172.31.30.24:50763 207.46.154.155:80 TIME_WAIT 0
  TCP 172.31.30.24:50784 111.221.29.100:443 ESTABLISHED 3104
  TCP 172.31.30.24:50847 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50851 172.31.16.196:7006 ESTABLISHED 2140
  TCP 172.31.30.24:50906 137.116.139.114:443 TIME_WAIT 0
  TCP 172.31.30.24:50949 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50950 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50956 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50959 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50960 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50963 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50964 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50965 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50966 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50967 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50968 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50969 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50970 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50971 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50972 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50973 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50974 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50975 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50976 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50977 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50978 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50979 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50980 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50981 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50982 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50983 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50984 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50985 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50986 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50987 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50988 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50989 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50990 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50991 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50992 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50993 172.31.16.196:7006 TIME_WAIT 0
  TCP 172.31.30.24:50994 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50995 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50996 172.31.31.97:3306 TIME_WAIT 0
  TCP 172.31.30.24:50997 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50998 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:50999 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51000 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51001 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51002 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51003 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51004 172.31.23.164:80 TIME_WAIT 0
  TCP 172.31.30.24:51005 172.31.23.164:80 TIME_WAIT 0
---- 以下省略 172.31.30.24:51006 - 172.31.30.24:52500 .. TIME_WAIT 狀態

沒有留言: