AWS Beanstalk: 日誌 Logs 推送到 CloudWatch 之設定筆記

參考 Reference 章節的 AWS 文件，紀錄如何將部署在 AWS Beanstalk 的服務所產生的日誌 (logs)，設定 CloudWatch Logs agent 來自動推送至 CloudWatch，而不用每次發生問題得自己花時間去撈 logs……。

提示：當初是因為 Beanstalk 預設安裝的 CloudWatch Logs agent 並沒有紀錄服務所產生的日誌，所以參考其文件手動設定，測試環境為 Go 1 running on 64bit Amazon Linux/2.8.0。

準備 Go 語言所寫的範例服務：

package main

import (
   "fmt"
   "log"
   "net/http"
   "os"
)

func main() {
   port := os.Getenv("PORT")
   if len(port) <= 0 {
      port = "8080"
   }

   l := log.New(os.Stdout, "", log.LstdFlags|log.LUTC)
   le := log.New(os.Stderr, "", log.LstdFlags|log.LUTC)

   h := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
      s := req.URL.String()
      l.Printf("New request: %s", s)
      fmt.Fprintf(w, "Request path: %s", s)
   })

   l.Printf("Start listening...:" + port)
   if err := http.ListenAndServe(":"+port, h); err != nil {
      le.Printf("Listen failed, %v", err)
   }
}

一般日誌會寫入到 os.Stdout，而錯誤日誌寫入到 os.Stderr，Beanstalk 平台會將這些日誌內容，會分別寫入到 /var/log/web-1.log 以及 /var/log/web-1.error.log 這兩個檔案中。

要注意的是日誌 (log) 格式，在以上範例中 prefix 為空白字元，參數 log.LstdFlags 以及 log.LUTC 紀錄 UTC 的日期與時間，注意日誌其格式會像是以下：

2018/05/15 13:26:17 Start listening...:8080
2018/05/15 13:26:21 New request: /
2018/05/15 13:26:22 New request: /favicon.ico
2018/05/15 13:26:27 New request: /hello
2018/05/15 13:26:28 New request: /favicon.ico

從官方文件中，下載 nginx (Go, Ruby, Node.js, and Docker) configuration files 這份預設的設定檔範例。

原則上僅提取這兩個檔案：

cwl-setup.config
- 初始設定 CloudWatch Logs agent，可直接使用不需要修改它
cwl-webrequest-metrics.config
- 設定 nginx 日誌要推送到 CloudWatch，可根據這份調整來設定自己的服務推送

根據 cwl-webrequest-metrics.config，調整其內容設定，將 /var/log/web-1.log 以及 /var/log/web-1.error.log 等其他日誌推送至 CloudWatch，另外儲存成 cwl-applogs-metrics.config。

Mappings:
  CWLogs:
    AppLog:
      LogFile: "/var/log/web-1.log"
      TimestampFormat: "%Y/%m/%d %H:%M:%S"
      TimeZone: "UTC"
      MultiLineStartPattern : "{datetime_format}"
    ErrLog:
      LogFile: "/var/log/web-1.error.log"
      TimestampFormat: "%Y/%m/%d %H:%M:%S"
      TimeZone: "UTC"
      MultiLineStartPattern : "{datetime_format}"

Outputs:
  AppCWLogGroup:
    Description: "The name of the Cloudwatch Logs Log Group created for this environments web server access logs. You can specify this by setting the value for the environment variable: WebRequestCWLogGroup. Please note: if you update this value, then you will need to go and clear out the old cloudwatch logs group and delete it through Cloudwatch Logs."
    Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0AppLogGroup"}

Resources :
  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0AppLogGroup:    ## Must have prefix:  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0
    Type: "AWS::Logs::LogGroup"
    DependsOn: AWSEBBeanstalkMetadata
    DeletionPolicy: Retain     ## this is required
    Properties:
      LogGroupName:
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: AppCWLogGroup
          DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "applogs"]]}
      RetentionInDays: 14

  ## Register the files/log groups for monitoring
  AWSEBAutoScalingGroup:
    Metadata:
      "AWS::CloudFormation::Init":
        CWLogsAgentConfigSetup:
          files:
            ## any .conf file put into /tmp/cwlogs/conf.d will be added to the cwlogs config (see cwl-agent.config)
            "/tmp/cwlogs/conf.d/applogs.conf":
              content : |
                [applogs]
                file = `{"Fn::FindInMap":["CWLogs", "AppLog", "LogFile"]}`
                log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0AppLogGroup" }`
                log_stream_name = {instance_id}
                datetime_format = `{"Fn::FindInMap":["CWLogs", "AppLog", "TimestampFormat"]}`
                time_zone = `{"Fn::FindInMap":["CWLogs", "AppLog", "TimeZone"]}`
                multi_line_start_pattern = `{"Fn::FindInMap":["CWLogs", "AppLog", "MultiLineStartPattern"]}`
                [errlogs]
                file = `{"Fn::FindInMap":["CWLogs", "ErrLog", "LogFile"]}`
                log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0AppLogGroup" }`
                log_stream_name = {instance_id}
                datetime_format = `{"Fn::FindInMap":["CWLogs", "ErrLog", "TimestampFormat"]}`
                time_zone = `{"Fn::FindInMap":["CWLogs", "ErrLog", "TimeZone"]}`
                multi_line_start_pattern = `{"Fn::FindInMap":["CWLogs", "ErrLog", "MultiLineStartPattern"]}`
              mode  : "000400"
              owner : root
              group : root

Line1-12: 參數與數值宣告
Line7: MultiLineStartPattern 設定每行開頭格式，採用 {datetime_format} (根據日誌格式設定)
Line29: DefaultValue 可設定 Group 名稱
Line40-54: CloudWatch Logs agent 設定檔內容，關於其格式參考官方文件：CloudWatch Logs Agent Reference

參考之前的 Beanstalk 部署筆記，將這些 configs 加入至 .ebextensions/ 資料夾，並且打包成 source bundle。

接著上傳部署，會發現在 CloudWatch > Logs 不會有任何的日誌資料出現，若查看該服務的 Logs - awslogs.log，會發現 CloudWatch 的錯誤：

2018-05-11 11:20:52,028 - cwlogs.push.batch - INFO - 2859 - Thread-2279 - Creating log group xsgGameDev-env/apps.log.
2018-05-11 11:20:52,136 - cwlogs.push.batch - WARNING - 2859 - Thread-2279 - CreateLogGroup failed with exception An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::592499666128:assumed-role/aws-elasticbeanstalk-ec2-role/i-005950796e6f0f89b is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:ap-northeast-1:592499666128:log-group:xsgGameDev-env/apps.log:log-stream:
2018-05-11 11:20:52,136 - cwlogs.push.batch - WARNING - 2859 - Thread-2279 - An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::592499666128:assumed-role/aws-elasticbeanstalk-ec2-role/i-005950796e6f0f89b is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:ap-northeast-1:592499666128:log-group:xsgGameDev-env/apps.log:log-stream:
2018-05-11 11:20:52,136 - cwlogs.push.batch - WARNING - 2859 - Thread-2279 - Method "_setup_resources" has failed for the last time.
2018-05-11 11:20:52,136 - cwlogs.threads - ERROR - 2859 - Thread-2279 - Exception caught in 
Traceback (most recent call last):
  File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/threads.py", line 58, in run
    self._run()
  File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py", line 1403, in _run
    self._publish_event_batch()
  File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py", line 1210, in _publish_event_batch
    self.sequence_token = self._put_log_events(self.event_batch)
  File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py", line 1294, in _put_log_events
    self._setup_resources
  File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/retry.py", line 116, in wrapper
    'attempts.' % (f.__name__, retry))
RuntimeError: Method "_setup_resources" has failed after 5 unsuccessful attempts.
2018-05-11 11:21:52,527 - cwlogs.push.reader - INFO - 2859 - Thread-2280 - Reader is leaving as requested...
2018-05-11 11:21:55,507 - cwlogs.push.stream - INFO - 2859 - Thread-1 - Removing dead reader [3ce9a002f611ba22d298aefdda3eec47, /var/log/web-1.error.log]
2018-05-11 11:21:55,507 - cwlogs.push.stream - INFO - 2859 - Thread-1 - Removing dead publisher [3ce9a002f611ba22d298aefdda3eec47, /var/log/web-1.error.log]

原因是執行環境沒有寫入 CloudWatch 的權限 (AccessDeniedException)，因此需要在 IAM 設定中賦予其權限。

從 Beanstalk 設定中，找到該環境所屬的 instance profile 名稱：

Beanstalk 安全性設定頁面
在 IAM > Roles 找到該 profile 設定

IAM Roles 管理頁面

Add inline policy > JSON 新增其權限：

  {
     "Version": "2012-10-17",
     "Statement": [
        {
           "Effect": "Allow",
           "Action": [
              "logs:CreateLogStream",
              "logs:DescribeLogGroups",
              "logs:DescribeLogStreams",
              "logs:GetLogEvents",
              "logs:PutRetentionPolicy",
              "logs:PutLogEvents",
              "logs:CreateLogGroup"
           ],
           "Resource": "*"
        }
     ]
  }