跳到主要内容

开启/关闭可观测性系统

为便于后续排查故障、定位问题，请在正式上线使用前开启可观测。

一、前置约束​

适用版本：ONES版本 >= 6.5.0 || 6.1 LTS >= 6.1.94 且底座为 K3s（K8s1.16.7 集群不支持）。

二、开启可观测性系统​

1.配置启动开关​

执行下面命令进入运行中的 ones 容器，修改配置文件

ones-ai-k8s.sh
vi config/private.yaml
# 添加如下配置：
onesTelemetryExportEndpoint: "https://hybridcloud.ones.cn/telemetry/otlp/"
onesTelemetryExportLogEnabled: true
onesTelemetryClickhouseEnable: true
otlpExportGrpcAddr: "otlp-collector-service.ones-telemetry:4317"
grafanaLocalNodePort: "31380"
grafanaLocalHost: "${IP}:31380" # Grafana 对外访问入口, 填写实际的IP和NodePort
# 上述配置默认启用混合云上报，需如不启用混合云上报, 请注释或删除如下两个配置:
# onesTelemetryExportEndpoint: "https://hybridcloud.ones.cn/telemetry/otlp/"
# onesTelemetryExportLogEnabled: true

2.配置安全选项（可选）​

默认访问 http://${IP}:31380 grafana 页面时无账号密码，如有安全要求，请再添加如下2个配置:

grafanaLocalBasicAuthUser: "" # example: "admin"; Grafana BasicAuth username.
grafanaLocalBasicAuthPassword: "" # example: "sToIgoAV4m2ho"; for password admin run:  openssl passwd -6 admin

3.卸载旧组件（仅升级场景）​

从 6.5.0 或 LTS 6.1.94 版本以下升级上来的环境，需要先卸载旧组件：

make delete-ones-logging RETAIN_DATA=false
make delete-monitoring-system RETAIN_DATA=false
make delete-prometheus RETAIN_DATA=false
make delete-fluent-bit RETAIN_DATA=false
make delete-ones-alert RETAIN_DATA=false
# 若集群使用本地存储，且无需保留监控数据，可以执行下面命令删除数据
# 默认 ONES 数据目录是 /data/ones，需kubectl describe deploy installer-api | grep Path 确认路径
rm -r /data/ones/ones-local-storage/ones-logging/ones-logging
rm -r /data/ones/ones-local-storage/others-static-pvc/monitoring-system

4. 安装可观测性系统​

# 执行如下命令使配置生效
make setup-ones-telemetry
make setup-ones

#检测所有pod状态为Running，表示开启动作完成
kubectl get pod -n ones-telemetry
kubectl get svc -n ones-telemetry | egrep 'grafana|otlp'
exit

#curl测试返回301表示正常
curl -sSI --max-time 5 http://127.0.0.1:31380

三、功能验证与信息采集​

1. 页面访问​

访问 http://${IP}:31380 Grafana 页面可看到监控数据；分析故障时，可截图故障点的监控信息提供给 ONES 工程师。

2.日志生成与获取​

在如下目录看到新的日志：

# 本地存储模式下默认 ONES 数据目录是 /data/ones
cd /data/ones/ones-local-storage/ones-logging/ones-telemetry/data-volume-ones-logging-statefulset-0/logs/

# 分析故障时，可打包该文件、提供给 ONES 工程师
tar cvf  "/tmp/logs-$(hostname)-$(date '+%Y%m%d').tar" ./
# 生成的离线压缩文件在 `/tmp/` 目录下，例如 `/tmp/logs-test1-20241008.tar`

3. 诊断文件生成与获取​

页面操作后，可在如下目录看到新的离线诊断文件：

# 本地存储模式下默认 ONES 数据目录是 /data/ones
cd /data/ones/ones-local-storage/otlp-collector/ones-telemetry/data-volume-otlp-collector-statefulset-0/files

# 分析故障时，可打包该文件、提供给 ONES 工程师
tar cvf  "/tmp/telemetry-$(hostname)-$(date '+%Y%m%d').tar" ./
# 生成的离线压缩文件在 `/tmp/` 目录下，例如 `/tmp/telemetry-test1-20241008.tar`

四、可观测性系统访问端口配置​

1、修改监控访问端口​

ONES 可观测性系统默认开启指标监控。通过配置 grafanaLocalNodePort 控制监控页面的访问端口。

修改 config/private.yaml 配置文件

grafanaLocalNodePort: "31380"
grafanaLocalHost: "${IP}:31380"

执行命令应用配置

make setup-ones-telemetry

使用 ONES 服务的 ip + grafanaLocalNodePort 访问 grafana 控制台。

2、关闭监控访问​

修改 config/private.yaml 配置文件

# 注释如下配置
# grafanaLocalNodePort: "31380"
# grafanaLocalHost: "${IP}:31380"

执行命令应用配置

make setup-ones-telemetry

五、卸载可观测性系统​

进入运行中的 ones 容器

ones-ai-k8s.sh

修改 config/private.yaml 配置文件，注释或删除上述新增配置。

执行如下命令，关闭可观测性系统并删除数据

make setup-ones # 应用配置
make delete-ones-telemetry RETAIN_DATA=false # 卸载可观测性系统

六、配置告警通知​

ONES 支持开启alertmanager监控告警，以如下几种方式为例，参考说明如下：

# 首先进入运行中的 ones 容器
ones-ai-k8s.sh

# 参考下述样例修改配置文件
vim config/private.yaml

# 保存修改，应用配置下发生效
make setup-ones-telemetry
kubectl delete pod -n ones-telemetry   vmalertmanager-main-0 

# 跟进告警日志
kubectl logs -f -n ones-telemetry   vmalertmanager-main-0  alertmanager

1 使用Webhook 接收Alertmanager告警​

# 以下分别创建两组 receivers webhook接收地址(可以创建多组receivers)
# 将P0-P3级别告警分别发送:Critical receivers
# 添加如下配置

alertmanagerConfig: |
  "global":
    "resolve_timeout": "1m"
  "inhibit_rules":
  - "equal":
    - "namespace"
    - "alertname"
    "source_match":
      "severity": "critical"
    "target_match_re":
      "severity": "warning|info"
  - "equal":
    - "namespace"
    - "alertname"
    "source_match":
      "severity": "warning"
    "target_match_re":
      "severity": "info"
  "receivers":
  - "name": "Default"
    "webhook_configs":
    - url: 'http://webhook-address:8129/Alert'
  - "name": "Critical"
    "webhook_configs":
    - url: 'http://webhook-address:8130/Alert'
  "route":
    "group_by":
    - "namespace"
    "group_interval": "1m"
    "group_wait": "30s"
    "receiver": "Default"
    "repeat_interval": "12h"
    "routes":
    - "match":
        "alertname": "Default"
      "receiver": "Default"
    - "match":
        "priority": "P0"
      "receiver": "Critical"
      "repeat_interval": "10m"
    - "match":
        "priority": "P1"
      "receiver": "Critical"
      "repeat_interval": "30m"
    - "match":
        "priority": "P2"
      "receiver": "Critical"
      "repeat_interval": "2h"
  "templates":
  - '/etc/alertmanager/config/default.tmpl'

2 使用邮件Smtp 发送Alertmanager告警​

# 在Alertmanager使用邮箱通知，只需要定义好SMTP相关的配置，
# 并且在receiver中定义接收方的邮件地址即可。在Alertmanager中我们可以直接在配置文件的global中定义全局的SMTP配置
# 如果使用QQ邮箱作为发件箱，默认SSL协议，请在global段添加"smtp_require_tls": false
# 添加如下配置

 alertmanagerConfig: |
   "global":
     "resolve_timeout": "5m"
     "smtp_smarthost": 'smtp.163.com:25'
     "smtp_from": 'example@163.com'
     "smtp_auth_username": 'example@163.com'
     "smtp_auth_password": 'yourpassword'
   "inhibit_rules":
   - "equal":
     - "namespace"
     - "alertname"
     "source_match":
       "severity": "critical"
     "target_match_re":
       "severity": "warning|info"
   - "equal":
     - "namespace"
     - "alertname"
     "source_match":
       "severity": "warning"
     "target_match_re":
       "severity": "info"
   "receivers":
   - "name": "Default"
     "email_configs":
     - "to": "example@163.com"
   - "name": "Watchdog"
   - "name": "Critical"
     "email_configs":
     - "to": "example@163.com"
   "route":
     "group_by":
     - "namespace"
     "group_interval": "1m"
     "group_wait": "30s"
     "receiver": "Default"
     "repeat_interval": "12h"
     "routes":
     - "match":
         "alertname": "Watchdog"
       "receiver": "Watchdog"
     - "match":
         "priority": "P0"
       "receiver": "Critical"
       "repeat_interval": "10m"
     - "match":
         "priority": "P1"
       "receiver": "Critical"
       "repeat_interval": "30m"
     - "match":
         "priority": "P2"
       "receiver": "Critical"
       "repeat_interval": "2h"
   "templates":
   - '/etc/alertmanager/config/default.tmpl'

3、配置开启alertmanager监控告警​

下面示范使用企业微信群Bot告警机器人功能

# 以下分别创建两组 receivers，其中Critical设置两组webhook接收地址
# 发送集群内企业微信Bot WebhookService服务，由服务转发告警至企业微信Bot
# 将P0-P1级别告警分别发送:Critical receivers
# 将P2-P3级别告警分别发送:Default receivers
# 添加如下配置
enableAlertWechatRobot: true # 打开集群内企业微信告警机器人功能，由服务转发到企业微信Bot

alertmanagerConfig: |
  "global":
    "resolve_timeout": "1m"
  "inhibit_rules":
  - "equal":
    - "namespace"
    - "alertname"
    "source_match":
      "severity": "critical"
    "target_match_re":
      "severity": "warning|info"
  - "equal":
    - "namespace"
    - "alertname"
    "source_match":
      "severity": "warning"
    "target_match_re":
      "severity": "info"
  "receivers":
  - "name": "Default"
    "webhook_configs":
    - "url": "http://otlp-collector-service:8129/alerts"
      "send_resolved": true
    - "url": "http://wechatrobot-svc:13100/webhook?key=这里填写企微Bot Webhook Key"
      "send_resolved": true
  - "name": "Watchdog"
  - "name": "Critical"
    "webhook_configs":
    - "url": "http://otlp-collector-service:8129/alerts"
      "send_resolved": true
    - "url": "http://wechatrobot-svc:13100/webhook?key=这里填写企微Bot Webhook Key"
      "send_resolved": true
  "route":
    "group_by":
    - "namespace"
    "group_interval": "1m"
    "group_wait": "30s"
    "receiver": "Default"
    "repeat_interval": "12h"
    "routes":
    - "match":
        "alertname": "Watchdog"
      "receiver": "Watchdog"
    - "match":
        "priority": "P0"
      "receiver": "Critical"
      "repeat_interval": "10m"
    - "match":
        "priority": "P1"
      "receiver": "Critical"
      "repeat_interval": "30m"
    - "match":
        "priority": "P2"
      "receiver": "Critical"
      "repeat_interval": "2h"
  "templates":
  - "/etc/alertmanager/config/default.tmpl"

更多告警接收器配置方案，请参考 Alertmanager官方配置文档。

七、配置参数说明​

配置项类型示例描述

onesTelemetryLoggingEnabled

booltrue是否记录日志到本地，默认开启

grafanaLocalNodePort

number31380grafana 监控页面访问端口

grafanaLocalHost

string10.0.0.10:31380Grafana 对外访问入口

alertmanagerConfig

复合结构参考文档模版alertmanager告警配置

otlpExportGrpcAddr

stringotlp-collector-service.ones-telemetry:4317otlp 服务地址，配置后会开启链路追踪，识别并采集有报错的链路

onesTelemetryClickhouseEnable

boolfalse是否开启链路追踪数据写入 ClickHouse（otel_traces/otel_logs），用于集群内链路追踪看板，默认关闭

onesTelemetryExportEndpoint

stringhttps://hybridcloud.ones.cn/telemetry/otlp/混合云 otlp 服务地址，配置后会上报监控数据到 ONES 混合云

onesTelemetryExportLogEnabled

boolfalse是否上报报错链路日志到 ONES 混合云，默认关闭

onesTelemetryLoggingLogFileMaxAge

int7本地日志文件最长保存天数，默认7天

onesTelemetryLoggingLogTotalMaxSize

int4096本地日志文件最大大小，默认4096M

onesTelemetryLoggingLogDiskIOTimeThreshold

int90本地日志默认在磁盘I/O时间高于90%时停止写入，低配环境遇到该负载限制无法写日志时可调整为200

onesTelemetryFluentElasticsearchEnabled

boolfalse是否开启日志输出到 elasticsearch，默认关闭

onesTelemetryFluentCustomOutputConfigurations

string自定义日志输出配置