V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
zhuzhiqiang
V2EX  ›  信息安全

今天服务器 cpu 直接到 100% 日志中发现/test.html 一直被访问 这是被恶意访问了吗

  •  
  •   zhuzhiqiang · 2019-03-07 10:57:15 +08:00 · 6372 次点击
    这是一个创建于 1848 天前的主题,其中的信息可能已经有所发展或是发生改变。

    113.96.109.157 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.147.39.151 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 121.12.109.39 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 47.106.50.155 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 202.108.249.153 - - [07/Mar/2019:08:07:04 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 60.221.194.35 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 202.99.114.204 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 117.27.235.150 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 122.156.57.161 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 123.6.31.154 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 113.6.227.203 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 121.22.229.26 - - [07/Mar/2019:08:07:06 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.119.23 - - [07/Mar/2019:08:07:06 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 183.214.130.150 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.223.240.35 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.135.227 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.221.154.209 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.31.194.149 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 61.168.101.24 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 58.20.147.25 - - [07/Mar/2019:08:07:09 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.112.13.205 - - [07/Mar/2019:08:07:10 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.151.171 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.167.151.155 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 27.221.56.150 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 106.60.80.28 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.201.253.38 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 111.6.251.48 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 223.111.105.160 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 183.213.20.27 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.23.169.196 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 112.29.216.161 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 124.239.234.163 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 61.163.8.22 - - [07/Mar/2019:08:07:14 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.190.214.147 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 140.205.253.144 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 111.48.30.40 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 27.221.92.164 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1

    第 1 条附言  ·  2019-03-07 12:39:35 +08:00
    这一定是恶意访问了吧,已经 UA 403 还有什么办法解决吗

    36.104.139.38 - - [07/Mar/2019:12:36:22 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 36.104.139.3 "-" "-" "-" "0.000"sendfileon
    47.105.29.90 - - [07/Mar/2019:12:36:23 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 47.105.29.67 "-" "-" "-" "0.000"sendfileon
    27.221.92.73 - - [07/Mar/2019:12:36:23 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 27.221.92.117 "-" "-" "-" "0.000"sendfileon
    101.200.28.193 - - [07/Mar/2019:12:36:24 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" - "-" "-" "-" "0.000"sendfileon
    14.204.147.209 - - [07/Mar/2019:12:36:25 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 14.204.147.197 "-" "-" "-" "0.000"sendfileon
    61.168.101.152 - - [07/Mar/2019:12:36:25 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 61.168.101.131 "-" "-" "-" "0.000"sendfileon
    58.216.118.23 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 58.216.118.3 "-" "-" "-" "0.000"sendfileon
    118.31.194.184 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" - "-" "-" "-" "0.000"sendfileon
    121.42.17.201 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 121.42.17.197 "-" "-" "-" "0.000"sendfileon
    111.6.243.37 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 111.6.243.3 "-" "-" "-" "0.000"sendfileon
    117.91.192.201 - - [07/Mar/2019:12:36:27 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 117.91.192.197 "-" "-" "-" "0.000"sendfileon
    120.223.242.31 - - [07/Mar/2019:12:36:28 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 120.223.242.3 "-" "-" "-" "0.000"sendfileon
    第 3 条附言  ·  2019-03-07 18:54:19 +08:00
    问题找到了,CPU 跑满并不是这个恶意访问造成的。 [逃跑]
    38 条回复    2019-04-03 17:35:23 +08:00
    Humorce
        1
    Humorce  
       2019-03-07 11:00:33 +08:00
    所以这个页面是干什么的呢
    zhuzhiqiang
        2
    zhuzhiqiang  
    OP
       2019-03-07 11:01:50 +08:00
    @Humorce 没这个页面 返回的 404
    Humorce
        3
    Humorce  
       2019-03-07 11:07:41 +08:00
    域名是新买的?

    这个时间密度,先更改配置 Block 掉此 UA 吧
    zhuzhiqiang
        4
    zhuzhiqiang  
    OP
       2019-03-07 12:40:58 +08:00
    @Humorce 403 了 还有什么好的解决办法吗 还一直在访问 [捂脸]
    blless
        5
    blless  
       2019-03-07 12:41:55 +08:00 via Android
    404 都能跑满 CPU 吗
    fiht
        6
    fiht  
       2019-03-07 12:49:29 +08:00
    Go-http-client 这个是 go 语言程序呀
    zhuzhiqiang
        7
    zhuzhiqiang  
    OP
       2019-03-07 12:52:18 +08:00
    @blless 我觉得也不至于啊 大佬 看下 CPU 和 TCP
    Steps
        8
    Steps  
       2019-03-07 13:06:22 +08:00
    是否使用了负载均衡?

    我的站跟你一模一样的情况,我过滤了 UA 直接给 503 了

    现在一共跑了 一千多万次吧。。。
    claysec
        9
    claysec  
       2019-03-07 13:16:54 +08:00
    @zhuzhiqiang 接个 cdn 让他慢慢跑呗
    zhuzhiqiang
        10
    zhuzhiqiang  
    OP
       2019-03-07 13:20:19 +08:00
    @Steps 没有使用均衡负载服务 就 Nginx 做了个热备
    boris1993
        11
    boris1993  
       2019-03-07 13:21:09 +08:00 via Android   ❤️ 2
    返回个 gzip 炸弹?
    zhuzhiqiang
        12
    zhuzhiqiang  
    OP
       2019-03-07 13:22:47 +08:00
    @Steps 老哥你的也是这个 UA 吗
    Vhc
        13
    Vhc  
       2019-03-07 13:25:19 +08:00
    1、这个访问频次并不高,CPU 占用和这一毛钱关系也没有。
    2、千万不要屏蔽 "Go-http-client/1.1" 这个 UA
    dbpe
        14
    dbpe  
       2019-03-07 13:27:42 +08:00
    新知识..GZIp Boom..
    gamexg
        15
    gamexg  
       2019-03-07 13:31:11 +08:00
    跳转到 ubuntu iso ?
    gamexg
        16
    gamexg  
       2019-03-07 13:31:42 +08:00
    @gamexg #15 额,开源社区钱不多,还是跳转到微软 iso 吧。
    zhuzhiqiang
        17
    zhuzhiqiang  
    OP
       2019-03-07 13:37:24 +08:00
    @Vhc 大佬 怎么说
    CallMeReznov
        18
    CallMeReznov  
       2019-03-07 13:46:55 +08:00
    @Vhc 为什么不能屏蔽?
    Steps
        19
    Steps  
       2019-03-07 13:53:06 +08:00
    @Vhc #13 不屏蔽? 你告诉我这样的请求量
    已不间断运行: 61 天 18 小时 34 分钟




    只是一个 503 的错误页面,61 天 867G 流量,该如何处理?
    LanAiFaZuo
        20
    LanAiFaZuo  
       2019-03-07 13:54:17 +08:00
    我昨天到今天也是 cpu 爆满,用的宝塔。不知道是不是被黑了。
    Steps
        21
    Steps  
       2019-03-07 14:15:59 +08:00
    求高人解决!
    ryd994
        22
    ryd994  
       2019-03-07 15:07:59 +08:00
    直接让 web 服务器给他一个静态页面不就好了。Nginx 和 Apache 都可以轻松实现这个目的。
    这点请求量,简单的静态请求不可能打满 CPU。要么有别的恶意请求混在里面。要么这些连接有鬼,比如 HTTP slow post 之类的。tcpdump 抓包分析吧。
    Newbing
        23
    Newbing  
       2019-03-07 15:30:32 +08:00
    我也遇到了,每天几十万次的请求。我后面建立了一个默认的页面。
    Steps
        24
    Steps  
       2019-03-07 15:59:45 +08:00
    @ryd994 #22 就算是静态页面 只有 2b 一天几十万次,也有 几个 G 的流量啊
    Steps
        25
    Steps  
       2019-03-07 16:22:04 +08:00
    @zhuzhiqiang #12 我是姐姐,还有,我的 问题和你一模一样的
    ryd994
        26
    ryd994  
       2019-03-07 17:20:53 +08:00 via Android
    @Steps 从你贴的 log 估测每秒 5 次,往上加一个数量级,50*24*3600 几百 MB 顶天了。你看到的这些都是烟幕弹。每秒 5 次的请求是不可能打出 100%CPU 的。
    你可以直接使用 limit_req 控制频率。也可以写个脚本查 404 的频率,再把恶意 IP 加到 ipset 里让 iptables 过滤。

    你先把 log 过滤一遍,看还有其他什么鬼。然后还是抓包分析。

    不必屏蔽 UA:1.已经是 404 了,屏蔽 UA 也减少不了多少压力 2.换个 UA 还不简单?
    opengps
        27
    opengps  
       2019-03-07 17:25:57 +08:00
    404 不应该导致 cpu 爆满吧
    SakuraKuma
        28
    SakuraKuma  
       2019-03-07 18:18:43 +08:00 via Android
    fail2ban 检查到这个 test 封 ip
    Marsss
        29
    Marsss  
       2019-03-07 18:24:56 +08:00 via iPhone
    触发验证码给他
    Steps
        30
    Steps  
       2019-03-07 18:32:55 +08:00
    @ryd994 #26 我并没有 100% CPU,我只是跑了很多 流量出来。。



    这里有请求的次数 503 错误就是被 UA 封闭的
    Steps
        31
    Steps  
       2019-03-07 18:33:54 +08:00
    @ryd994 #26
    @SakuraKuma #28 问题就是 IP 从不重复。。这个很害怕的。。
    zhuzhiqiang
        32
    zhuzhiqiang  
    OP
       2019-03-07 18:57:37 +08:00
    @CallMeReznov
    @Humorce
    @LanAiFaZuo
    @Marsss
    @Newbing
    @SakuraKuma
    @Steps
    @Vhc
    @blless
    @boris1993
    感谢各位 问题已经找到了
    不是恶意访问造成的 CPU 100 如果不是 cpu100 还真发现不了这个恶意访问 [捂脸]
    Steps
        33
    Steps  
       2019-03-08 13:58:48 +08:00
    @zhuzhiqiang #32 问题是什么?

    恶意访问不管了吗?
    zhuzhiqiang
        34
    zhuzhiqiang  
    OP
       2019-03-08 15:33:12 +08:00
    @Steps 恶意访问我也没办法 IP 动态的
    导致 CPU100 的问题是 业务写的有问题 SQL 慢查询 [逃跑]
    zhuzhiqiang
        35
    zhuzhiqiang  
    OP
       2019-03-08 15:40:37 +08:00
    @Steps 姐姐 我刚刚 awk 统计了下 nginx log /test.html 5 个小时跑了 269247 次[捂脸]
    Steps
        36
    Steps  
       2019-03-08 21:06:47 +08:00
    @zhuzhiqiang #35 是的,我也是 1 天差不多 40 多万次吧。。。

    很可怕,不晓得这是怎么回事!
    Steps
        37
    Steps  
       2019-03-12 09:10:15 +08:00
    /t/543424

    看下这个,是否你也开了?
    WanJiJun
        38
    WanJiJun  
       2019-04-03 17:35:23 +08:00
    是阿里云动态 CDN 搞的吧,我之前也这样,后来看 https://www.v2ex.com/t/543424,关掉动态加速恢复正常了。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   我们的愿景   ·   实用小工具   ·   2826 人在线   最高记录 6543   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 35ms · UTC 15:23 · PVG 23:23 · LAX 08:23 · JFK 11:23
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.