{"id":34264,"date":"2026-11-05T10:00:00","date_gmt":"2026-11-05T02:00:00","guid":{"rendered":"https:\/\/cn.hostease.com\/blog\/?p=34264"},"modified":"2026-10-27T18:00:33","modified_gmt":"2026-10-27T10:00:33","slug":"ai-crawler-traffic-control-robots-ua","status":"publish","type":"post","link":"https:\/\/cn.hostease.com\/blog\/tutorials\/ai-crawler-traffic-control-robots-ua\/","title":{"rendered":"AI \u8bad\u7ec3\u722c\u866b\u6293\u53d6\u91cf\u7edf\u8ba1\u4e0e\u5e26\u5bbd\u6cbb\u7406\uff1arobots.txt\u3001UA \u9650\u901f\u4e0e\u8bbf\u95ee\u65e5\u5fd7\u6838\u5bf9"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/cn.hostease.com\/blog\/wp-content\/uploads\/2026\/10\/ai-crawler-traffic-control-robots-ua-cover.webp\" alt=\"AI \u8bad\u7ec3\u722c\u866b\u6293\u53d6\u91cf\u4e0e\u5e26\u5bbd\u6cbb\u7406\u7684\u6574\u4f53\u5173\u7cfb\u793a\u610f\" loading=\"lazy\" \/><\/p>\n<p>\u6253\u5f00\u670d\u52a1\u5668\u8d26\u5355\u65f6\uff0c\u5f88\u591a\u7ad9\u957f\u4f1a\u770b\u5230\u4e00\u4e2a\u5c34\u5c2c\u7684\u73b0\u8c61\uff1a\u8bbf\u5ba2\u6570\u6ca1\u6da8\uff0c\u51fa\u53e3\u6d41\u91cf\u5374\u5148\u6da8\u4e86\u3002\u8fc7\u53bb\u4e00\u5e74\uff0cAI \u8bad\u7ec3\u722c\u866b\uff08\u4e3a\u6a21\u578b\u8bad\u7ec3\u800c\u81ea\u52a8\u6293\u53d6\u7f51\u9875\u5185\u5bb9\u7684\u673a\u5668\u4eba\u7a0b\u5e8f\uff09\u4ece\u96f6\u661f\u7684\u6d4b\u8bd5\u8bf7\u6c42\u53d8\u6210\u4e86\u7a33\u5b9a\u6d41\u91cf\uff0c\u4e00\u4e2a\u65e5 IP \u4e94\u5343\u5de6\u53f3\u7684\u5185\u5bb9\u7ad9\uff0c\u4e00\u4e2a\u6708\u88ab\u8bad\u7ec3\u722c\u866b\u62c9\u8d70\u51e0\u5341 GB \u5e76\u4e0d\u7f55\u89c1\u3002\u8fd9\u7bc7\u6587\u7ae0\u6559\u4f60\u5982\u4f55\u7528\u8bbf\u95ee\u65e5\u5fd7\u628a\u6293\u53d6\u91cf\u7b97\u6e05\u695a\uff0c\u518d\u901a\u8fc7 robots.txt \u58f0\u660e\u3001Nginx \u91cc\u7684 UA\uff08User-Agent\uff0c\u722c\u866b\u81ea\u62a5\u7684\u8eab\u4efd\u5b57\u7b26\u4e32\uff09\u9650\u901f\u548c\u7f13\u5b58\u7b56\u7565\uff0c\u628a\u5e26\u5bbd\uff08\u5355\u4f4d\u65f6\u95f4\u5185\u53ef\u4f20\u8f93\u7684\u6570\u636e\u91cf\uff09\u538b\u56de\u53ef\u63a7\u8303\u56f4\u3002\u6587\u4e2d\u547d\u4ee4\u53ef\u4ee5\u76f4\u63a5\u5728\u670d\u52a1\u5668\u4e0a\u6267\u884c\uff0c\u4e0d\u9700\u8981\u989d\u5916\u7684\u5546\u4e1a\u5206\u6790\u5de5\u5177\u3002<\/p>\n<h2>\u5148\u628a\u8d26\u7b97\u6e05\u695a\uff1a\u4e09\u7c7b\u8bf7\u6c42\u5fc5\u987b\u5206\u5f00\u7edf\u8ba1<\/h2>\n<p>\u4e00\u4efd\u5e38\u89c1\u7684 access.log \u91cc\u6df7\u7740\u4e09\u79cd\u5b8c\u5168\u4e0d\u540c\u7684\u8bbf\u95ee\u3002\u641c\u7d22\u5f15\u64ce\u8718\u86db\u6293\u7684\u662f HTML \u548c\u5c11\u91cf\u9759\u6001\u8d44\u6e90\uff0c\u5355\u6b21\u8bf7\u6c42\u5c0f\u3001\u9891\u7387\u7a33\u5b9a\uff1b\u8bad\u7ec3\u722c\u866b\u5f80\u5f80\u6309\u6574\u7ad9\u6279\u91cf\u62c9\u53d6\uff0c\u5355\u6b21\u8bf7\u6c42\u4e0d\u5927\u4f46\u603b\u91cf\u60ca\u4eba\uff1b\u771f\u5b9e\u7528\u6237\u5219\u7531\u9875\u9762\u5e26\u52a8\u51e0\u5341\u4e2a CSS\u3001JavaScript \u548c\u56fe\u7247\u8bf7\u6c42\u3002\u628a\u4e09\u8005\u6df7\u5728\u4e00\u8d77\u770b\uff0c\u4f60\u4f1a\u5f97\u5230\u201c\u6d41\u91cf\u6da8\u4e86\u4f46\u4e0d\u77e5\u9053\u6da8\u5728\u54ea\u91cc\u201d\u7684\u7ed3\u8bba\uff0c\u4e5f\u5c31\u65e0\u6cd5\u51b3\u5b9a\u8be5\u9650\u901f\u8c01\u3001\u8be5\u7f13\u5b58\u8c01\u3002<\/p>\n<p>\u5148\u6309\u5929\u6c47\u603b\u51fa\u53e3\u5b57\u8282\u6570\uff0c\u628a\u5f02\u5e38\u65e5\u671f\u5708\u51fa\u6765\u3002Nginx \u9ed8\u8ba4 combined \u683c\u5f0f\u91cc\uff0c\u7b2c\u5341\u4e2a\u5b57\u6bb5\u5c31\u662f\u54cd\u5e94\u5b57\u8282\u6570\uff1a<\/p>\n<pre><code class=\"language-bash\">awk '{gsub(\/\\[\/,&quot;&quot;,$(4)); split($(4),a,&quot;:&quot;); b[a[1]]+=$(10)}\n     END {for (d in b) printf &quot;%s %.2f GB\\n&quot;, d, b[d]\/1073741824}' \\\n     \/var\/log\/nginx\/access.log | sort -k2 -nr | head -7\n<\/code><\/pre>\n<p>\u67d0\u5929\u6d41\u91cf\u6bd4\u5e73\u5e38\u9ad8\u51fa\u4e00\u622a\uff0c\u57fa\u672c\u53ef\u4ee5\u786e\u8ba4\u88ab\u6279\u91cf\u6293\u53d6\u3002\u518d\u6309 UA \u805a\u5408\uff0c\u770b\u662f\u8c01\u5728\u62c9\uff1a<\/p>\n<pre><code class=\"language-bash\">awk -F'&quot;' '{k=$(6); sub(\/\\\/[0-9.]+.*\/,&quot;&quot;,k); if (k==&quot;&quot;) k=&quot;-&quot;; c[k]++}\n     END {for (k in c) print c[k], k}' \/var\/log\/nginx\/access.log | sort -rn | head -10\n<\/code><\/pre>\n<p>\u8f93\u51fa\u91cc\u51fa\u73b0 GPTBot\u3001ClaudeBot\u3001CCBot\u3001Bytespider \u8fd9\u7c7b\u540d\u5b57\u65f6\uff0c\u8bf4\u660e\u8bad\u7ec3\u6293\u53d6\u5df2\u7ecf\u5f00\u59cb\u3002\u771f\u6b63\u7684\u5224\u65ad\u4f9d\u636e\u662f\u5360\u6bd4\u800c\u4e0d\u662f\u540d\u5b57\uff1a\u6309\u6bcf\u5929\u4e09\u4e07\u6b21\u6293\u53d6\u3001\u5e73\u5747\u6bcf\u6b21 128 KB \u4f30\u7b97\uff0c\u4e00\u5929\u5c31\u662f\u7ea6 3.7 GB\uff0c\u4e00\u4e2a\u6708\u8d85\u8fc7 100 GB\uff1b\u5982\u679c\u7ad9\u70b9\u56fe\u7247\u6ca1\u505a\u538b\u7f29\uff0c\u8fd9\u4e2a\u6570\u5b57\u8fd8\u4f1a\u7ffb\u500d\u3002\u8fd9\u4e2a\u91cf\u7ea7\u8db3\u4ee5\u8ba9<a href=\"https:\/\/cn.hostease.com\/shared-hosting.html\">\u5171\u4eab\u4e3b\u673a<\/a>\u4e0a\u7684\u7ad9\u70b9\u89e6\u53d1\u6d41\u91cf\u544a\u8b66\uff0c\u4e5f\u8db3\u4ee5\u8ba9\u6309\u6708\u8ba1\u8d39\u7684\u72ec\u670d\uff08\u72ec\u7acb<a href=\"https:\/\/cn.hostease.com\/dedicated-server\/\">\u7269\u7406\u670d\u52a1\u5668<\/a>\uff09\u8d26\u5355\u53d8\u5f97\u96be\u770b\u3002<\/p>\n<h2>\u65e5\u5fd7\u91cc\u7684\u8bc1\u636e\u94fe\uff1aUA \u53ef\u4ee5\u4f2a\u9020\uff0c\u6765\u6e90\u8981\u4ea4\u53c9\u9a8c\u8bc1<\/h2>\n<p>UA \u5b57\u7b26\u4e32\u6ca1\u6709\u4efb\u4f55\u5f3a\u5236\u7ea6\u675f\uff0c\u4e00\u6761 curl -A &#8220;GPTBot&#8221; \u7684\u8bf7\u6c42\u5c31\u80fd\u628a\u81ea\u5df1\u4f2a\u88c5\u6210\u8bad\u7ec3\u722c\u866b\uff0c\u53cd\u8fc7\u6765\u4e5f\u4e00\u6837\uff0c\u771f\u5b9e\u722c\u866b\u4e5f\u53ef\u80fd\u501f\u7528\u901a\u7528 UA\u3002\u6240\u4ee5\u4ec5\u51ed UA \u505a\u9ed1\u540d\u5355\uff0c\u65e2\u62e6\u4e0d\u51c6\u4e5f\u5bb9\u6613\u8bef\u4f24\u3002\u6bd4\u8f83\u7a33\u59a5\u7684\u505a\u6cd5\u662f\u540c\u65f6\u6838\u5bf9\u4e09\u9879\u4fe1\u606f\uff1a\u53cd\u5411 DNS \u89e3\u6790\u51fa\u7684\u57df\u540d\u3001\u6765\u6e90 IP \u662f\u5426\u843d\u5728\u5b98\u65b9\u516c\u5e03\u7684\u5730\u5740\u6bb5\u3001\u4ee5\u53ca\u8bf7\u6c42\u8282\u594f\u662f\u5426\u7b26\u5408\u722c\u866b\u7279\u5f81\u3002<\/p>\n<p>\u8bf7\u6c42\u8282\u594f\u662f\u6700\u5bb9\u6613\u89c2\u5bdf\u7684\u4fe1\u53f7\uff1a\u771f\u5b9e\u7528\u6237\u4f1a\u5728\u9875\u9762\u505c\u7559\u51e0\u79d2\uff0c\u6293\u53d6\u7a0b\u5e8f\u5219\u5e76\u53d1\u3001\u8fde\u7eed\u3001\u4e0d\u5e26 Referer \u5730\u626b\u5168\u7ad9\u3002<\/p>\n<pre><code class=\"language-bash\">awk '{print $(1)}' \/var\/log\/nginx\/access.log | sort | uniq -c | sort -rn | head -20\ngrep -c '\\&quot; 429 ' \/var\/log\/nginx\/access.log\ngrep -c '\\&quot; 403 ' \/var\/log\/nginx\/access.log\n<\/code><\/pre>\n<p>\u540c\u4e00 IP \u5728\u51e0\u5206\u949f\u5185\u4ea7\u751f\u4e0a\u5343\u6b21\u8bf7\u6c42\uff0c\u4e14\u51e0\u4e4e\u4e0d\u5e26 Referer\uff0c\u8fd9\u7c7b\u6d41\u91cf\u65e0\u8bba UA \u5199\u7684\u662f\u4ec0\u4e48\uff0c\u90fd\u5e94\u8be5\u8fdb\u5165\u9650\u901f\u540d\u5355\u3002\u53cd\u8fc7\u6765\uff0c\u5982\u679c\u67d0\u4e2a IP \u6bcf\u5929\u56fa\u5b9a\u53ea\u6293\u51e0\u5341\u4e2a\u9875\u9762\u3001\u95f4\u9694\u5747\u5300\uff0c\u591a\u534a\u662f\u641c\u7d22\u5f15\u64ce\u7684\u5e38\u89c4\u6536\u5f55\uff0c\u538b\u5f97\u592a\u72e0\u4f1a\u76f4\u63a5\u5f71\u54cd\u81ea\u7136\u6d41\u91cf\u3002<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/cn.hostease.com\/blog\/wp-content\/uploads\/2026\/10\/ai-crawler-traffic-control-robots-ua-log-analysis.webp\" alt=\"\u8bbf\u95ee\u65e5\u5fd7\u6309 UA \u4e0e\u72b6\u6001\u7801\u5206\u5c42\u7684\u7edf\u8ba1\u793a\u610f\" loading=\"lazy\" \/><\/p>\n<h2>robots.txt \u80fd\u58f0\u660e\u4ec0\u4e48\uff0c\u6321\u4e0d\u4f4f\u4ec0\u4e48<\/h2>\n<p>robots.txt \u5728 2022 \u5e74\u88ab\u6536\u5f55\u4e3a RFC 9309\uff0c\u5b9a\u4f4d\u662f\u201c\u8bf7\u6c42\u4f60\u4e0d\u8981\u6293\u201d\uff0c\u9075\u5b88\u4e0e\u5426\u7531\u722c\u866b\u65b9\u51b3\u5b9a\u3002\u5bf9\u5408\u89c4\u7684\u4e3b\u6d41\u722c\u866b\uff0c\u5b83\u4ecd\u662f\u6700\u4f4e\u6210\u672c\u3001\u89c1\u6548\u6700\u5feb\u7684\u58f0\u660e\u65b9\u5f0f\uff0c\u6309\u7528\u9014\u5206\u7ec4\u5199\u6e05\u695a\u6bd4\u4e00\u53e5 Disallow: \/ \u66f4\u6709\u610f\u4e49\uff1a<\/p>\n<pre><code>User-agent: *\nAllow: \/\nDisallow: \/search\nDisallow: \/*?s=\n\nUser-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: CCBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/\n<\/code><\/pre>\n<p>\u8fd9\u91cc\u6709\u51e0\u4e2a\u5bb9\u6613\u641e\u6df7\u7684\u70b9\u3002Google-Extended \u7ba1\u7684\u53ea\u662f Gemini \u8bad\u7ec3\u7528\u9014\uff0c\u4e0d\u5f71\u54cd Googlebot \u7684\u641c\u7d22\u6536\u5f55\uff1bGPTBot \u5bf9\u5e94 OpenAI \u7684\u8bad\u7ec3\u6293\u53d6\uff0cCCBot \u5bf9\u5e94 Common Crawl \u7684\u516c\u5f00\u8bed\u6599\uff0c\u4e24\u8005\u88ab\u5141\u8bb8\u540e\u4f1a\u8fdb\u5165\u4e0b\u6e38\u591a\u4e2a\u6570\u636e\u96c6\uff0c\u6536\u7d27\u8fd9\u4e24\u4e2a token \u7684\u6548\u679c\u901a\u5e38\u6bd4\u5c01\u6740\u5341\u51e0\u4e2a\u5c0f\u722c\u866b\u66f4\u660e\u663e\u3002<\/p>\n<p>\u53e6\u4e00\u4e2a\u8bef\u533a\u662f\u628a robots.txt \u5f53\u4f5c\u8bbf\u95ee\u63a7\u5236\uff0c\u751a\u81f3\u8ba9\u670d\u52a1\u5668\u5bf9\u5b83\u8fd4\u56de 403\uff0c\u4ee5\u4e3a\u8fd9\u6837\u722c\u866b\u5c31\u8bfb\u4e0d\u5230\u89c4\u5219\u3002\u534f\u8bae\u91cc\u6ca1\u6709\u8fd9\u5c42\u542b\u4e49\uff1a4xx \u72b6\u6001\u4f1a\u88ab\u89e3\u91ca\u4e3a\u201c\u6ca1\u6709\u4efb\u4f55\u9650\u5236\u201d\uff0c\u7b49\u4e8e\u7ed9\u51fa\u6700\u5bbd\u677e\u7684\u8bb8\u53ef\uff1b\u53ea\u6709 5xx \u624d\u88ab\u89c6\u4e3a\u201c\u6682\u65f6\u4e0d\u53ef\u7528\u201d\u3002\u771f\u8981\u62e6\u622a\uff0c\u5fc5\u987b\u843d\u5728 Web \u670d\u52a1\u5668\u6216\u5e94\u7528\u5c42\u3002<\/p>\n<h2>\u5728 Nginx \u5c42\u505a UA \u914d\u989d\uff1a429 \u6bd4 403 \u66f4\u5408\u9002<\/h2>\n<p>\u62e6\u622a\u653e\u5728 Nginx \u6700\u7701\u8d44\u6e90\uff0c\u8bf7\u6c42\u8fd8\u6ca1\u8fdb PHP \u5c31\u88ab\u6321\u4f4f\u3002\u7528 map \u628a\u8bad\u7ec3\u722c\u866b\u6807\u8bb0\u51fa\u6765\uff0c\u7ed9\u5b83\u4eec\u5355\u72ec\u7684\u9650\u901f\u6876\uff0c\u666e\u901a\u7528\u6237\u8d70\u7a7a key\u3001\u4e0d\u8ba1\u6570\uff1a<\/p>\n<pre><code class=\"language-nginx\">map $http_user_agent $train_bot {\n    default              &quot;&quot;;\n    &quot;~*GPTBot&quot;           $binary_remote_addr;\n    &quot;~*ClaudeBot&quot;        $binary_remote_addr;\n    &quot;~*CCBot&quot;            $binary_remote_addr;\n    &quot;~*Bytespider&quot;       $binary_remote_addr;\n}\n\nlimit_req_zone  $train_bot zone=trainbot:10m rate=20r\/m;\nlimit_conn_zone $train_bot zone=trainconn:10m;\n\nserver {\n    limit_req        zone=trainbot burst=10 nodelay;\n    limit_req_status 429;\n    limit_conn       trainconn 4;\n    add_header       Retry-After 600 always;\n}\n<\/code><\/pre>\n<p>rate=20r\/m \u8868\u793a\u6bcf\u4e2a IP \u6bcf\u5206\u949f\u4e8c\u5341\u4e2a\u8bf7\u6c42\uff0cburst=10 \u5141\u8bb8\u5341\u4e2a\u77ac\u65f6\u7a81\u53d1\u3002\u6574\u7ad9\u4e24\u4e07\u4e2a\u9875\u9762\u6309\u8fd9\u4e2a\u901f\u7387\u6293\u5b8c\u4e00\u8f6e\u7ea6\u9700\u5341\u4e03\u5c0f\u65f6\uff0c\u5bf9\u8bad\u7ec3\u8bed\u6599\u5f71\u54cd\u6709\u9650\uff0c\u5bf9\u7ad9\u70b9\u5219\u662f\u6570\u91cf\u7ea7\u7684\u6d41\u91cf\u4e0b\u964d\u3002\u641c\u7d22\u5f15\u64ce\u8718\u86db\u4e0d\u5728\u540d\u5355\u91cc\uff0c\u6536\u5f55\u8282\u594f\u5b8c\u5168\u4e0d\u53d7\u5f71\u54cd\u3002<\/p>\n<p>\u72b6\u6001\u7801\u8981\u9009\u5bf9\u3002\u8fd4\u56de 403 \u65f6\uff0c\u4e0d\u5c11\u722c\u866b\u4f1a\u6301\u7eed\u91cd\u8bd5\uff0c\u751a\u81f3\u8f6e\u6362 IP \u7ee7\u7eed\u6293\uff1b\u8fd4\u56de 429 \u5e76\u9644\u4e0a Retry-After\uff0c\u7b49\u4e8e\u660e\u786e\u544a\u8bc9\u5bf9\u65b9\u201c\u7a0d\u540e\u518d\u6765\u201d\uff0c\u5408\u89c4\u722c\u866b\u901a\u5e38\u4f1a\u9000\u907f\u3002\u6ce8\u610f limit_req_status \u7684\u9ed8\u8ba4\u503c\u662f 503\uff0c\u5fc5\u987b\u663e\u5f0f\u6539\u6210 429\uff0c\u5426\u5219\u65e5\u5fd7\u91cc\u4f1a\u51fa\u73b0\u5927\u91cf\u770b\u8d77\u6765\u50cf\u670d\u52a1\u6545\u969c\u7684\u8bb0\u5f55\u3002\u9650\u901f\u7684\u66f4\u591a\u5199\u6cd5\u53ef\u4ee5\u53c2\u8003\u7ad9\u5185\u7684 <a href=\"https:\/\/cn.hostease.com\/blog\/guides\/nginx-limit-req-rate-limiting\/\">Nginx \u9650\u901f\u914d\u7f6e\u5b9e\u8df5<\/a>\uff0c\u5982\u679c\u7ad9\u70b9\u524d\u9762\u8fd8\u6709\u4e00\u5c42\u53cd\u5411\u4ee3\u7406\uff0c\u9700\u8981\u786e\u8ba4\u9650\u901f\u4f7f\u7528\u7684\u662f\u771f\u5b9e\u5ba2\u6237\u7aef\u5730\u5740\u800c\u4e0d\u662f\u4ee3\u7406\u5730\u5740\uff0c\u76f8\u5173\u914d\u7f6e\u89c1 <a href=\"https:\/\/cn.hostease.com\/blog\/tutorials\/nginx-reverse-proxy-load-balancing-guide\/\">Nginx \u53cd\u5411\u4ee3\u7406\u4e0e\u8d1f\u8f7d\u5747\u8861<\/a>\u3002<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/cn.hostease.com\/blog\/wp-content\/uploads\/2026\/10\/ai-crawler-traffic-control-robots-ua-bot-throttle.webp\" alt=\"\u8bad\u7ec3\u722c\u866b\u88ab\u5355\u72ec\u7eb3\u5165\u9650\u901f\u6876\u7684\u7f51\u7edc\u8def\u5f84\u793a\u610f\" loading=\"lazy\" \/><\/p>\n<h2>\u7f13\u5b58\u4e0e\u56de\u6e90\uff1a\u628a\u91cd\u590d\u6293\u53d6\u7684\u4ee3\u4ef7\u964d\u4e0b\u6765<\/h2>\n<p>\u9650\u901f\u89e3\u51b3\u7684\u662f\u201c\u6293\u5f97\u592a\u5feb\u201d\uff0c\u7f13\u5b58\u89e3\u51b3\u7684\u662f\u201c\u6bcf\u6293\u4e00\u6b21\u90fd\u56de\u6e90\u201d\u3002\u8bad\u7ec3\u722c\u866b\u5927\u90e8\u5206\u8bf7\u6c42\u4e0d\u5e26 Cookie\u3001\u4e5f\u4e0d\u770b\u767b\u5f55\u6001\uff0c\u5b83\u4eec\u8981\u7684\u5c31\u662f\u516c\u5f00\u9875\u9762\u7684\u5b8c\u6574 HTML\uff0c\u8fd9\u7c7b\u8bf7\u6c42\u5929\u7136\u9002\u5408\u547d\u4e2d CDN\uff08\u5185\u5bb9\u5206\u53d1\u7f51\u7edc\uff0c\u628a\u9759\u6001\u5185\u5bb9\u7f13\u5b58\u5728\u79bb\u7528\u6237\u66f4\u8fd1\u7684\u8fb9\u7f18\u8282\u70b9\uff09\u6216\u670d\u52a1\u5668\u672c\u5730\u7684\u53cd\u5411\u4ee3\u7406\u7f13\u5b58\u3002<\/p>\n<p>\u5148\u7ed9\u9759\u6001\u8d44\u6e90\u4e00\u4e2a\u957f\u671f\u7f13\u5b58\uff0c\u518d\u628a HTML \u7684\u7f13\u5b58\u65f6\u95f4\u8bbe\u6210\u5341\u5206\u949f\u7ea7\u522b\uff0c\u65e2\u80fd\u5438\u6536\u6279\u91cf\u6293\u53d6\uff0c\u53c8\u4e0d\u4f1a\u8ba9\u6587\u7ae0\u66f4\u65b0\u957f\u671f\u4e0d\u751f\u6548\uff1a<\/p>\n<pre><code class=\"language-nginx\">location ~* \\.(css|js|jpg|png|webp|woff2)$ {\n    expires 180d;\n    add_header Cache-Control &quot;public, immutable&quot;;\n}\n\nlocation ~* \\.html$ {\n    expires 10m;\n    add_header Cache-Control &quot;public, max-age=600&quot;;\n}\n<\/code><\/pre>\n<p>\u7f13\u5b58\u662f\u5426\u751f\u6548\uff0c\u8981\u770b\u65e5\u5fd7\u683c\u5f0f\u91cc\u7684 $upstream_cache_status\uff0c\u7edf\u8ba1 HIT \u4e0e MISS \u6bd4\u4f8b\uff1a<\/p>\n<pre><code class=\"language-bash\">log_format  cache  '$remote_addr - $upstream_cache_status [$time_local] '\n                    '&quot;$request&quot; $status $body_bytes_sent &quot;$http_user_agent&quot;';\n\nawk '{print $(3)}' \/var\/log\/nginx\/access.log | sort | uniq -c | sort -rn\n<\/code><\/pre>\n<p>\u547d\u4e2d\u7387\u4ece\u4e09\u6210\u63d0\u5230\u516b\u6210\uff0c\u610f\u5473\u7740\u540c\u6837\u7684\u6293\u53d6\u91cf\u4e0b\u56de\u6e90\u8bf7\u6c42\u5c11\u4e86\u8fd1\u4e03\u6210\uff0c\u6570\u636e\u5e93\u548c PHP \u8fdb\u7a0b\u7684\u538b\u529b\u540c\u6b65\u4e0b\u964d\u3002\u4ec0\u4e48\u65f6\u5019\u8be5\u8ba9 CDN \u627f\u62c5\u6293\u53d6\u3001\u4ec0\u4e48\u65f6\u5019\u5fc5\u987b\u56de\u6e90\uff0c\u7ad9\u5185\u8fd9\u4e24\u7bc7\u8bb2\u5f97\u66f4\u7ec6\uff1a<a href=\"https:\/\/cn.hostease.com\/blog\/guides\/cdn-origin-role-after-heavy-media\/\">\u91cd\u5a92\u4f53\u573a\u666f\u4e0b\u7684 CDN \u4e0e\u6e90\u7ad9\u5206\u5de5<\/a>\u3001<a href=\"https:\/\/cn.hostease.com\/blog\/guides\/multi-layer-cache-architecture\/\">\u591a\u5c42\u7f13\u5b58\u67b6\u6784\u7684\u53d6\u820d<\/a>\u3002<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/cn.hostease.com\/blog\/wp-content\/uploads\/2026\/10\/ai-crawler-traffic-control-robots-ua-cache-path.webp\" alt=\"\u722c\u866b\u8bf7\u6c42\u547d\u4e2d\u8fb9\u7f18\u7f13\u5b58\u4e0e\u56de\u6e90\u7684\u4e24\u6761\u8def\u5f84\u5bf9\u6bd4\" loading=\"lazy\" \/><\/p>\n<h2>\u6cbb\u7406\u6548\u679c\u7684\u6838\u5bf9\u6307\u6807<\/h2>\n<p>\u6539\u5b8c\u914d\u7f6e\u522b\u53ea\u770b\u6d41\u91cf\u66f2\u7ebf\uff0c\u56db\u4e2a\u6570\u5b57\u624d\u80fd\u8bf4\u660e\u95ee\u9898\uff1a<\/p>\n<ul>\n<li>\u8bad\u7ec3\u722c\u866b\u7684\u5b57\u8282\u5360\u6bd4\uff1a\u6309 UA \u6c47\u603b\uff0c\u591a\u6570\u5185\u5bb9\u7ad9\u6cbb\u7406\u524d\u5728 25%\u201340% \u4e4b\u95f4\uff0c\u538b\u5230 10% \u4ee5\u4e0b\u624d\u7b97\u771f\u6b63\u89c1\u6548<\/li>\n<li>429 \u6bd4\u4f8b\uff1a\u7a33\u5b9a\u5728\u6293\u53d6\u603b\u91cf\u7684 5%\u201315% \u8bf4\u660e\u9650\u901f\u89c4\u5219\u547d\u4e2d\u4e86\uff1b\u63a5\u8fd1\u96f6\u901a\u5e38\u662f UA \u5339\u914d\u5199\u9519\uff0c\u8d85\u8fc7\u4e00\u534a\u5219\u8bf4\u660e\u6863\u4f4d\u8fc7\u4e25<\/li>\n<li>\u7f13\u5b58\u547d\u4e2d\u7387\u4e0e\u56de\u6e90\u6bd4\uff1a\u547d\u4e2d\u7387\u63d0\u5347\u540e\uff0c<a href=\"https:\/\/cn.hostease.com\/blog\/guides\/ttfb-hosting-optimization\/\">TTFB\uff08\u9996\u5b57\u8282\u65f6\u95f4\uff09\u4f18\u5316<\/a> \u91cc\u63d0\u5230\u7684 P95 \u6307\u6807\u5e94\u540c\u6b65\u4e0b\u964d<\/li>\n<li>\u641c\u7d22\u5f15\u64ce\u8718\u86db\u7684\u6210\u529f\u7387\uff1aGooglebot\u3001Bingbot \u7684 200 \u6bd4\u4f8b\u8981\u4fdd\u6301\u5728 99% \u4ee5\u4e0a\uff0c\u4f4e\u4e8e\u8fd9\u4e2a\u6570\u5c31\u662f\u8bef\u4f24<\/li>\n<\/ul>\n<h2>\u56db\u4e2a\u5bb9\u6613\u8e29\u7a7a\u7684\u7ec6\u8282<\/h2>\n<ul>\n<li>\u53cd\u5411 DNS \u6821\u9a8c\u4e0d\u8981\u653e\u5728\u6bcf\u4e2a\u8bf7\u6c42\u4e0a\uff1a\u89e3\u6790\u4e00\u6b21\u8981\u51e0\u6beb\u79d2\uff0c\u53ea\u5728\u9650\u901f\u89e6\u53d1\u540e\u8bb0\u5f55\u5e76\u6838\u5b9e<\/li>\n<li>\u9650\u901f\u952e\u8981\u533a\u5206\u534f\u8bae\u7248\u672c\uff1a$binary_remote_addr \u5bf9 IPv4 \u662f 4 \u5b57\u8282\u3001IPv6 \u662f 16 \u5b57\u8282\uff0c10m \u5171\u4eab\u5185\u5b58\u6309\u5341\u4e07\u7ea7\u5730\u5740\u89c4\u5212\u8db3\u591f<\/li>\n<li>\u65e5\u5fd7\u4e5f\u8981\u8f6e\u8f6c\uff1a7 \u5929\u539f\u59cb\u65e5\u5fd7\u8db3\u591f\u5b9a\u4f4d\u95ee\u9898\uff0c\u672a\u538b\u7f29\u7684 access.log \u5355\u65e5\u5c31\u80fd\u6da8\u5230\u51e0\u767e MB<\/li>\n<li>\u7edf\u8ba1\u53e3\u5f84\u8981\u8986\u76d6\u6240\u6709\u57df\u540d\uff1a\u56fe\u7247\u8d70\u72ec\u7acb\u57df\u540d\u65f6\uff0c\u8fd9\u90e8\u5206\u6d41\u91cf\u4e0d\u4f1a\u51fa\u73b0\u5728\u4e3b\u7ad9\u65e5\u5fd7\u91cc\uff1b\u8de8\u533a\u57df\u8bbf\u95ee\u8fd8\u9700\u8981\u770b DNS\uff08\u57df\u540d\u89e3\u6790\u7cfb\u7edf\uff0c\u628a\u57df\u540d\u7ffb\u8bd1\u6210 IP \u5730\u5740\u7684\u670d\u52a1\uff09\u89e3\u6790\u8bb0\u5f55\uff0c\u76f8\u5173\u505a\u6cd5\u89c1 <a href=\"https:\/\/cn.hostease.com\/blog\/guides\/cross-border-dns-optimization-authoritative-ttl-multi-line\/\">\u8de8\u5883 DNS \u4f18\u5316<\/a><\/li>\n<\/ul>\n<p>\u5982\u679c\u4f60\u7684\u7ad9\u70b9\u8dd1\u5728\u72ec\u670d\uff08\u72ec\u7acb\u7269\u7406\u670d\u52a1\u5668\uff09\u4e0a\uff0c\u9650\u901f\u548c\u7f13\u5b58\u90fd\u7531\u81ea\u5df1\u638c\u63a7\uff0c\u6539\u5b8c\u914d\u7f6e reload \u5c31\u80fd\u9a8c\u8bc1\uff1b\u5982\u679c\u662f\u4e91\u670d\u52a1\u5668\uff08\u53ef\u5f39\u6027\u4f38\u7f29\u7684\u865a\u62df\u673a\u5b9e\u4f8b\uff09\uff0c\u8bb0\u5f97\u628a\u5e26\u5bbd\u544a\u8b66\u548c\u6293\u53d6\u91cf\u7edf\u8ba1\u4e00\u8d77\u770b\uff0c\u7a81\u53d1\u6d41\u91cf\u7684\u8ba1\u8d39\u65b9\u5f0f\u5f80\u5f80\u548c\u56fa\u5b9a\u5e26\u5bbd\u4e0d\u540c\u3002\u5728 Hostease \u7684 <a href=\"https:\/\/cn.hostease.com\/vps\/\">VPS<\/a>\uff08<a href=\"https:\/\/cn.hostease.com\/vps\/\">\u865a\u62df\u4e13\u7528\u670d\u52a1\u5668<\/a>\uff09\u4e0a\uff0cNginx \u7684\u7ad9\u70b9\u914d\u7f6e\u7531 root \u7edf\u4e00\u7ba1\u7406\uff0c\u4e0a\u9762\u8fd9\u5957 map\u3001limit_req \u548c\u7f13\u5b58\u89c4\u5219\u53ef\u4ee5\u76f4\u63a5\u5199\u8fdb \/etc\/nginx\/conf.d \u4e0b\u7684\u7ad9\u70b9\u6587\u4ef6\uff0creload \u4e4b\u540e\u7acb\u523b\u751f\u6548\u3002<\/p>\n<h2>\u603b\u7ed3\uff1a\u628a\u6293\u53d6\u6cbb\u7406\u53d8\u6210\u6bcf\u6708\u4e00\u6b21\u7684\u4f8b\u884c\u52a8\u4f5c<\/h2>\n<p>\u6293\u53d6\u91cf\u6cbb\u7406\u4e0d\u662f\u4e00\u6b21\u6027\u4efb\u52a1\uff1a\u6a21\u578b\u5728\u66f4\u65b0\uff0c\u722c\u866b\u5728\u6362 IP\uff0cUA \u5b57\u7b26\u4e32\u968f\u65f6\u4f1a\u53d8\uff0c\u4eca\u5929\u6709\u6548\u7684\u9ed1\u540d\u5355\u4e09\u4e2a\u6708\u540e\u5927\u6982\u7387\u6f0f\u6389\u4e00\u534a\u6d41\u91cf\u3002\u7701\u4e8b\u7684\u505a\u6cd5\u662f\u628a\u4e09\u4ef6\u4e8b\u56fa\u5316\u6210\u6d41\u7a0b\uff0c\u6bcf\u6708\u8dd1\u4e00\u6b21\u6309 UA \u7684\u5b57\u8282\u6c47\u603b\uff0c\u6bcf\u5b63\u5ea6\u590d\u6838 robots.txt \u8865\u9f50\u65b0\u51fa\u73b0\u7684\u8bad\u7ec3\u722c\u866b token\uff0c\u6bcf\u6b21\u8c03\u6574\u6863\u4f4d\u540e\u7528 429 \u6bd4\u4f8b\u548c\u641c\u7d22\u5f15\u64ce\u6210\u529f\u7387\u4e92\u76f8\u6821\u9a8c\u3002<\/p>\n<p>\u5982\u679c\u4f60\u9700\u8981\u4e00\u4e2a\u660e\u786e\u7684\u8d77\u70b9\uff0c\u5efa\u8bae\u5148\u53ea\u505a\u4e24\u4ef6\u4e8b\uff1a\u6309 UA \u6c47\u603b\u4e00\u6b21\u8bbf\u95ee\u65e5\u5fd7\uff0c\u7b97\u51fa\u8bad\u7ec3\u722c\u866b\u7684\u5360\u6bd4\uff1b\u518d\u7ed9\u8fd9\u51e0\u4e2a UA \u52a0\u4e0a 20r\/m \u7684\u9650\u901f\u6876\u3002\u8fd9\u4e24\u6b65\u4e0d\u52a8\u67b6\u6784\u3001\u56de\u6eda\u6210\u672c\u51e0\u4e4e\u4e3a\u96f6\uff0c\u5374\u80fd\u628a\u6d41\u91cf\u7ed3\u6784\u770b\u6e05\u695a\uff0c\u4e5f\u4e3a\u540e\u7eed\u8981\u4e0d\u8981\u4e0a CDN\u3001\u8981\u4e0d\u8981\u8c03\u6574\u4e3b\u673a\u89c4\u683c\u63d0\u4f9b\u4f9d\u636e\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u4ece\u8bbf\u95ee\u65e5\u5fd7\u7edf\u8ba1 AI \u8bad\u7ec3\u722c\u866b\u7684\u6293\u53d6\u91cf\u4e0e\u6d41\u91cf\u5360\u6bd4\uff0c\u518d\u7528 robots.txt \u58f0\u660e\u3001Nginx UA \u9650\u901f\u548c\u7f13\u5b58\u7b56\u7565\u628a\u5e26\u5bbd\u538b\u56de\u53ef\u63a7\u8303\u56f4\uff0c\u9644\u53ef\u6267\u884c\u547d\u4ee4\u4e0e\u6838\u5bf9\u6307\u6807\u3002<\/p>\n","protected":false},"author":2,"featured_media":34260,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4099],"tags":[4017,393,5480,5479,4974],"class_list":["post-34264","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tutorials","tag-ai","tag-nginx","tag-robotstxt","tag-5479","tag-4974"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/cn.hostease.com\/blog\/wp-content\/uploads\/2026\/10\/ai-crawler-traffic-control-robots-ua-cover.webp","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/posts\/34264","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/comments?post=34264"}],"version-history":[{"count":1,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/posts\/34264\/revisions"}],"predecessor-version":[{"id":34266,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/posts\/34264\/revisions\/34266"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/media\/34260"}],"wp:attachment":[{"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/media?parent=34264"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/categories?post=34264"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cn.hostease.com\/blog\/wp-json\/wp\/v2\/tags?post=34264"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}