{"id":490256,"date":"2023-09-19T14:41:37","date_gmt":"2023-09-19T14:41:37","guid":{"rendered":"https:\/\/oneproxy.pro\/?post_type=docs&#038;p=490256"},"modified":"2023-09-26T10:28:44","modified_gmt":"2023-09-26T10:28:44","slug":"proxies-for-web-scraping","status":"publish","type":"docs","link":"https:\/\/oneproxy.pro\/cn\/docs\/proxies-for-web-scraping\/","title":{"rendered":"\u5982\u4f55\u4f7f\u7528\u4ee3\u7406\u8fdb\u884c\u7f51\u9875\u6293\u53d6\uff1f"},"content":{"rendered":"<p>\u7f51\u7edc\u6293\u53d6\u5df2\u53d1\u5c55\u6210\u4e3a\u5404\u79cd\u4e1a\u52a1\u5e94\u7528\u7a0b\u5e8f\u7684\u5173\u952e\u5de5\u5177\uff0c\u5305\u62ec\u4f46\u4e0d\u9650\u4e8e\u6570\u636e\u5206\u6790\u3001\u673a\u5668\u5b66\u4e60\u7b97\u6cd5\u548c\u6f5c\u5728\u5ba2\u6237\u83b7\u53d6\u3002\u5c3d\u7ba1\u5b83\u5f88\u6709\u4ef7\u503c\uff0c\u4f46\u4e00\u81f4\u7684\u5927\u89c4\u6a21\u6570\u636e\u68c0\u7d22\u63d0\u51fa\u4e86\u8bb8\u591a\u6311\u6218\u3002\u5176\u4e2d\u5305\u62ec\u7f51\u7ad9\u6240\u6709\u8005\u7684\u5bf9\u7b56\uff0c\u4f8b\u5982 IP \u7981\u4ee4\u3001\u9a8c\u8bc1\u7801\u548c\u871c\u7f50\u3002\u4ee3\u7406\u4e3a\u8fd9\u4e9b\u95ee\u9898\u63d0\u4f9b\u4e86\u5f3a\u5927\u7684\u89e3\u51b3\u65b9\u6848\u3002\u5728\u672c\u6307\u5357\u4e2d\uff0c\u6211\u4eec\u6df1\u5165\u63a2\u8ba8\u4ec0\u4e48\u662f\u7f51\u7edc\u6293\u53d6\u548c\u4ee3\u7406\u670d\u52a1\u5668\u3001\u5b83\u4eec\u5728\u7f51\u7edc\u6293\u53d6\u4e2d\u7684\u4f5c\u7528\u3001\u5404\u79cd\u4ee3\u7406\u7c7b\u578b\u4ee5\u53ca\u5982\u4f55\u6709\u6548\u5730\u6d4b\u8bd5\u5b83\u4eec\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u7f51\u9875\u6293\u53d6\u7684\u590d\u6742\u6027<\/h2>\n\n\n\n<p>\u7f51\u7edc\u6293\u53d6\u662f\u4e00\u79cd\u4ee5\u7f16\u7a0b\u65b9\u5f0f\u4ece\u5728\u7ebf\u8d44\u6e90\u4e2d\u63d0\u53d6\u4fe1\u606f\u7684\u6280\u672f\u3002\u8fd9\u901a\u5e38\u6d89\u53ca HTTP \u8bf7\u6c42\u6216\u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u6765\u4ece\u591a\u4e2a\u7f51\u9875\u722c\u884c\u548c\u68c0\u7d22\u6570\u636e\u3002\u6570\u636e\u901a\u5e38\u4ee5\u7ed3\u6784\u5316\u5f62\u5f0f\u5b58\u50a8\uff0c\u4f8b\u5982\u7535\u5b50\u8868\u683c\u6216\u6570\u636e\u5e93\u3002<\/p>\n\n\n\n<p>\u8fd9\u662f\u4f7f\u7528 Python \u6293\u53d6\u6570\u636e\u7684\u7b80\u5355\u4ee3\u7801\u7247\u6bb5 <code data-no-translation=\"\">requests<\/code> \u56fe\u4e66\u9986\uff1a<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>Python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\" data-no-translation=\"\"><span class=\"hljs-keyword\">import<\/span> requests\n\nresponse = requests.get(<span class=\"hljs-string\">\"http:\/\/example.com\/data\"<\/span>)\ndata = response.text  <span class=\"hljs-comment\"># This would contain the HTML content of the page<\/span>\n<\/code><\/div><\/div><\/pre>\n\n\n\n<p>\u81ea\u52a8\u6293\u53d6\u7cfb\u7edf\u53ef\u6839\u636e\u7528\u6237\u5b9a\u4e49\u7684\u53c2\u6570\u5feb\u901f\u6536\u96c6\u6570\u636e\uff0c\u4ece\u800c\u63d0\u4f9b\u7ade\u4e89\u4f18\u52bf\u3002\u7136\u800c\uff0c\u7f51\u7ad9\u7684\u591a\u6837\u6027\u9700\u8981\u5e7f\u6cdb\u7684\u6280\u80fd\u548c\u5de5\u5177\u6765\u8fdb\u884c\u6709\u6548\u7684\u7f51\u7edc\u6293\u53d6\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u7f51\u9875\u6293\u53d6\u4e2d\u8bc4\u4f30\u4ee3\u7406\u7684\u6807\u51c6<\/h2>\n\n\n\n<p>\u5728\u8bc4\u4f30\u7f51\u7edc\u6293\u53d6\u4efb\u52a1\u7684\u4ee3\u7406\u65f6\uff0c\u8bf7\u5173\u6ce8\u4e09\u4e2a\u4e3b\u8981\u6807\u51c6\uff1a\u901f\u5ea6\u3001\u53ef\u9760\u6027\u548c\u5b89\u5168\u6027\u3002<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>\u6807\u51c6<\/th><th>\u91cd\u8981\u6027<\/th><th>\u6d4b\u8bd5\u5de5\u5177<\/th><\/tr><\/thead><tbody><tr><td>\u901f\u5ea6<\/td><td>\u5ef6\u8fdf\u548c\u8d85\u65f6\u4f1a\u4e25\u91cd\u5f71\u54cd\u6293\u53d6\u4efb\u52a1\u3002<\/td><td>cURL\uff0cfast.com<\/td><\/tr><tr><td>\u53ef\u9760\u6027<\/td><td>\u4e00\u81f4\u7684\u6b63\u5e38\u8fd0\u884c\u65f6\u95f4\u5bf9\u4e8e\u786e\u4fdd\u4e0d\u95f4\u65ad\u7684\u6570\u636e\u6536\u96c6\u81f3\u5173\u91cd\u8981\u3002<\/td><td>\u5185\u90e8\u6b63\u5e38\u8fd0\u884c\u65f6\u95f4\u62a5\u544a\u3001\u7b2c\u4e09\u65b9\u76d1\u63a7\u5de5\u5177<\/td><\/tr><tr><td>\u5b89\u5168<\/td><td>\u654f\u611f\u6570\u636e\u5e94\u52a0\u5bc6\u4e14\u4fdd\u5bc6\u3002<\/td><td>SSL \u5b9e\u9a8c\u5ba4\u3001Qualys SSL \u5b9e\u9a8c\u5ba4<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">\u901f\u5ea6<\/h3>\n\n\n\n<p>\u4f7f\u7528\u901f\u5ea6\u6162\u7684\u4ee3\u7406\u53ef\u80fd\u4f1a\u56e0\u5ef6\u8fdf\u548c\u8d85\u65f6\u800c\u4f7f\u60a8\u7684\u7f51\u7edc\u6293\u53d6\u9762\u4e34\u98ce\u9669\u3002\u4e3a\u4e86\u786e\u4fdd\u6700\u4f73\u6027\u80fd\uff0c\u8bf7\u8003\u8651\u4f7f\u7528 cURL \u6216 fast.com \u7b49\u5de5\u5177\u8fdb\u884c\u5b9e\u65f6\u901f\u5ea6\u6d4b\u8bd5\u3002<\/p>\n\n\n\n<p>\u5f53\u7136\uff0c\u4e86\u89e3\u5982\u4f55\u6d4b\u91cf\u4ee3\u7406\u670d\u52a1\u5668\u7684\u901f\u5ea6\u548c\u6027\u80fd\u5bf9\u4e8e\u786e\u4fdd\u60a8\u7684\u7f51\u7edc\u6293\u53d6\u4efb\u52a1\u9ad8\u6548\u53ef\u9760\u81f3\u5173\u91cd\u8981\u3002\u4ee5\u4e0b\u662f\u4f7f\u7528 cURL \u548c fast.com \u6d4b\u91cf\u4ee3\u7406\u670d\u52a1\u5668\u7684\u52a0\u8f7d\u65f6\u95f4\u548c\u6027\u80fd\u5206\u6570\u7684\u6307\u5357\u3002<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">\u4f7f\u7528 cURL \u6d4b\u91cf\u4ee3\u7406\u901f\u5ea6<\/h4>\n\n\n\n<p>cURL \u662f\u4e00\u4e2a\u547d\u4ee4\u884c\u5de5\u5177\uff0c\u7528\u4e8e\u4f7f\u7528\u5404\u79cd\u7f51\u7edc\u534f\u8bae\u4f20\u8f93\u6570\u636e\u3002\u5b83\u5bf9\u4e8e\u901a\u8fc7\u6d4b\u91cf\u4e0b\u8f7d\u7f51\u9875\u6240\u9700\u7684\u65f6\u95f4\u6765\u6d4b\u8bd5\u4ee3\u7406\u670d\u52a1\u5668\u7684\u901f\u5ea6\u975e\u5e38\u6709\u7528\u3002<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><p><strong>\u901a\u8fc7\u4ee3\u7406\u7684 cURL \u8bf7\u6c42\u7684\u57fa\u672c\u8bed\u6cd5\uff1a<\/strong><\/p><pre><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>\u5df4\u4ec0<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-bash\" data-no-translation=\"\">curl -x http:\/\/your.proxy.server:port <span class=\"hljs-string\">\"http:\/\/target.website.com\"<\/span>\n<\/code><\/div><\/div><\/pre><\/li>\n\n\n\n<li><p><strong>\u4f7f\u7528 cURL \u6d4b\u91cf\u65f6\u95f4\uff1a<\/strong>\n\u60a8\u53ef\u4ee5\u4f7f\u7528 <code data-no-translation=\"\">-o<\/code> \u6807\u5fd7\u4e22\u5f03\u8f93\u51fa\u548c <code data-no-translation=\"\">-w<\/code> \u6253\u5370\u65f6\u95f4\u8be6\u7ec6\u4fe1\u606f\u7684\u6807\u5fd7\u5982\u4e0b\uff1a<\/p><pre><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>\u5df4\u4ec0<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-bash\" data-no-translation=\"\">curl -x http:\/\/your.proxy.server:port <span class=\"hljs-string\">\"http:\/\/target.website.com\"<\/span> -o \/dev\/null -w <span class=\"hljs-string\">\"Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total}\\n\"<\/span>\n<\/code><\/div><\/div><\/pre><p>\u8fd9\u5c06\u4e3a\u60a8\u63d0\u4f9b\u4ee5\u4e0b\u6307\u6807\uff1a<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>\u8fde\u63a5\uff1a<\/strong> \u5efa\u7acb\u4e0e\u670d\u52a1\u5668\u7684 TCP \u8fde\u63a5\u6240\u82b1\u8d39\u7684\u65f6\u95f4\u3002<\/li>\n\n\n\n<li><strong>TTFB\uff08\u7b2c\u4e00\u4e2a\u5b57\u8282\u7684\u65f6\u95f4\uff09\uff1a<\/strong> \u8fde\u63a5\u5efa\u7acb\u540e\u63a5\u6536\u7b2c\u4e00\u4e2a\u5b57\u8282\u6240\u82b1\u8d39\u7684\u65f6\u95f4\u3002<\/li>\n\n\n\n<li><strong>\u603b\u65f6\u95f4\uff1a<\/strong> \u64cd\u4f5c\u6240\u82b1\u8d39\u7684\u603b\u65f6\u95f4\u3002<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><p><strong>\u4e86\u89e3\u7ed3\u679c\uff1a<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>\u65f6\u95f4\u8d8a\u77ed\u901a\u5e38\u610f\u5473\u7740\u4ee3\u7406\u901f\u5ea6\u8d8a\u5feb\u3002<\/li>\n\n\n\n<li>\u5f02\u5e38\u9ad8\u7684\u65f6\u95f4\u53ef\u80fd\u610f\u5473\u7740\u4ee3\u7406\u4e0d\u53ef\u9760\u6216\u62e5\u585e\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\">\u4f7f\u7528 Fast.com \u6d4b\u91cf\u4ee3\u7406\u901f\u5ea6<\/h4>\n\n\n\n<p>Fast.com \u662f\u4e00\u4e2a\u57fa\u4e8e\u7f51\u7edc\u7684\u5de5\u5177\uff0c\u7528\u4e8e\u6d4b\u91cf\u60a8\u7684\u4e92\u8054\u7f51\u901f\u5ea6\u3002\u867d\u7136\u5b83\u4e0d\u76f4\u63a5\u6d4b\u91cf\u4ee3\u7406\u7684\u901f\u5ea6\uff0c\u4f46\u60a8\u53ef\u4ee5\u624b\u52a8\u4f7f\u7528\u5b83\u6765\u68c0\u67e5\u8fde\u63a5\u5230\u4ee3\u7406\u670d\u52a1\u5668\u65f6\u7684\u901f\u5ea6\u3002<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><p><strong>\u624b\u52a8\u6d4b\u8bd5\uff1a<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>\u5c06\u60a8\u7684\u7cfb\u7edf\u8bbe\u7f6e\u4e3a\u4f7f\u7528\u4ee3\u7406\u670d\u52a1\u5668\u3002<\/li>\n\n\n\n<li>\u6253\u5f00\u7f51\u7edc\u6d4f\u89c8\u5668\u5e76\u8f6c\u5230 <a href=\"https:\/\/fast.com\/\" target=\"_new\" rel=\"noopener nofollow\">\u5feb\u7f51<\/a>.<\/li>\n\n\n\n<li>\u5355\u51fb\u201c\u5f00\u59cb\u201d\u5f00\u59cb\u901f\u5ea6\u6d4b\u8bd5\u3002<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><p><strong>\u4e86\u89e3\u7ed3\u679c\uff1a<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>\u8f83\u9ad8\u7684 Mbps \u5206\u6570\u610f\u5473\u7740\u66f4\u5feb\u7684\u4e92\u8054\u7f51\u901f\u5ea6\uff0c\u4ece\u800c\u8868\u660e\u4ee3\u7406\u901f\u5ea6\u66f4\u5feb\u3002<\/li>\n\n\n\n<li>\u8f83\u4f4e\u7684 Mbps \u5206\u6570\u53ef\u80fd\u610f\u5473\u7740\u4ee3\u7406\u901f\u5ea6\u8f83\u6162\u6216\u6d41\u91cf\u8f83\u9ad8\u3002<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><p><strong>\u81ea\u52a8\u5316\u6d4b\u8bd5\uff1a<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li>Fast.com \u6709\u4e00\u4e2a\u53ef\u7528\u4e8e\u81ea\u52a8\u5316\u6d4b\u8bd5\u7684 API\uff0c\u4f46\u5b83\u53ef\u80fd\u65e0\u6cd5\u76f4\u63a5\u901a\u8fc7\u4ee3\u7406\u5de5\u4f5c\u3002\u4e3a\u6b64\uff0c\u60a8\u9700\u8981\u989d\u5916\u7684\u7f16\u7a0b\u6765\u901a\u8fc7\u4ee3\u7406\u8def\u7531 Fast.com API \u8bf7\u6c42\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\">\u6c47\u603b\u8868<\/h4>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>\u65b9\u6cd5<\/th><th>\u6307\u6807<\/th><th>\u53ef\u81ea\u52a8\u5316<\/th><th>\u76f4\u63a5\u4ee3\u7406\u6d4b\u91cf<\/th><\/tr><\/thead><tbody><tr><td>\u5377\u66f2<\/td><td>TTFB\u3001\u8fde\u63a5\u65f6\u95f4\u3001\u603b\u65f6\u95f4<\/td><td>\u662f\u7684<\/td><td>\u662f\u7684<\/td><\/tr><tr><td>\u5feb\u7f51<\/td><td>\u4e92\u8054\u7f51\u901f\u5ea6\uff08\u4ee5 Mbps \u4e3a\u5355\u4f4d\uff09<\/td><td>\u53ef\u4ee5\u901a\u8fc7\u9644\u52a0\u7f16\u7801\u5b9e\u73b0<\/td><td>\u4e0d<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>\u901a\u8fc7\u5229\u7528 cURL \u548c fast.com \u7b49\u5de5\u5177\uff0c\u60a8\u53ef\u4ee5\u5168\u9762\u6d4b\u91cf\u4ee3\u7406\u670d\u52a1\u5668\u7684\u6027\u80fd\uff0c\u4ece\u800c\u5728\u8bbe\u7f6e\u7f51\u9875\u6293\u53d6\u67b6\u6784\u65f6\u505a\u51fa\u660e\u667a\u7684\u51b3\u5b9a\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u53ef\u9760\u6027<\/h3>\n\n\n\n<p>\u9009\u62e9\u4ee5\u5176\u6b63\u5e38\u8fd0\u884c\u65f6\u95f4\u548c\u53ef\u9760\u6027\u800c\u95fb\u540d\u7684\u4ee3\u7406\u3002\u4e00\u81f4\u7684\u64cd\u4f5c\u53ef\u786e\u4fdd\u60a8\u7684\u7f51\u7edc\u6293\u53d6\u5de5\u4f5c\u4e0d\u53d7\u963b\u788d\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u5b89\u5168<\/h3>\n\n\n\n<p>\u9009\u62e9\u4e00\u4e2a\u52a0\u5bc6\u60a8\u7684\u6570\u636e\u7684\u5b89\u5168\u4ee3\u7406\u3002\u4f7f\u7528 SSL Labs \u6216 Qualys SSL Labs \u8bc4\u4f30 SSL \u8bc1\u4e66\u5e76\u83b7\u5f97\u5b89\u5168\u8bc4\u7ea7\u3002<\/p>\n\n\n\n<p>\u6301\u7eed\u76d1\u63a7\u5bf9\u4e8e\u786e\u4fdd\u60a8\u9009\u62e9\u7684\u4ee3\u7406\u968f\u7740\u65f6\u95f4\u7684\u63a8\u79fb\u59cb\u7ec8\u7b26\u5408\u60a8\u6240\u9700\u7684\u6807\u51c6\u81f3\u5173\u91cd\u8981\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u8ba1\u7b97\u6240\u9700\u7684\u4ee3\u7406\u6570\u91cf<\/h2>\n\n\n\n<p>\u6240\u9700\u4ee3\u7406\u6570\u91cf\u7684\u8ba1\u7b97\u516c\u5f0f\u4e3a\uff1a<\/p>\n\n\n\n<math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mtext>\u4ee3\u7406\u6570\u91cf<\/mtext><mo>=<\/mo><mfrac><mtext>\u6bcf\u79d2\u8bf7\u6c42\u6570<\/mtext><mtext>\u6bcf\u4e2a\u4ee3\u7406\u6bcf\u79d2\u7684\u8bf7\u6c42\u6570<\/mtext><\/mfrac><\/mrow><annotation encoding=\"application\/x-tex\">\\text{\u4ee3\u7406\u6570\u91cf} = \\frac{\\text{\u6bcf\u79d2\u8bf7\u6c42\u6570}}{\\text{\u6bcf\u4e2a\u4ee3\u7406\u6bcf\u79d2\u8bf7\u6c42\u6570}}<\/annotation><\/semantics><\/math>\n\n\n\n<p><\/p>\n\n\n\n<p>\u4f8b\u5982\uff0c\u5982\u679c\u60a8\u6bcf\u79d2\u9700\u8981 100 \u4e2a\u8bf7\u6c42\uff0c\u5e76\u4e14\u6bcf\u4e2a\u4ee3\u7406\u53ef\u4ee5\u5bb9\u7eb3 10 \u4e2a\u8bf7\u6c42\uff0c\u5219\u60a8\u5c06\u9700\u8981 10 \u4e2a\u4ee3\u7406\u3002\u6293\u53d6\u76ee\u6807\u9875\u9762\u7684\u9891\u7387\u7531\u591a\u79cd\u56e0\u7d20\u51b3\u5b9a\uff0c\u5305\u62ec\u8bf7\u6c42\u9650\u5236\u3001\u7528\u6237\u6570\u91cf\u548c\u76ee\u6807\u7f51\u7ad9\u7684\u5bb9\u5fcd\u65f6\u95f4\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u7528\u4e8e\u4ee3\u7406\u6d4b\u8bd5\u548c\u7f51\u9875\u6293\u53d6\u7684\u5de5\u5177<\/h2>\n\n\n\n<p>\u5404\u79cd\u8f6f\u4ef6\u548c\u5e93\u53ef\u4ee5\u534f\u52a9\u4ee3\u7406\u8bc4\u4f30\u548c\u7f51\u7edc\u6293\u53d6\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u522e\u75e7<\/strong>\uff1a\u57fa\u4e8e Python \u7684\u7f51\u9875\u6293\u53d6\u6846\u67b6\uff0c\u5177\u6709\u5185\u7f6e\u4ee3\u7406\u7ba1\u7406\u3002<\/li>\n\n\n\n<li><strong>\u7852<\/strong>\uff1a\u4e00\u79cd\u81ea\u52a8\u5316\u6d4f\u89c8\u5668\u4ea4\u4e92\u7684\u5de5\u5177\uff0c\u5bf9\u4e8e\u6293\u53d6\u548c\u4ee3\u7406\u6d4b\u8bd5\u975e\u5e38\u6709\u7528\u3002<\/li>\n\n\n\n<li><strong>\u67e5\u5c14\u65af\u00b7\u666e\u7f57\u514b\u65af<\/strong>\uff1a\u7528\u4e8e\u8c03\u8bd5\u548c\u76d1\u63a7\u5ba2\u6237\u7aef\u548c\u670d\u52a1\u5668\u4e4b\u95f4\u7684 HTTP \u6d41\u91cf\u3002<\/li>\n\n\n\n<li><strong>\u7f8e\u4e3d\u7684\u6c64<\/strong>\uff1a\u7528\u4e8e\u89e3\u6790 HTML \u548c XML \u6587\u6863\u7684 Python \u5e93\uff0c\u901a\u5e38\u4e0e\u5176\u4ed6\u6293\u53d6\u5de5\u5177\u7ed3\u5408\u4f7f\u7528\u3002<\/li>\n<\/ul>\n\n\n\n<p>\u5f53\u7136\uff0c\u63d0\u4f9b\u4ee3\u7801\u793a\u4f8b\u5c06\u4f7f\u60a8\u66f4\u5b9e\u9645\u5730\u4e86\u89e3\u5982\u4f55\u5c06\u8fd9\u4e9b\u5de5\u5177\u5e94\u7528\u4e8e\u7f51\u7edc\u6293\u53d6\u9879\u76ee\u3002\u4ee5\u4e0b\u662f\u6bcf\u4e2a\u7684\u4ee3\u7801\u7247\u6bb5\uff1a<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scrapy\uff1a\u4ee3\u7406\u7ba1\u7406\u548c\u7f51\u9875\u6293\u53d6<\/h3>\n\n\n\n<p>Scrapy \u662f\u4e00\u4e2a Python \u6846\u67b6\uff0c\u53ef\u7b80\u5316\u7f51\u9875\u6293\u53d6\u4efb\u52a1\u5e76\u63d0\u4f9b\u5185\u7f6e\u4ee3\u7406\u7ba1\u7406\u529f\u80fd\u3002\u4e0b\u9762\u662f\u4e00\u4e2a\u793a\u4f8b\u4ee3\u7801\u7247\u6bb5\uff0c\u6f14\u793a\u4e86\u5982\u4f55\u5728 Scrapy \u4e2d\u8bbe\u7f6e\u4ee3\u7406\u3002<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>Python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\" data-no-translation=\"\"><span class=\"hljs-keyword\">import<\/span> scrapy\n\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title class_\">MySpider<\/span>(scrapy.Spider):\n    name = <span class=\"hljs-string\">'myspider'<\/span>\n    \n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title function_\">start_requests<\/span>(<span class=\"hljs-params\">self<\/span>):\n        url = <span class=\"hljs-string\">'http:\/\/example.com\/data'<\/span>\n        <span class=\"hljs-keyword\">yield<\/span> scrapy.Request(url, self.parse, meta={<span class=\"hljs-string\">'proxy'<\/span>: <span class=\"hljs-string\">'http:\/\/your.proxy.address:8080'<\/span>})\n        \n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title function_\">parse<\/span>(<span class=\"hljs-params\">self, response<\/span>):\n        <span class=\"hljs-comment\"># Your parsing logic here<\/span>\n<\/code><\/div><\/div><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Selenium\uff1a\u7f51\u9875\u6293\u53d6\u548c\u4ee3\u7406\u914d\u7f6e<\/h3>\n\n\n\n<p>Selenium \u5728\u6d4f\u89c8\u5668\u81ea\u52a8\u5316\u65b9\u9762\u5f88\u53d7\u6b22\u8fce\uff0c\u5728\u6293\u53d6\u9700\u8981\u4ea4\u4e92\u6216\u5177\u6709 AJAX \u52a0\u8f7d\u5185\u5bb9\u7684\u7f51\u7ad9\u65f6\u7279\u522b\u6709\u7528\u3002\u60a8\u8fd8\u53ef\u4ee5\u5728 Selenium \u4e2d\u8bbe\u7f6e\u4ee3\u7406\uff0c\u5982\u4e0b\u6240\u793a\uff1a<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>Python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\" data-no-translation=\"\"><span class=\"hljs-keyword\">from<\/span> selenium <span class=\"hljs-keyword\">import<\/span> webdriver\n\nPROXY = <span class=\"hljs-string\">'your.proxy.address:8080'<\/span>\nchrome_options = webdriver.ChromeOptions()\nchrome_options.add_argument(<span class=\"hljs-string\">f'--proxy-server=<span class=\"hljs-subst\">{PROXY}<\/span>'<\/span>)\n\ndriver = webdriver.Chrome(options=chrome_options)\ndriver.get(<span class=\"hljs-string\">'http:\/\/example.com\/data'<\/span>)\n\n<span class=\"hljs-comment\"># Your scraping logic here<\/span>\n<\/code><\/div><\/div><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Charles Proxy\uff1aHTTP \u76d1\u63a7\uff08\u6ce8\u610f\uff1a\u4e0d\u662f\u57fa\u4e8e\u4ee3\u7801\u7684\u5de5\u5177\uff09<\/h3>\n\n\n\n<p>Charles Proxy \u4e0d\u80fd\u901a\u8fc7\u4ee3\u7801\u8fdb\u884c\u7f16\u7a0b\uff0c\u56e0\u4e3a\u5b83\u662f\u4e00\u4e2a\u7528\u4e8e\u8c03\u8bd5\u5ba2\u6237\u7aef\u548c\u670d\u52a1\u5668\u4e4b\u95f4\u7684 HTTP \u6d41\u91cf\u7684\u5e94\u7528\u7a0b\u5e8f\u3002\u60a8\u53ef\u4ee5\u5728\u8ba1\u7b97\u673a\u4e0a\u8fdb\u884c\u8bbe\u7f6e\u5e76\u914d\u7f6e\u7cfb\u7edf\u8bbe\u7f6e\u4ee5\u901a\u8fc7 Charles \u8def\u7531\u6d41\u91cf\u3002\u8fd9\u5c06\u5141\u8bb8\u60a8\u76d1\u89c6\u3001\u62e6\u622a\u548c\u4fee\u6539\u8bf7\u6c42\u548c\u54cd\u5e94\u4ee5\u8fdb\u884c\u8c03\u8bd5\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Beautiful Soup\uff1a\u4f7f\u7528 Python \u8fdb\u884c HTML \u89e3\u6790<\/h3>\n\n\n\n<p>Beautiful Soup \u662f\u4e00\u4e2a\u7528\u4e8e\u89e3\u6790 HTML \u548c XML \u6587\u6863\u7684 Python \u5e93\u3002\u867d\u7136\u5b83\u672c\u8eab\u4e0d\u652f\u6301\u4ee3\u7406\uff0c\u4f46\u5b83\u53ef\u4ee5\u4e0e\u5176\u4ed6\u5de5\u5177\u7ed3\u5408\u4f7f\u7528\uff0c\u4f8b\u5982 <code data-no-translation=\"\">requests<\/code> \u83b7\u53d6\u6570\u636e\u3002\u8fd9\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u4f8b\u5b50\uff1a<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><div class=\"bg-black rounded-md mb-4\"><div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>Python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>\u590d\u5236\u4ee3\u7801<\/button><\/div><div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\" data-no-translation=\"\"><span class=\"hljs-keyword\">from<\/span> bs4 <span class=\"hljs-keyword\">import<\/span> BeautifulSoup\n<span class=\"hljs-keyword\">import<\/span> requests\n\nresponse = requests.get(<span class=\"hljs-string\">'http:\/\/example.com\/data'<\/span>)\nsoup = BeautifulSoup(response.text, <span class=\"hljs-string\">'html.parser'<\/span>)\n\n<span class=\"hljs-keyword\">for<\/span> item <span class=\"hljs-keyword\">in<\/span> soup.select(<span class=\"hljs-string\">'.item-class'<\/span>):  <span class=\"hljs-comment\"># Replace '.item-class' with the actual class name<\/span>\n    <span class=\"hljs-built_in\">print<\/span>(item.text)\n<\/code><\/div><\/div><\/pre>\n\n\n\n<p>\u8fd9\u4e9b\u53ea\u662f\u57fa\u672c\u793a\u4f8b\uff0c\u4f46\u5e94\u8be5\u4e3a\u60a8\u63d0\u4f9b\u4e00\u4e2a\u826f\u597d\u7684\u8d77\u70b9\uff0c\u8ba9\u60a8\u66f4\u6df1\u5165\u5730\u4e86\u89e3\u7f51\u9875\u6293\u53d6\u9879\u76ee\u4e2d\u6bcf\u4e2a\u5de5\u5177\u7684\u529f\u80fd\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u603b\u4e4b<\/h2>\n\n\n\n<p>\u4ee3\u7406\u662f\u9ad8\u6548\u7f51\u7edc\u6293\u53d6\u4e0d\u53ef\u6216\u7f3a\u7684\u5de5\u5177\uff0c\u524d\u63d0\u662f\u60a8\u4ed4\u7ec6\u9009\u62e9\u5e76\u6d4b\u8bd5\u5b83\u4eec\u3002\u901a\u8fc7\u672c\u6307\u5357\uff0c\u60a8\u53ef\u4ee5\u63d0\u5347\u7f51\u7edc\u6293\u53d6\u5b9e\u8df5\uff0c\u786e\u4fdd\u6570\u636e\u5b8c\u6574\u6027\u548c\u5b89\u5168\u6027\u3002\u5404\u79cd\u5de5\u5177\u9002\u7528\u4e8e\u6240\u6709\u6280\u80fd\u6c34\u5e73\uff0c\u6709\u52a9\u4e8e\u6293\u53d6\u8fc7\u7a0b\u548c\u4ee3\u7406\u9009\u62e9\u3002<\/p>","protected":false},"excerpt":{"rendered":"<p>Web scraping has evolved into a critical tool for various business applications, including but not limited to data analytics, machine learning algorithms, and lead acquisition. Despite its value, consistent and large-scale data retrieval presents numerous challenges. These include countermeasures from website owners, such as IP bans, CAPTCHAs, and honeypots. Proxies offer a powerful solution to [&hellip;]<\/p>\n","protected":false},"featured_media":0,"menu_order":0,"template":"","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"docs-categories":[56],"class_list":["post-490256","docs","type-docs","status-publish","hentry","docs-categories-proxy-use-cases"],"acf":{"faq_title":"Frequently Asked Questions (FAQs) on Web Scraping and Proxy Servers","faq_items":[{"question":"What is Web Scraping?","answer":"<span>Web scraping is a technique used to extract data from websites. This is typically done programmatically through code, using languages like Python, and tools like Scrapy and Selenium.<\/span>"},{"question":"What is a Proxy Server?","answer":"<span>A proxy server acts as an intermediary between your computer and the internet. It receives requests from your end, forwards them to the web, receives the response, and then forwards it back to you.<\/span>"},{"question":"Why Use Proxy Servers in Web Scraping?","answer":"<span>Proxy servers help you bypass restrictions such as IP bans or rate limits, making your web scraping tasks more efficient and less likely to be interrupted by anti-scraping measures.<\/span>"},{"question":"How Do I Set Up a Proxy with Scrapy?","answer":"You can add the following line within your Scrapy spider to set up a proxy:\r\n<div class=\"bg-black rounded-md mb-4\">\r\n<div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>Copy code<\/button><\/div>\r\n<div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\"><span class=\"hljs-keyword\">yield<\/span> scrapy.Request(url, self.parse, meta={<span class=\"hljs-string\">'proxy'<\/span>: <span class=\"hljs-string\">'http:\/\/your.proxy.address:8080'<\/span>})\r\n<\/code><\/div>\r\n<\/div>"},{"question":"How Do I Use Selenium with a Proxy?","answer":"You can configure Selenium to use a proxy like so:\r\n<div class=\"bg-black rounded-md mb-4\">\r\n<div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>Copy code<\/button><\/div>\r\n<div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\">chrome_options = webdriver.ChromeOptions()\r\nchrome_options.add_argument(<span class=\"hljs-string\">f'--proxy-server=<span class=\"hljs-subst\">{PROXY}<\/span>'<\/span>)\r\n<\/code><\/div>\r\n<\/div>"},{"question":"Can Charles Proxy Be Used for Web Scraping?","answer":"<span>Charles Proxy is mainly used for debugging and inspecting HTTP traffic. It is not generally used for web scraping, but it can be useful for diagnosing issues during the scraping process.<\/span>"},{"question":"How Do I Use Beautiful Soup to Parse HTML?","answer":"Here's a quick sample code snippet:\r\n<div class=\"bg-black rounded-md mb-4\">\r\n<div class=\"flex items-center relative text-gray-200 bg-gray-800 px-4 py-2 text-xs font-sans justify-between rounded-t-md\"><span>python<\/span><button class=\"flex ml-auto gap-2\"><svg stroke=\"currentColor\" fill=\"none\" stroke-width=\"2\" viewbox=\"0 0 24 24\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon-sm\" height=\"1em\" width=\"1em\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 4h2a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h2\"><\/path><rect x=\"8\" y=\"2\" width=\"8\" height=\"4\" rx=\"1\" ry=\"1\"><\/rect><\/svg>Copy code<\/button><\/div>\r\n<div class=\"p-4 overflow-y-auto\"><code class=\"!whitespace-pre hljs language-python\">soup = BeautifulSoup(response.text, <span class=\"hljs-string\">'html.parser'<\/span>)\r\n<span class=\"hljs-keyword\">for<\/span> item <span class=\"hljs-keyword\">in<\/span> soup.select(<span class=\"hljs-string\">'.item-class'<\/span>):\r\n<span class=\"hljs-built_in\">print<\/span>(item.text)\r\n<\/code><\/div>\r\n<\/div>"},{"question":"How Do I Measure the Speed of a Proxy?","answer":"<span>You can use tools like cURL or fast.com to measure the load time and performance score of a proxy server.<\/span>"},{"question":"How Do I Evaluate the Reliability of a Proxy?","answer":"<span>The reliability of a proxy can be assessed through uptime statistics and through third-party monitoring tools that measure the downtime of a proxy server.<\/span>"},{"question":"How Do I Ensure the Security of My Data?","answer":"<span>Choose a proxy that offers strong encryption methods. You can use SSL Labs or Qualys SSL Labs to evaluate the SSL certificate and security rating of a proxy server.<\/span>"},{"question":"How Many Proxies Do I Need for Web Scraping?","answer":"You can use the formula:\r\n\r\n<math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mtext>Number\u00a0of\u00a0Proxies<\/mtext><mo>=<\/mo><mfrac><mtext>Number\u00a0of\u00a0Requests\u00a0Per\u00a0Second<\/mtext><mtext>Requests\u00a0Per\u00a0Proxy\u00a0Per\u00a0Second<\/mtext><\/mfrac><\/mrow><annotation encoding=\"application\/x-tex\">\\text{Number of Proxies} = \\frac{\\text{Number of Requests Per Second}}{\\text{Requests Per Proxy Per Second}}<\/annotation><\/semantics><\/math>\r\n\r\nto calculate the number of proxies you'll need for your web scraping project."}]},"_links":{"self":[{"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/docs\/490256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/docs"}],"about":[{"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/types\/docs"}],"version-history":[{"count":0,"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/docs\/490256\/revisions"}],"wp:attachment":[{"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/media?parent=490256"}],"wp:term":[{"taxonomy":"docs-categories","embeddable":true,"href":"https:\/\/oneproxy.pro\/cn\/wp-json\/wp\/v2\/docs-categories?post=490256"}],"curies":[{"name":"\u53ef\u6e7f\u6027\u7c89\u5242","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}