{"id":491326,"date":"2023-11-07T05:06:42","date_gmt":"2023-11-07T05:06:42","guid":{"rendered":"https:\/\/oneproxy.pro\/?p=491326"},"modified":"2024-08-27T06:51:14","modified_gmt":"2024-08-27T06:51:14","slug":"proxy-rotation-with-python","status":"publish","type":"post","link":"https:\/\/oneproxy.pro\/my\/guides\/proxy-rotation-with-python\/","title":{"rendered":"Teknik Lanjutan untuk Putaran Proksi dengan Python"},"content":{"rendered":"\n

Creating an efficient proxy rotation mechanism is essential when dealing with large-scale web scraping or data mining tasks. While the early stages of web scraping projects or minimal-scale crawls might suffice with a basic setup, the real challenge arises when scaling up. To mitigate risks such as IP blocking and to ensure the robustness of your scraping infrastructure, utilizing a sophisticated proxy rotation system becomes imperative.<\/p>\n\n\n\n

For such purposes, the use of a professional proxy service provider like OneProxy becomes invaluable. With a diverse pool of data center proxy servers, such services can vastly enhance the reliability and efficiency of your scraping tasks.<\/p>\n\n\n\n

Below, we delve into the development of a more advanced proxy rotator using Python and Beautiful Soup, leveraging the services from OneProxy for optimal results.<\/p>\n\n\n\n

\"Proxy<\/figure>\n\n\n\n

Preliminary Setup<\/h2>\n\n\n\n

Before you begin, ensure that you have Beautiful Soup and the requests<\/code> library installed in your Python environment. These tools will enable you to parse HTML content and manage HTTP requests easily.<\/p>\n\n\n\n

Our proxy rotation script will fetch public proxies from OneProxy’s free proxy pool, which can be accessed at OneProxy Free Proxy List<\/a>. This list is updated regularly, offering a fresh set of proxies for various needs.<\/p>\n\n\n\n

Basic Fetch Code<\/h2>\n\n\n\n

First, we need to establish the basic code to fetch the HTML content from the OneProxy’s free proxy list. We use a user-agent string to emulate a web browser, which helps in bypassing basic user-agent based bot detections.<\/p>\n\n\n\n

# -*- coding: utf-8 -*-\nfrom bs4 import BeautifulSoup\nimport requests\nurl = https:\/\/oneproxy.pro\/free-proxy\/\n\ndef fetch_proxies(url):\n    header = {\n        'User-Agent': 'Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_11_2) ' +\n        'AppleWebKit\/601.3.9 (KHTML, like Gecko) Version\/9.0.2 Safari\/601.3.9'\n    }\n    response = requests.get(url, headers=header)\n    return response.content\n<\/code><\/pre><\/div>\n\n\n\n
This function simply retrieves the HTML content from the provided URL.<\/p>\n\n\n\n
Parsing the Proxy List<\/h2>\n\n\n\n
The BeautifulSoup<\/code> library will parse the HTML content to extract the proxies. The proxies are typically listed within a table structure on the web page, identified by specific HTML tags and attributes.<\/p>\n\n\n\n
def parse_proxies(html_content):\n    soup = BeautifulSoup(html_content, 'lxml')\n    proxy_table = soup.select_one('#proxy-list-table')  # Replace with the correct ID\n    proxies = []\n    for row in proxy_table.select('tr'):\n        columns = row.select('td')\n        if columns:\n            ip, port = columns[0].get_text(), columns[1].get_text()\n            proxies.append({'ip': ip, 'port': port})\n    return proxies<\/code><\/pre><\/div>\n\n\n\n
Rotating Proxies<\/h2>\n\n\n\n
The following function orchestrates the proxy rotation by randomly selecting an available proxy from the fetched list:<\/p>\n\n\n\n
from random import choice\n\ndef rotate_proxies(proxies):\n    if proxies:\n        return choice(proxies)\n    else:\n        return None<\/code><\/pre><\/div>\n\n\n\n
Putting It All Together<\/h2>\n\n\n\n
Combining all the functions, the final script integrates proxy fetching, parsing, and rotation, providing a seamless proxy rotation system.<\/p>\n\n\n\n
# -*- coding: utf-8 -*-\nimport requests\nfrom bs4 import BeautifulSoup\nfrom random import choice\n\n# Functions previously defined: fetch_proxies, parse_proxies, rotate_proxies\n\nproxies = []  # This will hold our list of proxies\n\ndef refresh_proxies():\n    global proxies\n    proxies = parse_proxies(fetch_proxies('https:\/\/oneproxy.pro\/free-proxy\/'))\n\ndef get_random_proxy():\n    if not proxies:\n        refresh_proxies()\n    return rotate_proxies(proxies)\n\n# Main execution\nrefresh_proxies()\nproxy = get_random_proxy()\nprint(proxy['ip'], proxy['port'])<\/code><\/pre><\/div>\n\n\n\n
Professional Scaling with OneProxy<\/h2>\n\n\n\n
For production environments where the scale extends to thousands of requests, free proxy pools may not suffice due to reliability and speed considerations. At this juncture, a rotating proxy service<\/a> becomes essential.<\/p>\n\n\n\n
OneProxy offers a robust solution with features such as:<\/p>\n\n\n\n