Complex Setup<\/strong>: Selenium setup methods can be complex.<\/li>\n<\/ul>\n\n\n\nWeb Scraping Sample with Selenium<\/h3>\n\n\n\n As with Puppeteer, let’s go through a tutorial on web scraping with Selenium using the same target site. We’ll extract the book previews from the Crime and Thriller category of the Danube website.<\/p>\n\n\n\n
Danube Store: Crime and Thrillers<\/strong><\/p>\n\n\n\nStep 1: Import the Necessary Modules and Configure Selenium<\/strong><\/p>\n\n\n\nimport time\nfrom selenium import webdriver\nfrom selenium.webdriver.common.by import By\n\noptions = webdriver.ChromeOptions()\noptions.add_argument("--headless")<\/code><\/pre><\/div>\n\n\n\nStep 2: Initialize the Chrome WebDriver<\/strong><\/p>\n\n\n\ndriver = webdriver.Chrome(options=options)<\/code><\/pre><\/div>\n\n\n\nStep 3: Navigate to the Target Website<\/strong><\/p>\n\n\n\ntime.sleep(1)\ncrime_n_thrillers = driver.find_element(By.CSS_SELECTOR, "ul[class='sidebar-list'] > li")\ncrime_n_thrillers.click()\ntime.sleep(1)\nbooks = driver.find_elements(By.CSS_SELECTOR, "div.shop-content li.preview")<\/code><\/pre><\/div>\n\n\n\nStep 4: Click on the Crime & Thrillers Category and Extract Book Previews<\/strong><\/p>\n\n\n\ntime.sleep(1)\ncrime_n_thrillers = driver.find_element(By.CSS_SELECTOR, "ul[class='sidebar-list'] > li")\ncrime_n_thrillers.click()\ntime.sleep(1)\nbooks = driver.find_elements(By.CSS_SELECTOR, "div.shop-content li.preview")<\/code><\/pre><\/div>\n\n\n\nStep 5: Define a Function to Extract Data from Each Book Preview<\/strong><\/p>\n\n\n\ndef extract(element):\n title = element.find_element(By.CSS_SELECTOR, "div.preview-title").text\n author = element.find_element(By.CSS_SELECTOR, "div.preview-author").text\n rating = element.find_element(By.CSS_SELECTOR, "div.preview-details p.preview-rating").text\n price = element.find_element(By.CSS_SELECTOR, "div.preview-details p.preview-price").text\n return {"title": title, "author": author, "rating": rating, "price": price}<\/code><\/pre><\/div>\n\n\n\nStep 6: Loop Through the Previews, Extract the Data, and Quit the Driver<\/strong><\/p>\n\n\n\nextracted_data = []\nfor element in books:\n data = extract(element)\n extracted_data.append(data)\n\nprint(extracted_data)\ndriver.quit()<\/code><\/pre><\/div>\n\n\n\nExpected Output<\/h3>\n\n\n\n Running the above code will produce an output similar to the following:<\/p>\n\n\n\n
[\n {'title': 'Does the Sun Also Rise?', 'author': 'Ernst Doubtingway', 'rating': '\u2605\u2605\u2605\u2605\u2606', 'price': '$9.95'},\n {'title': 'The Insiders', 'author': 'E. S. Hilton', 'rating': '\u2605\u2605\u2605\u2605\u2606', 'price': '$9.95'},\n {'title': 'A Citrussy Clock', 'author': 'Bethany Urges', 'rating': '\u2605\u2605\u2605\u2605\u2605', 'price': '$9.95'}\n]<\/code><\/pre><\/div>\n\n\n\nAdditional Selenium Example: Taking a Screenshot<\/h3>\n\n\n\n In addition to scraping data, Selenium can also be used to take screenshots of web pages. Here’s an example of how to take a screenshot of a web page using Selenium.<\/p>\n\n\n\n
Step 1: Import the Necessary Modules and Configure Selenium<\/strong><\/p>\n\n\n\nfrom selenium import webdriver\n\noptions = webdriver.ChromeOptions()\noptions.add_argument("--headless")<\/code><\/pre><\/div>\n\n\n\nStep 2: Initialize the Chrome WebDriver<\/strong><\/p>\n\n\n\ndriver = webdriver.Chrome(options=options)<\/code><\/pre><\/div>\n\n\n\nStep 3: Navigate to the Target Website<\/strong><\/p>\n\n\n\nurl = "https:\/\/example.com"\ndriver.get(url)<\/code><\/pre><\/div>\n\n\n\nStep 4: Take a Screenshot<\/strong><\/p>\n\n\n\ndriver.save_screenshot("example_screenshot.png")<\/code><\/pre><\/div>\n\n\n\nStep 5: Quit the Driver<\/strong><\/p>\n\n\n\ndriver.quit()<\/code><\/pre><\/div>\n\n\n\nSelenium is a versatile tool for web automation tasks, including web scraping and taking screenshots. Its support for multiple programming languages and browsers, along with its scalability, makes it a powerful choice for various automation needs. Whether you’re extracting data or generating reports, Selenium provides the capabilities to automate your tasks efficiently.<\/p>\n\n\n\n
Puppeteer vs. Selenium: Speed Comparison<\/h2>\n\n\n\n Is Puppeteer faster than Selenium? The answer is yes\u2014Puppeteer is generally faster than Selenium.<\/p>\n\n\n\n
To compare the speed of Puppeteer and Selenium, we used the Danube-store sandbox and ran the scripts presented above 20 times, averaging the execution times.<\/p>\n\n\n\n
Selenium Speed Test<\/h3>\n\n\n\n We used the time<\/code> module in Python to measure the execution time of the Selenium script. The start time was recorded at the beginning and the end time at the end of the script. The difference between these times provided the total execution duration.<\/p>\n\n\n\nHere is the complete script used for Selenium:<\/p>\n\n\n\n
import time\nfrom selenium import webdriver\nfrom selenium.webdriver.common.by import By\nfrom selenium.webdriver.support.ui import WebDriverWait\nfrom selenium.webdriver.support import expected_conditions as EC\n\ndef extract(element):\n title = element.find_element(By.CSS_SELECTOR, "div.preview-title").text\n author = element.find_element(By.CSS_SELECTOR, "div.preview-author").text\n rating = element.find_element(By.CSS_SELECTOR, "div.preview-details p.preview-rating").text\n price = element.find_element(By.CSS_SELECTOR, "div.preview-details p.preview-price").text\n return {"title": title, "author": author, "rating": rating, "price": price}\n\n# Start the timer\nstart_time = time.time()\n\noptions = webdriver.ChromeOptions()\noptions.add_argument("--headless")\n\n# Create a new instance of the Chrome driver\ndriver = webdriver.Chrome(options=options)\n\nurl = "https:\/\/danube-webshop.herokuapp.com\/"\ndriver.get(url)\n\n# Click on the Crime & Thrillers category\ntime.sleep(1)\ncrime_n_thrillers = driver.find_element(By.CSS_SELECTOR, "ul[class='sidebar-list'] > li")\ncrime_n_thrillers.click()\ntime.sleep(1)\n\n# Extract the book previews\nbooks = driver.find_elements(By.CSS_SELECTOR, "div.shop-content li.preview")\n\nextracted_data = []\nfor element in books:\n data = extract(element)\n extracted_data.append(data)\n\nprint(extracted_data)\n\n# End the timer\nend_time = time.time()\nprint(f"The whole script took: {end_time - start_time:.4f} seconds")\n\ndriver.quit()<\/code><\/pre><\/div>\n\n\n\nPuppeteer Speed Test<\/h3>\n\n\n\n For the Puppeteer script, we used the Date<\/code> object to measure the execution time. The start time was recorded at the beginning and the end time at the end of the script. The difference between these times provided the total execution duration.<\/p>\n\n\n\nHere is the complete script used for Puppeteer:<\/p>\n\n\n\n
const puppeteer = require('puppeteer');\n\nasync function main() {\n const start = Date.now();\n\n const browser = await puppeteer.launch({ headless: true });\n const page = await browser.newPage();\n await page.goto('https:\/\/danube-webshop.herokuapp.com\/', { waitUntil: 'networkidle2' });\n\n await page.waitForSelector('ul.sidebar-list');\n\n await Promise.all([\n page.waitForNavigation(),\n page.click("ul[class='sidebar-list'] > li > a"),\n ]);\n\n await page.waitForSelector("li[class='preview']");\n const books = await page.evaluateHandle(\n () => [...document.querySelectorAll("li[class='preview']")]\n );\n\n const processed_data = await page.evaluate(elements => {\n let data = [];\n elements.forEach(element => {\n let title = element.querySelector("div.preview-title").innerHTML;\n let author = element.querySelector("div.preview-author").innerHTML;\n let rating = element.querySelector("div.preview-details > p.preview-rating").innerHTML;\n let price = element.querySelector("div.preview-details > p.preview-price").innerHTML;\n\n let result = { title, author, rating, price };\n data.push(result);\n });\n return data;\n }, books);\n\n console.log(processed_data);\n await page.close();\n await browser.close();\n\n const end = Date.now();\n console.log(`Execution time: ${(end - start) \/ 1000} seconds`);\n}\n\nmain();<\/code><\/pre><\/div>\n\n\n\nPerformance Test Results<\/h3>\n\n\n\n The performance tests showed that Puppeteer is about 60% faster than Selenium. This speed advantage makes Puppeteer a more suitable choice for projects requiring high-speed web scraping and automation, especially when working with Chromium-based browsers.<\/p>\n\n\n\n
Speed Results Summary:<\/strong><\/p>\n\n\n\nThe chart below illustrates the performance difference between Puppeteer and Selenium:<\/p>\n\n\n\n
Scaling up Puppeteer applications for projects requiring fast, efficient web scraping is the optimal choice in this context.<\/p>\n\n\n\n
Puppeteer vs. Selenium: Which Is Better?<\/h2>\n\n\n\n So which one is better between Selenium and Puppeteer for scraping? There isn’t a direct answer to that question since it depends on multiple factors, such as long-term library support, cross-browser support, and your web scraping needs.<\/p>\n\n\n\n
Puppeteer is faster, but compared to Selenium, it supports fewer browsers. Selenium also supports more programming languages compared to Puppeteer.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n Although using Puppeteer or Selenium is a good option for web scraping, scaling up and optimizing your web scraping project can be challenging because advanced anti-bot measures can detect and block these libraries. The best way to avoid this is by using a web scraping API, like OneProxy.<\/p>\n\n\n\n
Using Puppeteer with Proxy Servers<\/h3>\n\n\n\n To use Puppeteer with a proxy server, you can pass the proxy settings in the args<\/code> option when launching the browser instance. Here\u2019s an example:<\/p>\n\n\n\nconst puppeteer = require('puppeteer');\n\nasync function main() {\n const proxyServer = 'http:\/\/your-proxy-server:port';\n \n const browser = await puppeteer.launch({\n headless: true,\n args: [`--proxy-server=${proxyServer}`]\n });\n\n const page = await browser.newPage();\n await page.goto('https:\/\/example.com', { waitUntil: 'networkidle2' });\n\n \/\/ Perform your web scraping tasks here\n\n await browser.close();\n}\n\nmain();<\/code><\/pre><\/div>\n\n\n\nUsing Selenium with Proxy Servers<\/h3>\n\n\n\n To use Selenium with a proxy server, you can set the proxy options using the webdriver.Proxy<\/code> class. Here\u2019s an example:<\/p>\n\n\n\nfrom selenium import webdriver\nfrom selenium.webdriver.common.proxy import Proxy, ProxyType\n\nproxy = Proxy()\nproxy.proxy_type = ProxyType.MANUAL\nproxy.http_proxy = "your-proxy-server:port"\nproxy.ssl_proxy = "your-proxy-server:port"\n\ncapabilities = webdriver.DesiredCapabilities.CHROME\nproxy.add_to_capabilities(capabilities)\n\noptions = webdriver.ChromeOptions()\noptions.add_argument("--headless")\n\ndriver = webdriver.Chrome(desired_capabilities=capabilities, options=options)\ndriver.get("https:\/\/example.com")\n\n# Perform your web scraping tasks here\n\ndriver.quit()<\/code><\/pre><\/div>\n\n\n\nUsing proxy servers with Puppeteer and Selenium can help bypass IP-based restrictions and reduce the risk of getting blocked, enhancing the efficiency of your web scraping tasks. OneProxy’s rotating proxies<\/a> can further optimize this process, providing a seamless scraping experience.<\/p>\n","protected":false},"excerpt":{"rendered":"Are you trying to decide between Puppeteer and Selenium for web scraping? Both are powerful browser automation frameworks, and making the right choice depends on your specific scraping needs and available resources. To help you make an informed decision, we’ve highlighted the key differences between Puppeteer and Selenium in the table below. Afterward, we will […]<\/p>\n","protected":false},"author":1,"featured_media":505389,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[92],"tags":[],"class_list":["post-505385","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-info"],"acf":{"faq_title":"Frequently Asked Questions (FAQ)","faq_items":[{"question":"What are Puppeteer and Selenium?","answer":"Puppeteer and Selenium are both browser automation frameworks used for web scraping, testing, and automating browser tasks. Puppeteer is a Node.js library that controls Chrome or Chromium over the DevTools Protocol, while Selenium is an open-source tool that supports various browsers and programming languages through its WebDriver API.<\/span>"},{"question":"Which is faster, Puppeteer or Selenium?","answer":"Puppeteer is generally faster than Selenium. However, the speed difference can vary depending on the specific tasks and configurations used in your web scraping or automation projects.<\/span>"},{"question":"What are the main advantages of Puppeteer?","answer":"Puppeteer is known for its ease of use, speed, and ability to automate tasks in headless mode by default. It supports Chromium and has a strong event-driven architecture that eliminates the need for manual sleep calls in code.<\/span>"},{"question":"What are the limitations of Puppeteer?","answer":"Puppeteer supports fewer browsers compared to Selenium and primarily focuses on JavaScript, though unofficial ports for other languages like Python and PHP exist.<\/span>"},{"question":"How can I use Puppeteer with a proxy server?","answer":"You can configure Puppeteer to use a proxy server by passing the proxy settings in the <\/span>args<\/code> option when launching the browser.<\/span>"},{"question":"What are the main advantages of Selenium?","answer":"Selenium supports multiple programming languages (Python, Java, JavaScript, Ruby, C#) and can automate various browsers, including Firefox, Edge, Safari, and custom browsers like QtWebKit. It also allows for extensive scalability through techniques like setting up cloud servers with different browser settings.<\/span>"},{"question":"What are the limitations of Selenium?","answer":"Selenium can be more complex to set up compared to Puppeteer, especially when configuring it for different browsers and environments.<\/span>"},{"question":"How can I use Selenium with a proxy server?","answer":"You can set up a proxy server in Selenium using the <\/span>webdriver.Proxy<\/code> class.<\/span>"},{"question":"How was the speed comparison between Puppeteer and Selenium conducted?","answer":"We ran the same web scraping tasks on the Danube-store sandbox using both Puppeteer and Selenium. Each script was executed 20 times, and the average execution times were calculated to compare the performance.<\/span>"},{"question":"What were the results of the speed comparison?","answer":"The results showed that Puppeteer is about 60% faster than Selenium, making it a better choice for high-speed web scraping and automation tasks.<\/span>"},{"question":"How can I avoid getting blocked by anti-bot measures?","answer":"OneProxy can help you avoid getting blocked. OneProxy handles anti-bot bypassing, provides rotating proxies, headless browsers, automatic retries, and more, ensuring a seamless web scraping experience.<\/span>"}]},"_links":{"self":[{"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/posts\/505385","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/comments?post=505385"}],"version-history":[{"count":3,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/posts\/505385\/revisions"}],"predecessor-version":[{"id":505393,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/posts\/505385\/revisions\/505393"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/media\/505389"}],"wp:attachment":[{"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/media?parent=505385"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/categories?post=505385"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oneproxy.pro\/in\/wp-json\/wp\/v2\/tags?post=505385"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"Puppeteer](\"https:\/\/oneproxy.pro\/wp-content\/uploads\/2024\/05\/puppeteer-logo.png\" "\\"\\""){kind=logo}
<\/figure>\n\n\n\n![\"Selenium](\"https:\/\/oneproxy.pro\/wp-content\/uploads\/2024\/05\/Selenium_Logo-2048x501.png\" "\\"\\""){kind=logo}
<\/figure>\n\n\n\n![\"Puppeteer](\"https:\/\/oneproxy.pro\/wp-content\/uploads\/2024\/05\/pupeteer-vs-selenium.png\" "\\"\\"")
<\/figure>\n\n\n\n