Python Web Scraping with Static Proxies

By Nicholas St. Germain —

Introduction

Web scraping offers access to valuable insights from platforms like LinkedIn and Zillow. However, repeated requests from a single IP address trigger rate limiting, bans, and verification challenges. Proxy rotation solves this by distributing requests across multiple IP addresses, making scrapers appear as regular users rather than automated tools.

Why Use Rotating Proxies in Python?

Avoid Rate Limits

Websites throttle or block IPs making excessive requests. Rotating proxies spreads request volume across addresses.

Prevent Bans

Sites identify and ban IPs performing automated activities. Changing IPs regularly avoids detection.

Maintain Data Flow

Scraping workflows halt when IPs get blocked midway. Proxies keep operations running continuously.

Change Your User-Agent

Websites detect the default Python user-agent string (python-requests). Modifying headers makes scripts appear browser-like:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
                  " AppleWebKit/537.36 (KHTML, like Gecko)"
                  " Chrome/58.0.3029.110 Safari/537.3"
}

Example 1: Scraping LinkedIn with Rotating Proxies

Objective: Collect public job postings from LinkedIn Challenge: Rate-limiting and IP bans from repeated requests

import requests
from bs4 import BeautifulSoup
import random

proxy_list = [
    "http://user1:pass1@proxy1.statproxies.com:3128",
    "http://user2:pass2@proxy2.statproxies.com:3128",
    "http://user3:pass3@proxy3.statproxies.com:3128"
]

headers = {
    "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/58.0.3029.110 Safari/537.3")
}

def fetch_linkedin_jobs(url):
    proxy = random.choice(proxy_list)
    print(f"Using proxy: {proxy}")

    try:
        response = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers=headers,
            timeout=10
        )
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")

        jobs = soup.select(".base-card__full-link")
        job_titles = [job.get_text(strip=True) for job in jobs]
        return job_titles
    except requests.exceptions.RequestException as e:
        print(f"Request failed with proxy {proxy}: {e}")
        return []

if __name__ == "__main__":
    url = "https://www.linkedin.com/jobs/search/?keywords=python"
    jobs = fetch_linkedin_jobs(url)
    for title in jobs[:5]:
        print("-", title)

Key Features:

  • Random proxy selection for each request
  • Custom user-agent headers for browser mimicry
  • BeautifulSoup extraction targeting job card classes

Example 2: Scraping Zillow for Real Estate Listings

Objective: Gather property addresses and prices from Zillow Challenge: IP throttling after multiple requests without rotation

import requests
from bs4 import BeautifulSoup
import time

proxy_list = [
    "http://user1:pass1@proxy1.statproxies.com:3128",
    "http://user2:pass2@proxy2.statproxies.com:3128",
    "http://user3:pass3@proxy3.statproxies.com:3128"
]

headers = {
    "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/58.0.3029.110 Safari/537.3")
}

def get_zillow_listings(url, proxy):
    print(f"Using proxy: {proxy}")
    try:
        resp = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers=headers,
            timeout=10
        )
        resp.raise_for_status()

        soup = BeautifulSoup(resp.text, "html.parser")
        listings = soup.select(".list-card-info")

        data = []
        for listing in listings:
            address = listing.select_one(".list-card-addr")
            price = listing.select_one(".list-card-price")

            data.append({
                "address": address.get_text(strip=True) if address else "N/A",
                "price": price.get_text(strip=True) if price else "N/A"
            })
        return data

    except requests.exceptions.RequestException as e:
        print(f"Request failed: {e}")
        return []

if __name__ == "__main__":
    url = "https://www.zillow.com/homes/for_sale/"
    for proxy in proxy_list:
        listings = get_zillow_listings(url, proxy)
        for item in listings[:3]:
            print(f"Address: {item['address']} | Price: {item['price']}")
        time.sleep(3)  # Delay between requests

Implementation Details:

  • Sequential proxy iteration across requests
  • Delays between calls to reduce detection likelihood
  • Targeted HTML element parsing for property information

Conclusion

For consistent web scraping without interruptions, proxy rotation combined with proper headers prevents rate limiting and bans. ISP proxies offer unlimited bandwidth, datacenter-level speed, and residential-proxy stealth for scalable scraping operations.