Week 9

Week 9: Web Scraping & Requests - 40 Python Snippets

This week covers HTTP requests, BeautifulSoup, extracting web data, and regex for pattern matching. Below are 40 Python snippets categorized into 4 topics (10 per topic).

1️⃣ HTTP Requests with `requests` (10 Snippets)

1. Install the `requests` Library

pip install requests

2. Sending a GET Request

import requests

response = requests.get("https://jsonplaceholder.typicode.com/todos/1")
print(response.json())  # Output JSON response

3. Sending a POST Request

import requests

data = {"title": "New Task", "completed": False}
response = requests.post("https://jsonplaceholder.typicode.com/todos", json=data)
print(response.json())

4. Sending a PUT Request (Updating Data)

data = {"title": "Updated Task"}
response = requests.put("https://jsonplaceholder.typicode.com/todos/1", json=data)
print(response.json())

5. Sending a DELETE Request

response = requests.delete("https://jsonplaceholder.typicode.com/todos/1")
print(response.status_code)  # 200 means success

6. Handling HTTP Headers

headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.json())

7. Handling Timeouts in Requests

try:
    response = requests.get("https://httpbin.org/delay/3", timeout=2)
except requests.exceptions.Timeout:
    print("Request timed out!")

8. Downloading an Image

url = "https://via.placeholder.com/150"
response = requests.get(url)

with open("image.jpg", "wb") as file:
    file.write(response.content)

9. Using Session for Persistent Requests

session = requests.Session()
session.get("https://httpbin.org/cookies/set/sessioncookie/123456789")
response = session.get("https://httpbin.org/cookies")
print(response.text)

10. Handling Redirects in Requests

response = requests.get("https://httpbin.org/redirect/1", allow_redirects=True)
print(response.url)

2️⃣ Parsing HTML with BeautifulSoup (10 Snippets)

11. Install BeautifulSoup

pip install beautifulsoup4 requests

12. Basic HTML Parsing with BeautifulSoup

from bs4 import BeautifulSoup

html = "<html><body><h1>Hello, World!</h1></body></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.h1.text)  # Output: Hello, World!

13. Fetching and Parsing a Web Page

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)

14. Extracting All Links from a Page

soup = BeautifulSoup(response.text, "html.parser")

for link in soup.find_all("a"):
    print(link.get("href"))

15. Extracting All Image URLs from a Page

for img in soup.find_all("img"):
    print(img.get("src"))

16. Extracting Text from a Page

text = soup.get_text()
print(text)

17. Finding Elements by Class Name

elements = soup.find_all("div", class_="example-class")
for element in elements:
    print(element.text)

18. Finding Elements by ID

element = soup.find(id="unique-id")
print(element.text)

19. Extracting Table Data

table = soup.find("table")
rows = table.find_all("tr")

for row in rows:
    cols = row.find_all("td")
    print([col.text for col in cols])

20. Handling JavaScript-Rendered Pages (Using Selenium)

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://example.com")
html = driver.page_source
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)
driver.quit()

3️⃣ Extracting Data from Web Pages (10 Snippets)

21. Extracting Specific Elements

paragraphs = soup.find_all("p")
for p in paragraphs:
    print(p.text)

22. Extracting Meta Tags

for meta in soup.find_all("meta"):
    print(meta.attrs)

23. Extracting Structured Data (JSON-LD)

import json

script = soup.find("script", type="application/ld+json")
if script:
    json_data = json.loads(script.string)
    print(json_data)

24. Extracting Data from Lists

ul = soup.find("ul")
for li in ul.find_all("li"):
    print(li.text)

25. Extracting Headings (H1, H2, H3, etc.)

for i in range(1, 4):
    for heading in soup.find_all(f"h{i}"):
        print(heading.text)

26. Extracting Data from Multiple Pages (Pagination Handling)

page_number = 1
while True:
    url = f"https://example.com/page/{page_number}"
    response = requests.get(url)
    
    if "No more pages" in response.text:
        break

    page_number += 1

27. Web Scraping with User-Agent Spoofing

headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get("https://example.com", headers=headers)
print(response.status_code)

28. Extracting Links with Specific Text

links = soup.find_all("a", string="Click Here")
for link in links:
    print(link["href"])

29. Extracting Email Addresses from Web Pages

import re

emails = re.findall(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+", soup.text)
print(emails)

30. Saving Extracted Data to a CSV File

import csv

data = [["Title", "Link"], ["Example", "https://example.com"]]

with open("data.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerows(data)

4️⃣ Regex (`re` module) for Pattern Matching (10 Snippets)

31. Importing the `re` Module

import re

32. Finding All Words in a String

text = "The quick brown fox jumps over the lazy dog."
words = re.findall(r"\b\w+\b", text)
print(words)

33. Extracting Phone Numbers

text = "Call me at 123-456-7890 or 987-654-3210."
phones = re.findall(r"\d{3}-\d{3}-\d{4}", text)
print(phones)

34. Validating Email Addresses

email = "user@example.com"
if re.match(r"[^@]+@[^@]+\.[^@]+", email):
    print("Valid Email")

(And more regex snippets…)

PreviousAssignment NextAssignment

Last updated 9 months ago