以下是7个Python爬虫的简单示例，每个案例都包括了必要的代码和简要说明。
- 简单的网页爬取
import requests
url = 'http://example.com'
response = requests.get(url)
print(response.text)
- 使用BeautifulSoup解析HTML
from bs4 import BeautifulSoup
import requests
url = 'http://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
print(soup.title.text)
- 使用lxml解析XML或HTML
from lxml import etree
import requests
url = 'http://example.com'
response = requests.get(url)
tree = etree.HTML(response.text)
print(tree.xpath('//title/text()'))
- 使用Scrapy框架创建一个简单的爬虫
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['http://example.com']
def parse(self, response):
# 提取信息的逻辑
pass
- 使用Selenium模拟用户行为爬取动态网页
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://example.com')
print(driver.page_source)
driver.quit()
- 使用pymongo将数据存储到MongoDB
from pymongo import MongoClient
import requests
url = 'http://example.com'
response = requests.get(url)
client = MongoClient('mongodb://localhost:27017/')
db = client['mydatabase']
collection = db['mycollection']
collection.insert_one({'data': response.text})
- 使用Proxy中转请求
import requests
url = 'http://example.com'
proxies = {
'http': 'http://10.10.1.10:3128',
'https': 'http://10.10.1.10:1080',
}
response = requests.get(url, proxies=proxies)
print(response.text)
这些例子提供了爬虫开发中常见的场景，包括简单的网页爬取、使用BeautifulSoup解析HTML、使用lxml解析XML/HTML、使用Scrapy框架、使用Selenium模拟用户行为、使用pymongo存储数据到MongoDB以及使用Proxy中转请求。每个案例都是一个入门级的爬虫实例，可以帮助开发者理解基本的爬虫技术。