爬虫综合案例-使用Scrapy爬虫框架

以下是一个使用Scrapy爬虫框架的简单示例，用于抓取一个网站上的书籍信息。

首先，创建一个新的Scrapy项目：

scrapy startproject bookscrawler

然后，定义你的Item：

# bookscrawler/items.py
 
import scrapy
 
class BookItem(scrapy.Item):
    title = scrapy.Field()
    author = scrapy.Field()
    price = scrapy.Field()

接着，编写爬虫：

# bookscrawler/spiders/bookspider.py
 
import scrapy
from bookscrawler.items import BookItem
 
class BookSpider(scrapy.Spider):
    name = 'bookspider'
    allowed_domains = ['books.com']
    start_urls = ['http://books.com/books']
 
    def parse(self, response):
        book_selectors = response.css('.book-listing')
        for book_selector in book_selectors:
            item = BookItem()
 
            title = book_selector.css('.book-title::text').extract_first()
            author = book_selector.css('.book-author::text').extract_first()
            price = book_selector.css('.book-price::text').extract_first()
 
            item['title'] = title
            item['author'] = author
            item['price'] = price
 
            yield item
 
        next_page_url = response.css('.next-page::attr(href)').extract_first()
        if next_page_url is not None:
            yield response.follow(next_page_url, self.parse)

最后，设置管道来处理Item：

# bookscrawler/pipelines.py
 
class BookPipeline(object):
    def process_item(self, item, spider):
        with open('books.csv', 'a') as f:
            f.write("{},{},{}\n".format(item['title'].encode('utf-8'), item['author'].encode('utf-8'), item['price'].encode('utf-8')))
        return item

在settings.py中启用管道：

ITEM_PIPELINES = {
    'bookscrawler.pipelines.BookPipeline': 300,
}

现在，运行爬虫：

scrapy crawl bookspider

这个简单的例子展示了如何使用Scrapy框架来创建一个爬取书籍信息的爬虫。它定义了一个Item来存储书籍数据，爬虫BookSpider用于抓取书籍列表页面，并通过管道将数据保存到CSV文件中。这个例子教会了如何组织爬虫代码，并提供了一个基本的数据抓取和存储的实践。