爬虫第三式：某某二手车数据抓取 -二级页面

本文主要是介绍爬虫第三式：某某二手车数据抓取 -二级页面，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

本章我们就已经彻底跨入高级阶段了，ready？

文章底部有全部代码

接下来我们就带着之前的学习内容，来学习高级阶段了，这次讲的主要内容如下

二级页面爬取、数据持久化MySQL

首先我们先来看一下任务：

【1】爬取地址 某某二手车网 - 我要买车 https://www.某某.com/bj/buy/ 
【2】爬取目标 所有汽车的 汽车名称、行驶里程、排量、变 速箱、价格 
【3】爬取分析 *********一级页面需抓取*********** 1、车辆详情页的链接 *********二级页面需抓取*********** 1、汽车名称 2、行驶里程 3、排量 4、变速箱 5、价格

实现步骤：

第一步 确定响应内容中是否存在所需抓取数据

在这里插入图片描述
我们查看网页源代码也是可以看到的：

所以这样的就不是动态加载的

第二步查看要爬取的URL地址，并获取规律：https://www.guazi.com/bj/buy/

第1页: https://www.guazi.com/bj/buy/o1/#bread
第2页: https://www.guazi.com/bj/buy/o2/#bread
…
第n页: https://www.guazi.com/bj/buy/o{}/#bread

第三步写正则表达式
一级页面正则表达式：

<li data-scroll-track=.*?href="(.*?)"

二级页面正则表达式：

<div class="product-textbox">.*?<h1 class="titlebox">(.*?)</h1>.*?<li class="two"><span>(.*?)</span>.*?<li class="three"><span>(.*?)</span>.*?<li class="last"><span>(.*?)</span>.*?<span class="price-num">(.*?)</span>

这个正则表达式是会变得，我之前爬过的表达式不是这样的，大家在爬取的时候还是要看一下，这个正则是否对，不对的话还要根据当时的HTML页面进行修改

第四步代码实现

`我们总体的思路还是：定义功能函数,减少重复代码`

导入模块：

import requests
import re
import time
import random

定义常用变量,url,headers及计数等：

class GuaziSpider:def __init__(self):self.url = 'https://www.guazi.com/bj/buy/o{}/#bread'self.headers = {'Cookie': 'uuid=bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633; cityDomain=bj; ganji_uuid=4748850580491413739853; antipas=4306793i04Y993u91R5Q45485152; lg=1; track_id=154140903399362560; clueSourceCode=%2A%2300; user_city_id=12; Hm_lvt_bf3ee5b290ce731c7a4ce7a617256354=1607691171,1607691369,1607691399,1607905256; guazitrackersessioncadata=%7B%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%7D; sessionid=23660523-0a05-4211-e3ef-c618fd61b415; lng_lat=116.41119_39.89243; gps_type=1; close_finance_popup=2020-12-14; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22pz_baidu%22%2C%22ca_n%22%3A%22pcbiaoti%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22%22%2C%22ca_campaign%22%3A%22%22%2C%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22ca_transid%22%3A%22%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22track_id%22%3A%22154140903399362560%22%2C%22display_finance_flag%22%3A%22-%22%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%22bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633%22%2C%22ca_city%22%3A%22bj%22%2C%22sessionid%22%3A%2223660523-0a05-4211-e3ef-c618fd61b415%22%7D; preTime=%7B%22last%22%3A1607905259%2C%22this%22%3A1605602948%2C%22pre%22%3A1605602948%7D; Hm_lpvt_bf3ee5b290ce731c7a4ce7a617256354=1607905260','User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36',}

注意！！！！！！

这个cookie
第一没有’s’ ，就是cookie
第二自己设置自己的不要复制我的，因为电脑不一样，访问网站的的cookie也不一样，具体怎么找cookie如下

先点进某某二手车网站 — 我要买车 — F12

在这里插入图片描述

像这样，找到3也就是cookie字样，复制所有的cookie里面的东西，从头到尾，
User-Agent也需要自己设置，现在去网上搜一下，或者可以在这个cookie下面也有你浏览器的User-Agent，也可以用(推荐) 后面我们使用更简单的方法设置User-Agent

获取响应内容函数获取html

def get_html(self, url):html = requests.get(url=url, headers=self.headers).content.decode('utf-8', 'ignore')# 多级页面或者需要多次发请求，需要额外定义功能函数时，我们使用 return htmlreturn html

使用正则表达式来解析页面，提取数据

def parse_html(self, one_url):"""爬虫逻辑函数"""one_html = self.get_html(url=one_url)one_regex = '<li data-scroll-track=.*?href="(.*?)"'href_list = self.re_func(regex=one_regex, html=one_html)for href in href_list:two_url = 'https://www.guazi.com' + href# 获取一辆汽车详情页的具体数据self.get_one_car_info(two_url)# 控制数据抓取的频率time.sleep(random.uniform(0, 1))

获取一辆汽车的具体数据

def get_one_car_info(self, two_url):# 名称、行驶里程、排量、变速箱、价格two_html = self.get_html(url=two_url)two_regex = '<div class="product-textbox">.*?<h1 class="titlebox">(.*?)</h1>.*?<li class="two"><span>(.*?)</span>.*?<li class="three"><span>(.*?)</span>.*?<li class="last"><span>(.*?)</span>.*?<span class="price-num">(.*?)</span>'car_info_list = self.re_func(regex=two_regex, html=two_html)# 获取具体数据item = {}item['name'] = car_info_list[0][0].strip().split('\r\n')[0].strip()item['km'] = car_info_list[0][1].strip()item['displace'] = car_info_list[0][2].strip()item['type'] = car_info_list[0][3].strip()item['price'] = car_info_list[0][4].strip()print(item)

程序入口函数，用来控制整体逻辑

def run(self):for o in range(1, 6):one_url = self.url.format(o)self.parse_html(one_url=one_url)

最后上传所有代码（不存储数据库的）：

import requests
import re
import time
import randomclass GuaziSpider:def __init__(self):self.url = 'https://www.guazi.com/bj/buy/o{}/#bread'self.headers = {'Cookie': 'uuid=bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633; cityDomain=bj; ganji_uuid=4748850580491413739853; antipas=4306793i04Y993u91R5Q45485152; lg=1; track_id=154140903399362560; clueSourceCode=%2A%2300; user_city_id=12; Hm_lvt_bf3ee5b290ce731c7a4ce7a617256354=1607691171,1607691369,1607691399,1607905256; guazitrackersessioncadata=%7B%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%7D; sessionid=23660523-0a05-4211-e3ef-c618fd61b415; lng_lat=116.41119_39.89243; gps_type=1; close_finance_popup=2020-12-14; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22pz_baidu%22%2C%22ca_n%22%3A%22pcbiaoti%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22%22%2C%22ca_campaign%22%3A%22%22%2C%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22ca_transid%22%3A%22%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22track_id%22%3A%22154140903399362560%22%2C%22display_finance_flag%22%3A%22-%22%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%22bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633%22%2C%22ca_city%22%3A%22bj%22%2C%22sessionid%22%3A%2223660523-0a05-4211-e3ef-c618fd61b415%22%7D; preTime=%7B%22last%22%3A1607905259%2C%22this%22%3A1605602948%2C%22pre%22%3A1605602948%7D; Hm_lpvt_bf3ee5b290ce731c7a4ce7a617256354=1607905260','User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36',}def get_html(self, url):"""请求功能函数: 获取html"""html = requests.get(url=url, headers=self.headers).content.decode('utf-8', 'ignore')return htmldef re_func(self, regex, html):"""解析功能函数: 正则解析得到列表"""pattern = re.compile(regex, re.S)r_list = pattern.findall(html)return r_listdef parse_html(self, one_url):"""爬虫逻辑函数"""one_html = self.get_html(url=one_url)one_regex = '<li data-scroll-track=.*?href="(.*?)"'href_list = self.re_func(regex=one_regex, html=one_html)for href in href_list:two_url = 'https://www.guazi.com' + href# 获取一辆汽车详情页的具体数据self.get_one_car_info(two_url)# 控制数据抓取的频率time.sleep(random.uniform(0, 1))def get_one_car_info(self, two_url):"""获取一辆汽车的具体数据"""# 名称、行驶里程、排量、变速箱、价格two_html = self.get_html(url=two_url)two_regex = '<div class="product-textbox">.*?<h1 class="titlebox">(.*?)</h1>.*?<li class="two"><span>(.*?)</span>.*?<li class="three"><span>(.*?)</span>.*?<li class="last"><span>(.*?)</span>.*?<span class="price-num">(.*?)</span>'car_info_list = self.re_func(regex=two_regex, html=two_html)# 获取具体数据item = {}item['name'] = car_info_list[0].strip().split('\r\n')[0].strip()item['km'] = car_info_list[0].strip()item['displace'] = car_info_list[0].strip()item['type'] = car_info_list[0].strip()item['price'] = car_info_list[0].strip()print(item)def run(self):for o in range(1, 6):one_url = self.url.format(o)self.parse_html(one_url=one_url)if __name__ == '__main__':spider = GuaziSpider()spider.run()

大家可以自己运行一下试试
接下来就是数据的存储了，我们爬去完了以后想要存储数据，可以存到数据库、csv等里面，保持数据的持久化存储，接下来我们先介绍MySQL存储：
看一下MySQL存储的大概步骤：

使用excute()方法将数据存入数据库思路


import pymysql    # 在Python中导入pymysql模块，我们使用Python连接数据库# 在 __init__() 中连接数据库并创建游标对象
# __init__(self)：self.db = pymysql.connect('IP',... ...)self.cursor = self.db.cursor()# 在 save_html() 中将所抓取的数据处理成列表，使用execute()方法写入
# save_html(self,r_list):self.cursor.execute('sql',[data1])self.db.commit()# 在run() 中等数据抓取完成后关闭游标及断开数据库连接
# run(self):self.cursor.close()self.db.close()

MySQL存储一定要看这里！！！！！！！！！！！！！！

当然我们要在MySQL数据创建一个数据库用来存储此次爬取的数据，代码如下，直接复制粘贴即可：
create database maoyandb charset utf8;
use maoyandb;
create table maoyantab(
name varchar(200),
star varchar(500),
time varchar(100)
)charset=utf8;

现在我们就将改写的MySQL存储所有代码奉上，同样的cookie和User-Agent用自己浏览器的：

import requests
import re
import time
import random
import pymysqlclass GuaziSpider:def __init__(self):self.url = 'https://www.guazi.com/bj/buy/o{}/#bread'self.headers = {'Cookie': 'uuid=bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633; cityDomain=bj; ganji_uuid=4748850580491413739853; antipas=4306793i04Y993u91R5Q45485152; lg=1; track_id=154140903399362560; clueSourceCode=%2A%2300; user_city_id=12; Hm_lvt_bf3ee5b290ce731c7a4ce7a617256354=1607691171,1607691369,1607691399,1607905256; guazitrackersessioncadata=%7B%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%7D; sessionid=23660523-0a05-4211-e3ef-c618fd61b415; lng_lat=116.41119_39.89243; gps_type=1; close_finance_popup=2020-12-14; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22pz_baidu%22%2C%22ca_n%22%3A%22pcbiaoti%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22%22%2C%22ca_campaign%22%3A%22%22%2C%22ca_kw%22%3A%22%25e7%2593%259c%25e5%25ad%2590%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22ca_transid%22%3A%22%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22track_id%22%3A%22154140903399362560%22%2C%22display_finance_flag%22%3A%22-%22%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%22bcef00ae-5a0b-4b1e-9afb-6f5a01e7a633%22%2C%22ca_city%22%3A%22bj%22%2C%22sessionid%22%3A%2223660523-0a05-4211-e3ef-c618fd61b415%22%7D; preTime=%7B%22last%22%3A1607905259%2C%22this%22%3A1605602948%2C%22pre%22%3A1605602948%7D; Hm_lpvt_bf3ee5b290ce731c7a4ce7a617256354=1607905260','User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36',}self.db = pymysql.connect('localhost', 'root', '123456', 'guazidb', charset='utf8')self.cur = self.db.cursor()def get_html(self, url):"""请求功能函数: 获取html"""html = requests.get(url=url, headers=self.headers).content.decode('utf-8', 'ignore')return htmldef re_func(self, regex, html):"""解析功能函数: 正则解析得到列表"""pattern = re.compile(regex, re.S)r_list = pattern.findall(html)return r_listdef parse_html(self, one_url):"""爬虫逻辑函数"""one_html = self.get_html(url=one_url)one_regex = '<li data-scroll-track=.*?href="(.*?)"'href_list = self.re_func(regex=one_regex, html=one_html)for href in href_list:two_url = 'https://www.guazi.com' + href# 获取一辆汽车详情页的具体数据self.get_one_car_info(two_url)# 控制数据抓取的频率time.sleep(random.uniform(0, 1))def get_one_car_info(self, two_url):"""获取一辆汽车的具体数据"""# 名称、行驶里程、排量、变速箱、价格two_html = self.get_html(url=two_url)two_regex = '<div class="product-textbox">.*?<h2 class="titlebox">(.*?)</h2>.*?<li class="two"><span>(.*?)</span>.*?<li class="three"><span>(.*?)</span>.*?<li class="last"><span>(.*?)</span>.*?<span class="price-num">(.*?)</span>'car_info_list = self.re_func(regex=two_regex, html=two_html)# 获取具体数据item = {}item['name'] = car_info_list[0][0].strip().split('\r\n')[0].strip()item['km'] = car_info_list[0][1].strip()item['displace'] = car_info_list[0][2].strip()item['type'] = car_info_list[0][3].strip()item['price'] = car_info_list[0][4].strip()print(item)li = [item['name'], item['km'], item['displace'], item['type'], item['price']]ins = 'insert into guazitab values(%s,%s,%s,%s,%s)'self.cur.execute(ins, li)self.db.commit()def run(self):for o in range(1, 3):one_url = self.url.format(o)self.parse_html(one_url=one_url)# 断开数据库self.cur.close()self.db.close()if __name__ == '__main__':spider = GuaziSpider()spider.run()