用Python脚本保存网页

本文主要是介绍用Python脚本保存网页，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

1. 需求

因为一些原因，希望把自己在csdn上的blog都自动备份到本地。

这涉及到两个方面：

保存网页：即把一个网页的内容保存到本地文件；——这里做了简化，即保存整个网页，而不是网页中blog的真正内容的那一部分；
分析出所有blog的网址：网址是前一步的输入，为了导出所有的blog，则需要自动分析出自己所有blog的网址。

按照Scrum的方法，我们首先进行Technique Story，即对一些关键技术进行预研。——这是可选的步骤，如果对这一块的技术领域非常熟悉，自然就不用technique story。

2. 保存一个网页的内容

通常Python教材会介绍下面的方法，即urllib.urlretrieve()方法：

>>> import urllib
>>> filename="/home/flying-bird/test.html"
>>> addr="http://blog.csdn.net/a_flying_bird/article/details/38780569"
>>> urllib.urlretrieve(addr, filename)
('/home/flying-bird/test.html', <httplib.HTTPMessage instance at 0xb6ea4c4c>)
>>> f = open(filename,"r")
>>> f.read()
'<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor="white">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'
>>> f.close()
>>>

但我们打开保存的本地文件，发现获取网页内容失败了，提示403 Forbidden。

参考 “Python抓取中文网页”： http://blog.csdn.net/nevasun/article/details/7331644 ，可以解决上述问题。对应代码如下：

>>> import sys, urllib2
>>> addr="http://blog.csdn.net/a_flying_bird/article/details/38780569"
>>> filename="/home/flying-bird/test.html"
>>> headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
>>> req = urllib2.Request(addr, headers = headers)
>>> content = urllib2.urlopen(req).read()
>>> type = sys.getfilesystemencoding()
>>> data = content.decode("UTF-8").encode(type)
>>> f = open(filename, "wb")
>>> f.write(data)
>>> f.close()
>>>

其中urllib2是urllib的改进版本，分别对应的帮助页面：

urllib：https://docs.python.org/2.7/library/urllib.html
urllib2：https://docs.python.org/2.7/library/urllib2.html

题外话：之前有过一段时间，无法正常访问Python网站，最近恢复正常了哦。希望google也尽快恢复。

2. 分析所有的blog网址

在可以保存指定网址的页面之后，我们就需要自动分析出所有blog的地址。

为此，我们可以首先查看博客入口页面http://blog.csdn.net/a_flying_bird的源码，发现会有如下的部分：

<div class="list_item article_item"><div class="article_title">   <span class="ico ico_type_Original"></span><h1><span class="link_title"><a href="/a_flying_bird/article/details/38930901">用Python脚本保存网页</a></span></h1></div>

以及：

<div id="papelist" class="pagelist"><span> 92条数据  共7页</span><strong>1</strong> <a href="/u0/article/list/2">2</a> <a href="/u0/article/list/3">3</a> <a href="/u0/article/list/4">4</a> <a href="/u0/article/list/5">5</a> <a href="/u0/article/list/6">...</a> <a href="/u0/article/list/2">下一页</a> <a href="/u0/article/list/7">尾页</a> 
</div>

由此，通过解析HTML文件，可以知道blog入口页面的文件列表（文件名和网址）；以及博客剩下的文章所在的网页。据此，就可以把所有blog的标题及网址都自动分析处理。

如果有xml解析的经验，可能会想到用XML来解析上面的HTML源码。但我们在分析上面的源码的时候，发现它并没有遵循严格的XML语法。这也是XML和HTML的区别，尽管XHTML提出了很多年，但实际的情况还是不容乐观。为此，就需要用HTML的语法来分析。

在Python中，提供了HTMLParser。经过分析blog页面的HTML源码特征，我们使用了下面的代码：

#!/usr/bin/env python
#encoding: utf-8import htmllib
import re
import formatter'''
Example:<span class="link_title"><a href="/a_flying_bird/article/details/38930901">用Python脚本保存网页</a>
</span>
'''blogUrls = []
blogNames = []class BlogItemsParser(htmllib.HTMLParser):def __init__(self):htmllib.HTMLParser.__init__(self, formatter.NullFormatter())self.isArticleItem = Falseself.pattern = "^http://blog.csdn.net/a_flying_bird/article/details/[\\d]+$"def start_a(self, attributes):#print "len=", len(attributes), ": ", attributesif len(attributes) != 1:returnattr_name, attr_value = attributes[0]#print "name: %s, value: %s" % (attr_name, attr_value)if attr_name != "href":returnif re.match(self.pattern, attr_value):blogUrls.append(attr_value)self.isArticleItem = Truedef end_a(self):self.isArticleItem = Falsedef handle_data(self, text):if self.isArticleItem:blogNames.append(text.strip())def getContent():'''We export a blog page manually.'''filename = "/home/flying-bird/examples/python/export_csdn_blog/blog.html"f = open(filename, "r")content = f.read()f.close()return contentif (__name__ == "__main__"):content = getContent()parser = BlogItemsParser()parser.feed(content)parser.close()len = len(blogUrls)for i in range(0, len):print blogUrls[i], blogNames[i]

运行效果：

flying-bird@flyingbird:~/examples/python/export_csdn_blog$ ./export-csdn-blog.py 
http://blog.csdn.net/a_flying_bird/article/details/38930901 用Python脚本保存网页
http://blog.csdn.net/a_flying_bird/article/details/38876477 ListView的例子
http://blog.csdn.net/a_flying_bird/article/details/38780569 滑动页面的一个例子
http://blog.csdn.net/a_flying_bird/article/details/38776553 转  提问的智慧
http://blog.csdn.net/a_flying_bird/article/details/38711751 C语言的一个正则表达式pcre
http://blog.csdn.net/a_flying_bird/article/details/38690025 一个C语言小程序
http://blog.csdn.net/a_flying_bird/article/details/38666177 查看Android虚拟机上的文件
http://blog.csdn.net/a_flying_bird/article/details/38665897 数据存储之SharedPreferences
http://blog.csdn.net/a_flying_bird/article/details/38665387 获取RunningTaskInfo
http://blog.csdn.net/a_flying_bird/article/details/38590093 Android EditText的使用方法
http://blog.csdn.net/a_flying_bird/article/details/38563305 Android TextView的使用方法
http://blog.csdn.net/a_flying_bird/article/details/38542253 谈谈Java的对象（续）
http://blog.csdn.net/a_flying_bird/article/details/38541855 谈谈Java的对象
http://blog.csdn.net/a_flying_bird/article/details/38519965 搭建gtest环境
http://blog.csdn.net/a_flying_bird/article/details/38497919 Ubuntu频繁报错
flying-bird@flyingbird:~/examples/python/export_csdn_blog$

3. 获取blog页面个数

前面的代码是分析了一个页面中的blog列表。当然blog较多，会分多个页面展示。通过分析网址，发现形式如下：

http://blog.csdn.net/u0/article/list/2

因此，我们可以从2开始，构造这种网址，直到出现非法页面。——对于我们的应用场景而言，这种处理方法是可以接受的。

但是，实际验证发现这种偷懒的方法行不通。好在注意到第一页blog最下面在跳转其他页面的时候，有如下信息：

<div id="papelist" class="pagelist"><span> 92条数据  共7页</span><strong>1</strong> <a href="http://blog.csdn.net/u0/article/list/2">2</a> <a href="http://blog.csdn.net/u0/article/list/3">3</a> <a href="http://blog.csdn.net/u0/article/list/4">4</a> <a href="http://blog.csdn.net/u0/article/list/5">5</a> <a href="http://blog.csdn.net/u0/article/list/6">...</a> <a href="http://blog.csdn.net/u0/article/list/2">下一页</a> <a href="http://blog.csdn.net/u0/article/list/7">尾页</a> 
</div>

注意到“尾页”即可，从这里可以获取最后一页的序号。因此，范围缩小为仅分析如下一句话：

<a href="http://blog.csdn.net/u0/article/list/7">尾页</a>

虽然在前面代码中的class中，可以增加一个处理，获取页面数据。但从代码可读性上来讲，把多个功能糅合在一起，可能并不是一个好主意。因此，我们另外再创建一个class，专门获取blog的页面个数。——尽管代码规模有一定增加，而且还有重复代码的嫌疑。当然，性能在此不是我们的考量方面。

代码重构如下：——关注class BlogPagesParser和getBlogPagesCount()。

#!/usr/bin/env python
#encoding: utf-8import htmllib
import re
import formatter'''
Example:<span class="link_title"><a href="/a_flying_bird/article/details/38930901">用Python脚本保存网页</a>
</span>
'''blogUrls = []
blogNames = []class BlogItemsParser(htmllib.HTMLParser):def __init__(self):htmllib.HTMLParser.__init__(self, formatter.NullFormatter())self.isArticleItem = Falseself.pattern = "^http://blog.csdn.net/a_flying_bird/article/details/[\\d]+$"def start_a(self, attributes):#print "len=", len(attributes), ": ", attributesif len(attributes) != 1:returnattr_name, attr_value = attributes[0]#print "name: %s, value: %s" % (attr_name, attr_value)if attr_name != "href":returnif re.match(self.pattern, attr_value):blogUrls.append(attr_value)self.isArticleItem = Truedef end_a(self):self.isArticleItem = Falsedef handle_data(self, text):if self.isArticleItem:blogNames.append(text.strip())class BlogPagesParser(htmllib.HTMLParser):def __init__(self):htmllib.HTMLParser.__init__(self, formatter.NullFormatter())self.isBlogPageNumber = Falseself.pattern = "^http://blog.csdn.net/u0/article/list/([\\d]+)$"self.tempPageCount = 1 # Only 1 blog page by defaultdef start_a(self, attributes):#print "len=", len(attributes), ": ", attributesif len(attributes) != 1:returnattr_name, attr_value = attributes[0]#print "name: %s, value: %s" % (attr_name, attr_value)if attr_name != "href":returnmatch = re.match(self.pattern, attr_value)if not match:returnself.isBlogPageNumber = Trueself.tempPageCount = match.group(1)def end_a(self):self.isBlogPageNumber = Falsedef handle_data(self, text):if not self.isBlogPageNumber:returntext = text.strip()#print "text: ", textif text == "尾页":print "got it: ", self.tempPageCount#else:#    print "fail :("def get_page_count(self):return self.tempPageCountdef getContent():'''We export a blog page manually.'''filename = "/home/flying-bird/examples/python/export_csdn_blog/blog.html"f = open(filename, "r")content = f.read()f.close()return contentdef getBlogItems():content = getContent()parser = BlogItemsParser()parser.feed(content)parser.close()len = len(blogUrls)for i in range(0, len):print blogUrls[i], blogNames[i]def getBlogPagesCount():content = getContent()parser = BlogPagesParser()parser.feed(content)pageCount = parser.get_page_count()parser.close()print "blog pages: ", pageCountif (__name__ == "__main__"):getBlogPagesCount()

4. 打通全部环节

至此，所有的技术点都已经验证通过，是时候把所有要素串在一起了。思路再梳理一遍：

给定个人blog的入口页面地址；
读取这个页面的内容；
读取这个页面中包含的blog列表（地址, 文章标题)
读取blog的总页面个数
遍历2开始的各个blog页面，获取每个页面的blog列表。至此，获取了所有的blog列表。
依次获取每个blog页面的内容，并保存到本地的指定目录下。

最后的代码如下：

#!/usr/bin/env python
#encoding: utf-8'''
Save all of my csdn blogs, for backup only.Input: 1. blogUrl: URL of my blog.2. userName: the registered user name of csdn, such as "a_flying_bird".2. userId: User id corresponding userName, such as "u0", which is allocated by csdn.The steps:1. Read the content of blogUrl.2. Parse all the blog items in blogUrl.3. Get the page number (N) from blogUrl.4. Parse all the blog items in page 2..NAnd now we get all the blog items.5. Read each blog, and save to local.TODO:
1. Read only the blog's content, not the page's content.
2. Update/Replace all of the hyper-links in the blogs.Examples:
1. Blog item in a blog page:<span class="link_title"><a href="/a_flying_bird/article/details/38930901">用Python脚本保存网页</a></span>
2. Blog page number in blogUrl:<div id="papelist" class="pagelist"><span> 92条数据  共7页</span><strong>1</strong> <a href="http://blog.csdn.net/u0/article/list/2">2</a> <a href="http://blog.csdn.net/u0/article/list/3">3</a> <a href="http://blog.csdn.net/u0/article/list/4">4</a> <a href="http://blog.csdn.net/u0/article/list/5">5</a> <a href="http://blog.csdn.net/u0/article/list/6">...</a> <a href="http://blog.csdn.net/u0/article/list/2">下一页</a> <a href="http://blog.csdn.net/u0/article/list/7">尾页</a> </div>
3. blogUrl, i.e., URL of one person's blog entrance:http://blog.csdn.net/a_flying_birdHere, we consider "a_flying_bird" as the user name.
4. Blog URLs from second page:http://blog.csdn.net/u0/article/list/2Here, we consider "u0" as the user id.Exception:Q: urllib2.HTTPError: HTTP Error 502: Bad GatewayA: Add some sleep between each reading
'''import htmllib
import urllib2
import sys
import re
import formatter
import string
import timedef readContentFromUrl(url, filename=None):'''If filename is not null, save the content of url to this file.url: for example, http://blog.csdn.net/a_flying_bird/article/details/38780569'''print "readContentFromUrl(), url: ", urlheaders = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}  req = urllib2.Request(url, headers = headers)  content = urllib2.urlopen(req).read()  type = sys.getfilesystemencoding()  data = content.decode("UTF-8").encode(type)  if filename is not None:f = open(filename, "wb")  f.write(data)  f.close()  return dataclass BlogItemsParser(htmllib.HTMLParser):def __init__(self):htmllib.HTMLParser.__init__(self, formatter.NullFormatter())self.isArticleItem = Falseself.pattern = None # before feed(), must call set_patter() fristself.blogUrls = []self.blogNames = []def set_pattern(self, blogItemPattern):self.pattern = blogItemPatterndef start_a(self, attributes):#print "len=", len(attributes), ": ", attributesif len(attributes) != 1:returnattr_name, attr_value = attributes[0]if attr_name != "href":return#print "name: %s, value: %s, pattern: %s" % (attr_name, attr_value, self.pattern)if re.match(self.pattern, attr_value):self.blogUrls.append(attr_value)self.isArticleItem = Truedef end_a(self):self.isArticleItem = Falsedef handle_data(self, text):if self.isArticleItem:#s = text.strip() # debug the title with specifial '&'#print "title: BEGIN--<" + s + ">--END"self.blogNames.append(text.strip())def get_blog_urls(self):return self.blogUrlsdef get_blog_names(self):return self.blogNamesclass BlogPagesParser(htmllib.HTMLParser):def __init__(self):htmllib.HTMLParser.__init__(self, formatter.NullFormatter())self.isBlogPageNumber = Falseself.pattern = None # before feed, must call set_pattern() first.self.pageCount = 1 # Only 1 blog page by defaultdef set_pattern(self, moreUrlPattern):self.pattern = moreUrlPatterndef start_a(self, attributes):#print "len=", len(attributes), ": ", attributesif len(attributes) != 1:returnattr_name, attr_value = attributes[0]#print "name: %s, value: %s" % (attr_name, attr_value)if attr_name != "href":returnmatch = re.match(self.pattern, attr_value)if not match:returnself.isBlogPageNumber = Trueself.pageCount = match.group(1)def end_a(self):self.isBlogPageNumber = Falsedef handle_data(self, text):if not self.isBlogPageNumber:returntext = text.strip()#print "text: ", text#TODO CHS now, you can change it manually.if text == "尾页":print "got the page count: ", self.pageCount#else:#    print "fail :("def get_page_count(self):return string.atoi(self.pageCount)# title is used for debugging.
def getBlogItems(title, content, blogItemRefPattern):parser = BlogItemsParser()parser.set_pattern(blogItemRefPattern)parser.feed(content)blogUrls = parser.get_blog_urls()blogNames = parser.get_blog_names()parser.close()#print "blog items for ", title#count = len(blogUrls)#for i in range(0, count):#    print blogUrls[i], blogNames[i]return blogUrls, blogNamesdef getBlogPagesCount(moreBlogUrl, content, blogPagesPattern):parser = BlogPagesParser()parser.set_pattern(blogPagesPattern)parser.feed(content)pageCount = parser.get_page_count()parser.close()return pageCountdef export_csdn_blog(userName, userId, savedDirectory):blogUrls = []blogNames = []if savedDirectory[-1] != '/':savedDirectory = savedDirectory + '/'blogUrl = "http://blog.csdn.net/" + userNamemoreBlogUrl = "http://blog.csdn.net/" + userId + "/article/list/"blogItemHrefPattern = "^/" + userName + "/article/details/[\\d]+$"blogPagesPattern = "^/" + userId + "/article/list/([\\d]+)$"# Read the content of blogUrl.filename = None # "/home/flying-bird/examples/python/export_csdn_blog/blog.html" # for debugging onlycontent = readContentFromUrl(blogUrl, filename)#print content#Parse all the blog items in blogUrl.tmpBlogUrls, tmpBlogNames = getBlogItems(blogUrl, content, blogItemHrefPattern)blogUrls = blogUrls + tmpBlogUrlsblogNames = blogNames + tmpBlogNames# Get the page number (N) from blogUrl.pageCount = getBlogPagesCount(moreBlogUrl, content, blogPagesPattern)# Parse all the blog items in page 2..Nfor i in range(2, pageCount):url = moreBlogUrl + ("%d" % i)print "i = %d, url = %s" % (i, url)content = readContentFromUrl(url)tmpBlogUrls, tmpBlogNames = getBlogItems(url, content, blogItemHrefPattern)blogUrls = blogUrls + tmpBlogUrlsblogNames = blogNames + tmpBlogNames# Read each blog, and save to local.count = len(blogUrls)for i in range(0, count):url = "http://blog.csdn.net" + blogUrls[i]filename = savedDirectory + blogNames[i] + ".html"print "url=%s, filename=%s" % (url, filename)readContentFromUrl(url, filename)time.sleep(30) # unit: secondsprint "DONE"def usage(proccessName):print "Usage: %s userName userId savedDirectory" % (processName,)print "For example:"print "    userName: a_flying_bird"print "    userId: u0"print "    savedDirectory: /home/csdn/"if (__name__ == "__main__"):argc = len(sys.argv)if argc == 1: # for debugging only.userName = "a_flying_bird"userId = "u0"savedDirectory = "/home/flying-bird/csdn/"elif argc == 4:userName = sys.argv[1]userId = sys.argv[2]savedDirectory = sys.argv[3]else:usage(sys.argv[0])sys.exit(-1)#TODO check the directory, or mkdir if neccessaryexport_csdn_blog(userName, userId, savedDirectory)

效果图：