python 解析html基础 HTMLParser库,方法,及代码实例

class HTMLParser.HTMLParser

An HTMLParser instance is fed HTML data and calls handler methodswhen start tags, end tags, text, comments, and other markup elements areencountered. The user should subclass HTMLParser and override its methods to implement the desired behavior.

The HTMLParser class is instantiated without arguments.

Unlike the parser in htmllib, this parser does not check that end tags match start tags or call the end-tag handler for elements which are closedimplicitly by closing an outer element.

exception HTMLParser.HTMLParseError

HTMLParser is able to handle broken markup, but in some cases it might raise this exception when it encounters an error while parsing.This exception provides three attributes: msg is a briefmessage explaining the error, lineno is the number of the line onwhich the broken construct was detected, and offset is the number ofcharacters into the line at which the construct starts.

            下面仅就重点进行解释，与html解析的pythonlib除了HTMLParser，还有htmllib，htmllib不同于HTMLParser的一点是htmllib基于SGMLParser

#!/usr/bin/python
#a simple htmlparser demo

from HTMLParser import HTMLParser

class SimpleParser(HTMLParser):
    def handle_starttag(self,tag,attr): print tag
    def handle_endtag(self,tag): print tag
    def handle_data(self,data): print data
simpleparser = SimpleParser()
simpleparser.feed("<title></title>")

这是一个很简单的例子，但是HTMLParser的基本原理也就是这样，当然，为了更好的完成工作我们还要处理一些非标准的html，因为现在浏览器的

发展是在不断迎合识别各种非标准的html，所以造成了今天我们对html的解析变得异常复杂，我们会看到<title>...然后没有终止标记这样的非标准格式，

当然，像xhtml这种语言是比较严格的，不会允许这种情况的出现，下面给出一个在《python基础网络编程》的例子，这个例子中除了处理这种不标准

的html标记外，还同时用

HTMLParser.handle_entityref(name)

This method is called to process a named character reference of the form&name; (e.g. &gt;), where name is a general entity reference(e.g. ‘gt‘).

下面是代码:

#!/usr/bin/env python
from HTMLParser import HTMLParser
#handle entity like &
from htmlentitydefs import entitydefs
import sys

class TitleParser(HTMLParser):
	def __init__(self):
		self.taglevels = []
                #处理title ul li 只有开始无结束标记的情况
                self.handledtags = ['title', 'ul', 'li']
		self.processing = None
		HTMLParser.__init__(self)
	
	def handle_starttag(self, tag, attrs):
		if len(self.taglevels) and self.taglevels[-1] == tag:
			self.handle_endtag(tag)
		
		self.taglevels.append(tag)
		if tag in self.handledtags:
			self.data = ''
			self.processing = tag
			if tag == 'ul':
				#do print

	def handle_data(self, data):
		if self.processing:
			self.title += data
	
	def handle_endtag(self, tag):
		if not tag in self.taglevels:
			return

		while len(self.taglevels):
			starttag = self.taglevels.pop()

			if starttag in self.handledtags:
				self.finishprocessing(starttag)

			if starttag == tag:
				break

	def cleanse(self):
		self.data = re.sub('\s+', ' ', self.data)

	def finishprocessing(self, tag):
		self.cleanse()
		if tag == 'title' and tag == self.processing:
			print "Dom title", self.data
		elif tag == 'ul':
			print "List ended"
		elif tag == 'li' and tag == self.processing:
			print "List item", self.data
		self.processing = None

	def gettitle(self):
		return self.title
	
        #处理特殊值，如果在映射表中有对应的，即采用映射的值，否则为字面值
	def handle_entityref(self, name):
		if entitydefs.has_key(name):
			self.handle_data(entitydefs[name])
		else:
			self.handle_data('&' + name + ';')
	
	def handle_charref(self, name):
		try:
			charnum = int(name)
		except ValueError:
			return

		if charnum < 1 or charnum > 255:
			return

		self.handle_data(chr(charnum))
	

fd = open(sys.argv[1])
tp = TitleParser()
tp.feed(fd.read())
print tp.gettitle()