R自动数据收集第二章HTML笔记2(主要关于htmlTreeParse函数)

时间：2016-12-30 23:20:40 阅读：685 评论：0 收藏：0 [点我收藏+]

包含以下几个小的知识点

1htmlTreeParse函数源码和一些参数

2hander的写法

3关于missing函数

4关于if-else语句中else语句的花括号问题

5关于checkHandlerNames函数

6关于GeneralHandlerNames属性

7关于match函数

8关于inherits函数

9关于on.exit函数

===============================================================================

还是尝试去阅读下该函数的源码：

htmlTreeParse函数

function (file, ignoreBlanks = TRUE, handlers = NULL, replaceEntities = FALSE,
asText = FALSE, trim = TRUE, validate = FALSE, getDTD = TRUE,
isURL = FALSE, asTree = FALSE, addAttributeNamespaces = FALSE,
useInternalNodes = FALSE, isSchema = FALSE, fullNamespaceInfo = FALSE,
encoding = character(), useDotNames = length(grep("^\\.",
names(handlers)))>0, xinclude = TRUE, addFinalizer = TRUE,
error = htmlErrorHandler, isHTML = TRUE, options = integer(),
parentFirst = FALSE)
{
#asText=T则file作为XML文本处理
isMissingAsText = missing(asText)
#地址大于一个时要中止程序并抛出异常
if(length(file)>1){
file = paste(file, collapse ="\n")
if(!missing(asText)&&!asText)
stop(structure(list(message ="multiple URLs passed to xmlTreeParse. If this is the content of the file, specify asText = TRUE"),
class= c("MultipleURLError","XMLParserError",
"simpleError","error","condition")))
asText = TRUE
}
#当isURL非空时且是XML的时候，才修改URL为数值型
#比如这里isURL=1
if(missing(isURL)&&!asText)
isURL <- length(grep("^(http|ftp|file)://", file, useBytes = TRUE,
perl = TRUE))
#isHTML 默认为 TRUE
if(isHTML){
validate = FALSE
getDTD = FALSE
isSchema = FALSE
docClass ="HTMLInternalDocument"
}
else docClass = character()
#checkHandlerNames返回的是一个逻辑值，其作用是
checkHandlerNames(handlers,"DOM")
if(missing(fullNamespaceInfo)&& inherits(handlers,"RequiresNamespaceInfo"))
fullNamespaceInfo = TRUE
oldValidate = xmlValidity()
xmlValidity(validate)
on.exit(xmlValidity(oldValidate))
if(!asText && isURL == FALSE){
if(file.exists(file)== FALSE)
if(!missing(asText)&& asText == FALSE){
e = simpleError(paste("File", file,"does not exist"))
class(e)= c("FileNotFound",class(e))
stop(e)
}
else asText <- TRUE
}
if(asText && length(file)>1)
file = paste(file, collapse ="\n")
old = setEntitySubstitution(replaceEntities)
on.exit(setEntitySubstitution(old), add = TRUE)
if(asText && length(grep(sprintf("^%s?\\s*<",BOMRegExp),
file, perl = TRUE, useBytes = TRUE))==0){
if(!isHTML ||(isMissingAsText &&!inherits(file,"AsIs"))){
e = simpleError(paste("XML content does not seem to be XML:",
sQuote(file)))
class(e)= c("XMLInputError",class(e))
(if(isHTML)
warning
else stop)(e)
}
}
if(!is.logical(xinclude)){
xinclude =as.logical(xinclude)
}
if(!asText &&!isURL)
file = path.expand(as.character(file))
if(useInternalNodes && trim){
prevBlanks =.Call("RS_XML_setKeepBlanksDefault",0L,
PACKAGE ="XML")
on.exit(.Call("RS_XML_setKeepBlanksDefault", prevBlanks,
PACKAGE ="XML"), add = TRUE)
}
.oldErrorHandler = setXMLErrorHandler(error)
on.exit(.Call("RS_XML_setStructuredErrorHandler",.oldErrorHandler,
PACKAGE ="XML"), add = TRUE)
if(length(options))
options = sum(options)
ans <-.Call("RS_XML_ParseTree",as.character(file), handlers,
as.logical(ignoreBlanks),as.logical(replaceEntities),
as.logical(asText),as.logical(trim),as.logical(validate),
as.logical(getDTD),as.logical(isURL),as.logical(addAttributeNamespaces),
as.logical(useInternalNodes),as.logical(isHTML),as.logical(isSchema),
as.logical(fullNamespaceInfo),as.character(encoding),
as.logical(useDotNames), xinclude, error, addFinalizer,
as.integer(options),as.logical(parentFirst), PACKAGE ="XML")
if(!missing(handlers)&& length(handlers)&&!as.logical(asTree))
return(handlers)
if(!isSchema && length(class(ans)))
class(ans)= c(docClass, oldClass(class(ans)))
if(inherits(ans,"XMLInternalDocument"))
addDocFinalizer(ans, addFinalizer)
elseif(!getDTD &&!isSchema){
class(ans)= oldClass("XMLDocumentContent")
}
ans
}
<environment: namespace:XML>

查看整个htmlTreeParse函数的源代码，可以用到handler的地方并不多。

我们看一下xmlTreeParse {XML}中对于该参数的描述：

参数handlers：

Optional collection of functions(函数集) used to map the different XML nodes to R objects. Typically, this is a named list of functions, and a closure(闭包) can be used to provide local data. This provides a way of filtering the tree as it is being created in R, adding or removing nodes, and generally processing them as they are constructed in the C code.

In a recent addition to the package (version 0.99-8), if this is specified as a single function object, we call that function for each node (of any type) in the underlying DOM tree. It is invoked with（调用） the new node（新节点） and its parent node（父节点）. This applies to regular nodes（普通节点） and also comments（注释）, processing instructions, CDATA nodes, etc. So this function must be sufficiently general to handle them all.

asTree参数

this only applies when on passes a value for the handlers argument and is used then to determine whether the DOM tree should be returned or the handlers object.

该参数在有传入处理器函数的时候要设置为TRUE，此时，函数的返回值是处理后的DOM树，而不是handlers函数对象本身

useDotNames参数

a logical value indicating whether to use the newer format for identifying general element function handlers with the ‘.‘ prefix, e.g. .text, .comment, .startElement. If this is FALSE, then the older format text, comment, startElement, ... are used. This causes problems when there are indeed nodes named text or comment or startElement as a node-specific handler are confused with the corresponding general handler of the same name(如果函数名叫startElement，而有个节点(自定义标签吧)，也交startElement就会让函数困惑滴). Using TRUE means that your list of handlers should have names that use the ‘.‘ prefix for these general element handlers. This is the preferred way to write new code.

TRUE是使用新的格式(函数以.开头)来区分

FLALSE是用旧的格式

实际上配合参数里的这一句很容易理解：

useDotNames = length(grep("^\\.", names(handlers))) > 0

取得处理函数的名字，看看其是否以点开头，如果长度不大于0，就是没有，就是FALSE咯

isURL参数

indicates whether the file argument refers to a URL (accessible via ftp or http) or a regular file on the system. If asText is TRUE, this should not be specified. The function attempts to determine whether the data source is a URL by using grep to look for http or ftp at the start of the string. The libxml parser handles the connection to servers, not the R facilities (e.g. scan).

hander的写法——在文档的Details中：

The handlers argument is used similarly to those specified in xmlEventParse. When an XML tag (element) is processed, we look for a function in this collection with the same name as the tag‘s name. If this is not found, we look for one named startElement. If this is not found, we use the default built in converter(变换器). The same works for comments, entity references, cdata, processing instructions, etc. The default entries should be named comment, startElement, externalEntity, processingInstruction, text, cdata and namespace. All but the last should take the XMLnode as their first argument. In the future, other information may be passed via ..., for example, the depth in the tree, etc. Specifically, the second argument will be the parent node into which they are being added, but this is not currently implemented, so should have a default value (NULL).

当一个标签被处理时，在函数集里1先找和标签同名的函数，2找startElement，最后才找默认的函数

当一个注释等被处理时，也是一样。那么也就是说，比如处理注释的时候，先找叫comment的函数呗，

所以，handler中的函数命名(列表的组件名)是需要讲规律的

node必须作为这些函数的第一个参数。

嗯，这一段讲的很好~，把handler函数怎么编写讲清楚了。

这个博客，也提到了这些参数的含义，有空可以看看

http://blog.163.com/zhoulili1987619@126/blog/static/35308201201531511273389/

=======================================================================

#test
#导包
library(XML)
#链接
url <-"http://www.r-datacollection.com/materials/html/fortunes.html"
#handers函数
h2 <- list(
startElement = function(node,...){
name <- xmlName(node)
if(name %in% c("div","title")){NULL}else{node}
},
comment = function(node){NULL}
)
#正式开始，函数参数
file=url
ignoreBlanks = TRUE
handlers = h2
replaceEntities = FALSE
asText = FALSE
trim = TRUE
validate = FALSE
getDTD = TRUE
isURL = FALSE
asTree = TRUE
addAttributeNamespaces = FALSE
useInternalNodes = FALSE
isSchema = FALSE
fullNamespaceInfo = FALSE
encoding = character()
useDotNames = length(grep("^\\.", names(handlers)))>0
xinclude = TRUE
addFinalizer = TRUE
error = XML:::htmlErrorHandler
isHTML = TRUE
options = integer()
parentFirst = FALSE
#函数体部分
#asText参数没有传入
#我们将其从missing(asText)改为TRUE
isMissingAsText = TRUE
#file的长度是否大于1，大于1如果asText未传入则要报错：传入了多个URL
#不大于1，则跳过
if(length(file)>1){
file = paste(file, collapse ="\n")
if(!missing(asText)&&!asText)
stop(structure(list(message ="multiple URLs passed to xmlTreeParse. If this is the content of the file, specify asText = TRUE"),
class= c("MultipleURLError","XMLParserError",
"simpleError","error","condition")))
asText = TRUE
}
# 本来是if (missing(isURL) && !asText)
# 我们修改为if (TURE && !asText)
#isURL参数没有传递且asText参数为假才执行
if(TRUE &&!asText)
isURL <- length(grep("^(http|ftp|file)://", file, useBytes = TRUE,
perl = TRUE))
#只要有http|ftp|file中的一个协议开头，比如http://，就是URL了。
# 此时isURL=1
#是否为HTML，是的
if(isHTML){
validate = FALSE
getDTD = FALSE #从默认值T改为了F
isSchema = FALSE
docClass ="HTMLInternalDocument"
}else{docClass = character()}#否则创建空的
# class(docClass)
#返回TRUE通过检验了，否则函数会中止并抛出异常
XML:::checkHandlerNames(handlers,"DOM")
#fullNamespaceInfo参数whether to provide the namespace URI and prefix on each node
#其实就是是否在节点面前带上URI信息
#fullNamespaceInfo为空且handlers含有该属性，才执行这一步
#missing(fullNamespaceInfo)被替换为TRUE
#handlers并没有RequiresNamespaceInfo属性,所以不执行
if( TRUE && inherits(handlers,"RequiresNamespaceInfo"))
fullNamespaceInfo = TRUE
#以下两行的结果都是integer(0)
#奇妙~，先保存原有的配置
oldValidate = XML:::xmlValidity()
#当前的配置
XML:::xmlValidity(validate)
#还原原来的配置
on.exit(XML:::xmlValidity(oldValidate))
#asText为假，且isURL是假的
#结果为TFALSE，我们不必管他
if(!asText && isURL == FALSE){
if(file.exists(file)== FALSE)
#如果本地文件不存在
if(!missing(asText)&& asText == FALSE){
#抛出异常，文件不存在
e = simpleError(paste("File", file,"does not exist"))
class(e)= c("FileNotFound",class(e))
stop(e)
}
else asText <- TRUE
}
#此时asText是FALSE，这个跟我们无关
if(asText && length(file)>1)
file = paste(file, collapse ="\n")
#replaceEntities的默认值是FALSE
old = XML:::setEntitySubstitution(replaceEntities)
#old的值是FALSE
on.exit(XML:::setEntitySubstitution(old), add = TRUE)
#BOMRegExp是一个内置的常量吧
#看名字应该是基于BOM的正则表达式
#因为是FALSE，所以我们也先不去管它
if(asText && length(grep(
sprintf("^%s?\\s*<", XML:::BOMRegExp),
file, perl = TRUE, useBytes = TRUE
)
)==0)
{
if(!isHTML ||(isMissingAsText &&!inherits(file,"AsIs"))){
e = simpleError(paste("XML content does not seem to be XML:",
sQuote(file)))
class(e)= c("XMLInputError",class(e))
(if(isHTML)
warning
else stop)(e)
}
}
#xinclude默认值是TRUE
#以下三个if都是F，所以不管了
if(!is.logical(xinclude)){
xinclude =as.logical(xinclude)
}
if(!asText &&!isURL)
file = path.expand(as.character(file))
if(useInternalNodes && trim){
prevBlanks =.Call("RS_XML_setKeepBlanksDefault",0L,
PACKAGE ="XML")
on.exit(.Call("RS_XML_setKeepBlanksDefault", prevBlanks,
PACKAGE ="XML"), add = TRUE)
}
.oldErrorHandler = XML:::setXMLErrorHandler(error)
#所以，这种点开头的命名是什么鬼？
# class(.oldErrorHandler)
# [1] "list"
on.exit(.Call("RS_XML_setStructuredErrorHandler",.oldErrorHandler,
PACKAGE ="XML"), add = TRUE)
# length(options)是0，所以不执行
if(length(options))
options = sum(options)
#调用一个叫做RS_XML_ParseTree的函数
getAnywhere("RS_XML_ParseTree")
ans <-.Call("RS_XML_ParseTree",as.character(file), handlers,
as.logical(ignoreBlanks),as.logical(replaceEntities),
as.logical(asText),as.logical(trim),as.logical(validate),
as.logical(getDTD),as.logical(isURL),as.logical(addAttributeNamespaces),
as.logical(useInternalNodes),as.logical(isHTML),as.logical(isSchema),
as.logical(fullNamespaceInfo),as.character(encoding),
as.logical(useDotNames), xinclude, error, addFinalizer,
as.integer(options),as.logical(parentFirst), PACKAGE ="XML")
print(ans)
print("-------我是可爱的分割线------------------")
#这里的missing(handlers)我们就不改了哈
#毕竟只要有默认值，他都觉得是TRUE
#和我们的确传递了处理函数的效果TRUE是一样的
if(!missing(handlers)&& length(handlers)&&!as.logical(asTree))
return("呵呵")
if(!isSchema && length(class(ans)))
class(ans)= c(docClass, oldClass(class(ans)))
if(inherits(ans,"XMLInternalDocument")){
addDocFinalizer(ans, addFinalizer)
}elseif(!getDTD &&!isSchema){
print("看我的类型")
print(class(ans))
class(ans)= oldClass("XMLDocumentContent")
print(class(ans))
}
print("看条件判断的逻辑值")
print("1")
print(!missing(handlers)&& length(handlers)&&!as.logical(asTree))
print("2")
print(!isSchema && length(class(ans)))
print("3")
print(inherits(ans,"XMLInternalDocument"))
print("4")
print(!getDTD &&!isSchema)
print(class(ans))
ans

用来调试理解该函数的代码：

注释1：关于missing函数

missing(asText)的含义并不是判断asText是否缺失，而是判断asText作为函数的形参，是否已经在函数被调用的时候传入了实参，所以即便形参asText是有默认值的，missing(asText)的返回值结果仍然是TRUE

#例子1
testMissing<-function(a=TRUE,b=FALSE){
if(missing(b))
return("b is missing")
else"b is here "+b
}
testMissing(F)
# [1] "b is missing"
#例子2
if(missing(b))
return("b is missing")
# Error in missing(b) : ‘missing‘ can only be used for arguments
#例子3
b=NULL
if(missing(b))
return("b is missing")
#没有输出

注释2 关于if-else语句中else语句的花括号问题

例子1

if(isHTML){
validate = FALSE
getDTD = FALSE #从默认值T改为了F
isSchema = FALSE
docClass ="HTMLInternalDocument"
}else{docClass = character()}#否则创建空的

或者

}else docClass = character()

前面的花括号很重要，在交互式模式，R语法分析器用else前面的右花括号来推断这是一个if-else结构，而不是if结构

见《R语言编程艺术中文版》P20

例子2

function(){
if(isHTML){
validate = FALSE
getDTD = FALSE #从默认值T改为了F
isSchema = FALSE
docClass ="HTMLInternalDocument"
}
else docClass = character()#否则创建空的
}

如果在函数体内，这样写是OK的。
否则，会报错：

Error: unexpected ‘else‘ in " else"

注释3.1：checkHandlerNames(handlers, "DOM")函数

事实上，必须要借助getAnywhere(checkHandlerNames)的帮助，才能获得其源码

我们可以在最后一行可以看到：<environment: namespace:XML>

所以我们可以通过XML:::checkHandlerNames()调用它

function (handlers, id ="SAX")
{
if(is.null(handlers)) #为空，则返回TRUE
return(TRUE)
ids = names(handlers) #取出handlers中的函数名
i = match(ids,GeneralHandlerNames) #匹配，返回逻辑值向量
prob = any(!is.na(i))#任一个回空才是TRUE
if(prob){
warning("future versions of the XML package will require names of general handler functions
to be prefixed by a . to distinguish them from handlers for nodes with those names. This _may_ affect the ",
paste(names(handlers)[!is.na(i)], collapse =", "))
}
#任意一个handler中的函数不是函数类型，则抛出异常
if(any(w <-!sapply(handlers,is.function)))
warning("some handlers are not functions: ", paste(names(handlers[w]),
collapse =", "))
#返回TRUE，后续代码继续运行
!prob
}
<environment: namespace:XML>

注释3.2关于GeneralHandlerNames属性

还真是人如其名呀，常用处理函数名

> XML:::GeneralHandlerNames
$SAX
[1]"text" "startElement"
[3]"endElement" "comment"
[5]"startDocument" "endDocument"
[7]"processingInstruction""entityDeclaration"
[9]"externalEntity"
$DOM
[1]"text" "startElement"
[3]"comment" "entity"
[5]"cdata" "processingInstruction"

分别有SAX和DOM两种方式，想起学Java时的DOM和SAX解析没？

注释3.3关于match函数

> h2 <- list(
+ startElement = function(node,...){
+ name <- xmlName(node)
+ if(name %in% c("div","title")){NULL}else{node}
+ },
+ comment = function(node){NULL}
+)
> handlers<-h2
> ids = names(handlers)
> ids
[1]"startElement""comment"
> i = match(ids, XML:::GeneralHandlerNames)
> i
[1] NA NA
>?match
starting httpd help server ... done
>!is.na(i)#如果i中有NA，则返回F，没有NA则返回T
[1] FALSE FALSE
> prob = any(!is.na(i))#当i中的任意一个元素都不是NA的时候，prob才返回T
> prob
[1] FALSE

match returns a vector of the positions of (first) matches of its first argument in its second.

math函数实际上返回的是：第一个参数中的元素如果匹配了第二个参数中的值，那么到底匹配的是第二个元素中的第几个，即其位置。

> testMatch<-c("a","c")
> testSet <- c(‘a‘,‘b‘,‘c‘)
> match(testMatch,testSet)
[1]13

事实上，看警告信息，未来版本的XML将要求普通的处理函数前面要加一个前缀点(.)，以区别于节点他们名字的处理函数(到底是哪些呢？)

future versions of the XML package will require names of general handler functions

to be prefixed by a . to distinguish them from handlers for nodes with those names. This _may_ affect the

所以我觉得，这一个match和if判断，其实是一个预设吧，毕竟id=‘SAX‘压根没有用到，应该是作者的编码还没打算在当前版本增加该设定。

其实在useDotNames中已经提到了

注释4关于inherits函数

inherits indicates whether its first argument inherits from any of the classes specified in the what argument. If which is TRUE then an integer vector of the same length as what is returned. Each element indicates the position in the class(x) matched by the element of what; zero indicates no match. If which is FALSE then TRUE is returned by inherits if any of the names in what match with any class.

what, value

a character vector naming classes. value can also be NULL.

inherits函数指明第一个参数是否继承了what参数中的任何一个classs类型

如果which为真，则inherits 返回一个同what一样长度的整型向量，每一个元素表示该位置的元素匹配了what的类型，如果数字是0，则表示不匹配

如果which为假，则当what中任意一个名字匹配了类型时，inherits 返回真

x <-10
class(x)# "numeric"
oldClass(x)# NULL
#看了下文档，我个人觉得oldClass是S语言的余毒啊！！！
inherits(x,"a")#FALSE
class(x)<- c("a","b")
# x
# [1] 10
# attr(,"class")
# [1] "a" "b"
#即，x为10这个变量被赋予了两个class，分别名为"a"和 "b"
inherits(x,"a")#TRUE
inherits(x,"a", TRUE)# 1
inherits(x, c("a","b","c"), TRUE)# 1 2 0

注释5.1：几行难懂的代码

#以下两行的结果都是integer(0)
#奇妙~，先保存原有的配置
oldValidate = XML:::xmlValidity()
#使用当前的配置
XML:::xmlValidity(validate)
#还原
on.exit(XML:::xmlValidity(oldValidate))

先看下xmlValidity函数

> XML:::xmlValidity()
integer(0)
> getAnywhere(xmlValidity)
A single object matching ‘xmlValidity’ was found
It was found in the following places
namespace:XML
with value
function (val = integer(0))
{
.Call("RS_XML_getDefaultValiditySetting",as.integer(val),
PACKAGE ="XML")
}
<environment: namespace:XML>

无奈没有更多的信息

RS_XML_getDefaultValiditySetting，看命名应该是获取默认正确的设定

我们再来看：

注释5.2.1：关于on.exit函数

on.exit records the expression given as its argument as needing to be executed when the current function exits (either naturally or as the result of an error). This is useful for resetting graphical parameters or performing other cleanup actions.

on.exit记录作为其参数的表达式，在当前的函数退出(自然结束或者出错)时执行，他通常用来充值图形参数，或者执行清理行为。

例子1