首页 > 其他 > 详细

Hadoop基础-Hadoop日志滚动

时间:2018-05-28 00:13:43      阅读:340      评论:0      收藏:0      [点我收藏+]

               Hadoop基础-Hadoop日志滚动

                                作者:尹正杰

版权声明:原创作品,谢绝转载!否则将追究法律责任。

 

 

 

一.查看日志镜像文件(如:fsimage_0000000000000000767)内容

1>.用"hdfs oiv"命令下载镜像文件格式为XML,操作如下:

[yinzhengjie@s101 ~]$ ll
total 0
drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ ls -lh /home/yinzhengjie/hadoop/dfs/name/current/ | grep fsimage
-rw-rw-r--. 1 yinzhengjie yinzhengjie 1.2K May 27 06:02 fsimage_0000000000000000767
-rw-rw-r--. 1 yinzhengjie yinzhengjie   62 May 27 06:02 fsimage_0000000000000000767.md5
-rw-rw-r--. 1 yinzhengjie yinzhengjie 1.4K May 27 07:58 fsimage_0000000000000000932
-rw-rw-r--. 1 yinzhengjie yinzhengjie   62 May 27 07:58 fsimage_0000000000000000932.md5
[yinzhengjie@s101 ~]$ hdfs oiv -i ./hadoop/dfs/name/current/fsimage_0000000000000000767 -o yinzhengjie.xml -p XML
[yinzhengjie@s101 ~]$ ll
total 8
drwxrwxr-x. 4 yinzhengjie yinzhengjie   35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie   96 May 25 22:05 shell
-rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ sz yinzhengjie.xml 
rz
 zmodem trl+C ?

  100%       4 KB    4 KB/s 00:00:01       0 Errors

[yinzhengjie@s101 ~]$ 
技术分享图片
<?xml version="1.0"?>
<fsimage><NameSection>
<genstampV1>1000</genstampV1><genstampV2>1019</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741839</lastAllocatedBlockId><txid>767</txid></NameSection>
<INodeSection><lastInodeId>16414</lastInodeId><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>1527331031268</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16387</id><type>FILE</type><name>xrsync.sh</name><replication>3</replication><mtime>1527308253459</mtime><atime>1527330550802</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission><blocks><block><id>1073741826</id><genstamp>1002</genstamp><numBytes>700</numBytes></block>
</blocks>
</inode>
<inode><id>16389</id><type>FILE</type><name>hadoop-2.7.3.tar.gz</name><replication>3</replication><mtime>1527310784699</mtime><atime>1527310775186</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission><blocks><block><id>1073741827</id><genstamp>1003</genstamp><numBytes>134217728</numBytes></block>
<block><id>1073741828</id><genstamp>1004</genstamp><numBytes>79874467</numBytes></block>
</blocks>
</inode>
<inode><id>16402</id><type>DIRECTORY</type><name>shell</name><mtime>1527331084147</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16403</id><type>DIRECTORY</type><name>awk</name><mtime>1527332686407</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16404</id><type>DIRECTORY</type><name>sed</name><mtime>1527332624472</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16405</id><type>DIRECTORY</type><name>grep</name><mtime>1527332592029</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16406</id><type>FILE</type><name>yinzhengjie.sh</name><replication>3</replication><mtime>1527331084161</mtime><atime>1527331084147</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16409</id><type>FILE</type><name>1.txt</name><replication>3</replication><mtime>1527332587208</mtime><atime>1527332587194</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16410</id><type>FILE</type><name>2.txt</name><replication>3</replication><mtime>1527332592042</mtime><atime>1527332592029</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16411</id><type>FILE</type><name>zabbix.sql</name><replication>3</replication><mtime>1527332604168</mtime><atime>1527332604154</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16412</id><type>FILE</type><name>nagios.sh</name><replication>3</replication><mtime>1527332624486</mtime><atime>1527332624472</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16413</id><type>FILE</type><name>keepalive.sh</name><replication>3</replication><mtime>1527332677350</mtime><atime>1527332677335</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16414</id><type>FILE</type><name>nginx.conf</name><replication>3</replication><mtime>1527332686421</mtime><atime>1527332686407</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter></SnapshotSection>
<INodeDirectorySection><directory><parent>16385</parent><inode>16389</inode><inode>16402</inode><inode>16387</inode></directory>
<directory><parent>16402</parent><inode>16403</inode><inode>16405</inode><inode>16404</inode><inode>16406</inode></directory>
<directory><parent>16403</parent><inode>16413</inode><inode>16414</inode></directory>
<directory><parent>16404</parent><inode>16412</inode><inode>16411</inode></directory>
<directory><parent>16405</parent><inode>16409</inode><inode>16410</inode></directory>
</INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SnapshotDiffSection><diff><inodeid>16385</inodeid></diff></SnapshotDiffSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId></CacheManagerSection>
</fsimage
yinzhengjie.xml 文件内容(推荐看这个版本,用的是XML格式)
技术分享图片
[yinzhengjie@s101 ~]$ cat yinzhengjie.xml 
<?xml version="1.0"?>
<fsimage><NameSection>
<genstampV1>1000</genstampV1><genstampV2>1019</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741839</lastAllocatedBlockId><txid>767</txid></NameSection>
<INodeSection><lastInodeId>16414</lastInodeId><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>1527331031268</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16387</id><type>FILE</type><name>xrsync.sh</name><replication>3</replication><mtime>1527308253459</mtime><atime>1527330550802</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission><blocks><block><id>1073741826</id><genstamp>1002</genstamp><numBytes>700</numBytes></block>
</blocks>
</inode>
<inode><id>16389</id><type>FILE</type><name>hadoop-2.7.3.tar.gz</name><replication>3</replication><mtime>1527310784699</mtime><atime>1527310775186</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission><blocks><block><id>1073741827</id><genstamp>1003</genstamp><numBytes>134217728</numBytes></block>
<block><id>1073741828</id><genstamp>1004</genstamp><numBytes>79874467</numBytes></block>
</blocks>
</inode>
<inode><id>16402</id><type>DIRECTORY</type><name>shell</name><mtime>1527331084147</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16403</id><type>DIRECTORY</type><name>awk</name><mtime>1527332686407</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16404</id><type>DIRECTORY</type><name>sed</name><mtime>1527332624472</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16405</id><type>DIRECTORY</type><name>grep</name><mtime>1527332592029</mtime><permission>yinzhengjie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16406</id><type>FILE</type><name>yinzhengjie.sh</name><replication>3</replication><mtime>1527331084161</mtime><atime>1527331084147</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16409</id><type>FILE</type><name>1.txt</name><replication>3</replication><mtime>1527332587208</mtime><atime>1527332587194</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16410</id><type>FILE</type><name>2.txt</name><replication>3</replication><mtime>1527332592042</mtime><atime>1527332592029</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16411</id><type>FILE</type><name>zabbix.sql</name><replication>3</replication><mtime>1527332604168</mtime><atime>1527332604154</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16412</id><type>FILE</type><name>nagios.sh</name><replication>3</replication><mtime>1527332624486</mtime><atime>1527332624472</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16413</id><type>FILE</type><name>keepalive.sh</name><replication>3</replication><mtime>1527332677350</mtime><atime>1527332677335</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
<inode><id>16414</id><type>FILE</type><name>nginx.conf</name><replication>3</replication><mtime>1527332686421</mtime><atime>1527332686407</atime><perferredBlockSize>134217728</perferredBlockSize><permission>yinzhengjie:supergroup:rw-r--r--</permission></inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter></SnapshotSection>
<INodeDirectorySection><directory><parent>16385</parent><inode>16389</inode><inode>16402</inode><inode>16387</inode></directory>
<directory><parent>16402</parent><inode>16403</inode><inode>16405</inode><inode>16404</inode><inode>16406</inode></directory>
<directory><parent>16403</parent><inode>16413</inode><inode>16414</inode></directory>
<directory><parent>16404</parent><inode>16412</inode><inode>16411</inode></directory>
<directory><parent>16405</parent><inode>16409</inode><inode>16410</inode></directory>
</INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SnapshotDiffSection><diff><inodeid>16385</inodeid></diff></SnapshotDiffSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId></CacheManagerSection>
</fsimage>
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ cat yinzhengjie.xml 

2>.镜像文件的作用

  通过查看上面的XML文件,可以明显的知道镜像文件是存放的是目录结构,文件属性等信息,说到这就不说不提一下镜像文件的md5校验文件了,这个校验文件是为了判断镜像文件是否被修改。fsimage文件是namenode中关于元数据的镜像,一般称为检查点。

技术分享图片

 

二.查看编辑日志文件内容

1>.使用oev命令查询hadoop的编辑日志文件,操作如下:

[yinzhengjie@s101 ~]$ ls -lh /home/yinzhengjie/hadoop/dfs/name/current/ | grep edits | tail -5
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 08:33 edits_0000000000000001001-0000000000000001002
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 08:34 edits_0000000000000001003-0000000000000001004
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 08:35 edits_0000000000000001005-0000000000000001006
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 08:36 edits_0000000000000001007-0000000000000001008
-rw-rw-r--. 1 yinzhengjie yinzhengjie 1.0M May 27 08:36 edits_inprogress_0000000000000001009
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ ll
total 8
drwxrwxr-x. 4 yinzhengjie yinzhengjie   35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie   96 May 25 22:05 shell
-rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ hdfs oev -i ./hadoop/dfs/name/current/edits_0000000000000001007-0000000000000001008 -o edits.xml -p XML
[yinzhengjie@s101 ~]$ ll
total 12
-rw-rw-r--. 1 yinzhengjie yinzhengjie  315 May 27 08:39 edits.xml
drwxrwxr-x. 4 yinzhengjie yinzhengjie   35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie   96 May 25 22:05 shell
-rw-rw-r--. 1 yinzhengjie yinzhengjie 4934 May 27 08:10 yinzhengjie.xml
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ cat edits.xml 
<?xml version="1.0" encoding="UTF-8"?>
<EDITS>
  <EDITS_VERSION>-63</EDITS_VERSION>
  <RECORD>
    <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>1007</TXID>
    </DATA>
  </RECORD>
  <RECORD>
    <OPCODE>OP_END_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>1008</TXID>
    </DATA>
  </RECORD>
</EDITS>
[yinzhengjie@s101 ~]$ 

 2>.查看正在使用的编辑日志文件

  哪个文件是正在使用的编辑日志文件呢?估计你已经看出来了,就是带有“edits_inprogress_*”字样的文件。

技术分享图片

技术分享图片
[yinzhengjie@s101 ~]$ ll
total 0
drwxrwxr-x. 4 yinzhengjie yinzhengjie 35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie 96 May 25 22:05 shell
[yinzhengjie@s101 ~]$ ls -lh /home/yinzhengjie/hadoop/dfs/name/current/ | grep edits | tail -5
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 08:59 edits_0000000000000001053-0000000000000001054
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 09:00 edits_0000000000000001055-0000000000000001056
-rw-rw-r--. 1 yinzhengjie yinzhengjie   42 May 27 09:01 edits_0000000000000001057-0000000000000001058
-rw-rw-r--. 1 yinzhengjie yinzhengjie  912 May 27 09:02 edits_0000000000000001059-0000000000000001071
-rw-rw-r--. 1 yinzhengjie yinzhengjie 1.0M May 27 09:02 edits_inprogress_0000000000000001072
[yinzhengjie@s101 ~]$ hdfs oev -i ./hadoop/dfs/name/current/edits_inprogress_0000000000000001072 -o yinzhengjie.xml -p XML
[yinzhengjie@s101 ~]$ ll
total 4
drwxrwxr-x. 4 yinzhengjie yinzhengjie  35 May 25 19:08 hadoop
drwxrwxr-x. 2 yinzhengjie yinzhengjie  96 May 25 22:05 shell
-rw-rw-r--. 1 yinzhengjie yinzhengjie 205 May 27 09:03 yinzhengjie.xml
[yinzhengjie@s101 ~]$ more yinzhengjie.xml 
<?xml version="1.0" encoding="UTF-8"?>
<EDITS>
  <EDITS_VERSION>-63</EDITS_VERSION>
  <RECORD>
    <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
    <DATA>
      <TXID>1072</TXID>
    </DATA>
  </RECORD>
</EDITS>
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ hdfs oev -i ./hadoop/dfs/name/current/edits_inprogress_0000000000000001072 -o yinzhengjie.xml -p XML

 3>.编辑日志的作用

  顾名思义,编辑日志当然是记录对文件或者目录的编辑信息啦,比如删除目录,修改文件等信息都会被该文件记录。

 

三.手动对编辑日志进行滚动

1>.

 

2>.

Hadoop基础-Hadoop日志滚动

原文:https://www.cnblogs.com/yinzhengjie/p/9098092.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!