首页 > 系统服务 > 详细

Ubuntu及RHEL linux环境下多端口捆绑实战验证(基于模式4和1)

时间:2017-03-03 19:27:32      阅读:468      评论:0      收藏:0      [点我收藏+]

 互联网上关于Linux下多网卡捆绑的配置说明文章和博客一搜就是一大把,但纸上得来终觉浅;加之前几天重装了一台HP DL585机器进行多网卡捆绑时总觉得配置没撒问题就是捆绑不成功;刚好手上有台三千兆网卡的机器,曾昨天下午闲暇时间特意实战测试和验证了一下多端口捆绑模式4和模式1,其它模式及模式间的区别和差异请自行搜索和查阅。

具体实战情况请继续。。。

实战环境(1):

Ubuntu系统环境下

 

1、 网络拓扑结构图如下:

技术分享

 

说明:线上数据分别对应交换机端口Gi1/0/1Gi1/0/2Gi1/0/7Gi1/0/11,默认所有端口均在VLAN1内。

IP 配置规划说明:

C2960S VLAN1 MANAGE IP:192.168.4.1/255.255.255.0

Notebook IP:192.168.4.87/255.255.255.0

ubuntu-desktop eth0 IP:192.168.4.100/255.255.255.0 作为管理IP以便ssh远程连接操作

bond0 (eth1,eth2) IP:192.168.4.200/255.255.255.0

 

2、 Ubuntu系统环境信息:

root@ubuntu-desktop:~# cat /etc/issue

Ubuntu 12.04.4 LTS \n \l

root@ubuntu-desktop:~# uname -r

3.2.0-35-generic

root@ubuntu-desktop:~# uname -a

Linux ubuntu-desktop 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

 

3、 Ubuntu系统配置多网卡bonding

在保障机器可以上网的情况下安装ifenslave(如已经安装则不需要)

root@ubuntu-desktop:~#apt-get install ifenslave

修改接口配置信息如下:

root@ubuntu-desktop:~# cat /etc/network/interfaces

auto lo eth0 eth1 eth2

iface lo inet loopback

iface eth1 inet static

iface eth2 inet static

auto bond0

iface bond0 inet static

address 192.168.4.200

netmask 255.255.255.0

gateway 192.168.4.1

up ifenslave bond0 eth1 eth2

down ifenslave -d bond0 eth1 eth2

加载bonding

root@ubuntu-desktop:~# modprobe bonding

修改bonding模式和检测时间如下:

root@ubuntu-desktop:~# cat /etc/modules

# /etc/modules: kernel modules to load at boot time.

#

# This file contains the names of kernel modules that should be loaded

# at boot time, one per line. Lines beginning with "#" are ignoredn.

bonding mode=4 miimon=100

 

4、 交换机配置信息:

test_sw#sh run

interface Port-channel1

 description link_to_ubuntu_server_bind

 switchport mode access

interface GigabitEthernet1/0/1

 switchport mode access

 channel-group 1 mode active

interface GigabitEthernet1/0/2

 switchport mode access

 channel-group 1 mode active

interface GigabitEthernet1/0/7

 description link_to_notebook

 switchport mode access

interface GigabitEthernet1/0/11

 description link_to_ubuntu_server_mgmt

 switchport mode access

interface Vlan1

 ip address 192.168.4.1 255.255.255.0

test_sw#sh vlan

 

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

1    default                          active    Gi1/0/3, Gi1/0/4, Gi1/0/5

                                                Gi1/0/6, Gi1/0/7, Gi1/0/8

                                                Gi1/0/9, Gi1/0/10, Gi1/0/11

                                                Gi1/0/12, Gi1/0/13, Gi1/0/14

                                                Gi1/0/15, Gi1/0/16, Gi1/0/17

                                                Gi1/0/18, Gi1/0/19, Gi1/0/20

                                                Gi1/0/21, Gi1/0/22, Gi1/0/23

                                                Gi1/0/24, Gi1/0/25, Gi1/0/26

                                                Gi1/0/27, Gi1/0/28, Gi1/0/29

                                                Gi1/0/30, Gi1/0/31, Gi1/0/32

                                                Gi1/0/33, Gi1/0/34, Gi1/0/35

                                                Gi1/0/36, Gi1/0/37, Gi1/0/38

                                                Gi1/0/39, Gi1/0/40, Gi1/0/41

                                                Gi1/0/42, Gi1/0/43, Gi1/0/44

                                                Gi1/0/45, Gi1/0/46, Gi1/0/47

                                                Gi1/0/48, Gi1/0/49, Gi1/0/50

                                                Gi1/0/51, Gi1/0/52, Po1

配置完交换机后重启服务器

root@ubuntu-desktop:~# /etc/init.d/networking restart

 

5、 Ubuntu及交换机端口bonding校验

Ubuntu系统上校验

root@aaron-desktop:~# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

 

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer2 (0)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

 

802.3ad info

LACP rate: slow

Min links: 0

Aggregator selection policy (ad_select): stable

Active Aggregator Info:

        Aggregator ID: 1

        Number of ports: 2

        Actor Key: 17

        Partner Key: 1

        Partner Mac Address: f0:9e:63:f2:ec:80

 

Slave Interface: eth1

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4e

Aggregator ID: 1

Slave queue ID: 0

 

Slave Interface: eth2

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4f

Aggregator ID: 1

Slave queue ID: 0

交换机上校验

test_sw#sh int status    过滤了其它端口信息

Port      Name               Status       Vlan       Duplex  Speed Type

Gi1/0/1                      connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/2                      connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/7   link_to_notebook   connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/11  link_to_ubuntu_ser connected    1          a-full a-1000 10/100/1000BaseTX     

Po1       link_to_ubuntu_ser connected    1          a-full a-1000

 

test_sw#sh int port-channel 1

Port-channel1 is up, line protocol is up (connected)

  Hardware is EtherChannel, address is f09e.63f2.ec82 (bia f09e.63f2.ec82)

  Description: link_to_ubuntu_server_bind

  MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, link type is auto, media type is unknown

  input flow-control is off, output flow-control is unsupported

  Members in this channel: Gi1/0/1 Gi1/0/2

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     1631 packets input, 159398 bytes, 0 no buffer

     Received 216 broadcasts (190 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 190 multicast, 0 pause input

     0 input packets with dribble condition detected

     3020 packets output, 269535 bytes, 0 underruns

     0 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

说明:端口捆绑成功,带宽已提升为2GB,不但增加了带宽而且实现了负载分担功能!

 

网卡捆绑模式1验证及测试(模式1不需要交换机支持)

 

Ubuntu环境下修改捆绑模式:

root@ubuntu-desktop:~# cat /etc/modules

# /etc/modules: kernel modules to load at boot time.

#

# This file contains the names of kernel modules that should be loaded

# at boot time, one per line. Lines beginning with "#" are ignoredn.

bonding mode=1 miimon=100

 

交换机去掉port-channel配置

test_sw#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

test_sw(config-if-range)#no channel-group 1 mode active

test_sw(config-if-range)#end

test_sw#sh run int gi 1/0/1   

Building configuration...

Current configuration : 62 bytes

!

interface GigabitEthernet1/0/1

 switchport mode access

end

test_sw#sh run int gi 1/0/2

Building configuration...

Current configuration : 62 bytes

!

interface GigabitEthernet1/0/2

 switchport mode access

end

test_sw#sh ip int bri

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  192.168.4.1     YES manual up                    up      

FastEthernet0          unassigned      YES NVRAM  down                  down    

GigabitEthernet1/0/1   unassigned      YES unset  up                    up      

GigabitEthernet1/0/2   unassigned      YES unset  up                    up  

 

Ubuntu机器重启,重启后bonding恢复正常!

root@ubuntu-desktop:~# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

 

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth1

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

 

Slave Interface: eth1

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4e

Slave queue ID: 0

 

Slave Interface: eth2

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4f

Slave queue ID: 0

 

root@ubuntu-desktop:~# ifconfig

bond0     Link encap:Ethernet  HWaddr 00:11:0a:57:5f:4e  

          inet addr:192.168.4.200  Bcast:192.168.4.255  Mask:255.255.255.0

          inet6 addr: fe80::211:aff:fe57:5f4e/64 Scope:Link

          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

          RX packets:78 errors:0 dropped:2 overruns:0 frame:0

          TX packets:125 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:7645 (7.6 KB)  TX bytes:18267 (18.2 KB)

eth1      Link encap:Ethernet  HWaddr 00:11:0a:57:5f:4e  

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:76 errors:0 dropped:0 overruns:0 frame:0

          TX packets:125 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:7525 (7.5 KB)  TX bytes:18267 (18.2 KB)

eth2      Link encap:Ethernet  HWaddr 00:11:0a:57:5f:4e  

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:2 errors:0 dropped:2 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:120 (120.0 B)  TX bytes:0 (0.0 B)

接口shutdown测试

test_sw#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

test_sw(config)#int gi 1/0/1

test_sw(config-if)#shut

test_sw(config-if)#

*Mar  1 19:35:25.124: %LINK-5-CHANGED: Interface GigabitEthernet1/0/1, changed state to administratively down

*Mar  1 19:35:26.136: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down

root@ubuntu-desktop:~# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

 

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth1

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

 

Slave Interface: eth1

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4e

Slave queue ID: 0

 

Slave Interface: eth2

MII Status: down

Speed: Unknown

Duplex: Unknown

Link Failure Count: 1

Permanent HW addr: 00:11:0a:57:5f:4f

Slave queue ID: 0

test_sw(config-if)#no shut

*Mar  1 19:36:26.287: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

*Mar  1 19:36:27.289: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

 

Ubuntu端口捆绑恢复正常!

root@ubuntu-desktop:~# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth1

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth1

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4e

Slave queue ID: 0

Slave Interface: eth2

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 1

Permanent HW addr: 00:11:0a:57:5f:4f

Slave queue ID: 0

 

说明:端口模式1捆绑成功,只能提供冗余功能,不能增加带宽和实现负载分担功能!

 

实战环境(2):RHEL 5.7系统环境下

 

rhel5.7 x64环境下端口捆绑基本类似于ubuntu,经实战测试可实现相同功能;现给出RHEL系统下bonding配置信息

[root@RHEL5 ~]# cat /etc/redhat-release

Red Hat Enterprise Linux Server release 5.7 (Tikanga)

[root@RHEL5 ~]# uname -a

Linux RHEL5.7-bond 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@RHEL5 ~]# uname -r

2.6.18-274.el5

[root@RHEL5 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:11:0A:57:5F:4E  

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:11 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:3297 (3.2 KiB)  TX bytes:2052 (2.0 KiB)

 

eth1      Link encap:Ethernet  HWaddr 00:11:0A:57:5F:4F  

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:4 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:987 (987.0 b)  TX bytes:2052 (2.0 KiB)

 

eth2      Link encap:Ethernet  HWaddr C8:60:00:2A:75:99  

          inet addr:192.168.4.100  Bcast:192.168.4.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:127 errors:0 dropped:0 overruns:0 frame:0

          TX packets:123 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:12244 (11.9 KiB)  TX bytes:16505 (16.1 KiB)

          Interrupt:74 Memory:f7c00000-f7c20000

 

lo        Link encap:Local Loopback  

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:5911 errors:0 dropped:0 overruns:0 frame:0

          TX packets:5911 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:8227737 (7.8 MiB)  TX bytes:8227737 (7.8 MiB)

[root@RHEL5 ~]# cd /etc/sysconfig/network-scripts/

[root@RHEL5 network-scripts]# cat ifcfg-bond0

DEVICE=bond0

TYPE=Ethernet

BOOTPROTO=static

ONBOOT=yes

IPADDR=192.168.4.200

NETMASK=255.255.255.0

NETWORK=192.168.4.0

GATEWAY=192.168.4.1

USERCTL=no

[root@RHEL5 network-scripts]# cat ifcfg-eth0

# Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)

DEVICE=eth0

BOOTPROTO=static

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

 

[root@RHEL5 network-scripts]# cat ifcfg-eth1

# Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)

DEVICE=eth1

BOOTPROTO=static

ONBOOT=yes

MASTER=bond0

SLAVE=yes

USERCTL=no

[root@RHEL5 network-scripts]# vi /etc/modprobe.conf

添加:

alias bond0 bonding

options bond0 mode=4 miimon=100

 

交换机配置如上port-channel捆绑配置,略

 

RHEL5.7服务器重启

[root@RHEL5 network-scripts]# service network restart

RHEL5.7服务器重启后,信息校验

[root@RHEL5 ~]# ifconfig

bond0     Link encap:Ethernet  HWaddr 00:11:0A:57:5F:4E  

          inet addr:192.168.4.200  Bcast:192.168.4.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

          RX packets:236 errors:0 dropped:0 overruns:0 frame:0

          TX packets:100 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:25211 (24.6 KiB)  TX bytes:17630 (17.2 KiB)

 

eth0      Link encap:Ethernet  HWaddr 00:11:0A:57:5F:4E  

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:22 errors:0 dropped:0 overruns:0 frame:0

          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:4786 (4.6 KiB)  TX bytes:2962 (2.8 KiB)

 

eth1      Link encap:Ethernet  HWaddr 00:11:0A:57:5F:4E  

          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:215 errors:0 dropped:0 overruns:0 frame:0

          TX packets:90 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:20485 (20.0 KiB)  TX bytes:15556 (15.1 KiB)

 

eth2      Link encap:Ethernet  HWaddr C8:60:00:2A:75:99  

          inet addr:192.168.4.100  Bcast:192.168.4.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:1795 errors:0 dropped:0 overruns:0 frame:0

          TX packets:1379 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:159687 (155.9 KiB)  TX bytes:168819 (164.8 KiB)

          Interrupt:74 Memory:f7c00000-f7c20000

 

lo        Link encap:Local Loopback  

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:5921 errors:0 dropped:0 overruns:0 frame:0

          TX packets:5921 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:8231533 (7.8 MiB)  TX bytes:8231533 (7.8 MiB)

 

[root@RHEL5 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)

 

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer2 (0)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

 

802.3ad info

LACP rate: slow

Active Aggregator Info:

        Aggregator ID: 1

        Number of ports: 2

        Actor Key: 17

        Partner Key: 1

        Partner Mac Address: f0:9e:63:f2:ec:80

 

Slave Interface: eth0

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4e

Aggregator ID: 1

 

Slave Interface: eth1

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:11:0a:57:5f:4f

Aggregator ID: 1

 

test_sw#sh int status

Port      Name               Status       Vlan       Duplex  Speed Type

Gi1/0/1                      connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/2                      connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/7   link_to_notebook   connected    1          a-full a-1000 10/100/1000BaseTX

Gi1/0/11  link_to_ubuntu_ser connected    1          a-full a-1000 10/100/1000BaseTX

Po1       link_to_rhel5.7    connected    1          a-full a-1000

test_sw#sh int port-channel 1

Port-channel1 is up, line protocol is up (connected)

  Hardware is EtherChannel, address is f09e.63f2.ec82 (bia f09e.63f2.ec82)

  Description: link_to_rhel5.7

  MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, link type is auto, media type is unknown

  input flow-control is off, output flow-control is unsupported

  Members in this channel: Gi1/0/1 Gi1/0/2

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:01, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 2000 bits/sec, 1 packets/sec

  5 minute output rate 1000 bits/sec, 2 packets/sec

     7191 packets input, 913837 bytes, 0 no buffer

     Received 4508 broadcasts (4468 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 4468 multicast, 0 pause input

     0 input packets with dribble condition detected

     55261 packets output, 4793216 bytes, 0 underruns

     0 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

本文出自 “Bruce_tan” 博客,请务必保留此出处http://380281.blog.51cto.com/370281/1903058

Ubuntu及RHEL linux环境下多端口捆绑实战验证(基于模式4和1)

原文:http://380281.blog.51cto.com/370281/1903058

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!