Hbase安装与配置_hbase 10102端口

了解Hbase

HBase特性

• 数据容量大，单表可以有百亿行、百万列，数据矩阵横向和纵向两个维度所支持的数据量级都非常具有弹性
• 多版本，每一列存储的数据可以有多个version
• 稀疏性，为空的列并不占用存储空间，表可以设计的非常稀疏
• 读写强一致，非 “最终一致性” 的数据存储，使得它非常适合高速的计算聚合
• 自动分片，通过Region分散在集群中，当行数增长的时候，Region也会自动的切分和再分配
• Hadoop/HDFS集成，和HDFS开箱即用，不用太麻烦的衔接。扩展性强，只需要增加DataNode就可以增加存储空间
• 丰富的“简洁，高效”API，提供了Thrift/REST API，Java API等方式对HBase进行访问
• 块缓存，布隆过滤器，可以高效的列查询优化
• 操作管理，Hbase提供了内置的web界面来操作，还可以监控JMX指标
• 高可靠，保证了系统的容错能力，WAL机制使得数据写入时不会因为集群异常而导致写入数据丢失。故HBase选择了CAP中的CP
• 面向列的存储和权限控制，并支持独立检索，可以动态的增加列列式存储：其数据在表中是按照某列存储的，这样在查询只需要少数几个字段的时候，能大大减少读取的数据量
• 高性能，具备海量数据的随机访问和实时读写能力写方面：底层的 LSM 数据结构和 Rowkey
  有序排列等架构上的独特设计，使得HBase具有非常高的写入性能。读方面：region
  切分、主键索引和缓存机制使得HBase在海量数据下具备一定的随机读取性能，针对 Rowkey 的查询能够达到毫秒级别

综上，HBase是一个高可靠、高性能、面向列、可伸缩的分布式数据库，是谷歌Bigtable的开源实现。主要用来存储非结构化和半结构化的松散数据。HBase的目标是处理非常庞大的表，可以通过水平扩展的方式，利用廉价计算机集群处理由超过10亿行数据和数百万列元素组成的数据表。更多内容详见官方文档。

下载

注：博主使用的Hadoop版本为2.9.2，请注意版本问题。

下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.2.6/

安装

下载完成之后，将压缩包解压到/home/apps/目录下，将解压之后的文件名改为hbase

[root@master dev]# tar - zxvf hbase-2.2.6-bin.tar.gz
[root@master dev]# mkdir - p /home/apps
[root@master dev]# mv hbase-2.2.6 hbase
[root@master dev]# mv hbase /home/apps/
[root@master dev]# cd /home/apps/
[root@master home]# ll
total 12
drwxr-xr-x. 7 root   root   4096 Jan 21 17:13 hbase
drwxr-xr-x. 9 centos centos 4096 Dec 19  2017 sqoop
drwxr-xr-x. 8 root   root   4096 Jan 15 20:58 zookeeper
1
2
3
4
5
6
7
8
9
10

配置

hbase需要配置两个文件hbase-env.sh、hbase-site.xml文件

修改conf目录下面hbase-env.sh文件

[root@master home]# cd conf
[root@master conf]# vi hbase-env.sh
#!/usr/bin/env bash
#
#/**
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements.  See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership.  The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License.  You may obtain a copy of the License at
# *
# *     http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */

# Set environment variables here.

# This script sets variables multiple times over the course of starting an hbase process,
# so try to keep things idempotent unless you want to take an even deeper look
# into the startup scripts (bin/hbase, etc.)

# The java implementation to use.  Java 1.8+ required.
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_221

# Extra Java CLASSPATH elements.  Optional.
# The maximum amount of heap to use. Default is left to JVM default.
# export HBASE_HEAPSIZE=1G

# Uncomment below if you intend to use off heap cache. For example, to allocate 8G of
# offheap, set the value to "8G".
# export HBASE_OFFHEAPSIZE=1G

# Extra Java runtime options.
# Below are what we set by default.  May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://hbase.apache.org/book.html#performance
export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC"

# This enables basic gc logging to the .out file.
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"

# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .

# Uncomment one of the below three options to enable java garbage collection logging for the client processes.

# This enables basic gc logging to the .out file.
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"

# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .

# See the package documentation for org.apache.hadoop.hbase.io.hfile for other configurations
# needed setting up off-heap block caching.

# Uncomment and adjust to enable JMX exporting
# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
# NOTE: HBase provides an alternative JMX implementation to fix the random ports issue, please see JMX
# section in HBase Reference Guide for instructions.

# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"
# export HBASE_REST_OPTS="$HBASE_REST_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10105"

# File naming hosts on which HRegionServers will run.  $HBASE_HOME/conf/regionservers by default.
# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers

# Uncomment and adjust to keep all the Region Server pages mapped to be memory resident
#HBASE_REGIONSERVER_MLOCK=true
#HBASE_REGIONSERVER_UID="hbase"

# File naming hosts on which backup HMaster will run.  $HBASE_HOME/conf/backup-masters by default.
# export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters

# Extra ssh options.  Empty by default.
# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"

# Where log files are stored.  $HBASE_HOME/logs by default.
# export HBASE_LOG_DIR=${HBASE_HOME}/logs

# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
# export HBASE_REST_OPTS="$HBASE_REST_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8074"

# A string representing this instance of hbase. $USER by default.
# export HBASE_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HBASE_NICENESS=10

# The directory where pid files are stored. /tmp by default.
# export HBASE_PID_DIR=/var/hadoop/pids

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HBASE_SLAVE_SLEEP=0.1

# Tell HBase whether it should manage it's own instance of ZooKeeper or not.
export HBASE_MANAGES_ZK=false

# The default log rolling policy is RFA, where the log file is rolled as per the size defined for the
# RFA appender. Please refer to the log4j.properties file to see more details on this appender.
# In case one needs to do log rolling on a date change, one should set the environment property
# HBASE_ROOT_LOGGER to "<DESIRED_LOG LEVEL>,DRFA".
# For example:
# HBASE_ROOT_LOGGER=INFO,DRFA
# The reason for changing default to RFA is to avoid the boundary case of filling out disk space as
# DRFA doesn't put any cap on the log size. Please refer to HBase-5655 for more context.

# Tell HBase whether it should include Hadoop's lib when start up,
# the default value is false,means that includes Hadoop's lib.
# export HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP="true"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

修改hbase-env.sh

export HBASE_MANAGES_ZK=false （hbase有自带的zookeeper，设为false，将不在使用hbase自带的zookeeper）
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_221（jdk安装路径）

修改hbase-site.xml文件

[root@master conf]# vi hbase-site.xml
1

添加以下配置

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:9000/hbase</value>
</property>
<!--单机模式不需要配置，使用集群需要配置此项,值为true,表示为多节点分布-->
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<!-- zookeeper的端口号-->
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
</property>
<!-- 设置备份数 -->
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<!--设置zookeeper各节点，单节点不需要配置多个-->
<property>
    <name>hbase.zookeeper.quorm</name>
    <value>master:2181,slave1:2181,slave2:2181</value>
</property>
<!--设置端口号默认60000-->
<property>
      <name>hbase.master.port</name>
      <value>60000</value>
</property>
<!--指定zookeeper的dataDir路径（与zookeeper中的dataDir路径一样）-->
<property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/home/apps/zookeeper/data</value>
</property>
<!--指定zookeeper的dataLogDir路径（与zookeeper中的dataLogDir路径一样）-->
<property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/home/apps/zookeeper/logs</value>
</property>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

修改环境变量：

[root@master conf]# vi /etc/profile
1

添加配置

export HBASE_HOME=/home/hbase-1.4.13
export PATH=$PATH:$HBASE_HOME/bin
1
2

将环境变量立即生效

[root@master conf]# source /etc/profile
1

启动

注：启动hbase之前需要将Hadoop和zookeeper启动

[root@master ~]# start-hbase.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/apps/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /home/apps/hbase/logs/hbase-root-master-XAA01.out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/apps/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
: running regionserver, logging to /home/apps/hbase/logs/hbase-root-regionserver-XAA01.out
: SLF4J: Class path contains multiple SLF4J bindings.
: SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
: SLF4J: Found binding in [jar:file:/home/apps/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[root@master ~]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

查看是否启动成功

[root@master ~]# jps
4050 NameNode
12804 QuorumPeerMain
9141 HRegionServer
4503 SecondaryNameNode
4248 DataNode
9736 Jps
8985 HMaster
4892 NodeManager
4751 ResourceManager
1
2
3
4
5
6
7
8
9
10

出现HRegionServer，启动成功

结语：大数据Hadoop笔记 Hbase 安装与配置