荣耀彩票代理

IT技术互动交流平台

Hbase架构与理

来源:IT165收集  发布日期:2016-07-19 21:07:33

HBaseSHIApache HadoopZHONGDEYIGEZIXIANGMU,HbaseYITUOYUHadoopDEHDFSZUOWEIZUIJIBENCUNCHUJICHUDANYUAN,TONGGUOSHIYONGhadoopDEDFSGONGJUJIUKEYIKANDAOZHEIXIEZHEIXIESHUJUCUNCHUWENJIANJIADEJIEGOU,HAIKEYITONGGUOMap/ReduceDEKUANGJIA(SUANFA)DUIHBaseJINXINGCAOZUO

YI、 hbaseJIAGOU

1.概述。

HBaseSHIApache HadoopDESHUJUKU,NENGGOUDUIDAXINGSHUJUTIGONGSUIJI、SHISHIDEDUXIEFANGWEN。HBaseDEMUBIAOSHICUNCHUBINGCHULIDAXINGDESHUJU。HBaseSHIYIGEKAIYUANDE,FENBUSHIDE,DUOBANBENDE,MIANXIANGLIEDECUNCHUMOXING。TACUNCHUDESHISONGSANXINGSHUJU。

SHANGTUSHIhadoopDESHENGTAIXITONGMIAOSHU,hadoopSUOYOUYINGYONGDOUSHIGOUJIANYUhdfs(TATIGONGGAOKEKAODEDICENGCUNCHUZHICHI,JIHUYIJINGCHENGWEIFENBUSHIWENJIANCUNCHUXITONGSHISHISHANGDEGONGYEBIAOZHUN)ZHISHANGDEFENBUSHILIECUNCHUXITONG,ZHUYAOYONGYUHAILIANGJIEGOUHUASHUJUCUNCHU。

荣耀彩票代理HBaseSHIYIZHONGNoSQLSHUJUKU. NoSQLSHIYIGETONGYONGCIBIAOSHISHUJUKUBUSHIRDBMS ,HOUZHEZHICHI SQL ZUOWEIZHUYAOFANGWENSHOUDUAN。YOUXUDUOZHONG NoSQL SHUJUKU: BerkeleyDB SHIBENDI NoSQL SHUJUKULIZI, ER HBase SHIDAXINGFENBUSHISHUJUKU。 JISHUSHANGLAISHUO, HBase GENGXIANGSHI"SHUJUCUNCHU(Data Store)" DUOYU "SHUJUKU(Data Base)"。YINWEIQUESHAOHENDUORDBMSTEXING, RULIELEIXING,DIERSUOYIN,CHUFAQI,GAOJICHAXUNYUYANDENG

RANER, HBase YOUXUDUOTEZHENGTONGSHIZHICHIXIANXINGHUAHEMOKUAIHUAKUOCHONG。 HBase JIQUNTONGGUOZENGJIARegionServersJINXINGKUOCHONG。 TAKEYIFANGZAIPUTONGDEFUWUQIZHONG。LIRU,RUGUOJIQUNCONG10GEKUOCHONGDAO20GERegionServer,CUNCHUKONGJIANHECHULIRONGLIANGDOUTONGSHIFANBEI。 RDBMS YENENGHENHAOKUOCHONG, DANJINDUIYIGEDIAN - TEBIESHIDUIYIGEDANDUSHUJUKUFUWUQIDEDAXIAO - TONGSHI,WEILEGENGHAODEXINGNENG,XUYAOTESHUDEYINGJIANHECUNCHUSHEBEI。HbaseTEXING:

荣耀彩票代理QIANGYIZHIXINGDUXIE: HBase BUSHI "ZUIZHONGYIZHIXING(eventually consistent)" SHUJUCUNCHU. ZHEIRANGTAHENSHIHEGAOSUJISHUJUHELEIRENWU。

荣耀彩票代理ZIDONGFENPIAN(Automatic sharding):HBase BIAOTONGGUOregionFENBUZAIJIQUNZHONG。SHUJUZENGZHANGSHI,regionHUIZIDONGFENGEBINGZHONGXINFENBU。

荣耀彩票代理RegionServer ZIDONGGUZHANGZHUANYI

荣耀彩票代理Hadoop/HDFS JICHENG: HBase ZHICHIBENJIWAIHDFS ZUOWEITADEFENBUSHIWENJIANXITONG。

MapReduce: HBase TONGGUOMapReduceZHICHIDABINGFACHULI, HBase KEYITONGSHIZUOYUANHEMUBIAO.

Java KEHUDUAN API: HBase ZHICHIYIYUSHIYONGDE Java API JINXINGBIANCHENGFANGWEN.

Thrift/REST API:HBase YEZHICHIThriftHE REST ZUOWEIFEIJava QIANDUAN.

荣耀彩票代理Block Cache HE Bloom Filters: DUIYUDARONGLIANGCHAXUNYOUHUA, HBaseZHICHI Block Cache HE Bloom Filters。

YUNWEIGUANLI: HBaseTIGONGNEIZHIWANGYEYONGYUYUNWEISHIJIAOHEJMX DULIANG.

荣耀彩票代理QIANWENTIDAOHbaseSHIYIGELIESHICUNCHUDESHUJUKU,NEIMESHENMESHILIESHICUNCHU,TAYUCHUANTONGDERDBMSCAIYONGDEXINGSHICUNCHUYOUYOUSHENMEQUBIE?LIECUNCHUBUTONGYUCHUANTONGDEGUANXIXINGSHUJUKU,QISHUJUZAIBIAOZHONGSHIANXINGCUNCHUDE,LIEFANGSHISUODAILAIDEZHONGYAOHAOCHUZHIYIJIUSHI,YOUYUCHAXUNZHONGDEXUANZEGUIZESHITONGGUOLIELAIDINGYIDE,YINCIZHENGGESHUJUKUSHIZIDONGSUOYINHUADE。ANLIECUNCHUMEIGEZIDUANDESHUJUJUJICUNCHU,ZAICHAXUNZHIXUYAOSHAOSHUJIGEZIDUANDESHIHOU,NENGDADAJIANSHAODUQUDESHUJULIANG,YIGEZIDUANDESHUJUJUJICUNCHU,NEIJIUGENGRONGYIWEIZHEIZHONGJUJICUNCHUSHEJIGENGHAODEYASUO/JIEYASUANFA。ZHEIZHANGTUJIANGSHULECHUANTONGDEXINGCUNCHUHELIECUNCHUDEQUBIE:

2.hbase架构

ZHU:ZHUNQUEDESHUOWEIYUSHANGTUXIABANBUFENDEZUJIANYINGGAISHIhdfsERFEIhadoop,hbaseBINGBUYILAIYUhadoop,DANSHITAGOUJIANYUhdfsZHISHANG。

Zookeeper:
Zookeeper Quorum存储-ROOT-表地址、HMaster地址
HRegionServer把自己以Ephedral方式注册到Zookeeper中,HMaster随时感知各个HRegionServer的健康状况
Zookeeper避免HMaster单点问题
HMaster
HMaster没有单点问题,HBase中可以启动多个HMaster,通过Zookeeper的MasterElection机制保证总有一个Master在运行
主要负责Table和Region的管理工作:
1 管理用户对表的增删改查操作
2 管理HRegionServer的负载均衡,调整Region分布
3 Region Split后,负责新Region的分布
荣耀彩票代理 4 在HRegionServer停机后,负责失效HRegionServer上Region迁移

HRegionServer:
荣耀彩票代理 HBase中最核心的模块,主要负责响应用户I/O请求,向HDFS文件系统中读写数据

HRegionServer管理一些列HRegion对象;
每个HRegion对应Table中一个Region,HRegion由多个HStore组成;
每个HStore对应Table中一个Column Family的存储;
荣耀彩票代理 Column Family就是一个集中的存储单元,故将具有相同IO特性的Column放在一个Column Family会更高效

HStore:
HBase存储的核心。由MemStore和StoreFile组成。
荣耀彩票代理 MemStore是Sorted Memory Buffer。用户写入数据的流程:

Client写入 -> 存入MemStore,一直到MemStore满 -> Flush成一个StoreFile,直至增长到一定阈值 -> 出发Compact合并操作 -> 多个StoreFile合并成一个StoreFile,同时进行版本合并和数据删除 -> 当StoreFiles Compact后,逐步形成越来越大的StoreFile ->单个StoreFile大小超过一定阈值后,触发Split操作,把当前Region Split成2个Region,Region会下线,新Split出的2个孩子Region会被HMaster分配到相应的HRegionServer 上,使得原先1个Region的压力得以分流到2个Region上
由此过程可知,HBase只是增加数据,所有的更新和删除操作,都是在Compact阶段做的,所以,用户写操作只需要进入到内存即可立即返回,从而保证I/O高性能。

HLog
引入HLog原因:
在分布式系统环境中,无法避免系统出错或者宕机,一旦HRegionServer意外退出,MemStore中的内存数据就会丢失,引入HLog就是防止这种情况
工作机制:
每 个HRegionServer中都会有一个HLog对象,HLog是一个实现Write Ahead Log的类,每次用户操作写入Memstore的同时,也会写一份数据到HLog文件,HLog文件定期会滚动出新,并删除旧的文件(已持久化到 StoreFile中的数据)。当HRegionServer意外终止后,HMaster会通过Zookeeper感知,HMaster首先处理遗留的 HLog文件,将不同region的log数据拆分,分别放到相应region目录下,然后再将失效的region重新分配,领取到这些region的 HRegionServer在Load Region的过程中,会发现有历史HLog需要处理,因此会Replay HLog中的数据到MemStore中,然后flush到StoreFiles,完成数据恢复。

HBase存储格式
HBase中的所有数据文件都存储在Hadoop HDFS文件系统上,格式主要有两种:
1 HFile HBase中KeyValue数据的存储格式,HFile是Hadoop的二进制格式文件,实际上StoreFile就是对HFile做了轻量级包装,即StoreFile底层就是HFile
2 HLog File,HBase中WAL(Write Ahead Log) 的存储格式,物理上是Hadoop的Sequence File

HFile

HFile文件不定长,长度固定的块只有两个:Trailer和FileInfo
Trailer中指针指向其他数据块的起始点
File Info中记录了文件的一些Meta信息,例如:AVG_KEY_LEN,AVG_VALUE_LEN, LAST_KEY, COMPARATOR, MAX_SEQ_ID_KEY等
Data Index和Meta Index块记录了每个Data块和Meta块的起始点
Data Block是HBase I/O的基本单元,为了提高效率,HRegionServer中有基于LRU的Block Cache机制
每个Data块的大小可以在创建一个Table的时候通过参数指定,大号的Block有利于顺序Scan,小号Block利于随机查询
每个Data块除了开头的Magic以外就是一个个KeyValue对拼接而成, Magic内容就是一些随机数字,目的是防止数据损坏

HFileLIMIANDEMEIGEKeyValueDUIJIUSHIYIGEJIANDANDEbyteSHUZU。ZHEIGEbyteSHUZULIMIANBAOHANLEHENDUOXIANG,BINGQIEYOUGUDINGDEJIEGOU。

KeyLength和ValueLength:两个固定的长度,分别代表Key和Value的长度
Key部分:Row Length是固定长度的数值,表示RowKey的长度,Row 就是RowKey
Column Family Length是固定长度的数值,表示Family的长度
接着就是Column Family,再接着是Qualifier,然后是两个固定长度的数值,表示Time Stamp和Key Type(Put/Delete)
Value部分没有这么复杂的结构,就是纯粹的二进制数据

HLog文件就是一个普通的Hadoop Sequence File,Sequence File 的Key是HLogKey对象,HLogKey中记录了写入数据的归属信息,除了table和region名字外,同时还包括 sequence number和timestamp,timestamp是“写入时间”,sequence number的起始值为0,或者是最近一次存入文件系统中sequence number。
荣耀彩票代理 HLog Sequece File的Value是HBase的KeyValue对象,即对应HFile中的KeyValue

3.什么时候应该使用hbase

ZAIXUEXIYIMENXINJISHUDESHIHOUSHOUXIANXUYAOMINGBAI,WOMENDAODIYINGGAIZAISHENMESHIHOUSHIYONGTA,RANHOUCAISHIZENMEQUSHIYONGTA,HbaseBINGBUSHIHESUOYOUWENTI。

SHOUXIAN,QUEXINYOUZUGOUDUOSHUJU,RUGUOYOUSHANGYIHUOSHANGQIANYIXINGSHUJU,HBaseSHIHENHAODEBEIXUAN。RUGUOZHIYOUSHANGQIANHUOSHANGBAIWANXING,ZEYONGCHUANTONGDERDBMSKENENGSHIGENGHAODEXUANZE。YINWEISUOYOUSHUJUKEYIZAIYILIANGGEJIEDIANBAOCUN,JIQUNQITAJIEDIANKENENGXIANZHI。

QICI,QUEXINKEYIBUYILAISUOYOURDBMSDEEWAITEXING (e.g., LIESHUJULEIXING, DIERSUOYIN,SHIWU,GAOJICHAXUNYUYANDENG.) YIGEJIANLIZAIRDBMSSHANGYINGYONG,RUBUNENGJINTONGGUOGAIBIANYIGEJDBCQUDONGYIZHIDAOHBase。XIANGDUIYUYIZHI, XUKAOLVCONGRDBMS DAO HBaseSHIYICIWANQUANDEZHONGXINSHEJI。

荣耀彩票代理DISAN, QUEXINNIYOUZUGOUYINGJIAN。SHENZHI HDFS ZAIXIAOYU5GESHUJUJIEDIANSHI,GANBUHAOSHENMESHIQING (GENJURUHDFS KUAIFUZHIJUYOUQUESHENGZHI 3), HAIYAOJIASHANGYIGENameNode.

荣耀彩票代理HBase NENGZAIDANDUDEBIJIBENSHANGYUNXINGLIANGHAO。DANZHEIYINGJINDANGCHENGKAIFAPEIZHI。

4.目录表(.meta.和-root-)

-ROOT- BAOCUN .META. BIAOCUNZAINALIDEZONGJI. -ROOT- BIAOJIEGOURUXIA:

Key:

.META. region key (.META.,,1)

Values:

info:regioninfo (XULIEHUA.META.DE HRegionInfo SHILI )

荣耀彩票代理info:server ( BAOCUN .META.DERegionServerDEserver:port)

info:serverstartcode ( BAOCUN .META.DERegionServerJINCHENGDEQIDONGSHIJIAN)

.META. BAOCUNXITONGZHONGSUOYOUregionLIEBIAO。 .META.BIAOJIEGOURUXIA:

Key:

Region key GESHI ([table],[region start key],[region id])

Values:

info:regioninfo (XULIEHUA.META.DE HRegionInfo SHILI )

info:server ( BAOCUN .META.DERegionServerDEserver:port)

info:serverstartcode ( BAOCUN .META.DERegionServerJINCHENGDEQIDONGSHIJIAN)

YISHANGSHIGUANWANGWENDANGDUIYU.meta.HE-root-DEMIAOSHU,JIANERYANZHI,-root-ZHONGCUNCHULE.meta.DEWEIZHI,ERZAI.meta.ZHONGBAOCUNLEJUTISHUJU(region)DECUNCHUWEIZHI。RUTU:

Zookeeper中记录了-ROOT-表的location
客户端访问数据的流程:
Client -> Zookeeper -> -ROOT- -> .META.-> 用户数据表
多次网络操作,不过client端有cache缓存

a. QIDONGSHIXU

1.QIDONGSHIZHUFUWUQIDIAOYONGAssignmentManager.

2.AssignmentManager ZAIMETAZHONGCHAZHAOYIJINGCUNZAIDEQUYUFENPEI。

荣耀彩票代理3.RUGUOQUYUFENPEIHAIYOUXIAO(RU RegionServer HAIZAIXIAN),NEIMEFENPEIJIXUBAOCHI。

4.RUGUOQUYUFENPEISHIXIAO,LoadBalancerFactory BEIDIAOYONGLAIFENPEIQUYU。 DefaultLoadBalancer JIANGSUIJIFENPEIQUYUDAORegionServer.

5.META SUIRegionServer FENPEIGENGXIN(RUGUOXUYAO) , RegionServer QIDONGQUYUKAIQIDAIMA(RegionServer QIDONGSHIJINCHENG)

b.GUZHANGZHUANYI

DANGregionServerGUZHANGTUICHUSHI:

荣耀彩票代理1.QUYULIJIBUKEHUOQU,YINWEIQUYUFUWUQITUICHU。

2.ZHUFUWUQIHUIJIANCEDAOQUYUFUWUQITUICHU。

荣耀彩票代理3.QUYUFENPEIHUISHIXIAOBINGBEIZHONGXINFENPEI,RUTONGQIDONGSHIXU。

5.预写日志(wal)

MEIGERegionServerHUIJIANGGENGXIN(Puts, Deletes)XIANJILUDAOYUXIERIZHIZHONG(WAL),RANHOUJIANGQIGENGXINZAIStoreDEMemStoreLIMIAN。ZHEIYANGJIUBAOZHENGLEHBaseDEXIEDEKEKAOXING。RUGUOMEIYOUWAL,DANGRegionServerDANGDIAODESHIHOU,MemStoreHAIMEIYOUflush,StoreFileHAIMEIYOUBAOCUN,SHUJUJIUHUIDIUSHI。HLog SHIHBaseDEYIGEWALSHIXIAN,YIGERegionServerYOUYIGEHLogSHILI。

荣耀彩票代理WAL BAOCUNZAIHDFS DE /hbase/.logs/ LIMIAN,MEIGEregionYIGEWENJIAN。

ER、 SHUJUMOXING

1.概念视图

YIbigTableLUNWENZHONGDELIZILAISHUOMING,YOUYIGEMINGWEIwebtableDEBIAO,BAOHANLIANGGELIEZU:contentsHEanchor.ZAIZHEIGELIZILIMIAN,anchorYOULIANGGELIE (anchor:cssnsi.com,anchor:my.look.ca),contentsJINYOUYILIE(contents:html)

Row Key

Time Stamp

ColumnFamily contents

ColumnFamily anchor

"com.cnn.www"

t9

 

anchor:cnnsi.com = "CNN"

荣耀彩票代理"com.cnn.www"

t8

 

荣耀彩票代理anchor:my.look.ca = "CNN.com"

"com.cnn.www"

t6

contents:html = "<html>..."

 

"com.cnn.www"

t5

荣耀彩票代理contents:html = "<html>..."

 

"com.cnn.www"

t3

contents:html = "<html>..."

 

RowKey:行键,是表中每条记录的“主键”,方便快速查找,Rowkey的设计非常重要。
Column Family:列族,拥有一个名称(string),包含一个或者多个相关列
Column:属于某一个columnfamily,每条记录可动态添加
Version Number:类型为Long,默认值是系统时间戳,可由用户自定义
Value(Cell):一个cell由familyName:columnName唯一定义

2.物理视图

JINGUANZAIGAINIANSHITULI,BIAOKEYIBEIKANCHENGSHIYIGEXISHUDEXINGDEJIHE。DANZAIWULISHANG,TADESHIQUFENLIEZU CUNCHUDE。XINDEcolumnsKEYIBUJINGGUOSHENGMINGZHIJIEJIARUYIGELIEZU.

Row Key

Time Stamp

Column Family anchor

"com.cnn.www"

t9

anchor:cnnsi.com = "CNN"

荣耀彩票代理"com.cnn.www"

t8

anchor:my.look.ca = "CNN.com"

 

Row Key

Time Stamp

ColumnFamily "contents:"

"com.cnn.www"

t6

contents:html = "<html>..."

荣耀彩票代理"com.cnn.www"

t5

contents:html = "<html>..."

"com.cnn.www"

t3

荣耀彩票代理contents:html = "<html>..."

荣耀彩票代理ZHIDEZHUYIDESHIZAISHANGMIANDEGAINIANSHITUZHONGKONGBAIcellZAIWULISHANGSHIBUCUNCHUDE,YINWEIGENBENMEIYOUBIYAOCUNCHU。YINCIRUOYIGEQINGQIUWEIYAOHUOQUt8SHIJIANDEcontents:html,TADEJIEGUOJIUSHIKONG。XIANGSIDE,RUOQINGQIUWEIHUOQUt9SHIJIANDEanchor:my.look.ca,JIEGUOYESHIKONG。DANSHI,RUGUOBUZHIMINGSHIJIAN,JIANGHUIFANHUIZUIXINSHIJIANDEXING,MEIGEZUIXINDEDOUHUIFANHUI。LIRU,RUGUOQINGQIUWEIHUOQUXINGJIANWEI"com.cnn.www",MEIYOUZHIMINGSHIJIANCHUODEHUA,HUODONGDEJIEGUOSHIt6XIADEcontents:html,t9XIADEanchor:cnnsi.comHEt8XIAanchor:my.look.ca。

荣耀彩票代理DUIYUhbaseWOYIZHIYOUYIGEYIWEN,ZAIhbaseTIGONGLEXIUGAIHESHANCHUDEJIEKOU,DANSHIhdfsBENSHENHENNANSHIXIANXIUGAIHESHANCHU(KEYIJIANGWENJIANKUAICONGhdfsZHONGXIAZAI,JINXINGXIUGAIZAISHANGCHUAN),NEIMEhbaseSHIRUHESHIXIANKUAISUDESHANCHUYUXIUGAINI?SHIJISHANGZAIHBaseZHONG,XIUGAIHESHANCHUSHUJUDOUSHIZENGJIA1GEXINBANBENDESHUJU(SHIJIANCHUOWEIZUIXIN),JIUBANBENDESHUJUBINGMEIYOUFASHENGBIANHUA,ERSHIJISHANGDEXIUGAIHESHANCHUSHIZAIHfileDEHEBINGJIEDUANSHIXIANDE。

SAN、 HbaseYOUHUA

1. 预先分区

MORENQINGKUANGXIA,ZAICHUANGJIAN HBase BIAODESHIHOUHUIZIDONGCHUANGJIANYIGE Region FENQU,DANGDAORUSHUJUDESHIHOU,SUOYOUDE HBase KEHUDUANDOUXIANGZHEIYIGE Region XIESHUJU,ZHIDAOZHEIGE Region ZUGOUDALECAIJINXINGQIEFEN。YIZHONGKEYIJIAKUAIPILIANGXIERUSUDUDEFANGFASHITONGGUOYUXIANCHUANGJIANYIXIEKONGDE Regions,ZHEIYANGDANGSHUJUXIERU HBase SHI,HUIANZHAO Region FENQUQINGKUANG,ZAIJIQUNNEIZUOSHUJUDEFUZAIJUNHENG。

2. Rowkey优化

HBase ZHONG Rowkey SHIANZHAOZIDIANXUCUNCHU,YINCI,SHEJI Rowkey SHI,YAOCHONGFENLIYONGPAIXUTEDIAN,JIANGJINGCHANGYIQIDUQUDESHUJUCUNCHUDAOYIKUAI,JIANGZUIJINKENENGHUIBEIFANGWENDESHUJUFANGZAIYIKUAI。

荣耀彩票代理CIWAI,Rowkey RUOSHIDIZENGDESHENGCHENG,JIANYIBUYAOSHIYONGZHENGXUZHIJIEXIERU Rowkey,ERSHICAIYONG reverse DEFANGSHIFANZHUANRowkey,SHIDE Rowkey DAZHIJUNHENGFENBU,ZHEIYANGSHEJIYOUGEHAOCHUSHINENGJIANG RegionServer DEFUZAIJUNHENG,FOUZERONGYICHANSHENGSUOYOUXINSHUJUDOUZAIYIGE RegionServer SHANGDUIJIDEXIANXIANG,ZHEIYIDIANHAIKEYIJIEHE table DEYUQIEFENYIQISHEJI。

3. 减少列族数量

BUYAOZAIYIZHANGBIAOLIDINGYITAIDUODE ColumnFamily。MUQIAN Hbase BINGBUNENGHENHAODECHULICHAOGUO 2~3 GE ColumnFamily DEBIAO。YINWEIMOUGE ColumnFamily ZAI flush DESHIHOU,TALINJINDE ColumnFamily YEHUIYINGUANLIANXIAOYINGBEICHUFA flush,ZUIZHONGDAOZHIXITONGCHANSHENGGENGDUODE I/O。

4. 缓存策略

CHUANGJIANBIAODESHIHOU,KEYITONGGUO HColumnDescriptor.setInMemory(true) JIANGBIAOFANGDAO RegionServer DEHUANCUNZHONG,BAOZHENGZAIDUQUDESHIHOUBEI cache MINGZHONG。

5. 设置存储生命期

荣耀彩票代理CHUANGJIANBIAODESHIHOU,KEYITONGGUO HColumnDescriptor.setTimeToLive(int timeToLive) SHEZHIBIAOZHONGSHUJUDECUNCHUSHENGMINGQI,GUOQISHUJUJIANGZIDONGBEISHANCHU。

6. 硬盘配置

MEITAI RegionServer GUANLI 10~1000 GE Regions,MEIGE Region ZAI 1~2G,ZEMEITAI Server ZUISHAOYAO 10G,ZUIDAYAO1000*2G=2TB,KAOLV 3 BEIFEN,ZEYAO 6TB。FANGANYISHIYONG 3 KUAI 2TB YINGPAN,ERSHIYONG 12 KUAI 500G YINGPAN,DAIKUANZUGOUSHI,HOUZHENENGTIGONGGENGDADETUNTULV,GENGXILIDUDERONGYUBEIFEN,GENGKUAISUDEDANPANGUZHANGHUIFU。

7. 分配合适的内存给RegionServer服务

ZAIBUYINGXIANGQITAFUWUDEQINGKUANGXIA,YUEDAYUEHAO。LIRUZAI HBase DE conf MULUXIADE hbase-env.sh DEZUIHOUTIANJIA export HBASE_REGIONSERVER_OPTS="-Xmx16000m$HBASE_REGIONSERVER_OPTS”

荣耀彩票代理QIZHONG 16000m WEIFENPEIJI RegionServer DENEICUNDAXIAO。

8. 写数据的备份数

BEIFENSHUYUDUXINGNENGCHENGZHENGBI,YUXIEXINGNENGCHENGFANBI,QIEBEIFENSHUYINGXIANGGAOKEYONGXING。YOULIANGZHONGPEIZHIFANGSHI,YIZHONGSHIJIANG hdfs-site.xmlKAOBEIDAO hbase DE conf MULUXIA,RANHOUZAIQIZHONGTIANJIAHUOXIUGAIPEIZHIXIANG dfs.replication DEZHIWEIYAOSHEZHIDEBEIFENSHU,ZHEIZHONGXIUGAIDUISUOYOUDE HBase YONGHUBIAODOUSHENGXIAO,LINGWAIYIZHONGFANGSHI,SHIGAIXIE HBase DAIMA,RANG HBase ZHICHIZHENDUILIEZUSHEZHIBEIFENSHU,ZAICHUANGJIANBIAOSHI,SHEZHILIEZUBEIFENSHU,MORENWEI 3,CIZHONGBEIFENSHUZHIDUISHEZHIDELIEZUSHENGXIAO。

9. WAL(预写日志)

KESHEZHIKAIGUAN,BIAOSHI HBase ZAIXIESHUJUQIANYONGBUYONGXIANXIERIZHI,MORENSHIDAKAI,GUANDIAOHUITIGAOXINGNENG,DANSHIRUGUOXITONGCHUXIANGUZHANG(FUZECHARUDE RegionServer GUADIAO),SHUJUKENENGHUIDIUSHI。PEIZHI WAL ZAIDIAOYONG JavaAPI XIERUSHI,SHEZHI Put SHILIDEWAL,DIAOYONG Put.setWriteToWAL(boolean)。

10. 批量写

荣耀彩票代理HBase DE Put ZHICHIDANTIAOCHARU,YEZHICHIPILIANGCHARU,YIBANLAISHUOPILIANGXIEGENGKUAI,JIESHENGLAIHUIDEWANGLUOKAIXIAO。ZAIKEHUDUANDIAOYONGJavaAPI SHI,XIANJIANGPILIANGDE Put FANGRUYIGE Put LIEBIAO,RANHOUDIAOYONG HTable DE Put(Put LIEBIAO) HANSHULAIPILIANGXIE。

11. 客户端一次从服务器拉取的数量

荣耀彩票代理TONGGUOPEIZHIYICILAQUDEJIAODADESHUJULIANGKEYIJIANSHAOKEHUDUANHUOQUSHUJUDESHIJIAN,DANSHITAHUIZHANYONGKEHUDUANNEICUN。YOUSANGEDIFANGKEJINXINGPEIZHI:

荣耀彩票代理1)ZAI HBase DE conf PEIZHIWENJIANZHONGJINXINGPEIZHI hbase.client.scanner.caching;

2)TONGGUODIAOYONG HTable.setScannerCaching(intscannerCaching) JINXINGPEIZHI;

荣耀彩票代理3)TONGGUODIAOYONG Scan.setCaching(intcaching) JINXINGPEIZHI。SANZHEDEYOUXIANJIYUELAIYUEGAO。

12. RegionServer的请求处理I/O线程数

JIAOSHAODE IO XIANCHENGSHIYONGYUCHULIDANCIQINGQIUNEICUNXIAOHAOJIAOGAODE Big Put CHANGJING (DARONGLIANGDANCI Put HUOSHEZHILEJIAODA cache DEScan,JUNSHUYU Big Put) HUO ReigonServer DENEICUNBIJIAOJINZHANGDECHANGJING。

JIAODUODE IO XIANCHENG,SHIYONGYUDANCIQINGQIUNEICUNXIAOHAODI,TPS YAOQIU (MEIMIAOSHIWUCHULILIANG (TransactionPerSecond)) FEICHANGGAODECHANGJING。SHEZHIGAIZHIDESHIHOU,YIJIANKONGNEICUNWEIZHUYAOCANKAO。

荣耀彩票代理ZAI hbase-site.xml PEIZHIWENJIANZHONGPEIZHIXIANGWEI hbase.regionserver.handler.count。

13. Region的大小设置

PEIZHIXIANGWEI hbase.hregion.max.filesize,SUOSHUPEIZHIWENJIANWEI hbase-site.xml.,MORENDAXIAO 256M。

ZAIDANGQIAN ReigonServer SHANGDANGE Reigon DEZUIDACUNCHUKONGJIAN,DANGE Region CHAOGUOGAIZHISHI,ZHEIGE Region HUIBEIZIDONG splitCHENGGENGXIAODE Region。XIAO Region DUI split HE compaction YOUHAO,YINWEICHAIFEN Region HUO compact XIAO Region LIDEStoreFile SUDUHENKUAI,NEICUNZHANYONGDI。QUEDIANSHI split HE compaction HUIHENPINFAN,TEBIESHISHULIANGJIAODUODEXIAO Region BUTINGDIsplit, compaction,HUIDAOZHIJIQUNXIANGYINGSHIJIANBODONGHENDA,Region SHULIANGTAIDUOBUJINJIGUANLISHANGDAILAIMAFAN,SHENZHIHUIYINFAYIXIEHbase DE bug。YIBAN 512M YIXIADEDOUSUANXIAO Region。DA Region ZEBUTAISHIHEJINGCHANG split HE compaction,YINWEIZUOYICI compact HE split HUICHANSHENGJIAOZHANGSHIJIANDETINGDUN,DUIYINGYONGDEDUXIEXINGNENGCHONGJIFEICHANGDA。

CIWAI,DA Region YIWEIZHEJIAODADE StoreFile,compaction SHIDUINEICUNYESHIYIGETIAOZHAN。RUGUONIDEYINGYONGCHANGJINGZHONG,MOUGESHIJIANDIANDEFANGWENLIANGJIAODI,NEIMEZAICISHIZUO compact HE split,JINENGSHUNLIWANCHENG split HE compaction,YOUNENGBAOZHENGJUEDADUOSHUSHIJIANPINGWENDEDUXIEXINGNENG。compaction SHIWUFABIMIANDE,split KEYICONGZIDONGDIAOZHENGWEISHOUDONG。ZHIYAOTONGGUOJIANGZHEIGECANSHUZHIDIAODADAOMOUGEHENNANDADAODEZHI,BIRU 100G,JIUKEYIJIANJIEJINYONGZIDONG split(RegionServer BUHUIDUIWEIDAODA 100G DE Region ZUOsplit)。ZAIPEIHE RegionSplitter ZHEIGEGONGJU,ZAIXUYAO split SHI,SHOUDONG split。SHOUDONG split ZAILINGHUOXINGHEWENDINGXINGSHANGBIQIZIDONGsplit YAOGAOHENDUO,ERQIEGUANLICHENGBENZENGJIABUDUO,BIJIAOTUIJIAN online SHISHIXITONGSHIYONG。NEICUNFANGMIAN,XIAO Region ZAISHEZHImemstore DEDAXIAOZHISHANGBIJIAOLINGHUO,DA Region ZEGUODAGUOXIAODOUBUXING,GUODAHUIDAOZHI flush SHI app DE IO wait ZENGGAO,GUOXIAOZEYIN StoreFile GUODUOYINGXIANGDUXINGNENG。

14. 操作系统参数

Linux荣耀彩票代理系统最大可打开文件数一般默认的参数值是1024,如果你不进行修改并发量上来的时候会出现“Too Many Open Files”的错误,导致整个HBase不可运行,你可以用ulimit -n 命令进行修改,或者修改/etc/security/limits.conf和/proc/sys/fs/file-max 的参数,具体如何修改可以去Google 关键字 “linux limits.conf ”

15. Jvm配置

XIUGAI hbase-env.sh WENJIANZHONGDEPEIZHICANSHU,GENJUNIDEJIQIYINGJIANHEDANGQIANCAOZUOXITONGDEJVM(32/64WEI)PEIZHISHIDANGDECANSHU

HBASE_HEAPSIZE 4000 HBaseSHIYONGDE JVM DUIDEDAXIAO

荣耀彩票代理HBASE_OPTS "‐server ‐XX:+UseConcMarkSweepGC"JVM GC XUANXIANG

荣耀彩票代理HBASE_MANAGES_ZKfalse SHIFOUSHIYONGZookeeperJINXINGFENBUSHIGUANLI

16. 持久化

ZHONGQICAOZUOXITONGHOUHBaseZHONGSHUJUQUANWU,NIKEYIBUZUORENHEXIUGAIDEQINGKUANGXIA,CHUANGJIANYIZHANGBIAO,XIEYITIAOSHUJUJINXING,RANHOUJIANGJIQIZHONGQI,ZHONGQIHOUNIZAIJINRUHBaseDEshellZHONGSHIYONG list MINGLINGCHAKANDANGQIANSUOCUNZAIDEBIAO,YIGEDOUMEIYOULE。SHIBUSHIHENBEIJU?MEIYOUGUANXINIKEYIZAIhbase/conf/hbase-default.xmlZHONGSHEZHIhbase.rootdirDEZHI,LAISHEZHIWENJIANDEBAOCUNWEIZHIZHIDINGYIGEWENJIANJIA,LIRU:<value>file:///you/hbase-data/path</value>,NIJIANLIDEHBaseZHONGDEBIAOHESHUJUJIUZHIJIEXIEDAOLENIDECIPANSHANG,TONGYANGNIYEKEYIZHIDINGNIDEFENBUSHIWENJIANXITONGHDFSDELUJINGLIRU:hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR,ZHEIYANGJIUXIEDAOLENIDEFENBUSHIWENJIANXITONGSHANGLE。

17. 缓冲区大小

荣耀彩票代理hbase.client.write.buffer

ZHEIGECANSHUKEYISHEZHIXIERUSHUJUHUANCHONGQUDEDAXIAO,DANGKEHUDUANHEFUWUQIDUANCHUANSHUSHUJU,FUWUQIWEILETIGAOXITONGYUNXINGXINGNENGKAIPIYIGEXIEDEHUANCHONGQULAICHULITA,ZHEIGECANSHUSHEZHIRUGUOSHEZHIDEDALE,JIANGHUIDUIXITONGDENEICUNYOUYIDINGDEYAOQIU,ZHIJIEYINGXIANGXITONGDEXINGNENG。

18. 扫描目录表

荣耀彩票代理hbase.master.meta.thread.rescanfrequency

DINGYIDUOZHANGSHIJIANHMasterDUIXITONGBIAO root HE meta SAOMIAOYICI,ZHEIGECANSHUKEYISHEZHIDEZHANGYIXIE,JIANGDIXITONGDENENGHAO。

19. split/compaction时间间隔

荣耀彩票代理hbase.regionserver.thread.splitcompactcheckfrequency

荣耀彩票代理ZHEIGECANSHUSHIBIAOSHIDUOJIUQURegionServerFUWUQIYUNXINGYICIsplit/compactionDESHIJIANJIANGE,DANGRANsplitZHIQIANHUIXIANJINXINGYIGEcompactCAOZUO.ZHEIGEcompactCAOZUOKENENGSHIminorcompactYEKENENGSHImajor compact.compactHOU,HUICONGSUOYOUDEStoreXIADESUOYOUStoreFileWENJIANZUIDADENEIGEQUmidkey.ZHEIGEmidkeyKENENGBINGBUCHUYUQUANBUSHUJUDEmidZHONG.YIGErow-keyDEXIAMIANDESHUJUKENENGHUIKUABUTONGDEHRegion。

20. 缓存在JVM堆中分配的百分比

hfile.block.cache.size

荣耀彩票代理ZHIDINGHFile/StoreFile HUANCUNZAIJVMDUIZHONGFENPEIDEBAIFENBI,MORENZHISHI0.2,YISIJIUSHI20%,ERRUGUONISHEZHICHENG0,JIUBIAOSHIDUIGAIXUANXIANGPINGBI。

21. ZooKeeper客户端同时访问的并发连接数

hbase.zookeeper.property.maxClientCnxns

荣耀彩票代理ZHEIXIANGPEIZHIDEXUANXIANGJIUSHICONGzookeeperZHONGLAIDE,BIAOSHIZooKeeperKEHUDUANTONGSHIFANGWENDEBINGFALIANJIESHU,ZooKeeperDUIYUHBaseLAISHUOJIUSHIYIGERUKOUZHEIGECANSHUDEZHIKEYISHIDANGFANGDAXIE。

22. memstores占用堆的大小参数配置

荣耀彩票代理hbase.regionserver.global.memstore.upperLimit

荣耀彩票代理ZAIRegionServerZHONGSUOYOUmemstoresZHANYONGDUIDEDAXIAOCANSHUPEIZHI,MORENZHISHI0.4,BIAOSHI40%,RUGUOSHEZHIWEI0,JIUSHIDUIXUANXIANGJINXINGPINGBI。

23. Memstore中缓存写入大小

hbase.hregion.memstore.flush.size

荣耀彩票代理MemstoreZHONGHUANCUNDENEIRONGCHAOGUOPEIZHIDEFANWEIHOUJIANGHUIXIEDAOCIPANSHANG,LIRU:SHANCHUCAOZUOSHIXIANXIERUMemStoreLIZUOGEBIAOJI,ZHISHINEIGEvalue, column HUO familyDENGXIASHIYAOSHANCHUDE,HBaseHUIDINGQIDUICUNCHUWENJIANZUOYIGEmajor compaction,ZAINEISHIHBaseHUIBAMemStoreSHUARUYIGEXINDEHFileCUNCHUWENJIANZHONG。RUGUOZAIYIDINGSHIJIANFANWEINEIMEIYOUZUOmajor compaction,ERMemstoreZHONGCHAOCHUDEFANWEIJIUXIERUCIPANSHANGLE。

Tag标签:   
  • 专题推荐

About IT165 - 广告服务 - 隐私声明 - 版权申明 - 免责条款 - 网站地图 - 网友投稿 - 联系方式
本站内容来自于互联网,仅供用于网络技术学习,学习中请遵循相关法律法规