荣耀彩票代理

IT技术互动交流平台

HBase的Write Ahead Log (WAL) 整体架构 线程模型

作者:Yijian  来源:IT165收集  发布日期:2016-08-26 20:21:17

解决的问题

HBaseDEWrite Ahead Log (WAL)TIGONGLEYIZHONGGAOBINGFA、CHIJIUHUADERIZHIBAOCUNYUHUIFANGJIZHI。MEIYIGEYEWUSHUJUDEXIERUCAOZUO(PUT / DELETE)ZHIXINGQIAN,DOUHUIJIZHANGZAIWALZHONG。

RUGUOCHUXIANHBaseFUWUQIDANGJI,ZEKEYICONGWALZHONGHUIFANGZHIXINGZHIQIANMEIYOUWANCHENGDECAOZUO。

BENWENZHUYAOTANTAOHBaseDEWALJIZHI,RUHECONGXIANCHENGMOXING、XIAOXIJIZHIDECENGMIANSHANG,JIEJUEZHEIXIEWENTI:

荣耀彩票代理1. YOUYUDUOGEHBaseKEHUDUANKEYIDUIMOUYITAIHBase Region ServerFAQIBINGFADEYEWUSHUJUXIERUQINGQIU,YINCIWALYEYAOZHICHIBINGFADEDUOXIANCHENGRIZHIXIERU。——QUEBAORIZHIXIERUDEXIANCHENGANQUAN、GAOBINGFA。

荣耀彩票代理2. DUIYUDANGEHBaseKEHUDUAN,TAZAIWALZHONGDERIZHISHUNXU,YINGGAIYUZHEIGEKEHUDUANFAQIDEYEWUSHUJUXIERUQINGQIUDESHUNXUYIZHI。

荣耀彩票代理(DUIYUYISHANGLIANGDIANYAOQIU,DAJIAHENRONGYIXIANGDAO,YONGYIGEDUILIEJIUGAODINGLE。JIANXIAWENDEJIAGOUTU。)

荣耀彩票代理3. WEILEBAOZHENGGAOKEKAO,RIZHIBUJINYAOXIERUWENJIANXITONGDENEICUNHUANCUN,ERQIEYINGGAIJINKUAI、QIANGZHISHUADAOCIPANSHANG(JIWALDESyncCAOZUO)。DANSHISyncTAIPINFAN,XINGNENGHUIBIANCHA。SUOYI:

 (1) SyncYINGDANGZAIDUOGEHOUTAIXIANCHENGZHONGYIBUZHIXING

 (2) PINFANDEDUOGESync,KEYIHEBINGWEIYICISync——SHIDANGFANGSONGDUIKEKAOXINGDEYAOQIU,TIGAOXINGNENG。

架构图——线程模型、消息机制

荣耀彩票代理XIAMIANSHIWOHUADEHBase WALJIAGOUTU。WOZAITUSHANGJIALEBUSHAOZHUJIE,SUOYIZHEIZHANGTUYINGGAISHIZIJIESHIDE:

 Region Server RPC服务线程

ZHEIXIEXIANCHENGCHULIHBaseKEHUDUANTONGGUORPCFUWUDIAOYONG(SHIJISHANGSHIGoogle ProtobufFUWUDIAOYONG)FACHUDEYEWUSHUJUXIERUQINGQIU。ZAISHANGTUDELIZIZHONG,“Region Server RPCFUWUXIANCHENG1” ZUOLE3GERowDEAppendCAOZUO,HEYIGEQIANGZHISHUACIPANDESyncCAOZUO。

荣耀彩票代理SyncCAOZUOSHIWEILEQUEBAOZHIQIANDEAppendCAOZUO(BAOKUOSHEJIDEYEWUSHUJU)YIDINGKEKAODIJILUDAOLECIPANSHANGDERIZHIZHONG,RANHOUHBaseCAINENGZUOHOUXUXIANGDUIBUKEKAODEFUZACAOZUO,BIRUXIERUMemStore。——ZHEIJIUSHIWrite AheadDEYUYI。

荣耀彩票代理CONGJIAGOUTUZHONGKEJIAN,BINGFADEAppendCAOZUOZHISHIWANGDUILIEZHONGZENGJIALEAppendQINGQIUDUIXIANG。

ZHEILIDEDUILIESHIYIGELMAX Disrutpor RingBuffer(WODEZHEIPIANWENZHANGZUOLEJIESHAO),NIKEYIJIANDANLIJIEWEISHIYIGEWUSUOGAOBINGFADUILIE。

AppendDEJUTIDAIMARUXIA:

DUIYUSyncCAOZUO:

荣耀彩票代理(1)WANGDUILIELIFANGYIGESyncFutureDUIXIANG,DAIBIAOYICISyncCAOZUOQINGQIU。

MEIYIGESyncFutureDOUYOUYIGEZIZENGDESequence ID——ZHEISHIQUANJUWEIYIDE,YOULMAX DisrutporDUILIECHUANGJIAN。HOULAIDESyncFutureDESequence IDGENGGAO。

(2)调用SyncFuture.get()阻塞等待,直到后台线程(架构图中的SyncRunner)通知SyncFuture退出阻塞,表明WAL日志已经保存在了磁盘上。

WAL日志消费线程

WAL机制中,只有一个WAL日志消费线程,从队列中获取Append和Sync操作。这样一个多生产者,单消费者的模式,决定了WAL日志并发写入时日志的全局唯一顺序。

荣耀彩票代理1. DUIYUHUOQUDAODEAppendCAOZUO,ZHIJIEDIAOYONGHadoop Sequence File WriterJIANGZHEIGEAppendCAOZUO(BAOKUOYUANSHUJUHErow key, family, qualifier, timestamp, valueDENGYEWUSHUJU)XIERUWENJIAN。

    YINCIWALRIZHIWENJIANSHIYONGDESHIHadoop SequenceWENJIANGESHI。DANGRAN,TAYEKEYITIHUANCHENGQITACUNCHUGESHI,RUAvro。

    Hadoop SequenceWENJIANGESHIBUZAIZHEILILEISHU,QIZHUYAOTEDIANSHI:

   (1) ERJINZHIGESHI。row key, family, qualifier, timestamp, valueDENGHBase byte[]SHUJU,DOUYUANFENGBUDONGDISHUNXUXIERUWENJIAN。

荣耀彩票代理   (2) SequenceWENJIANZHONG,MEIGERUOGANXING,HUICHARUYIGE16ZIJIEDEMOSHUZUOWEIFENGEFU。ZHEIYANGRUGUOWENJIANSUNHUAI,DAOZHIMOUYIXINGCANQUEBUQUAN,KEYITONGGUOZHEIGEMOSHUFENGEFUTIAOGUOZHEIYIXING,JIXUDUQUXIAYIGEWANZHENGDEXING。

   (3) ZHICHIYASUO。KEYIANXINGYASUO。YEKEYIANKUAIYASUO(JIANGDUOXINGDACHENGYIGEKUAI)

2. DUIYUHUOQUDAODESyncCAOZUO,HUITIJIAOJIHOUTAISyncRunnerDEXIANCHENGCHI(JIANSHANGWENJIAGOUTU)YIBUZHIXING。

YISHANGDEthis.syncRunnersJIUSHISyncRunnerXIANCHENGCHI。KEYIKANDAO,TONGGUOJISUANsyncRunnerIndex,CAIYONGLEJIANDANDELUNXUNTIJIAOSUANFA。

另外,WAL日志消费线程,会尝试收集一批SyncFuture对象(即sync操作),一次提交给SyncRunner。

        SUOYI,ZAIYISHANGDAIMAZHONG,KEYIKANDAOCHUANRUoffer()FANGFADE,SHIthis.syncFuturesZHEIYISyncFutures[]SHUZU,ERBUSHIDANGESyncFutureDUIXIANG。

荣耀彩票代理        SHOUJIYIPICIZAITIJIAO,XINGNENGBIJIAOHAO。DANSHIDANGEPICIXUYAOJIZANDESyncFutureDUIXIANGYUEDUO,ZESyncDEJISHIXINGYUECHA,HUIDAOZHIQIANTAIRegion Server RPCFUWUXIANCHENGZUSAIZAISyncFuture.get()SHANGDESHIJIANJIUYUEZHANG。

        因此,这里存在吞吐量和及时性之间的平衡。HBase为了支持海量数据的写入,在这里更倾向于高吞吐量荣耀彩票代理,体现在了以下注释中。具体多少个SyncFuture构成一个批次,有一定的策略,在此不再累述。

SyncRunner线程

荣耀彩票代理1. CONGDUILIEZHONGHUOQUYIGEYOUWALRIZHIXIAOFEIXIANCHENGTIJIAODESyncFuture(XIATUHONGKUANGZHONGDEDAIMA)。

荣耀彩票代理2. DIAOYONGWENJIANXITONGAPI,ZHIXINGsync()CAOZUO(XIATULANKUANGZHONGDEDAIMA)

合并多次频繁的sync()操作,提高性能。

        SHANGWENTIDAO,WALRIZHIXIAOFEIXIANCHENGYICIHUITIJIAODUOGESyncFuture。DUICI,SyncRunnerXIANCHENGZHIHUILUOSHIZHIXINGQIZHONGZUIXINDESyncFuture(YEJIUSHISequence IDZUIDADENEIGE)SUODAIBIAODESyncCAOZUO。ERHULVEZHIQIANDESyncFuture。

        ZHEIJIUSHIXIATULVKUANGZHONGDEDAIMA。

荣耀彩票代理3. RUGUOsync()WANCHENG,HUOZHEYINWEISHANGMIANTIDAODEHEBINGHULVELEMOUYIGESyncFuture,NEIMEHUIDIAOYONGreleaseSyncFuture() ==> Object.notify()LAITONGZHISyncFutureZUSAITUICHU。

   ZHIQIANZUSAIZAISyncFuture.get()SHANGDERegion Server RPCFUWUXIANCHENGJIUKEYIJIXUWANGXIAZHIXINGLE。

荣耀彩票代理ZHICI,ZHENGGEWALXIERULIUCHENGWANCHENG。

总结

WOJUEDEDUIXIANCHENGBINGFAXIERUWENJIANSHI,YONGDUILIELAIXIEDIAO,BAOZHENGRIZHIXIERUDESHUNXU,ZHEIHAISHIBIJIAORONGYIXIANGDAODE。

DANSHI,TIGONGSync() APIQUEBAORIZHIXIERUDEKEKAOXING,TONGSHIBIMIANPINFANDESync()CAOZUOYINGXIANGXINGNENG。——ZHEISHIHBase WALSHIXIANDEYIDALIANGDIAN。

HOUXUWOZAIYANJIUYANJIUWALDEcheckpointHEDUQUWALHUIFANGJIZHI,ZAIHEDAJIAFENXIANG。

Tag标签:         
  • 专题推荐

About IT165 - 广告服务 - 隐私声明 - 版权申明 - 免责条款 - 网站地图 - 网友投稿 - 联系方式
本站内容来自于互联网,仅供用于网络技术学习,学习中请遵循相关法律法规