Master status for dsjhd113,60000,1402039442506 as of Fri Jun 06 18:20:08 CST 2014
Version Info:
===========================================================
HBase 0.96.0.2.0.6.0-76-hadoop2
Subversion git://c64-s11/grid/0/workspace/BIGTOP-bigwheel-GA-HDP_RPM_REPO/label/c64-s11/bigtop-0.5/build/hbase/rpm/BUILD/hbase-0.96.0.2.0.6.0 -r e6d7a56f72914d01e55c0478d74e5cfd3778f231
Compiled by jenkins on Thu Oct 17 18:15:20 PDT 2013
Hadoop 2.2.0.2.0.6.0-76
Subversion
[email protected]:hortonworks/hadoop.git -r 8656b1cfad13b03b29e98cad042626205e7a1c86
Compiled by jenkins on 2013-10-18T00:19Z
Tasks:
===========================================================
Task: RpcServer.handler=0,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=1,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=2,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=3,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=4,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=5,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=6,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=7,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=8,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=9,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=10,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=11,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=12,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=13,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52589: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=14,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=15,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=16,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=17,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=18,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52604: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=19,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52580: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=20,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=21,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=22,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=23,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=24,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=25,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52596: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=26,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=27,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52589: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=28,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=29,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=30,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=31,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=32,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=33,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=34,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=35,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=36,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=37,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=38,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52600: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=39,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=40,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=41,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=42,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=43,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=44,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52600: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=45,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=46,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=47,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=48,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52584: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=49,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=50,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=51,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=52,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=53,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=54,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52596: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=55,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=56,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: RpcServer.handler=57,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52577: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=58,port=60000
Status: RUNNING:Servicing call from 132.228.248.123:52604: ListTableDescriptorsByNamespace
Running for 10565s
Task: RpcServer.handler=59,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: Replication.RpcServer.handler=0,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: Replication.RpcServer.handler=1,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Task: Replication.RpcServer.handler=2,port=60000
Status: WAITING:Waiting for a call
Running for 10565s
Servers:
===========================================================
dsjhd076,60020,1402027922130: requestsPerSecond=285198.0, numberOfOnlineRegions=391, usedHeapMB=9940, maxHeapMB=16332, numberOfStores=535, numberOfStorefiles=1413, storefileUncompressedSizeMB=13194842, storefileSizeMB=2510416, compressionRatio=0.1903, memstoreSizeMB=877, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6201918, rootIndexSizeKB=33277, totalStaticIndexSizeKB=20600067, totalStaticBloomSizeKB=36594476, totalCompactingKVs=393668602, currentCompactedKVs=298781948, compactionProgressPct=0.7589682, coprocessors=[]
dsjhd055,60020,1402027951094: requestsPerSecond=328.0, numberOfOnlineRegions=389, usedHeapMB=8289, maxHeapMB=16332, numberOfStores=535, numberOfStorefiles=851, storefileUncompressedSizeMB=13355512, storefileSizeMB=2554818, compressionRatio=0.1913, memstoreSizeMB=621, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=4394552, rootIndexSizeKB=38034, totalStaticIndexSizeKB=20866881, totalStaticBloomSizeKB=36815194, totalCompactingKVs=479738188, currentCompactedKVs=479738188, compactionProgressPct=1.0, coprocessors=[]
dsjhd074,60020,1402039899044: requestsPerSecond=653.0, numberOfOnlineRegions=351, usedHeapMB=7170, maxHeapMB=16332, numberOfStores=508, numberOfStorefiles=788, storefileUncompressedSizeMB=12716455, storefileSizeMB=2356905, compressionRatio=0.1853, memstoreSizeMB=399, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5037169, rootIndexSizeKB=34313, totalStaticIndexSizeKB=20463008, totalStaticBloomSizeKB=32405318, totalCompactingKVs=505141420, currentCompactedKVs=505141420, compactionProgressPct=1.0, coprocessors=[]
dsjhd075,60020,1402027953331: requestsPerSecond=269.0, numberOfOnlineRegions=389, usedHeapMB=7767, maxHeapMB=16332, numberOfStores=557, numberOfStorefiles=883, storefileUncompressedSizeMB=13914907, storefileSizeMB=2629550, compressionRatio=0.1890, memstoreSizeMB=413, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6307135, rootIndexSizeKB=37493, totalStaticIndexSizeKB=22046104, totalStaticBloomSizeKB=37025777, totalCompactingKVs=659623182, currentCompactedKVs=659623182, compactionProgressPct=1.0, coprocessors=[]
dsjhd078,60020,1402027921335: requestsPerSecond=666.0, numberOfOnlineRegions=388, usedHeapMB=6692, maxHeapMB=16332, numberOfStores=540, numberOfStorefiles=887, storefileUncompressedSizeMB=13351402, storefileSizeMB=2555262, compressionRatio=0.1914, memstoreSizeMB=627, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=7102881, rootIndexSizeKB=38573, totalStaticIndexSizeKB=20689894, totalStaticBloomSizeKB=37597883, totalCompactingKVs=158498513, currentCompactedKVs=158498513, compactionProgressPct=1.0, coprocessors=[]
dsjhd057,60020,1402039842287: requestsPerSecond=431.0, numberOfOnlineRegions=351, usedHeapMB=2127, maxHeapMB=16332, numberOfStores=487, numberOfStorefiles=789, storefileUncompressedSizeMB=12426629, storefileSizeMB=2335082, compressionRatio=0.1879, memstoreSizeMB=656, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5288972, rootIndexSizeKB=33268, totalStaticIndexSizeKB=19578441, totalStaticBloomSizeKB=33462794, totalCompactingKVs=711157130, currentCompactedKVs=640427134, compactionProgressPct=0.9005424, coprocessors=[]
dsjhd068,60020,1402038513322: requestsPerSecond=474.0, numberOfOnlineRegions=351, usedHeapMB=3689, maxHeapMB=16332, numberOfStores=471, numberOfStorefiles=792, storefileUncompressedSizeMB=11800330, storefileSizeMB=2254462, compressionRatio=0.1911, memstoreSizeMB=362, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6098047, rootIndexSizeKB=32137, totalStaticIndexSizeKB=18195643, totalStaticBloomSizeKB=33492854, totalCompactingKVs=592761684, currentCompactedKVs=592761684, compactionProgressPct=1.0, coprocessors=[]
dsjhd062,60020,1402027917800: requestsPerSecond=592.0, numberOfOnlineRegions=389, usedHeapMB=8051, maxHeapMB=16332, numberOfStores=543, numberOfStorefiles=851, storefileUncompressedSizeMB=13974248, storefileSizeMB=2621347, compressionRatio=0.1876, memstoreSizeMB=869, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6727636, rootIndexSizeKB=36288, totalStaticIndexSizeKB=22118194, totalStaticBloomSizeKB=37155772, totalCompactingKVs=804952991, currentCompactedKVs=804952991, compactionProgressPct=1.0, coprocessors=[]
dsjhd059,60020,1402027941533: requestsPerSecond=308.0, numberOfOnlineRegions=389, usedHeapMB=8334, maxHeapMB=16332, numberOfStores=538, numberOfStorefiles=861, storefileUncompressedSizeMB=13199517, storefileSizeMB=2500137, compressionRatio=0.1894, memstoreSizeMB=426, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6914927, rootIndexSizeKB=36146, totalStaticIndexSizeKB=20710708, totalStaticBloomSizeKB=36002786, totalCompactingKVs=435381501, currentCompactedKVs=435381501, compactionProgressPct=1.0, coprocessors=[]
dsjhd066,60020,1402027920604: requestsPerSecond=484.0, numberOfOnlineRegions=390, usedHeapMB=6343, maxHeapMB=16332, numberOfStores=541, numberOfStorefiles=869, storefileUncompressedSizeMB=13245107, storefileSizeMB=2513471, compressionRatio=0.1898, memstoreSizeMB=521, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6413922, rootIndexSizeKB=36406, totalStaticIndexSizeKB=20851595, totalStaticBloomSizeKB=35851661, totalCompactingKVs=323319647, currentCompactedKVs=247970609, compactionProgressPct=0.76695186, coprocessors=[]
dsjhd052,60020,1402039731411: requestsPerSecond=594.0, numberOfOnlineRegions=351, usedHeapMB=6907, maxHeapMB=16332, numberOfStores=497, numberOfStorefiles=832, storefileUncompressedSizeMB=11519771, storefileSizeMB=2227394, compressionRatio=0.1934, memstoreSizeMB=1311, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6349749, rootIndexSizeKB=31350, totalStaticIndexSizeKB=17600323, totalStaticBloomSizeKB=33354160, totalCompactingKVs=1271691778, currentCompactedKVs=1271691778, compactionProgressPct=1.0, coprocessors=[]
dsjhd070,60020,1402027928334: requestsPerSecond=339.0, numberOfOnlineRegions=388, usedHeapMB=8344, maxHeapMB=16332, numberOfStores=527, numberOfStorefiles=865, storefileUncompressedSizeMB=13001803, storefileSizeMB=2506474, compressionRatio=0.1928, memstoreSizeMB=691, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6522753, rootIndexSizeKB=35687, totalStaticIndexSizeKB=19934802, totalStaticBloomSizeKB=37072136, totalCompactingKVs=591807087, currentCompactedKVs=591807087, compactionProgressPct=1.0, coprocessors=[]
dsjhd069,60020,1402032388543: requestsPerSecond=214.0, numberOfOnlineRegions=351, usedHeapMB=3036, maxHeapMB=16332, numberOfStores=501, numberOfStorefiles=783, storefileUncompressedSizeMB=12556044, storefileSizeMB=2344028, compressionRatio=0.1867, memstoreSizeMB=858, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6823330, rootIndexSizeKB=33864, totalStaticIndexSizeKB=20052582, totalStaticBloomSizeKB=32489808, totalCompactingKVs=274994501, currentCompactedKVs=274994501, compactionProgressPct=1.0, coprocessors=[]
dsjhd063,60020,1402039188138: requestsPerSecond=299.0, numberOfOnlineRegions=351, usedHeapMB=5791, maxHeapMB=16332, numberOfStores=505, numberOfStorefiles=795, storefileUncompressedSizeMB=12624689, storefileSizeMB=2378884, compressionRatio=0.1884, memstoreSizeMB=706, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5367856, rootIndexSizeKB=31968, totalStaticIndexSizeKB=20066251, totalStaticBloomSizeKB=33340458, totalCompactingKVs=492221600, currentCompactedKVs=492221600, compactionProgressPct=1.0, coprocessors=[]
dsjhd054,60020,1402027922467: requestsPerSecond=1076.0, numberOfOnlineRegions=388, usedHeapMB=11263, maxHeapMB=16332, numberOfStores=546, numberOfStorefiles=882, storefileUncompressedSizeMB=13467633, storefileSizeMB=2550587, compressionRatio=0.1894, memstoreSizeMB=1274, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6953014, rootIndexSizeKB=36103, totalStaticIndexSizeKB=21074984, totalStaticBloomSizeKB=36714006, totalCompactingKVs=275450057, currentCompactedKVs=275450057, compactionProgressPct=1.0, coprocessors=[]
dsjhd067,60020,1402027944119: requestsPerSecond=869.0, numberOfOnlineRegions=390, usedHeapMB=2848, maxHeapMB=16332, numberOfStores=552, numberOfStorefiles=878, storefileUncompressedSizeMB=13256892, storefileSizeMB=2514078, compressionRatio=0.1896, memstoreSizeMB=1216, storefileIndexSizeMB=0, readRequestsCount=54, writeRequestsCount=6255995, rootIndexSizeKB=36050, totalStaticIndexSizeKB=20885937, totalStaticBloomSizeKB=35550005, totalCompactingKVs=391266872, currentCompactedKVs=391266872, compactionProgressPct=1.0, coprocessors=[]
dsjhd060,60020,1402027925238: requestsPerSecond=618.0, numberOfOnlineRegions=389, usedHeapMB=10196, maxHeapMB=16332, numberOfStores=540, numberOfStorefiles=856, storefileUncompressedSizeMB=13242092, storefileSizeMB=2529191, compressionRatio=0.1910, memstoreSizeMB=591, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6639840, rootIndexSizeKB=35666, totalStaticIndexSizeKB=20587296, totalStaticBloomSizeKB=37031791, totalCompactingKVs=375974916, currentCompactedKVs=375974916, compactionProgressPct=1.0, coprocessors=[]
dsjhd056,60020,1402027921412: requestsPerSecond=570.0, numberOfOnlineRegions=389, usedHeapMB=9162, maxHeapMB=16332, numberOfStores=532, numberOfStorefiles=860, storefileUncompressedSizeMB=13242766, storefileSizeMB=2508637, compressionRatio=0.1894, memstoreSizeMB=977, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5874910, rootIndexSizeKB=35815, totalStaticIndexSizeKB=20704668, totalStaticBloomSizeKB=36409248, totalCompactingKVs=208286299, currentCompactedKVs=208286299, compactionProgressPct=1.0, coprocessors=[]
dsjhd064,60020,1402027918813: requestsPerSecond=153863.0, numberOfOnlineRegions=392, usedHeapMB=10299, maxHeapMB=16332, numberOfStores=555, numberOfStorefiles=1018, storefileUncompressedSizeMB=13796015, storefileSizeMB=2608225, compressionRatio=0.1891, memstoreSizeMB=297, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=7124943, rootIndexSizeKB=35382, totalStaticIndexSizeKB=21830978, totalStaticBloomSizeKB=36957582, totalCompactingKVs=135472294, currentCompactedKVs=130549503, compactionProgressPct=0.9636621, coprocessors=[]
dsjhd073,60020,1402039874233: requestsPerSecond=762.0, numberOfOnlineRegions=351, usedHeapMB=4429, maxHeapMB=16332, numberOfStores=498, numberOfStorefiles=798, storefileUncompressedSizeMB=12122837, storefileSizeMB=2321883, compressionRatio=0.1915, memstoreSizeMB=383, storefileIndexSizeMB=0, readRequestsCount=100, writeRequestsCount=5114202, rootIndexSizeKB=32562, totalStaticIndexSizeKB=18921431, totalStaticBloomSizeKB=33381664, totalCompactingKVs=471874244, currentCompactedKVs=471874244, compactionProgressPct=1.0, coprocessors=[]
dsjhd051,60020,1402036579439: requestsPerSecond=589.0, numberOfOnlineRegions=352, usedHeapMB=5045, maxHeapMB=16332, numberOfStores=492, numberOfStorefiles=796, storefileUncompressedSizeMB=12215192, storefileSizeMB=2322454, compressionRatio=0.1901, memstoreSizeMB=465, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=6889980, rootIndexSizeKB=32192, totalStaticIndexSizeKB=18926837, totalStaticBloomSizeKB=34115119, totalCompactingKVs=489759604, currentCompactedKVs=489759604, compactionProgressPct=1.0, coprocessors=[]
dsjhd061,60020,1402027921226: requestsPerSecond=0.0, numberOfOnlineRegions=389, usedHeapMB=4931, maxHeapMB=16332, numberOfStores=540, numberOfStorefiles=860, storefileUncompressedSizeMB=13503175, storefileSizeMB=2565486, compressionRatio=0.1900, memstoreSizeMB=227, storefileIndexSizeMB=0, readRequestsCount=218, writeRequestsCount=5671262, rootIndexSizeKB=37518, totalStaticIndexSizeKB=20970351, totalStaticBloomSizeKB=37459647, totalCompactingKVs=518045130, currentCompactedKVs=518045130, compactionProgressPct=1.0, coprocessors=[]
dsjhd058,60020,1402047076406: requestsPerSecond=1218.0, numberOfOnlineRegions=351, usedHeapMB=4283, maxHeapMB=16332, numberOfStores=480, numberOfStorefiles=811, storefileUncompressedSizeMB=11731762, storefileSizeMB=2265024, compressionRatio=0.1931, memstoreSizeMB=798, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=508542, rootIndexSizeKB=32525, totalStaticIndexSizeKB=17787321, totalStaticBloomSizeKB=34423162, totalCompactingKVs=353247546, currentCompactedKVs=353247546, compactionProgressPct=1.0, coprocessors=[]
dsjhd065,60020,1402027967439: requestsPerSecond=717.0, numberOfOnlineRegions=390, usedHeapMB=9718, maxHeapMB=16332, numberOfStores=544, numberOfStorefiles=872, storefileUncompressedSizeMB=13938670, storefileSizeMB=2626918, compressionRatio=0.1885, memstoreSizeMB=1120, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5130145, rootIndexSizeKB=36165, totalStaticIndexSizeKB=21887923, totalStaticBloomSizeKB=37974426, totalCompactingKVs=808162374, currentCompactedKVs=808162374, compactionProgressPct=1.0, coprocessors=[]
dsjhd077,60020,1402027915865: requestsPerSecond=580.0, numberOfOnlineRegions=388, usedHeapMB=6226, maxHeapMB=16332, numberOfStores=546, numberOfStorefiles=857, storefileUncompressedSizeMB=13911984, storefileSizeMB=2610002, compressionRatio=0.1876, memstoreSizeMB=887, storefileIndexSizeMB=0, readRequestsCount=38315885, writeRequestsCount=7521917, rootIndexSizeKB=35143, totalStaticIndexSizeKB=22130809, totalStaticBloomSizeKB=36778986, totalCompactingKVs=197310894, currentCompactedKVs=197310894, compactionProgressPct=1.0, coprocessors=[]
dsjhd053,60020,1402027976229: requestsPerSecond=389.0, numberOfOnlineRegions=389, usedHeapMB=8669, maxHeapMB=16332, numberOfStores=543, numberOfStorefiles=870, storefileUncompressedSizeMB=13402607, storefileSizeMB=2533951, compressionRatio=0.1891, memstoreSizeMB=372, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=5185420, rootIndexSizeKB=37047, totalStaticIndexSizeKB=21053319, totalStaticBloomSizeKB=36264735, totalCompactingKVs=431095832, currentCompactedKVs=431095832, compactionProgressPct=1.0, coprocessors=[]
Regions-in-transition:
===========================================================
Executors:
===========================================================
Status for executor: Executor-6-MASTER_TABLE_OPERATIONS-dsjhd113:60000
=======================================
0 events queued, 0 running
Status for executor: Executor-5-M_LOG_REPLAY_OPS-dsjhd113:60000
=======================================
0 events queued, 0 running
Status for executor: Executor-3-MASTER_SERVER_OPERATIONS-dsjhd113:60000
=======================================
0 events queued, 0 running
Status for executor: Executor-1-MASTER_OPEN_REGION-dsjhd113:60000
=======================================
0 events queued, 0 running
Status for executor: Executor-2-MASTER_CLOSE_REGION-dsjhd113:60000
=======================================
0 events queued, 0 running
Status for executor: Executor-4-MASTER_META_SERVER_OPERATIONS-dsjhd113:60000
=======================================
0 events queued, 0 running
Stacks:
===========================================================
Process Thread Dump:
112 active threads
Thread 1419 (IPC Client (71257789) connection to dsjhd113/132.228.248.123:60000 from hbase):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
java.io.FilterInputStream.read(FilterInputStream.java:116)
java.io.FilterInputStream.read(FilterInputStream.java:116)
org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:553)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1057)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:719)
Thread 1418 (IPC Client (71257789) connection to dsjhd061/132.228.248.71:60020 from hbase):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
java.io.FilterInputStream.read(FilterInputStream.java:116)
java.io.FilterInputStream.read(FilterInputStream.java:116)
org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:553)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1057)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:719)
Thread 1405 (843405603@qtp-1312568829-42):
State: RUNNABLE
Blocked count: 26
Waited count: 29
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165)
org.apache.hadoop.hbase.master.MasterDumpServlet.doGet(MasterDumpServlet.java:80)
javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1081)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
Thread 1382 (431225297@qtp-1312568829-37):
State: TIMED_WAITING
Blocked count: 15
Waited count: 248
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1432)
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.listTableDescriptorsByNamespace(MasterProtos.java:40612)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.listTableDescriptorsByNamespace(HConnectionManager.java:2054)
org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2268)
org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2265)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3124)
org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2264)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__catalogTables(MasterStatusTmplImpl.java:461)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:270)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:95)
javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
Thread 1198 (600134808@qtp-1312568829-34):
State: TIMED_WAITING
Blocked count: 25
Waited count: 490
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1432)
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.listTableDescriptorsByNamespace(MasterProtos.java:40612)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.listTableDescriptorsByNamespace(HConnectionManager.java:2054)
org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2268)
org.apache.hadoop.hbase.client.HBaseAdmin$22.call(HBaseAdmin.java:2265)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3124)
org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2264)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__catalogTables(MasterStatusTmplImpl.java:461)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:270)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:95)
javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
Thread 942 (MASTER_TABLE_OPERATIONS-dsjhd113:60000-0):
State: WAITING
Blocked count: 37
Waited count: 52
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6c13cbd0
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 409 (IPC Parameter Sending Thread #1):
State: TIMED_WAITING
Blocked count: 1514
Waited count: 669156
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 156 (CatalogJanitor-dsjhd113:60000):
State: TIMED_WAITING
Blocked count: 6643
Waited count: 9574
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 155 (dsjhd113,60000,1402039442506-BalancerChore):
State: TIMED_WAITING
Blocked count: 102117
Waited count: 183590
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 154 (dsjhd113,60000,1402039442506-ClusterStatusChore):
State: TIMED_WAITING
Blocked count: 353
Waited count: 503
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 123 (MASTER_SERVER_OPERATIONS-dsjhd113:60000-4):
State: WAITING
Blocked count: 248
Waited count: 246
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b240236
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 122 (MASTER_SERVER_OPERATIONS-dsjhd113:60000-3):
State: WAITING
Blocked count: 380
Waited count: 417
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b240236
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 121 (MASTER_SERVER_OPERATIONS-dsjhd113:60000-2):
State: WAITING
Blocked count: 279
Waited count: 254
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b240236
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 120 (MASTER_SERVER_OPERATIONS-dsjhd113:60000-1):
State: WAITING
Blocked count: 546
Waited count: 748
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b240236
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 119 (MASTER_SERVER_OPERATIONS-dsjhd113:60000-0):
State: WAITING
Blocked count: 249
Waited count: 236
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b240236
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Thread 115 (master:dsjhd113:60000.archivedHFileCleaner):
State: TIMED_WAITING
Blocked count: 631998
Waited count: 1243640
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 116 (snapshot-hfile-cleaner-cache-refresher):
State: TIMED_WAITING
Blocked count: 64
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 111 (master:dsjhd113:60000.oldLogCleaner):
State: TIMED_WAITING
Blocked count: 10113
Waited count: 15584
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 114 (snapshot-log-cleaner-cache-refresher):
State: TIMED_WAITING
Blocked count: 36
Waited count: 103
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 113 (master:dsjhd113:60000-EventThread):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@69e1e999
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 112 (master:dsjhd113:60000-SendThread(dsjhd113:2181)):
State: RUNNABLE
Blocked count: 7
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1072)
Thread 110 (dsjhd113,60000,1402039442506.splitLogManagerTimeoutMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 10494
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:662)
Thread 109 (org.apache.hadoop.hdfs.PeerCache@2b1bed2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 88
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:252)
org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:39)
org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:135)
java.lang.Thread.run(Thread.java:662)
Thread 102 (1294712427@qtp-1312568829-3-EventThread):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c5b55a9
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 101 (1294712427@qtp-1312568829-3-SendThread(dsjhd113:2181)):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1072)
Thread 97 (Timer-0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 354
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
State: RUNNABLE
Blocked count: 3
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 12 (master:dsjhd113:60000):
State: TIMED_WAITING
Blocked count: 897
Waited count: 105599
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56)
org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:734)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:604)
java.lang.Thread.run(Thread.java:662)
Thread 93 (JvmPauseMonitor):
State: TIMED_WAITING
Blocked count: 3
Waited count: 21118
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
java.lang.Thread.run(Thread.java:662)
Thread 92 (Replication.RpcServer.handler=2,port=60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43277a30
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 91 (Replication.RpcServer.handler=1,port=60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43277a30
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 90 (Replication.RpcServer.handler=0,port=60000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43277a30
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 89 (RpcServer.handler=59,port=60000):
State: WAITING
Blocked count: 8
Waited count: 1634
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 88 (RpcServer.handler=58,port=60000):
State: BLOCKED
Blocked count: 128
Waited count: 1866
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 87 (RpcServer.handler=57,port=60000):
State: TIMED_WAITING
Blocked count: 38
Waited count: 2006
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1432)
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:27308)
org.apache.hadoop.hbase.protobuf.ProtobufUtil.get(ProtobufUtil.java:1281)
org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:752)
org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:750)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
org.apache.hadoop.hbase.client.HTable.get(HTable.java:755)
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:134)
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:118)
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3103)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3123)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 86 (RpcServer.handler=56,port=60000):
State: WAITING
Blocked count: 200
Waited count: 2013
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 85 (RpcServer.handler=55,port=60000):
State: WAITING
Blocked count: 14
Waited count: 1647
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 84 (RpcServer.handler=54,port=60000):
State: BLOCKED
Blocked count: 131
Waited count: 1846
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 83 (RpcServer.handler=53,port=60000):
State: WAITING
Blocked count: 134
Waited count: 1870
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 82 (RpcServer.handler=52,port=60000):
State: WAITING
Blocked count: 14
Waited count: 1642
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 81 (RpcServer.handler=51,port=60000):
State: WAITING
Blocked count: 141
Waited count: 1882
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 80 (RpcServer.handler=50,port=60000):
State: WAITING
Blocked count: 247
Waited count: 2111
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 79 (RpcServer.handler=49,port=60000):
State: WAITING
Blocked count: 158
Waited count: 1912
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 78 (RpcServer.handler=48,port=60000):
State: BLOCKED
Blocked count: 9
Waited count: 1582
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 77 (RpcServer.handler=47,port=60000):
State: WAITING
Blocked count: 2
Waited count: 1624
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 76 (RpcServer.handler=46,port=60000):
State: WAITING
Blocked count: 383
Waited count: 2337
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 75 (RpcServer.handler=45,port=60000):
State: WAITING
Blocked count: 8
Waited count: 1635
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 74 (RpcServer.handler=44,port=60000):
State: BLOCKED
Blocked count: 139
Waited count: 1874
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 73 (RpcServer.handler=43,port=60000):
State: WAITING
Blocked count: 509
Waited count: 2583
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 72 (RpcServer.handler=42,port=60000):
State: WAITING
Blocked count: 7
Waited count: 1634
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 71 (RpcServer.handler=41,port=60000):
State: WAITING
Blocked count: 129
Waited count: 1876
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 70 (RpcServer.handler=40,port=60000):
State: WAITING
Blocked count: 244
Waited count: 2103
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 69 (RpcServer.handler=39,port=60000):
State: WAITING
Blocked count: 126
Waited count: 1869
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 68 (RpcServer.handler=38,port=60000):
State: BLOCKED
Blocked count: 129
Waited count: 1857
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 67 (RpcServer.handler=37,port=60000):
State: WAITING
Blocked count: 148
Waited count: 1888
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 66 (RpcServer.handler=36,port=60000):
State: WAITING
Blocked count: 145
Waited count: 1871
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 65 (RpcServer.handler=35,port=60000):
State: WAITING
Blocked count: 380
Waited count: 2375
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 64 (RpcServer.handler=34,port=60000):
State: WAITING
Blocked count: 7
Waited count: 1631
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 63 (RpcServer.handler=33,port=60000):
State: WAITING
Blocked count: 130
Waited count: 1878
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 62 (RpcServer.handler=32,port=60000):
State: WAITING
Blocked count: 121
Waited count: 1858
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 61 (RpcServer.handler=31,port=60000):
State: WAITING
Blocked count: 119
Waited count: 1856
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 60 (RpcServer.handler=30,port=60000):
State: WAITING
Blocked count: 119
Waited count: 1855
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 59 (RpcServer.handler=29,port=60000):
State: WAITING
Blocked count: 146
Waited count: 1910
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 58 (RpcServer.handler=28,port=60000):
State: WAITING
Blocked count: 128
Waited count: 1870
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 57 (RpcServer.handler=27,port=60000):
State: BLOCKED
Blocked count: 135
Waited count: 1838
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 56 (RpcServer.handler=26,port=60000):
State: WAITING
Blocked count: 247
Waited count: 2113
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 55 (RpcServer.handler=25,port=60000):
State: BLOCKED
Blocked count: 135
Waited count: 1853
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 54 (RpcServer.handler=24,port=60000):
State: WAITING
Blocked count: 132
Waited count: 1874
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 53 (RpcServer.handler=23,port=60000):
State: WAITING
Blocked count: 127
Waited count: 1872
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 52 (RpcServer.handler=22,port=60000):
State: WAITING
Blocked count: 373
Waited count: 2364
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 51 (RpcServer.handler=21,port=60000):
State: WAITING
Blocked count: 2
Waited count: 1621
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 50 (RpcServer.handler=20,port=60000):
State: WAITING
Blocked count: 252
Waited count: 2110
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 49 (RpcServer.handler=19,port=60000):
State: BLOCKED
Blocked count: 239
Waited count: 2034
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 48 (RpcServer.handler=18,port=60000):
State: BLOCKED
Blocked count: 388
Waited count: 2358
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 47 (RpcServer.handler=17,port=60000):
State: WAITING
Blocked count: 23
Waited count: 1658
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 46 (RpcServer.handler=16,port=60000):
State: WAITING
Blocked count: 133
Waited count: 1884
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 45 (RpcServer.handler=15,port=60000):
State: WAITING
Blocked count: 367
Waited count: 2333
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 44 (RpcServer.handler=14,port=60000):
State: WAITING
Blocked count: 127
Waited count: 1870
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 43 (RpcServer.handler=13,port=60000):
State: BLOCKED
Blocked count: 143
Waited count: 1860
Blocked on org.apache.hadoop.hbase.master.TableNamespaceManager@67e24f6d
Blocked by 87 (RpcServer.handler=57,port=60000)
Stack:
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:250)
org.apache.hadoop.hbase.master.HMaster.isTableNamespaceManagerReady(HMaster.java:3141)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3120)
org.apache.hadoop.hbase.master.HMaster.listTableDescriptorsByNamespace(HMaster.java:3024)
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38261)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Thread 42 (RpcServer.handler=12,port=60000):
State: WAITING
Blocked count: 134
Waited count: 1887
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 41 (RpcServer.handler=11,port=60000):
State: WAITING
Blocked count: 243
Waited count: 2096
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 40 (RpcServer.handler=10,port=60000):
State: WAITING
Blocked count: 365
Waited count: 2349
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 39 (RpcServer.handler=9,port=60000):
State: WAITING
Blocked count: 8
Waited count: 1630
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 38 (RpcServer.handler=8,port=60000):
State: WAITING
Blocked count: 245
Waited count: 2110
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 37 (RpcServer.handler=7,port=60000):
State: WAITING
Blocked count: 129
Waited count: 1876
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 36 (RpcServer.handler=6,port=60000):
State: WAITING
Blocked count: 127
Waited count: 1871
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 35 (RpcServer.handler=5,port=60000):
State: WAITING
Blocked count: 135
Waited count: 1888
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 34 (RpcServer.handler=4,port=60000):
State: WAITING
Blocked count: 0
Waited count: 1618
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 33 (RpcServer.handler=3,port=60000):
State: WAITING
Blocked count: 133
Waited count: 1882
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 32 (RpcServer.handler=2,port=60000):
State: WAITING
Blocked count: 120
Waited count: 1865
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 31 (RpcServer.handler=1,port=60000):
State: WAITING
Blocked count: 13
Waited count: 1638
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 30 (RpcServer.handler=0,port=60000):
State: WAITING
Blocked count: 362
Waited count: 2340
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54c707c1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1826)
Thread 13 (RpcServer.listener,port=60000):
State: RUNNABLE
Blocked count: 420
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:660)
Thread 26 (RpcServer.responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:847)
org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:830)
Thread 29 (main-EventThread):
State: WAITING
Blocked count: 890
Waited count: 16872
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56ce3b62
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
Thread 28 (main-SendThread(dsjhd112:2181)):
State: RUNNABLE
Blocked count: 5945
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1072)
Thread 25 (Timer for 'HBase' metrics system):
State: TIMED_WAITING
Blocked count: 1
Waited count: 1058
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 24 (ganglia):
State: WAITING
Blocked count: 1056
Waited count: 1057
Waiting on org.apache.hadoop.metrics2.impl.SinkQueue@67071c84
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.metrics2.impl.SinkQueue.waitForData(SinkQueue.java:114)
org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:83)
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:127)
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:86)
Thread 23 (RpcServer.reader=9,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 45
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 22 (RpcServer.reader=8,port=60000):
State: RUNNABLE
Blocked count: 43
Waited count: 46
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 21 (RpcServer.reader=7,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 48
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 20 (RpcServer.reader=6,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 49
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 19 (RpcServer.reader=5,port=60000):
State: RUNNABLE
Blocked count: 43
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 18 (RpcServer.reader=4,port=60000):
State: RUNNABLE
Blocked count: 43
Waited count: 50
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 17 (RpcServer.reader=3,port=60000):
State: RUNNABLE
Blocked count: 47
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 16 (RpcServer.reader=2,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 15 (RpcServer.reader=1,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 47
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 14 (RpcServer.reader=0,port=60000):
State: RUNNABLE
Blocked count: 42
Waited count: 43
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:552)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:538)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 5 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 580
Waited count: 579
Waiting on java.lang.ref.ReferenceQueue$Lock@3e152f4
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 888
Waited count: 878
Waiting on java.lang.ref.Reference$Lock@3e0d1329
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: WAITING
Blocked count: 19
Waited count: 12
Waiting on java.lang.Thread@7d9ccb73
Stack:
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1186)
java.lang.Thread.join(Thread.java:1239)
org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:89)
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:190)
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2787)
Master configuration:
===========================================================
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
<property><name>hbase.regionserver.catalog.timeout</name><value>600000</value><source>hbase-default.xml</source></property>
<property><name>hbase.status.publisher.class</name><value>org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher</value><source>hbase-default.xml</source></property>
<property><name>hbase.regions.slop</name><value>0.2</value><source>hbase-default.xml</source></property>
<property><name>dfs.image.transfer.bandwidthPerSec</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>hbase.zookeeper.leaderport</name><value>3888</value><source>hbase-default.xml</source></property>
<property><name>hbase.regionserver.info.port</name><value>60030</value><source>hbase-default.xml</source></property>
<property><name>dfs.block.access.token.lifetime</name><value>600</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.proxyuser.hive.hosts</name><value>dsjhd112</value><source>core-site.xml</source></property>
<property><name>fs.AbstractFileSystem.file.impl</name><value>org.apache.hadoop.fs.local.LocalFs</value><source>core-default.xml</source></property>
<property><name>hbase.rs.cacheblocksonwrite</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.domain.socket.path</name><value>/var/lib/hadoop-hdfs/dn_socket</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.ssl.keystores.factory.class</name><value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value><source>core-default.xml</source></property>
<property><name>hbase.rpc.server.engine</name><value>org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine</value><source>hbase-default.xml</source></property>
<property><name>hadoop.http.authentication.kerberos.keytab</name><value>${user.home}/hadoop.keytab</value><source>core-default.xml</source></property>
<property><name>io.seqfile.sorter.recordlimit</name><value>1000000</value><source>core-default.xml</source></property>
<property><name>hbase.data.umask.enable</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>s3.blocksize</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>hbase.master.dns.nameserver</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.num.checkpoints.retained</name><value>2</value><source>hdfs-default.xml</source></property>
<property><name>hbase.lease.recovery.dfs.timeout</name><value>64000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.delegation.token.renew-interval</name><value>86400000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.status.listener.class</name><value>org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value><source>hbase-default.xml</source></property>
<property><name>io.map.index.interval</name><value>128</value><source>core-default.xml</source></property>
<property><name>hfile.format.version</name><value>2</value><source>hbase-default.xml</source></property>
<property><name>nfs3.mountd.port</name><value>4242</value><source>core-default.xml</source></property>
<property><name>s3.client-write-packet-size</name><value>65536</value><source>core-default.xml</source></property>
<property><name>hbase.master.catalog.timeout</name><value>600000</value><source>hbase-default.xml</source></property>
<property><name>hbase.auth.token.max.lifetime</name><value>604800000</value><source>hbase-default.xml</source></property>
<property><name>hbase.hregion.majorcompaction.jitter</name><value>0.50</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.http-address</name><value>dsjhd112,dsjhd113:50070</value><source>hdfs-site.xml</source></property>
<property><name>ha.zookeeper.session-timeout.ms</name><value>5000</value><source>core-default.xml</source></property>
<property><name>hbase.master.loadbalancer.class</name><value>org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer</value><source>hbase-default.xml</source></property>
<property><name>hbase.hstore.flush.retries.number</name><value>120</value><source>hbase-site.xml</source></property>
<property><name>hadoop.hdfs.configuration.version</name><value>1</value><source>hdfs-default.xml</source></property>
<property><name>s3.replication</name><value>3</value><source>core-default.xml</source></property>
<property><name>hfile.block.bloom.cacheonwrite</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.balance.bandwidthPerSec</name><value>6250000</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.ssl.enabled</name><value>false</value><source>core-default.xml</source></property>
<property><name>dfs.journalnode.rpc-address</name><value>0.0.0.0:8485</value><source>hdfs-default.xml</source></property>
<property><name>hbase.lease.recovery.timeout</name><value>900000</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.readahead.bytes</name><value>4193404</value><source>hdfs-default.xml</source></property>
<property><name>ipc.client.connect.max.retries.on.timeouts</name><value>45</value><source>core-default.xml</source></property>
<property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-site.xml</source></property>
<property><name>dfs.client.failover.max.attempts</name><value>15</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.dir</name><value>/data/data1/namesecondary</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.proxyuser.hcat.groups</name><value>users</value><source>core-site.xml</source></property>
<property><name>hbase.regionserver.logroll.period</name><value>3600000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.replication.work.multiplier.per.iteration</name><value>2</value><source>hdfs-default.xml</source></property>
<property><name>fs.trash.interval</name><value>360</value><source>core-site.xml</source></property>
<property><name>ha.health-monitor.check-interval.ms</name><value>1000</value><source>core-default.xml</source></property>
<property><name>hadoop.jetty.logs.serve.aliases</name><value>true</value><source>core-default.xml</source></property>
<property><name>hadoop.http.authentication.kerberos.principal</name><value>HTTP/_HOST@LOCALHOST</value><source>core-default.xml</source></property>
<property><name>hbase.dynamic.jars.dir</name><value>${hbase.rootdir}/lib</value><source>hbase-default.xml</source></property>
<property><name>s3native.blocksize</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.dns.interface</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.edits.dir</name><value>${dfs.namenode.name.dir}</value><source>hdfs-default.xml</source></property>
<property><name>ha.health-monitor.sleep-after-disconnect.ms</name><value>1000</value><source>core-default.xml</source></property>
<property><name>dfs.encrypt.data.transfer</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.datanode.http.address</name><value>0.0.0.0:50075</value><source>hdfs-site.xml</source></property>
<property><name>hbase.hstore.blockingStoreFiles</name><value>20</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.write.stale.datanode.ratio</name><value>1.0f</value><source>hdfs-site.xml</source></property>
<property><name>dfs.client.use.datanode.hostname</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>hbase.hstore.bytes.per.checksum</name><value>16384</value><source>hbase-default.xml</source></property>
<property><name>hbase.client.write.buffer</name><value>2097152</value><source>hbase-default.xml</source></property>
<property><name>hadoop.security.instrumentation.requires.admin</name><value>false</value><source>core-default.xml</source></property>
<property><name>hadoop.security.authorization</name><value>false</value><source>core-site.xml</source></property>
<property><name>dfs.namenode.fs-limits.min-block-size</name><value>1048576</value><source>hdfs-default.xml</source></property>
<property><name>dfs.client.failover.connection.retries.on.timeouts</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.search.filter.group</name><value>(objectClass=group)</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.safemode.extension</name><value>30000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.zookeeper.property.clientPort</name><value>2181</value><source>hbase-site.xml</source></property>
<property><name>hbase.regionserver.info.port.auto</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>hbase.coprocessor.region.classes</name><value>org.apache.hadoop.hbase.security.access.AccessController</value><source>hbase-site.xml</source></property>
<property><name>dfs.datanode.sync.behind.writes</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.https.server.keystore.resource</name><value>ssl-server.xml</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.search.attr.group.name</name><value>cn</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.replication.min</name><value>1</value><source>hdfs-default.xml</source></property>
<property><name>hbase.master.logcleaner.ttl</name><value>600000</value><source>hbase-default.xml</source></property>
<property><name>fs.client.resolve.remote.symlinks</name><value>true</value><source>core-default.xml</source></property>
<property><name>s3native.bytes-per-checksum</name><value>512</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.optionallogflushinterval</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>tfile.fs.output.buffer.size</name><value>262144</value><source>core-default.xml</source></property>
<property><name>fs.AbstractFileSystem.hdfs.impl</name><value>org.apache.hadoop.fs.Hdfs</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.codecs</name><value>snappy</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.safemode.min.datanodes</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.uid.cache.secs</name><value>14400</value><source>core-default.xml</source></property>
<property><name>hbase.rootdir</name><value>hdfs://rtpha/apps/hbase/data</value><source>hbase-site.xml</source></property>
<property><name>dfs.client.https.need-auth</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.client.write.exclude.nodes.cache.expiry.interval.millis</name><value>600000</value><source>hdfs-default.xml</source></property>
<property><name>dfs.client.https.keystore.resource</name><value>ssl-client.xml</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.max.objects</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>hbase.zookeeper.dns.interface</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>hadoop.ssl.client.conf</name><value>ssl-client.xml</value><source>core-default.xml</source></property>
<property><name>zookeeper.session.timeout</name><value>30000</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.safemode.threshold-pct</name><value>1.0f</value><source>hdfs-site.xml</source></property>
<property><name>hbase.hstore.compaction.max</name><value>10</value><source>hbase-default.xml</source></property>
<property><name>mapred.task.id</name><value>hb_m_dsjhd113,60000,1402039442506</value><source>programatically</source></property>
<property><name>dfs.blocksize</name><value>134217728</value><source>hdfs-site.xml</source></property>
<property><name>hbase.security.authorization</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>hbase.master.info.bindAddress</name><value>0.0.0.0</value><source>hbase-default.xml</source></property>
<property><name>io.native.lib.available</name><value>true</value><source>core-default.xml</source></property>
<property><name>dfs.client-write-packet-size</name><value>65536</value><source>hdfs-default.xml</source></property>
<property><name>hbase.hstore.compactionThreshold</name><value>3</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.name.dir</name><value>/data/data1/namenode,/data/data2/namenode</value><source>hdfs-site.xml</source></property>
<property><name>dfs.ha.log-roll.period</name><value>120</value><source>hdfs-default.xml</source></property>
<property><name>io.storefile.bloom.block.size</name><value>131072</value><source>hbase-default.xml</source></property>
<property><name>dfs.client.failover.sleep.base.millis</name><value>500</value><source>hdfs-default.xml</source></property>
<property><name>dfs.datanode.directoryscan.threads</name><value>1</value><source>hdfs-default.xml</source></property>
<property><name>hbase.offheapcache.percentage</name><value>0</value><source>hbase-default.xml</source></property>
<property><name>dfs.permissions.enabled</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>hbase.thrift.maxQueuedRequests</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>dfs.support.append</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>hbase.thrift.minWorkerThreads</name><value>16</value><source>hbase-default.xml</source></property>
<property><name>ipc.client.connection.maxidletime</name><value>30000</value><source>core-site.xml</source></property>
<property><name>fs.checkpoint.size</name><value>0.5</value><source>core-site.xml</source></property>
<property><name>hbase.data.umask</name><value>000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.invalidate.work.pct.per.iteration</name><value>0.32f</value><source>hdfs-default.xml</source></property>
<property><name>dfs.blockreport.intervalMsec</name><value>21600000</value><source>hdfs-default.xml</source></property>
<property><name>fs.s3.sleepTimeSeconds</name><value>10</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.replication.considerLoad</name><value>true</value><source>hdfs-default.xml</source></property>
<property><name>dfs.client.block.write.retries</name><value>3</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.ssl.server.conf</name><value>ssl-server.xml</value><source>core-default.xml</source></property>
<property><name>hbase.balancer.period</name><value>300000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.name.dir.restore</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.datanode.hdfs-blocks-metadata.enabled</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>ha.zookeeper.parent-znode</name><value>/hadoop-ha</value><source>core-default.xml</source></property>
<property><name>io.seqfile.lazydecompress</name><value>true</value><source>core-default.xml</source></property>
<property><name>hbase.zookeeper.property.maxClientCnxns</name><value>300</value><source>hbase-default.xml</source></property>
<property><name>dfs.https.enable</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://dsjhd112:8485;dsjhd113:8485;dsjhd051:8485/rtpha</value><source>hdfs-site.xml</source></property>
<property><name>fail.fast.expired.active.master</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.replication</name><value>3</value><source>hdfs-site.xml</source></property>
<property><name>hbase.client.scanner.caching</name><value>100</value><source>hbase-site.xml</source></property>
<property><name>ipc.client.tcpnodelay</name><value>false</value><source>core-default.xml</source></property>
<property><name>hbase.ipc.client.tcpnodelay</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.accesstime.precision</name><value>0</value><source>hdfs-site.xml</source></property>
<property><name>s3.stream-buffer-size</name><value>4096</value><source>core-default.xml</source></property>
<property><name>hbase.client.retries.number</name><value>350</value><source>programatically</source></property>
<property><name>nfs3.server.port</name><value>2049</value><source>core-default.xml</source></property>
<property><name>hbase.client.scanner.timeout.period</name><value>60000</value><source>hbase-default.xml</source></property>
<property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value><source>core-site.xml</source></property>
<property><name>hbase.zookeeper.useMulti</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>hbase.zookeeper.dns.nameserver</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>io.file.buffer.size</name><value>131072</value><source>core-site.xml</source></property>
<property><name>hfile.index.block.max.size</name><value>131072</value><source>hbase-default.xml</source></property>
<property><name>hbase.regionserver.hlog.reader.impl</name><value>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.audit.loggers</name><value>default</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.txns</name><value>1000000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.zookeeper.property.dataDir</name><value>${hbase.tmp.dir}/zookeeper</value><source>hbase-default.xml</source></property>
<property><name>hbase.hregion.memstore.mslab.enabled</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>hbase.regionserver.dns.nameserver</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>hfile.block.index.cacheonwrite</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>hfile.block.cache.size</name><value>0.0</value><source>programatically</source></property>
<property><name>dfs.ha.tail-edits.period</name><value>60</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.authentication</name><value>simple</value><source>core-site.xml</source></property>
<property><name>fs.s3.buffer.dir</name><value>${hadoop.tmp.dir}/s3</value><source>core-default.xml</source></property>
<property><name>hbase.hstore.checksum.algorithm</name><value>CRC32</value><source>hbase-default.xml</source></property>
<property><name>hbase.master.info.port</name><value>60010</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.avoid.read.stale.datanode</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.client.file-block-storage-locations.timeout</name><value>60</value><source>hdfs-default.xml</source></property>
<property><name>hbase.rest.threads.min</name><value>2</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.drop.cache.behind.writes</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>tfile.fs.input.buffer.size</name><value>262144</value><source>core-default.xml</source></property>
<property><name>dfs.block.access.token.enable</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.journalnode.http-address</name><value>0.0.0.0:8480</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.proxyuser.hive.groups</name><value>users</value><source>core-site.xml</source></property>
<property><name>ftp.blocksize</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>hbase.local.dir</name><value>${hbase.tmp.dir}/local/</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.data.dir</name><value>/data/data1,/data/data2,/data/data3,/data/data4,/data/data5,/data/data6</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.policy.file</name><value>hbase-policy.xml</value><source>hbase-default.xml</source></property>
<property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>20000</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.rpc-address.rtpha.nn2</name><value>dsjhd113:8020</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.rpc-address.rtpha.nn1</name><value>dsjhd112:8020</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.max.extra.edits.segments.retained</name><value>10000</value><source>hdfs-default.xml</source></property>
<property><name>dfs.https.port</name><value>50470</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.replication.interval</name><value>3</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.https-address</name><value>dsjhd112,dsjhd113:50470</value><source>hdfs-site.xml</source></property>
<property><name>mapreduce.jobtracker.webinterface.trusted</name><value>false</value><source>core-site.xml</source></property>
<property><name>ipc.client.kill.max</name><value>10</value><source>core-default.xml</source></property>
<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.image.transfer.timeout</name><value>600000</value><source>hdfs-default.xml</source></property>
<property><name>zookeeper.znode.acl.parent</name><value>acl</value><source>hbase-default.xml</source></property>
<property><name>dfs.client.failover.sleep.max.millis</name><value>15000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.superuser</name><value>hbase</value><source>hbase-site.xml</source></property>
<property><name>dfs.default.chunk.view.size</name><value>32768</value><source>hdfs-default.xml</source></property>
<property><name>hbase.auth.key.update.interval</name><value>86400000</value><source>hbase-default.xml</source></property>
<property><name>hbase.server.compactchecker.interval.multiplier</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>hadoop.http.filter.initializers</name><value>org.apache.hadoop.http.lib.StaticUserWebFilter</value><source>core-default.xml</source></property>
<property><name>dfs.datanode.failed.volumes.tolerated</name><value>0</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.http.authentication.type</name><value>simple</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.regionSplitLimit</name><value>2147483647</value><source>hbase-default.xml</source></property>
<property><name>hbase.rest.port</name><value>8080</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.data.dir.perm</name><value>750</value><source>hdfs-site.xml</source></property>
<property><name>ipc.server.listen.queue.size</name><value>128</value><source>core-default.xml</source></property>
<property><name>file.stream-buffer-size</name><value>4096</value><source>core-default.xml</source></property>
<property><name>hbase.storescanner.parallel.seek.threads</name><value>10</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.fs-limits.max-directory-items</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>io.mapfile.bloom.size</name><value>1048576</value><source>core-default.xml</source></property>
<property><name>ftp.replication</name><value>3</value><source>core-default.xml</source></property>
<property><name>dfs.datanode.dns.nameserver</name><value>default</value><source>hdfs-default.xml</source></property>
<property><name>hbase.client.pause</name><value>100</value><source>hbase-default.xml</source></property>
<property><name>hadoop.proxyuser.hcat.hosts</name><value>dsjhd112</value><source>core-site.xml</source></property>
<property><name>dfs.replication.max</name><value>50</value><source>hdfs-site.xml</source></property>
<property><name>hbase.metrics.exposeOperationTimes</name><value>true</value><source>hbase-default.xml</source></property>
<property><name>dfs.stream-buffer-size</name><value>4096</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.backup.address</name><value>0.0.0.0:50100</value><source>hdfs-default.xml</source></property>
<property><name>dfs.hosts.exclude</name><value>/etc/hadoop/conf/dfs.exclude</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.util.hash.type</name><value>murmur</value><source>core-default.xml</source></property>
<property><name>dfs.block.access.key.update.interval</name><value>600</value><source>hdfs-default.xml</source></property>
<property><name>dfs.datanode.dns.interface</name><value>default</value><source>hdfs-default.xml</source></property>
<property><name>dfs.datanode.use.datanode.hostname</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>dfs.client.read.shortcircuit</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.backup.http-address</name><value>0.0.0.0:50105</value><source>hdfs-default.xml</source></property>
<property><name>hbase.snapshot.enabled</name><value>true</value><source>hbase-default.xml</source></property>
<property><name>hbase.rpc.timeout</name><value>60000</value><source>hbase-default.xml</source></property>
<property><name>ha.zookeeper.acl</name><value>world:anyone:rwcda</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.enable.retrycache</name><value>true</value><source>hdfs-default.xml</source></property>
<property><name>io.map.index.skip</name><value>0</value><source>core-default.xml</source></property>
<property><name>net.topology.node.switch.mapping.impl</name><value>org.apache.hadoop.net.ScriptBasedMapping</value><source>core-default.xml</source></property>
<property><name>fs.s3.maxRetries</name><value>4</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.logging.level</name><value>info</value><source>hdfs-default.xml</source></property>
<property><name>ha.failover-controller.new-active.rpc-timeout.ms</name><value>60000</value><source>core-default.xml</source></property>
<property><name>s3native.client-write-packet-size</name><value>65536</value><source>core-default.xml</source></property>
<property><name>hadoop.http.staticuser.user</name><value>dr.who</value><source>core-default.xml</source></property>
<property><name>ha.zookeeper.quorum</name><value>dsjhd051:2181,dsjhd112:2181,dsjhd113:2181</value><source>core-site.xml</source></property>
<property><name>hbase.ipc.client.fallback-to-simple-auth-allowed</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>hadoop.http.authentication.simple.anonymous.allowed</name><value>true</value><source>core-default.xml</source></property>
<property><name>hadoop.fuse.timer.period</name><value>5</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.num.extra.edits.retained</name><value>1000000</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.rpc.socket.factory.class.default</name><value>org.apache.hadoop.net.StandardSocketFactory</value><source>core-default.xml</source></property>
<property><name>hbase.hstore.blockingWaitTime</name><value>90000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.handler.count</name><value>100</value><source>hdfs-site.xml</source></property>
<property><name>fs.automatic.close</name><value>true</value><source>core-default.xml</source></property>
<property><name>dfs.datanode.directoryscan.interval</name><value>21600</value><source>hdfs-default.xml</source></property>
<property><name>hbase.coprocessor.abortonerror</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>hbase.zookeeper.quorum</name><value>dsjhd051,dsjhd112,dsjhd113</value><source>hbase-site.xml</source></property>
<property><name>dfs.client.file-block-storage-locations.num-threads</name><value>10</value><source>hdfs-default.xml</source></property>
<property><name>hbase.defaults.for.version</name><value>0.96.0.2.0.6.0-76-hadoop2</value><source>hbase-default.xml</source></property>
<property><name>hbase.hregion.max.filesize</name><value>10737418240</value><source>hbase-site.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.directory.search.timeout</name><value>10000</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.logroll.errors.tolerated</name><value>2</value><source>hbase-default.xml</source></property>
<property><name>dfs.bytes-per-checksum</name><value>512</value><source>hdfs-default.xml</source></property>
<property><name>hbase.status.published</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.max-retries</name><value>3</value><source>hdfs-default.xml</source></property>
<property><name>zookeeper.znode.parent</name><value>/hbase-unsecure</value><source>hbase-site.xml</source></property>
<property><name>ftp.stream-buffer-size</name><value>4096</value><source>core-default.xml</source></property>
<property><name>ha.health-monitor.rpc-timeout.ms</name><value>45000</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.avoid.write.stale.datanode</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.edits.noeditlogchannelflush</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.search.attr.member</name><value>member</value><source>core-default.xml</source></property>
<property><name>dfs.blockreport.initialDelay</name><value>120</value><source>hdfs-site.xml</source></property>
<property><name>io.compression.codec.bzip2.library</name><value>system-native</value><source>core-default.xml</source></property>
<property><name>hadoop.http.authentication.token.validity</name><value>36000</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.delegation.token.max-lifetime</name><value>604800000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.metrics.showTableName</name><value>true</value><source>hbase-default.xml</source></property>
<property><name>s3native.replication</name><value>3</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.hlog.writer.impl</name><value>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter</value><source>hbase-default.xml</source></property>
<property><name>hbase.zookeeper.property.syncLimit</name><value>5</value><source>hbase-default.xml</source></property>
<property><name>dfs.heartbeat.interval</name><value>3</value><source>hdfs-site.xml</source></property>
<property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value><source>core-default.xml</source></property>
<property><name>hbase.thrift.htablepool.size.max</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>net.topology.impl</name><value>org.apache.hadoop.net.NetworkTopology</value><source>core-default.xml</source></property>
<property><name>dfs.nameservices</name><value>rtpha</value><source>hdfs-site.xml</source></property>
<property><name>zookeeper.znode.rootserver</name><value>root-region-server</value><source>hbase-default.xml</source></property>
<property><name>hbase.regionserver.msginterval</name><value>3000</value><source>hbase-default.xml</source></property>
<property><name>ha.failover-controller.graceful-fence.rpc-timeout.ms</name><value>5000</value><source>core-default.xml</source></property>
<property><name>hbase.table.lock.enable</name><value>true</value><source>hbase-default.xml</source></property>
<property><name>hbase.rest.threads.max</name><value>100</value><source>hbase-default.xml</source></property>
<property><name>dfs.client.read.shortcircuit.streams.cache.size</name><value>4096</value><source>hdfs-site.xml</source></property>
<property><name>hbase.master.logcleaner.plugins</name><value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner,org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner</value><source>programatically</source></property>
<property><name>dfs.namenode.secondary.http-address</name><value>null</value><source>hdfs-site.xml</source></property>
<property><name>hbase.coprocessor.master.classes</name><value>org.apache.hadoop.hbase.security.access.AccessController</value><source>hbase-site.xml</source></property>
<property><name>ipc.client.idlethreshold</name><value>8000</value><source>core-site.xml</source></property>
<property><name>ipc.server.tcpnodelay</name><value>false</value><source>core-default.xml</source></property>
<property><name>ftp.bytes-per-checksum</name><value>512</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.stale.datanode.interval</name><value>30000</value><source>hdfs-site.xml</source></property>
<property><name>s3.bytes-per-checksum</name><value>512</value><source>core-default.xml</source></property>
<property><name>hbase.rpc.shortoperation.timeout</name><value>10000</value><source>hbase-default.xml</source></property>
<property><name>hbase.storescanner.parallel.seek.enable</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.ha.fencing.methods</name><value>shell(/bin/true)</value><source>hdfs-site.xml</source></property>
<property><name>dfs.namenode.http-address.rtpha.nn2</name><value>dsjhd113:50070</value><source>hdfs-site.xml</source></property>
<property><name>hbase.online.schema.update.enable</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.http-address.rtpha.nn1</name><value>dsjhd112:50070</value><source>hdfs-site.xml</source></property>
<property><name>fs.s3.block.size</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>dfs.client.failover.connection.retries</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>hbase.status.multicast.address.ip</name><value>226.1.1.3</value><source>hbase-default.xml</source></property>
<property><name>hadoop.rpc.protection</name><value>authentication</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.retrycache.expirytime.millis</name><value>600000</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.security.auth_to_local</name><value>
RULE:[2:$1@$0]([rn]m@.*)s/.*/yarn/
RULE:[2:$1@$0](jhs@.*)s/.*/mapred/
RULE:[2:$1@$0]([nd]n@.*)s/.*/hdfs/
RULE:[2:$1@$0](hm@.*)s/.*/hbase/
RULE:[2:$1@$0](rs@.*)s/.*/hbase/
DEFAULT</value><source>core-site.xml</source></property>
<property><name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name><value>${dfs.web.authentication.kerberos.principal}</value><source>hdfs-default.xml</source></property>
<property><name>dfs.journalnode.edits.dir</name><value>/grid/0/hdfs/journal</value><source>hdfs-site.xml</source></property>
<property><name>ftp.client-write-packet-size</name><value>65536</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.checksum.verify</name><value>true</value><source>hbase-default.xml</source></property>
<property><name>fs.defaultFS</name><value>hdfs://rtpha</value><source>because fs.default.name is deprecated</source></property>
<property><name>file.client-write-packet-size</name><value>65536</value><source>core-default.xml</source></property>
<property><name>hbase.tmp.dir</name><value>/opt/hadoop/hbase</value><source>hbase-site.xml</source></property>
<property><name>hbase.zookeeper.property.initLimit</name><value>10</value><source>hbase-default.xml</source></property>
<property><name>fs.trash.checkpoint.interval</name><value>0</value><source>core-default.xml</source></property>
<property><name>hbase.cluster.distributed</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>hadoop.http.authentication.signature.secret.file</name><value>${user.home}/hadoop-http-auth-signature-secret</value><source>core-default.xml</source></property>
<property><name>s3native.stream-buffer-size</name><value>4096</value><source>core-default.xml</source></property>
<property><name>hbase.thrift.maxWorkerThreads</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.edits.dir</name><value>${dfs.namenode.checkpoint.dir}</value><source>hdfs-site.xml</source></property>
<property><name>fs.permissions.umask-mode</name><value>022</value><source>hdfs-site.xml</source></property>
<property><name>hbase.hstore.compaction.kv.max</name><value>10</value><source>hbase-default.xml</source></property>
<property><name>hadoop.common.configuration.version</name><value>0.23.0</value><source>core-default.xml</source></property>
<property><name>hbase.bulkload.retries.number</name><value>0</value><source>hbase-default.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.ssl</name><value>false</value><source>core-default.xml</source></property>
<property><name>dfs.client.failover.proxy.provider.rtpha</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value><source>hdfs-site.xml</source></property>
<property><name>io.serializations</name><value>org.apache.hadoop.io.serializer.WritableSerialization</value><source>core-site.xml</source></property>
<property><name>fs.df.interval</name><value>60000</value><source>core-default.xml</source></property>
<property><name>hbase.rest.readonly</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>io.seqfile.compress.blocksize</name><value>1000000</value><source>core-default.xml</source></property>
<property><name>hbase.client.localityCheck.threadPoolSize</name><value>2</value><source>hbase-default.xml</source></property>
<property><name>ipc.client.connect.max.retries</name><value>50</value><source>core-site.xml</source></property>
<property><name>hadoop.security.groups.cache.secs</name><value>300</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.delegation.key.update-interval</name><value>86400000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.regionserver.port</name><value>60020</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name><value>0.75f</value><source>hdfs-default.xml</source></property>
<property><name>hbase.regionserver.handler.count</name><value>60</value><source>hbase-site.xml</source></property>
<property><name>hadoop.security.group.mapping.ldap.search.filter.user</name><value>(&(objectClass=user)(sAMAccountName={0}))</value><source>core-default.xml</source></property>
<property><name>dfs.image.compress</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>hbase.hregion.majorcompaction</name><value>604800000</value><source>hbase-site.xml</source></property>
<property><name>dfs.namenode.kerberos.internal.spnego.principal</name><value>${dfs.web.authentication.kerberos.principal}</value><source>hdfs-default.xml</source></property>
<property><name>fs.s3n.block.size</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>hbase.regionserver.optionalcacheflushinterval</name><value>3600000</value><source>hbase-default.xml</source></property>
<property><name>fs.ftp.host</name><value>0.0.0.0</value><source>core-default.xml</source></property>
<property><name>hadoop.security.group.mapping</name><value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value><source>core-default.xml</source></property>
<property><name>dfs.datanode.address</name><value>0.0.0.0:50010</value><source>hdfs-site.xml</source></property>
<property><name>dfs.datanode.https.address</name><value>0.0.0.0:50475</value><source>hdfs-default.xml</source></property>
<property><name>file.replication</name><value>1</value><source>core-default.xml</source></property>
<property><name>hbase.rest.filter.classes</name><value>org.apache.hadoop.hbase.rest.filter.GzipFilter</value><source>hbase-default.xml</source></property>
<property><name>hbase.config.read.zookeeper.config</name><value>false</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.drop.cache.behind.reads</name><value>false</value><source>hdfs-default.xml</source></property>
<property><name>hbase.regionserver.global.memstore.upperLimit</name><value>0.4</value><source>hbase-site.xml</source></property>
<property><name>hadoop.fuse.connection.timeout</name><value>300</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.work.around.non.threadsafe.getpwuid</name><value>false</value><source>core-default.xml</source></property>
<property><name>mapreduce.client.genericoptionsparser.used</name><value>true</value><source>programatically</source></property>
<property><name>hadoop.tmp.dir</name><value>/tmp/hadoop-${user.name}</value><source>core-default.xml</source></property>
<property><name>dfs.client.block.write.replace-datanode-on-failure.policy</name><value>DEFAULT</value><source>hdfs-default.xml</source></property>
<property><name>hbase.hregion.preclose.flush.size</name><value>5242880</value><source>hbase-default.xml</source></property>
<property><name>hbase.regionserver.global.memstore.lowerLimit</name><value>0.38</value><source>hbase-site.xml</source></property>
<property><name>dfs.cluster.administrators</name><value> hdfs</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.kerberos.kinit.command</name><value>kinit</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.fs-limits.max-blocks-per-file</name><value>1048576</value><source>hdfs-default.xml</source></property>
<property><name>dfs.webhdfs.enabled</name><value>true</value><source>hdfs-site.xml</source></property>
<property><name>dfs.datanode.du.reserved</name><value>1073741824</value><source>hdfs-site.xml</source></property>
<property><name>fs.default.name</name><value>hdfs://rtpha</value><source>programatically</source></property>
<property><name>file.bytes-per-checksum</name><value>512</value><source>core-default.xml</source></property>
<property><name>hbase.master.dns.interface</name><value>default</value><source>hbase-default.xml</source></property>
<property><name>dfs.client.block.write.replace-datanode-on-failure.enable</name><value>true</value><source>hdfs-default.xml</source></property>
<property><name>net.topology.script.number.args</name><value>100</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.decommission.interval</name><value>30</value><source>hdfs-default.xml</source></property>
<property><name>hbase.defaults.for.version.skip</name><value>true</value><source>hbase-site.xml</source></property>
<property><name>dfs.image.compression.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value><source>hdfs-default.xml</source></property>
<property><name>hadoop.ssl.hostname.verifier</name><value>DEFAULT</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.support.allow.format</name><value>true</value><source>hdfs-default.xml</source></property>
<property><name>ipc.client.connect.timeout</name><value>20000</value><source>core-default.xml</source></property>
<property><name>hbase.server.versionfile.writeattempts</name><value>3</value><source>hbase-default.xml</source></property>
<property><name>io.mapfile.bloom.error.rate</name><value>0.005</value><source>core-default.xml</source></property>
<property><name>dfs.permissions.superusergroup</name><value>hdfs</value><source>hdfs-site.xml</source></property>
<property><name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name><value>10737418240</value><source>hdfs-default.xml</source></property>
<property><name>hbase.regionserver.info.bindAddress</name><value>0.0.0.0</value><source>hbase-default.xml</source></property>
<property><name>hbase.master.hfilecleaner.plugins</name><value>org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner,org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner</value><source>programatically</source></property>
<property><name>ha.failover-controller.graceful-fence.connection.retries</name><value>1</value><source>core-default.xml</source></property>
<property><name>ha.health-monitor.connect-retry-interval.ms</name><value>1000</value><source>core-default.xml</source></property>
<property><name>hbase.security.authentication</name><value>simple</value><source>hbase-site.xml</source></property>
<property><name>io.seqfile.local.dir</name><value>${hadoop.tmp.dir}/io/local</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.check.period</name><value>60</value><source>hdfs-default.xml</source></property>
<property><name>tfile.io.chunk.size</name><value>1048576</value><source>core-default.xml</source></property>
<property><name>file.blocksize</name><value>67108864</value><source>core-default.xml</source></property>
<property><name>io.skip.checksum.errors</name><value>false</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.edits.journal-plugin.qjournal</name><value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value><source>hdfs-default.xml</source></property>
<property><name>hbase.zookeeper.peerport</name><value>2888</value><source>hbase-default.xml</source></property>
<property><name>dfs.datanode.handler.count</name><value>10</value><source>hdfs-default.xml</source></property>
<property><name>hbase.master.port</name><value>60000</value><source>hbase-default.xml</source></property>
<property><name>fs.ftp.host.port</name><value>21</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.decommission.nodes.per.interval</name><value>5</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.fs-limits.max-component-length</name><value>0</value><source>hdfs-default.xml</source></property>
<property><name>dfs.namenode.checkpoint.period</name><value>21600</value><source>hdfs-site.xml</source></property>
<property><name>fs.AbstractFileSystem.viewfs.impl</name><value>org.apache.hadoop.fs.viewfs.ViewFs</value><source>core-default.xml</source></property>
<property><name>dfs.namenode.retrycache.heap.percent</name><value>0.03f</value><source>hdfs-default.xml</source></property>
<property><name>ipc.client.fallback-to-simple-auth-allowed</name><value>false</value><source>core-default.xml</source></property>
<property><name>dfs.ha.namenodes.rtpha</name><value>nn1,nn2</value><source>hdfs-site.xml</source></property>
<property><name>hbase.status.multicast.address.port</name><value>6100</value><source>hbase-default.xml</source></property>
<property><name>hbase.client.keyvalue.maxsize</name><value>10485760</value><source>hbase-site.xml</source></property>
<property><name>dfs.datanode.ipc.address</name><value>0.0.0.0:8010</value><source>hdfs-site.xml</source></property>
<property><name>hadoop.ssl.require.client.cert</name><value>false</value><source>core-default.xml</source></property>
<property><name>hbase.hregion.memstore.block.multiplier</name><value>8</value><source>hbase-site.xml</source></property>
<property><name>dfs.datanode.max.transfer.threads</name><value>1024</value><source>hdfs-site.xml</source></property>
<property><name>hbase.server.thread.wakefrequency</name><value>10000</value><source>hbase-default.xml</source></property>
</configuration>
Recent regionserver aborts:
===========================================================
2014-06-06T17:10:24 Region server dsjhd058,60020,1402038676433 reported a fatal error:
ABORTING region server dsjhd058,60020,1402038676433: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing dsjhd058,60020,1402038676433 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:339)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1351)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5087)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Cause:
org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing dsjhd058,60020,1402038676433 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:339)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1351)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5087)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:235)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:991)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:838)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException): org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing dsjhd058,60020,1402038676433 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:339)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1351)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5087)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1650)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1708)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:5414)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:989)
... 2 more
2014-06-06T17:10:24 Region server dsjhd058,60020,1402038676433 reported a fatal error:
ABORTING region server dsjhd058,60020,1402038676433: IOE in log roller
Cause:
java.io.FileNotFoundException: File does not exist: hdfs://rtpha/apps/hbase/data/WALs/dsjhd058,60020,1402038676433/dsjhd058%2C60020%2C1402038676433.1402045785595
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:527)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:96)
at java.lang.Thread.run(Thread.java:662)
2014-06-06T17:10:25 Region server dsjhd058,60020,1402038676433 reported a fatal error:
ABORTING region server dsjhd058,60020,1402038676433: regionserver:60020-0x1466fded954006b regionserver:60020-0x1466fded954006b received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:401)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:319)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Logs
===========================================================
+++++++++++++++++++++++++++++++
/var/log/hbase/hbase-hbase-master-dsjhd113.log
+++++++++++++++++++++++++++++++
2014-06-06 18:06:13,309 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402044753437
2014-06-06 18:06:13,312 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048318944
2014-06-06 18:06:13,315 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048321383
2014-06-06 18:06:13,318 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048323708
2014-06-06 18:06:13,321 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048326167
2014-06-06 18:06:13,323 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048328583
2014-06-06 18:06:13,326 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048331641
2014-06-06 18:06:13,329 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048333978
2014-06-06 18:06:13,332 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048337079
2014-06-06 18:06:13,334 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd077%2C60020%2C1402027915865.1402045440987
2014-06-06 18:06:13,337 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd077%2C60020%2C1402027915865.1402045531965
2014-06-06 18:06:13,340 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd077%2C60020%2C1402027915865.1402045576448
2014-06-06 18:06:13,343 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd077%2C60020%2C1402027915865.1402045628744
2014-06-06 18:06:13,346 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd077%2C60020%2C1402027915865.1402045714430
2014-06-06 18:07:13,293 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd051%2C60020%2C1402036579439.1402045196462
2014-06-06 18:07:13,297 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402046845630
2014-06-06 18:07:13,300 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402046865873
2014-06-06 18:07:13,303 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402046873000
2014-06-06 18:07:13,306 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402046884848
2014-06-06 18:07:13,309 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402046969319
2014-06-06 18:07:13,313 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd052%2C60020%2C1402039731411.1402047028326
2014-06-06 18:07:13,315 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd053%2C60020%2C1402027976229.1402044988600
2014-06-06 18:07:13,318 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd053%2C60020%2C1402027976229.1402045038538
2014-06-06 18:07:13,321 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd053%2C60020%2C1402027976229.1402045219706
2014-06-06 18:07:13,324 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402044762886
2014-06-06 18:07:13,327 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd063%2C60020%2C1402039188138.1402045242796
2014-06-06 18:07:13,330 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd063%2C60020%2C1402039188138.1402045388005
2014-06-06 18:07:13,333 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd063%2C60020%2C1402039188138.1402045433680
2014-06-06 18:07:13,336 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd066%2C60020%2C1402027920604.1402045883546
2014-06-06 18:07:13,340 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402045757267
2014-06-06 18:07:13,343 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046061894
2014-06-06 18:07:13,346 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046197639
2014-06-06 18:07:13,349 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046215876
2014-06-06 18:07:13,352 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046240616
2014-06-06 18:07:13,355 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046294360
2014-06-06 18:07:13,358 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046449669
2014-06-06 18:07:13,361 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd069%2C60020%2C1402032388543.1402046475135
2014-06-06 18:07:13,366 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048340021
2014-06-06 18:07:13,370 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048342408
2014-06-06 18:07:13,373 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048345085
2014-06-06 18:07:13,377 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048347837
2014-06-06 18:08:13,288 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048349938
2014-06-06 18:08:13,291 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048351695
2014-06-06 18:08:13,294 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048354186
2014-06-06 18:10:05,365 DEBUG [AM.ZK.Worker-pool2-t394] master.AssignmentManager: Handling RS_ZK_REQUEST_REGION_SPLIT, server=dsjhd064,60020,1402027918813, region=5544bd37b36ea28e890ba839f4506af6, current_state={5544bd37b36ea28e890ba839f4506af6 state=OPEN, ts=1402048435327, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:05,366 DEBUG [AM.ZK.Worker-pool2-t394] zookeeper.ZKAssign: master:60000-0x2466f52fa5c01c6 Transitioning 5544bd37b36ea28e890ba839f4506af6 from RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REGION_SPLITTING
2014-06-06 18:10:05,369 DEBUG [AM.ZK.Worker-pool2-t394] zookeeper.ZKAssign: master:60000-0x2466f52fa5c01c6 Transitioned node 5544bd37b36ea28e890ba839f4506af6 from RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REGION_SPLITTING
2014-06-06 18:10:05,369 INFO [AM.ZK.Worker-pool2-t394] master.RegionStates: Transitioned {69f69a03e5ac38c7c12003a6aee3df39 state=OFFLINE, ts=1402049405369, server=null} to {69f69a03e5ac38c7c12003a6aee3df39 state=SPLITTING_NEW, ts=1402049405369, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:05,369 INFO [AM.ZK.Worker-pool2-t394] master.RegionStates: Transitioned {d1095a2fffc1a67c1fac9e700ef54679 state=OFFLINE, ts=1402049405369, server=null} to {d1095a2fffc1a67c1fac9e700ef54679 state=SPLITTING_NEW, ts=1402049405369, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:05,369 INFO [AM.ZK.Worker-pool2-t394] master.RegionStates: Transitioned {5544bd37b36ea28e890ba839f4506af6 state=OPEN, ts=1402048435327, server=dsjhd064,60020,1402027918813} to {5544bd37b36ea28e890ba839f4506af6 state=SPLITTING, ts=1402049405369, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:05,371 DEBUG [AM.ZK.Worker-pool2-t395] master.AssignmentManager: Handling RS_ZK_REGION_SPLITTING, server=dsjhd064,60020,1402027918813, region=5544bd37b36ea28e890ba839f4506af6, current_state={5544bd37b36ea28e890ba839f4506af6 state=SPLITTING, ts=1402049405369, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:07,335 DEBUG [AM.ZK.Worker-pool2-t396] master.AssignmentManager: Handling RS_ZK_REGION_SPLIT, server=dsjhd064,60020,1402027918813, region=5544bd37b36ea28e890ba839f4506af6, current_state={5544bd37b36ea28e890ba839f4506af6 state=SPLITTING, ts=1402049405371, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:07,335 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Transitioned {5544bd37b36ea28e890ba839f4506af6 state=SPLITTING, ts=1402049407335, server=dsjhd064,60020,1402027918813} to {5544bd37b36ea28e890ba839f4506af6 state=SPLIT, ts=1402049407335, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:07,335 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Offlined 5544bd37b36ea28e890ba839f4506af6 from dsjhd064,60020,1402027918813
2014-06-06 18:10:07,335 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Transitioned {69f69a03e5ac38c7c12003a6aee3df39 state=SPLITTING_NEW, ts=1402049407335, server=dsjhd064,60020,1402027918813} to {69f69a03e5ac38c7c12003a6aee3df39 state=OPEN, ts=1402049407335, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:07,335 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Onlined 69f69a03e5ac38c7c12003a6aee3df39 on dsjhd064,60020,1402027918813
2014-06-06 18:10:07,335 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Transitioned {d1095a2fffc1a67c1fac9e700ef54679 state=SPLITTING_NEW, ts=1402049407335, server=dsjhd064,60020,1402027918813} to {d1095a2fffc1a67c1fac9e700ef54679 state=OPEN, ts=1402049407335, server=dsjhd064,60020,1402027918813}
2014-06-06 18:10:07,336 INFO [AM.ZK.Worker-pool2-t396] master.RegionStates: Onlined d1095a2fffc1a67c1fac9e700ef54679 on dsjhd064,60020,1402027918813
2014-06-06 18:10:07,336 DEBUG [AM.ZK.Worker-pool2-t396] master.AssignmentManager: Handling SPLIT event for 5544bd37b36ea28e890ba839f4506af6; deleting node
2014-06-06 18:10:07,338 DEBUG [AM.ZK.Worker-pool2-t396] zookeeper.ZKAssign: master:60000-0x2466f52fa5c01c6 Deleted unassigned node 5544bd37b36ea28e890ba839f4506af6 in expected state RS_ZK_REGION_SPLIT
2014-06-06 18:10:07,338 INFO [AM.ZK.Worker-pool2-t396] master.AssignmentManager: Handled SPLIT event; parent=ERR_BT_D_BF001_201406,0000000000000002014060517561739900000000000000000800000000000000000004744400000005,1402048433136.5544bd37b36ea28e890ba839f4506af6., daughter a=ERR_BT_D_BF001_201406,0000000000000002014060517561739900000000000000000800000000000000000004744400000005,1402049404816.69f69a03e5ac38c7c12003a6aee3df39., daughter b=ERR_BT_D_BF001_201406,00000000000000020140605181642335,I074,I07000000008000000000000000000053,1402049404816.d1095a2fffc1a67c1fac9e700ef54679., on dsjhd064,60020,1402027918813
2014-06-06 18:10:25,948 DEBUG [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Deleting region BT_D_BF001_201406,46003636179312,1402045922425.c52d75455c6e36c8fb3243959f4cb1a0. because daughter splits no longer hold references
2014-06-06 18:10:25,948 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: ARCHIVING hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0
2014-06-06 18:10:25,952 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf]
2014-06-06 18:10:25,965 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/09188096e8e647a5990120b5f7583bfd, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/09188096e8e647a5990120b5f7583bfd
2014-06-06 18:10:25,969 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/233b2b45bef84a578b7e09221e593250, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/233b2b45bef84a578b7e09221e593250
2014-06-06 18:10:25,973 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/2782f43fd3794df5b3952069bb554df6, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/2782f43fd3794df5b3952069bb554df6
2014-06-06 18:10:25,977 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/3b45c8be193845ab8d46e3af5709eca4, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/3b45c8be193845ab8d46e3af5709eca4
2014-06-06 18:10:25,980 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/4472e97d5aa2403bbd5df3531ce15329, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/4472e97d5aa2403bbd5df3531ce15329
2014-06-06 18:10:25,984 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/6549a9f976e4407f82476b8852c9759d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/6549a9f976e4407f82476b8852c9759d
2014-06-06 18:10:25,988 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/6a4814415fa84da1b2da373704834a41, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/6a4814415fa84da1b2da373704834a41
2014-06-06 18:10:25,992 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/7718f6d4de0143f1b1425dc3363ed9df, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/7718f6d4de0143f1b1425dc3363ed9df
2014-06-06 18:10:25,996 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/81c34c3f26784f5daa3c2ddbe683933e, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/81c34c3f26784f5daa3c2ddbe683933e
2014-06-06 18:10:25,999 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/8ed842bbf31d4da7b7cf4ef3d1611e33, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/8ed842bbf31d4da7b7cf4ef3d1611e33
2014-06-06 18:10:26,003 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/a0a99388df4e49a580aa5b2ff6e8b707, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/a0a99388df4e49a580aa5b2ff6e8b707
2014-06-06 18:10:26,007 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/a4ef628cdb0f4161b840420871ab360d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/a4ef628cdb0f4161b840420871ab360d
2014-06-06 18:10:26,010 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/bf308a440f034cb78e5d3d8c94a3feb8, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/bf308a440f034cb78e5d3d8c94a3feb8
2014-06-06 18:10:26,014 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/d035722daf224d5289b155b17b005f7d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/d035722daf224d5289b155b17b005f7d
2014-06-06 18:10:26,018 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/d0d38e8c15c74bca90607753edd1bf23, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/d0d38e8c15c74bca90607753edd1bf23
2014-06-06 18:10:26,022 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/e11cbfc28a4743a6aabdafbb6c470303, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/e11cbfc28a4743a6aabdafbb6c470303
2014-06-06 18:10:26,026 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/f4d31cab976b4504b50829ca009a68f2, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/f4d31cab976b4504b50829ca009a68f2
2014-06-06 18:10:26,030 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/fa4c35d84bcf432a957587970e6d4867, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0/cf/fa4c35d84bcf432a957587970e6d4867
2014-06-06 18:10:26,032 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Deleted all region files in: hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/c52d75455c6e36c8fb3243959f4cb1a0
2014-06-06 18:10:26,039 INFO [CatalogJanitor-dsjhd113:60000] catalog.MetaEditor: Deleted BT_D_BF001_201406,46003636179312,1402045922425.c52d75455c6e36c8fb3243959f4cb1a0.
2014-06-06 18:10:26,043 DEBUG [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Deleting region DPI_ORIGINAL_DETAIL,0615144540420140530074247793;3,1401820613695.92e752bd4fc0afbe1dece267ffa8e21a. because daughter splits no longer hold references
2014-06-06 18:10:26,043 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: ARCHIVING hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a
2014-06-06 18:10:26,047 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D]
2014-06-06 18:10:26,059 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/0624d17a1dc64628adf703e0b78bf3c3, to hdfs://rtpha/apps/hbase/data/archive/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/0624d17a1dc64628adf703e0b78bf3c3
2014-06-06 18:10:26,063 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/1fbcb12740a843caa34e7132e313f8b4, to hdfs://rtpha/apps/hbase/data/archive/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/1fbcb12740a843caa34e7132e313f8b4
2014-06-06 18:10:26,067 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/70c5979cdc1b4eb2bac6045de144ea94, to hdfs://rtpha/apps/hbase/data/archive/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/70c5979cdc1b4eb2bac6045de144ea94
2014-06-06 18:10:26,071 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/79fb17a402d54e1b981441c337e6db6d, to hdfs://rtpha/apps/hbase/data/archive/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/79fb17a402d54e1b981441c337e6db6d
2014-06-06 18:10:26,075 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/ddfd947bad8247ef83365eb93455c318, to hdfs://rtpha/apps/hbase/data/archive/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a/D/ddfd947bad8247ef83365eb93455c318
2014-06-06 18:10:26,077 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Deleted all region files in: hdfs://rtpha/apps/hbase/data/data/default/DPI_ORIGINAL_DETAIL/92e752bd4fc0afbe1dece267ffa8e21a
2014-06-06 18:10:26,081 INFO [CatalogJanitor-dsjhd113:60000] catalog.MetaEditor: Deleted DPI_ORIGINAL_DETAIL,0615144540420140530074247793;3,1401820613695.92e752bd4fc0afbe1dece267ffa8e21a.
2014-06-06 18:10:26,087 INFO [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Scanned 9780 catalog row(s), gc'd 0 unreferenced merged region(s) and 2 unreferenced parent region(s)
2014-06-06 18:11:13,335 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd054%2C60020%2C1402027922467.1402045732230
2014-06-06 18:11:13,339 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd059%2C60020%2C1402027941533.1402045232669
2014-06-06 18:11:13,342 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd068%2C60020%2C1402038513322.1402045236289
2014-06-06 18:11:13,345 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd068%2C60020%2C1402038513322.1402045337919
2014-06-06 18:11:13,348 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd068%2C60020%2C1402038513322.1402045399892
2014-06-06 18:11:13,350 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd070%2C60020%2C1402027928334.1402045312779
2014-06-06 18:11:13,353 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd073%2C60020%2C1402039874233.1402044797547
2014-06-06 18:11:13,356 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd073%2C60020%2C1402039874233.1402045004966
2014-06-06 18:11:13,359 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd073%2C60020%2C1402039874233.1402045058240
2014-06-06 18:12:13,294 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd051%2C60020%2C1402036579439.1402045234168
2014-06-06 18:12:13,298 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd051%2C60020%2C1402036579439.1402045295397
2014-06-06 18:12:13,301 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd062%2C60020%2C1402027917800.1402045424873
2014-06-06 18:12:13,305 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd062%2C60020%2C1402027917800.1402045450854
2014-06-06 18:12:13,308 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd062%2C60020%2C1402027917800.1402045572219
2014-06-06 18:12:13,311 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd062%2C60020%2C1402027917800.1402045631206
2014-06-06 18:12:13,314 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd062%2C60020%2C1402027917800.1402045722152
2014-06-06 18:12:13,317 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048356263
2014-06-06 18:13:13,330 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048404306
2014-06-06 18:13:13,333 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048409431
2014-06-06 18:13:13,337 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048414324
2014-06-06 18:13:13,340 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048418192
2014-06-06 18:13:13,344 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048422871
2014-06-06 18:13:13,347 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048431169
2014-06-06 18:13:13,350 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048435798
2014-06-06 18:13:13,353 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048439181
2014-06-06 18:13:13,357 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048443573
2014-06-06 18:13:13,360 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048448330
2014-06-06 18:13:13,363 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048453232
2014-06-06 18:13:13,365 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048457409
2014-06-06 18:13:13,368 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048358607
2014-06-06 18:13:13,371 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048361006
2014-06-06 18:13:13,374 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048363409
2014-06-06 18:13:13,377 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048366086
2014-06-06 18:13:13,380 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048368136
2014-06-06 18:13:13,383 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048370503
2014-06-06 18:13:13,385 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048372858
2014-06-06 18:13:13,388 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048374780
2014-06-06 18:13:13,391 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048377961
2014-06-06 18:13:13,393 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048380427
2014-06-06 18:13:13,396 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048382715
2014-06-06 18:13:13,399 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048385145
2014-06-06 18:13:13,402 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048387269
2014-06-06 18:13:13,404 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048389348
2014-06-06 18:13:13,407 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048391668
2014-06-06 18:13:13,411 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048393897
2014-06-06 18:13:13,416 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048396363
2014-06-06 18:13:13,420 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048398455
2014-06-06 18:13:13,424 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048400758
2014-06-06 18:13:13,426 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048403003
2014-06-06 18:13:13,429 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048405264
2014-06-06 18:13:13,431 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048407693
2014-06-06 18:13:13,434 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048410246
2014-06-06 18:13:13,436 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048412354
2014-06-06 18:13:13,439 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048414252
2014-06-06 18:13:13,443 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048416296
2014-06-06 18:13:13,447 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048418081
2014-06-06 18:13:13,449 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048420247
2014-06-06 18:13:13,454 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048422449
2014-06-06 18:13:13,458 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048428306
2014-06-06 18:13:13,460 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048430604
2014-06-06 18:13:13,462 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048432636
2014-06-06 18:13:13,465 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048434667
2014-06-06 18:13:13,467 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048436255
2014-06-06 18:13:13,469 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048438110
2014-06-06 18:13:13,472 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048440067
2014-06-06 18:13:13,474 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048442099
2014-06-06 18:13:13,476 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048444210
2014-06-06 18:13:13,479 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048446472
2014-06-06 18:13:13,481 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048448642
2014-06-06 18:13:13,483 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048451055
2014-06-06 18:13:13,485 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048453212
2014-06-06 18:13:13,488 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048455437
2014-06-06 18:13:13,490 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048457451
2014-06-06 18:13:13,492 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048459399
2014-06-06 18:13:13,495 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048461584
2014-06-06 18:13:13,497 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048463747
2014-06-06 18:13:13,500 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048466067
2014-06-06 18:13:13,502 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048468292
2014-06-06 18:13:13,505 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048470343
2014-06-06 18:13:13,507 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048472592
2014-06-06 18:13:13,510 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048474951
2014-06-06 18:13:13,513 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048477238
2014-06-06 18:13:13,515 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048479335
2014-06-06 18:13:13,518 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048481455
2014-06-06 18:13:13,521 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048483610
2014-06-06 18:13:13,523 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048485697
2014-06-06 18:13:13,526 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048487881
2014-06-06 18:13:13,528 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048489715
2014-06-06 18:13:13,531 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048491548
2014-06-06 18:13:13,534 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048493310
2014-06-06 18:13:13,536 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048495628
2014-06-06 18:13:13,562 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048497786
2014-06-06 18:13:13,566 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048499379
2014-06-06 18:13:13,568 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048501067
2014-06-06 18:13:13,571 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048502819
2014-06-06 18:13:13,574 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048504555
2014-06-06 18:13:13,577 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048506785
2014-06-06 18:13:13,580 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048508694
2014-06-06 18:13:13,585 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048510729
2014-06-06 18:13:13,587 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048512935
2014-06-06 18:13:13,589 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048515890
2014-06-06 18:13:13,591 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048518710
2014-06-06 18:13:13,593 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048523891
2014-06-06 18:13:48,897 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:14:13,327 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048461989
2014-06-06 18:14:13,330 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048466817
2014-06-06 18:14:13,333 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048471631
2014-06-06 18:14:13,336 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048476684
2014-06-06 18:14:13,339 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048481330
2014-06-06 18:14:13,342 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048485970
2014-06-06 18:14:13,345 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048490385
2014-06-06 18:14:13,348 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048494612
2014-06-06 18:14:13,351 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048499111
2014-06-06 18:14:13,353 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048503095
2014-06-06 18:14:13,356 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048507639
2014-06-06 18:14:13,359 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048512413
2014-06-06 18:14:13,362 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048531567
2014-06-06 18:14:13,365 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048551423
2014-06-06 18:14:13,368 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048571173
2014-06-06 18:14:49,265 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:15:13,330 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048519249
2014-06-06 18:15:13,333 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048535326
2014-06-06 18:15:13,337 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048560010
2014-06-06 18:15:13,340 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048596472
2014-06-06 18:15:13,344 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048626528
2014-06-06 18:15:13,348 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048650702
2014-06-06 18:15:13,351 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048931511
2014-06-06 18:15:13,354 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048936590
2014-06-06 18:15:13,357 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048941809
2014-06-06 18:15:13,360 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048594603
2014-06-06 18:15:13,363 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048611087
2014-06-06 18:15:13,365 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048628148
2014-06-06 18:15:13,368 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048635438
2014-06-06 18:15:13,371 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048650860
2014-06-06 18:15:13,373 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048929446
2014-06-06 18:15:25,933 DEBUG [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Deleting region BT_D_BF001_201406,4600304159432142014060517055720500000000000000000808000000000000000005668700000006410174664,1402046055308.bf89ff438e6961193b0bb7b772e2fbd2. because daughter splits no longer hold references
2014-06-06 18:15:25,933 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: ARCHIVING hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2
2014-06-06 18:15:25,938 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf]
2014-06-06 18:15:25,952 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/18986793f0ea43599bebe6ad4c4df099, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/18986793f0ea43599bebe6ad4c4df099
2014-06-06 18:15:25,957 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/1c454591b5a2427fadabc5211d138c01, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/1c454591b5a2427fadabc5211d138c01
2014-06-06 18:15:25,960 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/1ca167a494e14a9f88c68b53e4dcc8cf, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/1ca167a494e14a9f88c68b53e4dcc8cf
2014-06-06 18:15:25,964 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/3af727f20bd1411aa45dd87736a2c3e9, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/3af727f20bd1411aa45dd87736a2c3e9
2014-06-06 18:15:25,968 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/3eeb9fe2252749af989b3b53e7eb49c9, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/3eeb9fe2252749af989b3b53e7eb49c9
2014-06-06 18:15:25,972 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/57701292ab1f41988ab3e32c7393871a, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/57701292ab1f41988ab3e32c7393871a
2014-06-06 18:15:25,977 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/5de7a228c52646a39e7e0d81e201ff9d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/5de7a228c52646a39e7e0d81e201ff9d
2014-06-06 18:15:25,981 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/5e4b9612a2334cf99d336ca1d510b129, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/5e4b9612a2334cf99d336ca1d510b129
2014-06-06 18:15:25,985 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/69301f0dd10245369ec7f6fc00280405, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/69301f0dd10245369ec7f6fc00280405
2014-06-06 18:15:25,990 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/8fced1ba62d945679f1f927e6b0c7b2f, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/8fced1ba62d945679f1f927e6b0c7b2f
2014-06-06 18:15:25,994 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/97bfe0707bb14738a59e8867f8a0d78d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/97bfe0707bb14738a59e8867f8a0d78d
2014-06-06 18:15:25,998 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/a72baaaec9f84302b96fe7fd92ce2077, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/a72baaaec9f84302b96fe7fd92ce2077
2014-06-06 18:15:26,003 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/c7e43d1b578e4564a0b05068b9e91957, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/c7e43d1b578e4564a0b05068b9e91957
2014-06-06 18:15:26,007 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/d11486209d214c5fb0e6b1d0f5d3578d, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/d11486209d214c5fb0e6b1d0f5d3578d
2014-06-06 18:15:26,011 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/f009edcfec28494d906061c49278df80, to hdfs://rtpha/apps/hbase/data/archive/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2/cf/f009edcfec28494d906061c49278df80
2014-06-06 18:15:26,013 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Deleted all region files in: hdfs://rtpha/apps/hbase/data/data/default/BT_D_BF001_201406/bf89ff438e6961193b0bb7b772e2fbd2
2014-06-06 18:15:26,034 INFO [CatalogJanitor-dsjhd113:60000] catalog.MetaEditor: Deleted BT_D_BF001_201406,4600304159432142014060517055720500000000000000000808000000000000000005668700000006410174664,1402046055308.bf89ff438e6961193b0bb7b772e2fbd2.
2014-06-06 18:15:26,039 DEBUG [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Deleting region ERR_BT_D_BF001_201406,0000000000000002014060517561739900000000000000000800000000000000000004744400000005,1402048433136.5544bd37b36ea28e890ba839f4506af6. because daughter splits no longer hold references
2014-06-06 18:15:26,040 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: ARCHIVING hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6
2014-06-06 18:15:26,043 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf]
2014-06-06 18:15:26,056 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/0fdc490810b94a10ad117421d60cbf2d, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/0fdc490810b94a10ad117421d60cbf2d
2014-06-06 18:15:26,060 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/6de1db61811b48809a733b21f032b6e4, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/6de1db61811b48809a733b21f032b6e4
2014-06-06 18:15:26,064 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/7c1ec330954b49198304646fe07d41c7, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/7c1ec330954b49198304646fe07d41c7
2014-06-06 18:15:26,068 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/96359a9a2d14442280a3365805a7ecee, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/96359a9a2d14442280a3365805a7ecee
2014-06-06 18:15:26,073 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/9c63d6e7dae94effb97c30a77cfe7269, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/9c63d6e7dae94effb97c30a77cfe7269
2014-06-06 18:15:26,077 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/a1fb57e26f9e47ee8f495a108ea5d75b, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/a1fb57e26f9e47ee8f495a108ea5d75b
2014-06-06 18:15:26,081 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/a936551fe9264d10bedaa839a31be3de, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/a936551fe9264d10bedaa839a31be3de
2014-06-06 18:15:26,085 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/ac4a1f7a101c4b389c4315f8765f53a9, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/ac4a1f7a101c4b389c4315f8765f53a9
2014-06-06 18:15:26,089 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/af04730876cb4560ba214930431ff91f, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/af04730876cb4560ba214930431ff91f
2014-06-06 18:15:26,093 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/dc65e7c7ef004bd980fa4d2f0da1b9e3, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/dc65e7c7ef004bd980fa4d2f0da1b9e3
2014-06-06 18:15:26,097 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/ecda798867b24016807157f22732d59e, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/ecda798867b24016807157f22732d59e
2014-06-06 18:15:26,102 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/eef82e61f779404eb5c42b0a18f8f54b, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/eef82e61f779404eb5c42b0a18f8f54b
2014-06-06 18:15:26,106 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/fd44da5f4b3e400486481c8c85dffea9, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/fd44da5f4b3e400486481c8c85dffea9
2014-06-06 18:15:26,110 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/fe7d0d3a048848e684e4f09d605a939b, to hdfs://rtpha/apps/hbase/data/archive/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6/cf/fe7d0d3a048848e684e4f09d605a939b
2014-06-06 18:15:26,112 DEBUG [CatalogJanitor-dsjhd113:60000] backup.HFileArchiver: Deleted all region files in: hdfs://rtpha/apps/hbase/data/data/default/ERR_BT_D_BF001_201406/5544bd37b36ea28e890ba839f4506af6
2014-06-06 18:15:26,117 INFO [CatalogJanitor-dsjhd113:60000] catalog.MetaEditor: Deleted ERR_BT_D_BF001_201406,0000000000000002014060517561739900000000000000000800000000000000000004744400000005,1402048433136.5544bd37b36ea28e890ba839f4506af6.
2014-06-06 18:15:26,117 INFO [CatalogJanitor-dsjhd113:60000] master.CatalogJanitor: Scanned 9778 catalog row(s), gc'd 0 unreferenced merged region(s) and 2 unreferenced parent region(s)
2014-06-06 18:15:49,833 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:16:13,292 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048946361
2014-06-06 18:16:13,296 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048950787
2014-06-06 18:16:13,299 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048954665
2014-06-06 18:16:13,302 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048959247
2014-06-06 18:16:13,305 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048964174
2014-06-06 18:16:13,308 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048968639
2014-06-06 18:16:13,311 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048973028
2014-06-06 18:16:13,314 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048977598
2014-06-06 18:16:13,316 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048983229
2014-06-06 18:16:13,319 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048987930
2014-06-06 18:16:13,322 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402048997754
2014-06-06 18:16:13,325 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049002092
2014-06-06 18:16:13,327 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049007197
2014-06-06 18:16:13,330 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049011687
2014-06-06 18:16:13,332 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049016886
2014-06-06 18:16:13,335 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049021827
2014-06-06 18:16:13,338 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049026762
2014-06-06 18:16:13,341 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049032132
2014-06-06 18:16:13,344 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049037021
2014-06-06 18:16:13,347 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048933338
2014-06-06 18:16:13,350 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048935814
2014-06-06 18:16:13,352 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048937911
2014-06-06 18:16:13,355 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048940232
2014-06-06 18:16:13,358 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048942766
2014-06-06 18:16:13,360 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048944719
2014-06-06 18:16:13,363 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048946386
2014-06-06 18:16:13,367 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048948435
2014-06-06 18:16:13,370 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048950236
2014-06-06 18:16:13,373 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048952115
2014-06-06 18:16:13,376 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048953784
2014-06-06 18:16:13,378 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048955682
2014-06-06 18:16:13,381 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048957714
2014-06-06 18:16:13,384 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048959789
2014-06-06 18:16:13,386 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048962263
2014-06-06 18:16:13,389 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048964314
2014-06-06 18:16:13,392 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048966244
2014-06-06 18:16:13,394 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048968079
2014-06-06 18:16:13,397 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048970142
2014-06-06 18:16:13,400 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048972136
2014-06-06 18:16:13,402 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048974098
2014-06-06 18:16:13,404 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048976149
2014-06-06 18:16:13,407 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048978357
2014-06-06 18:16:13,410 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048980465
2014-06-06 18:16:13,413 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048983378
2014-06-06 18:16:13,415 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048985782
2014-06-06 18:16:13,418 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048987914
2014-06-06 18:16:13,420 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048992125
2014-06-06 18:16:13,423 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048997124
2014-06-06 18:16:50,911 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:17:13,287 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402044995434
2014-06-06 18:17:13,290 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045016308
2014-06-06 18:17:13,293 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045076854
2014-06-06 18:17:13,296 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045228388
2014-06-06 18:17:13,299 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045259832
2014-06-06 18:17:13,302 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045394327
2014-06-06 18:17:13,305 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045439203
2014-06-06 18:17:13,308 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd057%2C60020%2C1402039842287.1402045556982
2014-06-06 18:17:13,311 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045137219
2014-06-06 18:17:13,313 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045230745
2014-06-06 18:17:13,316 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045271441
2014-06-06 18:17:13,343 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045395619
2014-06-06 18:17:13,346 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045440327
2014-06-06 18:17:13,349 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045549088
2014-06-06 18:17:13,352 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045622002
2014-06-06 18:17:13,355 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd060%2C60020%2C1402027925238.1402045718581
2014-06-06 18:17:13,358 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402044954679
2014-06-06 18:17:13,361 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402045036377
2014-06-06 18:17:13,364 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402045219697
2014-06-06 18:17:13,367 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402045245477
2014-06-06 18:17:13,370 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd074%2C60020%2C1402039899044.1402045393308
2014-06-06 18:17:13,373 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402048999129
2014-06-06 18:17:13,376 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049001457
2014-06-06 18:17:13,379 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049003628
2014-06-06 18:17:13,381 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049005940
2014-06-06 18:17:13,384 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049008308
2014-06-06 18:17:13,387 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049010366
2014-06-06 18:17:13,390 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049012487
2014-06-06 18:17:13,393 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049014811
2014-06-06 18:17:13,395 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049017100
2014-06-06 18:17:13,398 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049019403
2014-06-06 18:17:13,401 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049021840
2014-06-06 18:17:13,404 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049024068
2014-06-06 18:17:13,407 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049026394
2014-06-06 18:17:13,410 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049028753
2014-06-06 18:17:13,412 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049030958
2014-06-06 18:17:13,415 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049033623
2014-06-06 18:17:13,417 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049035965
2014-06-06 18:17:13,420 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049038330
2014-06-06 18:17:13,423 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049041236
2014-06-06 18:17:13,426 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049043631
2014-06-06 18:17:13,428 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049045793
2014-06-06 18:17:13,431 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049047912
2014-06-06 18:17:13,434 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049050293
2014-06-06 18:17:13,436 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049052605
2014-06-06 18:17:13,439 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049054781
2014-06-06 18:17:13,442 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049057191
2014-06-06 18:17:13,445 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049059395
2014-06-06 18:17:13,447 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049061781
2014-06-06 18:17:13,450 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049064109
2014-06-06 18:17:13,453 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049066260
2014-06-06 18:17:13,455 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049068577
2014-06-06 18:17:13,459 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049070894
2014-06-06 18:17:13,463 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049073286
2014-06-06 18:17:13,466 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049075554
2014-06-06 18:17:13,471 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049078156
2014-06-06 18:17:13,474 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049080419
2014-06-06 18:17:13,476 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049082884
2014-06-06 18:17:13,479 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049085224
2014-06-06 18:17:13,482 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049087507
2014-06-06 18:17:13,484 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049089716
2014-06-06 18:17:13,486 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049092246
2014-06-06 18:17:13,488 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049094645
2014-06-06 18:17:13,491 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049097072
2014-06-06 18:17:13,494 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049099773
2014-06-06 18:17:13,496 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049102351
2014-06-06 18:17:13,499 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049104753
2014-06-06 18:17:13,501 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049106711
2014-06-06 18:17:13,504 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049108796
2014-06-06 18:17:13,506 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049111658
2014-06-06 18:17:13,509 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049113986
2014-06-06 18:17:13,511 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049116420
2014-06-06 18:17:13,514 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049118914
2014-06-06 18:17:13,516 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049121116
2014-06-06 18:17:13,519 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049123268
2014-06-06 18:17:13,521 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049129255
2014-06-06 18:17:13,524 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049131495
2014-06-06 18:17:13,526 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049133745
2014-06-06 18:17:13,529 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049135759
2014-06-06 18:17:13,532 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049137916
2014-06-06 18:17:13,535 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049140298
2014-06-06 18:17:13,537 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049142508
2014-06-06 18:17:13,540 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049144665
2014-06-06 18:17:13,543 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd078%2C60020%2C1402027921335.1402045224394
2014-06-06 18:17:13,545 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd078%2C60020%2C1402027921335.1402045247047
2014-06-06 18:17:13,547 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd078%2C60020%2C1402027921335.1402045388045
2014-06-06 18:17:13,550 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd078%2C60020%2C1402027921335.1402045420942
2014-06-06 18:17:13,553 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd078%2C60020%2C1402027921335.1402045457402
2014-06-06 18:18:01,014 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:18:13,286 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd055%2C60020%2C1402027951094.1402045177206
2014-06-06 18:18:13,289 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd055%2C60020%2C1402027951094.1402045269050
2014-06-06 18:18:13,292 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd055%2C60020%2C1402027951094.1402045421126
2014-06-06 18:18:13,296 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd055%2C60020%2C1402027951094.1402045567664
2014-06-06 18:18:13,299 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd055%2C60020%2C1402027951094.1402045678757
2014-06-06 18:18:13,302 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049042715
2014-06-06 18:18:13,305 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049047512
2014-06-06 18:18:13,308 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049052445
2014-06-06 18:18:13,311 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049057573
2014-06-06 18:18:13,314 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049062520
2014-06-06 18:18:13,316 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049067235
2014-06-06 18:18:13,319 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049072382
2014-06-06 18:18:13,322 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049077598
2014-06-06 18:18:13,325 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049082878
2014-06-06 18:18:13,328 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049087658
2014-06-06 18:18:13,331 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049092838
2014-06-06 18:18:13,334 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049098118
2014-06-06 18:18:13,337 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049103936
2014-06-06 18:18:13,339 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049108385
2014-06-06 18:18:13,342 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049113782
2014-06-06 18:18:13,345 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049119079
2014-06-06 18:18:13,349 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049127594
2014-06-06 18:18:13,352 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049132910
2014-06-06 18:18:13,355 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049137472
2014-06-06 18:18:13,358 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049142244
2014-06-06 18:18:13,361 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049147238
2014-06-06 18:18:13,364 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049147110
2014-06-06 18:18:13,367 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049149138
2014-06-06 18:18:13,371 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049151344
2014-06-06 18:18:13,376 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049153622
2014-06-06 18:18:13,379 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049155987
2014-06-06 18:18:13,384 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049158145
2014-06-06 18:18:13,389 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049160435
2014-06-06 18:18:13,392 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049162560
2014-06-06 18:18:13,395 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049164362
2014-06-06 18:18:13,397 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049166423
2014-06-06 18:18:13,400 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049168796
2014-06-06 18:18:13,403 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049170833
2014-06-06 18:19:11,170 DEBUG [RpcServer.handler=57,port=60000] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to dsjhd061,60020,1402027921226
2014-06-06 18:19:13,291 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd054%2C60020%2C1402027922467.1402045803789
2014-06-06 18:19:13,295 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049151909
2014-06-06 18:19:13,298 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049156816
2014-06-06 18:19:13,302 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd064%2C60020%2C1402027918813.1402049161745
2014-06-06 18:19:13,305 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd066%2C60020%2C1402027920604.1402046117563
2014-06-06 18:19:13,308 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045229582
2014-06-06 18:19:13,311 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045268073
2014-06-06 18:19:13,315 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045394363
2014-06-06 18:19:13,317 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045440203
2014-06-06 18:19:13,320 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045559507
2014-06-06 18:19:13,323 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd075%2C60020%2C1402027953331.1402045624300
2014-06-06 18:19:13,326 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049173142
2014-06-06 18:19:13,329 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049175412
2014-06-06 18:19:13,332 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049177798
2014-06-06 18:19:13,335 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049180160
2014-06-06 18:19:13,337 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049182815
2014-06-06 18:19:13,340 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049185115
2014-06-06 18:19:13,343 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049187804
2014-06-06 18:19:13,346 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049190153
2014-06-06 18:19:13,349 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049192222
2014-06-06 18:19:13,352 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049194249
2014-06-06 18:19:13,354 DEBUG [master:dsjhd113:60000.oldLogCleaner] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: dsjhd076%2C60020%2C1402027922130.1402049196275