Hadoop學習總結之二 HDFS讀寫過程解析

hdfs開啟乙個檔案，需要在客戶端呼叫distributedfilesystem.open(path f, int buffersize)，其實現為：

public fsdatainputstream open(path f, int buffersize) throws ioexception while (curpos < endoff

&& curblk < blocks.length

&& results.size() < nrblockstoreturn);

//使用此locatedblock鍊錶構造乙個locatedblocks物件返回

return inode.createlocatedblocks(results);

通過rpc呼叫，在namenode得到的locatedblocks物件，作為成員變數構造dfsinputstream物件，最後包裝為fsdatainputstream返回給使用者。

檔案讀取的時候，客戶端利用檔案開啟的時候得到的fsdatainputstream.read(long position, byte buffer, int offset, int length)函式進行檔案讀操作。

fsdatainputstream會呼叫其封裝的dfsinputstream的read(long position, byte buffer, int offset, int length)函式，實現如下：

public int read(long position, byte buffer, int offset, int length)

throws ioexception finally catch (ioexception ie) catch (throwable t) finally finally catch(remoteexception re) catch (interruptedexception e) catch (throwable e) while (retry && --count >= 0);

return nodes;

locatefollowingblock中通過rpc呼叫namenode.addblock(src, clientname)函式

namenode的addblock函式實現如下：

public locatedblock addblock(string src,

string clientname) throws ioexception finally

if (mirrorout != null) catch (ioexception e) catch (ioexception e) {

handlemirrorouterror(e);

buf.position(endofheader);

int len = buf.getint();

offsetinblock += len;

int checksumlen = ((len + bytesperchecksum - 1)/bytesperchecksum)*

checksumsize;

int checksumoff = buf.position();

int dataoff = checksumoff + checksumlen;

byte pktbuf = buf.array();

buf.position(buf.limit()); // move to the end of the data.

//將資料寫入本地的block

out.write(pktbuf, dataoff, len);

/// flush entire packet before sending ack

flush();

// put in queue for pending acks

if (responder != null) {

((packetresponder)responder.getrunnable()).enqueue(seqno,

lastpacketinblock);

return payloadlen;

Hadoop學習總結之二 HDFS讀寫過程解析

Hadoop學習總結之二 HDFS讀寫過程解析

Hadoop集群搭建之二（測試hadoop集群）

JS學習總結之二

Hadoop學習總結之二 HDFS讀寫過程解析

Hadoop學習總結之二 HDFS讀寫過程解析

Hadoop集群搭建之二（測試hadoop集群）

JS學習總結之二

相關推薦