序列IO與並行IO的效率比較

2021-05-28 10:14:48 字數 3063 閱讀 3324

mpi_file_read_at函式與c語言中的fread函式,依次讀取乙個100m的檔案,讀取時間如下所示:

實驗一

[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed

there are 1500015 samples...

start test mpi_file_read....

in mpi002 mpi_file_read takes 1.566990137100 seconds...

start test fread....

in mpi002 fread takes 0.078104019165 seconds...    

實驗二

[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed

there are 1500000 samples...

start test mpi_file_read....

in mpi002 mpi_file_read takes 1.315994024277 seconds...

start test fread....

in mpi002 fread takes 1.259196043015 seconds...

兩次實驗具有不同的結果,原因是:

實驗一:mpi_file_read_at與fread讀取同一資料塊,mpi先讀,fread後讀,因此,mpi花費的時間較多

fread花費的時間較短,主要是受到了快取的影響;

實驗二:mpi_file_read_at與fread讀取不同資料塊,mpi先讀,fread後讀,因此,mpi花費的時間與fread花費的時間相差不多;

再來看實驗三和實驗四:

實驗三:啟動1個程序分別使用mpi_file_read_at函式和fread函式讀取100g資料檔案,只讀取其中的100m資料,實驗結果如下:

mpi_file_read_at:[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed

there are 1500000 samples...

in mpi002-0 mpi_file_read takes 1.155555009842 seconds...

fread:[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed

there are 1500000 samples...

in mpi002-0 fread takes 1.155176162720 seconds...

實驗四:啟動336個程序(14個節點x24)讀取100g資料檔案,每個程序分別使用mpi_file_read_at函式和fread函式讀取100m,而且都是各不相同的100m,部分實驗結果如下:

mpi_file_read_at:[zhangyang@mpi002 kmeans_exper]$ /usr/mpi/gcc/mvapich2-1.6/bin/mpiexec -np 336 -hosts mpi002,mpi003,mpi004,mpi005,mpi006,mpi007,mpi008,mpi009,mpi011,mpi012,mpi013,mpi014,mpi015,mpi016 /home/zhangyang/kmeans/1000g/kmeans_exper/mpi_c_fspeed

there are 1500000 samples...

in mpi002-224 mpi_file_read takes 20.344742059708 seconds...

in mpi002-280 mpi_file_read takes 20.536442995071 seconds...

in mpi002-84 mpi_file_read takes 20.720577955246 seconds...

.................

in mpi015-82 mpi_file_read takes 24.234681844711 seconds...

in mpi013-332 mpi_file_read takes 24.371000051498 seconds...

in mpi002-0 mpi_file_read takes 24.875329971313 seconds...

fread:[zhangyang@mpi002 kmeans_exper]$ /usr/mpi/gcc/mvapich2-1.6/bin/mpiexec -np 336 -hosts mpi002,mpi003,mpi004,mpi005,mpi006,mpi007,mpi008,mpi009,mpi011,mpi012,mpi013,mpi014,mpi015,mpi016 /home/zhangyang/kmeans/1000g/kmeans_exper/mpi_c_fspeed

there are 1500000 samples...

in mpi002-28 fread takes 20.996963024139 seconds...

in mpi002-238 fread takes 21.178472995758 seconds...

in mpi002-154 fread takes 21.282578945160 seconds...

in mpi012-247 fread takes 24.225381851196 seconds...

in mpi012-331 fread takes 24.246413946152 seconds...

in mpi002-0 fread takes 24.365787982941 seconds...

比較實驗三和實驗四的結果可以得出結論:讀取同樣大小的資料塊,單程序讀取的速度要比多程序並行讀取的速度快乙個數量級,因此,在並行條件下,資料讀取速度會受到很大的影響

標準IO與系統IO比較

分別使用標準io與系統io隨機寫入1000000個整數到檔案,比較哪種更快,為什麼?include include intmain fclose fwp real 0m0.113s 標準io user 0m0.100s sys 0m0.008s include include include inc...

累加計算的並行與序列效率比較

計算一幅影象的畫素值的平均值 測試環境 cpu intel r core tm i7 7700 cpu 3.60ghz x 8gpu nvidia geforce gtx 1050 1,在cuda gpu上計算累加和,先分段計算,在進行規約,分段計算裡面其實也是規約,然後除以總的畫素個數。2,在cp...

nio與io的比較

nio是new io的簡稱,從jdk1.4就被引入了。現在的jdk已經到了1.6了,可以說不是什麼新東西了。但其中的一些思想值得我來研究。這兩天,我研究了下其中的套接字部分,有一些心得,在此分享。首先先分析下 為什麼要nio套接字?nio的主要作用就是用來解決速度差異的。舉個例子 計算機處理的速度,...