mpi_file_read_at函式與c語言中的fread函式,依次讀取乙個100m的檔案,讀取時間如下所示:
實驗一:
[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed
there are 1500015 samples...
start test mpi_file_read....
in mpi002 mpi_file_read takes 1.566990137100 seconds...
start test fread....
in mpi002 fread takes 0.078104019165 seconds...
實驗二:
[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed
there are 1500000 samples...
start test mpi_file_read....
in mpi002 mpi_file_read takes 1.315994024277 seconds...
start test fread....
in mpi002 fread takes 1.259196043015 seconds...
兩次實驗具有不同的結果,原因是:
實驗一:mpi_file_read_at與fread讀取同一資料塊,mpi先讀,fread後讀,因此,mpi花費的時間較多
fread花費的時間較短,主要是受到了快取的影響;
實驗二:mpi_file_read_at與fread讀取不同資料塊,mpi先讀,fread後讀,因此,mpi花費的時間與fread花費的時間相差不多;
再來看實驗三和實驗四:
實驗三:啟動1個程序分別使用mpi_file_read_at函式和fread函式讀取100g資料檔案,只讀取其中的100m資料,實驗結果如下:
mpi_file_read_at:[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed
there are 1500000 samples...
in mpi002-0 mpi_file_read takes 1.155555009842 seconds...
fread:[zhangyang@mpi002 kmeans_exper]$ mpiexec -np 1 ./mpi_c_fspeed
there are 1500000 samples...
in mpi002-0 fread takes 1.155176162720 seconds...
實驗四:啟動336個程序(14個節點x24)讀取100g資料檔案,每個程序分別使用mpi_file_read_at函式和fread函式讀取100m,而且都是各不相同的100m,部分實驗結果如下:
mpi_file_read_at:[zhangyang@mpi002 kmeans_exper]$ /usr/mpi/gcc/mvapich2-1.6/bin/mpiexec -np 336 -hosts mpi002,mpi003,mpi004,mpi005,mpi006,mpi007,mpi008,mpi009,mpi011,mpi012,mpi013,mpi014,mpi015,mpi016 /home/zhangyang/kmeans/1000g/kmeans_exper/mpi_c_fspeed
there are 1500000 samples...
in mpi002-224 mpi_file_read takes 20.344742059708 seconds...
in mpi002-280 mpi_file_read takes 20.536442995071 seconds...
in mpi002-84 mpi_file_read takes 20.720577955246 seconds...
.................
in mpi015-82 mpi_file_read takes 24.234681844711 seconds...
in mpi013-332 mpi_file_read takes 24.371000051498 seconds...
in mpi002-0 mpi_file_read takes 24.875329971313 seconds...
fread:[zhangyang@mpi002 kmeans_exper]$ /usr/mpi/gcc/mvapich2-1.6/bin/mpiexec -np 336 -hosts mpi002,mpi003,mpi004,mpi005,mpi006,mpi007,mpi008,mpi009,mpi011,mpi012,mpi013,mpi014,mpi015,mpi016 /home/zhangyang/kmeans/1000g/kmeans_exper/mpi_c_fspeed
there are 1500000 samples...
in mpi002-28 fread takes 20.996963024139 seconds...
in mpi002-238 fread takes 21.178472995758 seconds...
in mpi002-154 fread takes 21.282578945160 seconds...
in mpi012-247 fread takes 24.225381851196 seconds...
in mpi012-331 fread takes 24.246413946152 seconds...
in mpi002-0 fread takes 24.365787982941 seconds...
比較實驗三和實驗四的結果可以得出結論:讀取同樣大小的資料塊,單程序讀取的速度要比多程序並行讀取的速度快乙個數量級,因此,在並行條件下,資料讀取速度會受到很大的影響
標準IO與系統IO比較
分別使用標準io與系統io隨機寫入1000000個整數到檔案,比較哪種更快,為什麼?include include intmain fclose fwp real 0m0.113s 標準io user 0m0.100s sys 0m0.008s include include include inc...
累加計算的並行與序列效率比較
計算一幅影象的畫素值的平均值 測試環境 cpu intel r core tm i7 7700 cpu 3.60ghz x 8gpu nvidia geforce gtx 1050 1,在cuda gpu上計算累加和,先分段計算,在進行規約,分段計算裡面其實也是規約,然後除以總的畫素個數。2,在cp...
nio與io的比較
nio是new io的簡稱,從jdk1.4就被引入了。現在的jdk已經到了1.6了,可以說不是什麼新東西了。但其中的一些思想值得我來研究。這兩天,我研究了下其中的套接字部分,有一些心得,在此分享。首先先分析下 為什麼要nio套接字?nio的主要作用就是用來解決速度差異的。舉個例子 計算機處理的速度,...