Ubuntu系统使用SPDK运行fio测试

  因最近需求所要,所以研究了一下SPDK运行Fio方式,本文主要讲解spdk_nvme的方式运行fio进行性能测试。若感兴趣可以看看官方文档【SPDK:程序员指南

一、拉取SPDK,以及准备环境

    下载SPDK软件包,--recursive参数是会循环拉去SPDK,相当于“git clone https://github.com/spdk/spdk”之后,在进行“git submodule update --init”。
    下载Fio,需注意Fio的版本需要同SPDK里面一致,可查看在SPDK路径“./examples/nvme/fio_plugin/”里面的“README.md”文件,按文件中的指令进行git拉取fio代码。

root@test-System-Product-Name:/home# git clone https://github.com/spdk/spdk --recursive
Cloning into 'spdk'...
remote: Enumerating objects: 181402, done.
remote: Counting objects: 100% (5702/5702), done.
remote: Compressing objects: 100% (2099/2099), done.
remote: Total 181402 (delta 3945), reused 4698 (delta 3569), pack-reused 175700
Receiving objects: 100% (181402/181402), 79.23 MiB | 29.00 KiB/s, done.
Resolving deltas: 100% (141210/141210), done.
Submodule 'dpdk' (https://github.com/spdk/dpdk.git) registered for path 'dpdk'
Submodule 'intel-ipsec-mb' (https://github.com/spdk/intel-ipsec-mb.git) registered for path 'intel-ipsec-mb'
Submodule 'isa-l' (https://github.com/spdk/isa-l.git) registered for path 'isa-l'
Submodule 'isa-l-crypto' (https://github.com/intel/isa-l_crypto) registered for path 'isa-l-crypto'
Submodule 'libvfio-user' (https://github.com/nutanix/libvfio-user.git) registered for path 'libvfio-user'
Submodule 'ocf' (https://github.com/Open-CAS/ocf.git) registered for path 'ocf'
Submodule 'xnvme' (https://github.com/OpenMPDK/xNVMe.git) registered for path 'xnvme'
Cloning into '/root/spdk/dpdk'...
......
root@test-System-Product-Name:/home# git clone https://github.com/axboe/fio
Cloning into 'fio'...
remote: Enumerating objects: 35674, done.
remote: Counting objects: 100% (35674/35674), done.
remote: Compressing objects: 100% (10580/10580), done.
remote: Total 35674 (delta 25041), reused 35351 (delta 24999), pack-reused 0
Receiving objects: 100% (35674/35674), 25.53 MiB | 2.67 MiB/s, done.
Resolving deltas: 100% (25041/25041), done.
root@test-System-Product-Name:/home# cd fio/

# git拉去的回事mester分支fio,若拉取固定版本需chekout
root@test-System-Product-Name:/home# git checkout fio-3.17
root@test-System-Product-Name:/home/fio# ./configure
Operating system              Linux
CPU                           x86_64
......

TCMalloc support              no
seed_buckets                  4
root@test-System-Product-Name:/home/fio#
root@test-System-Product-Name:/home/fio# make
FIO_VERSION = fio-3.35-30-gedaee
    CC crc/crc16.o
......
  LINK unittests/unittest
root@test-System-Product-Name:/home/fio#

二、编译SPDK

    编译SPDK完成后,可以查看SPDK下“/build/fio/”目录,里面是否生成spdk_bdev和spdk_nvme两个二进制文件。spdk_nvme是基于裸盘NVMe的fio_plugin,其特点为I/O通过SPDK用户态驱动直接访问裸盘,常用于评估SPDK用户态驱动在裸盘上的性能。spdk_bdev是基于bdev的fio_plugin,其特点为I/O测试基于SPDK块设备bdev之上,所有I/O经由块设备层bdev,再传送至裸盘设备。常用于评估SPDK块设备bdev的性能。

root@test-System-Product-Name:/home# cd spdk/
# 运行scripts目录下的pkgdep.sh文件获取SPDK所需的依赖包
root@test-System-Product-Name:/home/spdk# ./scripts/pkgdep.sh
root@test-System-Product-Name:/home/spdk# ls
app              autorun.sh    CODE_OF_CONDUCT.md  deprecation.md  dpdkbuild  include         isalbuild        libvfio-user  mdl_rules.rb  proto      scripts      vfiouserbuild
autobuild.sh     autotest.sh   CONFIG              doc             examples   intel-ipsec-mb  isa-l-crypto     LICENSE       mk            python     SECURITY.md  xnvme
autopackage.sh   build         configure           docker          fio.fio    ipsecbuild      isalcryptobuild  licenses      module        README.md  shared_lib   xnvmebuild
autorun_post.py  CHANGELOG.md  CONTRIBUTING.md     dpdk            go         isa-l           lib              Makefile      ocf           rpmbuild   test
root@test-System-Product-Name:/home/spdk#
# 这里fio需要选择刚拉取并make编译的fio路径,等待完成
root@test-System-Product-Name:/home/spdk# ./configure --with-fio=/home/fio
root@test-System-Product-Name:/home/spdk# make
root@test-System-Product-Name:/home/spdk# ls -l /home/spdk/build/fio/
total 18556
-rwxr-xr-x 1 root root 13215920 6月   8 15:22 spdk_bdev
-rwxr-xr-x 1 root root  5783136 6月   8 15:22 spdk_nvme
root@test-System-Product-Name:/home/spdk#

三、运行SPDK和fio

  3.1、SDPK安装完成后,需查看一下是否可以正常运行。可以运行SPDK目录下的“scripts/setup.sh”文件,当前驱动是否替换为SPDK的驱动。

root@test-System-Product-Name:/home/spdk# scripts/setup.sh
0000:03:00.0 (1dee 5216): Already using the uio_pci_generic driver
INFO: Requested 1024 hugepages but 5738 already allocated on node0
# status:可以查看SPDK驱动下盘的BDF号,以及状态
root@test-System-Product-Name:/home/spdk# scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        1 /      1
node0      2048kB     5738 /   5738

Type     BDF             Vendor Device NUMA    Driver           Device     Block devices
NVMe     0000:03:00.0    1dee   5216   0       uio_pci_generic  -          -
# reset:恢复nvme驱动
root@test-System-Product-Name:/home/spdk# scripts/setup.sh reset
0000:03:00.0 (1dee 5216): uio_pci_generic -> nvme
root@test-System-Product-Name:/home/spdk#

  3.2、运行fio文件,当前介绍的fio运行方式主要是以spdk_nvme作为驱动方式。NVMe的fio_plugin支持两种模式下的测试,一是本地的NVMe设备,即NVMe over PCIe;二是远端的NVMe设备,即NVMe over Fabrics。二者运行方式区别主要在于filename参数配置不同。下面是fio文件格式,和我们平常写fio的方式一样。
  需要注意:使用spdk_nvme方式需setup替换驱动;必须设置“thread=1”参数;以及盘的BDF号,也就是filename参数按这个格式进行编写“filename=trtype=PCIe traddr=0000.03.00.0 ns=1”。

[global]
ioengine=spdk
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=2
iodepth=128
size=4
numjobs=1
[job1]
rw=randread
filename=trtype=PCIe traddr=0000.03.00.0 ns=1
filename=trtype=TCP adrfam=IPv4 traddr=192.168.100.100 trsvcid=4420 ns=1

  3.3、终端演示运行
    方式一、fio文件只需写关键参数,filename放在外面运行,需注意此时filename需加引号“ '--filename=trtype=PCIe traddr=0000.03.00.0 ns=1' ”。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio /home/spdk/examples/nvme/fio_plugin/example_config.fio '--filename=trtype=PCIe traddr=0000.03.00.0 ns=1'
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][75.0%][r=325MiB/s][r=83.3k IOPS][eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=423667: Fri Jun  9 09:30:38 2023
  read: IOPS=85.7k, BW=335MiB/s (351MB/s)(670MiB/2002msec)
    slat (nsec): min=75, max=1513, avg=80.92, stdev= 6.39
    clat (usec): min=95, max=65450, avg=1483.68, stdev=1437.30
     lat (usec): min=95, max=65450, avg=1483.76, stdev=1437.30
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1045],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1647], 80.00th=[ 1811], 90.00th=[ 2008], 95.00th=[ 2180],
     | 99.00th=[ 2638], 99.50th=[ 2900], 99.90th=[18744], 99.95th=[46924],
     | 99.99th=[55313]
   bw (  KiB/s): min=312808, max=356632, per=100.00%, avg=342912.00, stdev=20345.47, samples=4
   iops        : min=78202, max=89158, avg=85728.00, stdev=5086.37, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=3.99%, 1000=13.23%
  lat (msec)   : 2=72.35%, 4=10.26%, 10=0.01%, 20=0.07%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=99.80%, sys=0.15%, ctx=3, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=171583,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=670MiB (703MB), run=2002-2002msec

    方式二、filename也放在fio文件里面,格式如上3.2所示。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio /home/spdk/examples/nvme/fio_plugin/example_config.fio
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][75.0%][r=347MiB/s][r=88.9k IOPS][eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=421306: Thu Jun  8 18:03:19 2023
  read: IOPS=88.5k, BW=346MiB/s (363MB/s)(692MiB/2002msec)
    slat (nsec): min=75, max=1503, avg=81.59, stdev= 9.00
    clat (usec): min=98, max=4054, avg=1436.34, stdev=440.02
     lat (usec): min=98, max=4054, avg=1436.43, stdev=440.02
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1045],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2008], 95.00th=[ 2180],
     | 99.00th=[ 2606], 99.50th=[ 2802], 99.90th=[ 3261], 99.95th=[ 3458],
     | 99.99th=[ 3818]
   bw (  KiB/s): min=348552, max=357032, per=100.00%, avg=354222.00, stdev=3911.61, samples=4
   iops        : min=87138, max=89258, avg=88555.50, stdev=977.90, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=4.01%, 1000=13.21%
  lat (msec)   : 2=72.45%, 4=10.30%, 10=0.01%
  cpu          : usr=100.00%, sys=0.00%, ctx=1, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=177238,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=692MiB (726MB), run=2002-2002msec

    方式三、写成我们平常使用的fio形式,即命令行形式运行。需注意当前filename后面参数需要加引号“ --filename='trtype=PCIe traddr=0000.03.00.0 ns=1' ”。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio --ioengine=spdk --bs=4k --rw=randread --thread=1 --group_reporting=1 --direct=1 --verify=0 --time_based=1 --ramp_time=0 --runtime=2 --iodepth=128 --numjobs=1 --size=1G --name=job --filename='trtype=PCIe traddr=0000.03.00.0 ns=1'
job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][-.-%][r=348MiB/s][r=89.0k IOPS][eta 00m:00s]
job: (groupid=0, jobs=1): err= 0: pid=423702: Fri Jun  9 09:41:45 2023
  read: IOPS=85.7k, BW=335MiB/s (351MB/s)(670MiB/2002msec)
    slat (nsec): min=74, max=1673, avg=86.59, stdev= 8.44
    clat (usec): min=93, max=66157, avg=1483.63, stdev=1428.75
     lat (usec): min=94, max=66157, avg=1483.72, stdev=1428.75
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1037],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1663], 80.00th=[ 1811], 90.00th=[ 2024], 95.00th=[ 2212],
     | 99.00th=[ 2638], 99.50th=[ 2900], 99.90th=[18744], 99.95th=[46924],
     | 99.99th=[54789]
   bw (  KiB/s): min=311424, max=358440, per=100.00%, avg=342920.00, stdev=21436.21, samples=4
   iops        : min=77856, max=89610, avg=85730.00, stdev=5359.05, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=4.05%, 1000=13.32%
  lat (msec)   : 2=72.05%, 4=10.41%, 10=0.01%, 20=0.07%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=99.30%, sys=0.55%, ctx=2, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=171587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=670MiB (703MB), run=2002-2002msec

    方式四、写成我们平常使用的fio形式,即命令行形式运行。和方式三不同的是,不需要LD_PRELOAD spdk_nvme,可以直接再“ioengine”这里写上spdk_nvme。需注意当前filename后面参数需要加引号“ --filename='trtype=PCIe traddr=0000.03.00.0 ns=1' ”,filename里面参数ns可以不写。

root@test-System-Product-Name:/home# /home/fio/fio --ioengine=/home/spdk/build/fio/spdk_nvme --bs=4k --rw=randread --thread=1 --group_reporting=1 --direct=1 --verify=0 --time_based=1 --ramp_time=0 --runtime=2 --iodepth=128 --numjobs=1 --size=1G --name=job --filename='trtype=PCIe traddr=0000.03.00.0'
job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][60.0%][r=349MiB/s][r=89.3k IOPS][eta 00m:02s]
job: (groupid=0, jobs=1): err= 0: pid=423758: Fri Jun  9 09:48:18 2023
  read: IOPS=71.1k, BW=278MiB/s (291MB/s)(556MiB/2002msec)
    slat (nsec): min=75, max=1398, avg=81.20, stdev= 7.31
    clat (usec): min=93, max=258819, avg=1789.98, stdev=5455.01
     lat (usec): min=93, max=258819, avg=1790.06, stdev=5455.01
    clat percentiles (usec):
     |  1.00th=[   635],  5.00th=[   766], 10.00th=[   873], 20.00th=[  1037],
     | 30.00th=[  1172], 40.00th=[  1303], 50.00th=[  1418], 60.00th=[  1532],
     | 70.00th=[  1647], 80.00th=[  1795], 90.00th=[  2024], 95.00th=[  2212],
     | 99.00th=[  2868], 99.50th=[ 18482], 99.90th=[ 92799], 99.95th=[125305],
     | 99.99th=[141558]
   bw (  KiB/s): min=177896, max=358232, per=100.00%, avg=284250.00, stdev=88507.99, samples=4
   iops        : min=44474, max=89558, avg=71062.50, stdev=22127.00, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=4.27%, 1000=13.40%
  lat (msec)   : 2=71.65%, 4=10.04%, 10=0.06%, 20=0.08%, 50=0.08%
  lat (msec)   : 100=0.34%, 250=0.06%, 500=0.01%
  cpu          : usr=99.65%, sys=0.15%, ctx=2, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=142252,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=556MiB (583MB), run=2002-2002msec

~文章已经结束了~

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 206,126评论 6 481
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 88,254评论 2 382
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 152,445评论 0 341
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 55,185评论 1 278
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 64,178评论 5 371
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 48,970评论 1 284
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 38,276评论 3 399
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 36,927评论 0 259
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 43,400评论 1 300
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 35,883评论 2 323
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 37,997评论 1 333
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 33,646评论 4 322
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 39,213评论 3 307
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,204评论 0 19
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,423评论 1 260
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 45,423评论 2 352
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 42,722评论 2 345

推荐阅读更多精彩内容