05-11-2019, 04:41 AM
Check storage performance with dd
<div><p>This article includes some example commands to show you how to get a <em>rough</em> estimate of hard drive and RAID array performance using the <em>dd</em> command. Accurate measurements would have to take into account things like <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://www.ibm.com/developerworks/community/blogs/ibmnas/entry/misalignment_can_be_twice_the_cost1?lang=en" target="_blank">write amplification</a> and <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://eklitzke.org/efficient-file-copying-on-linux" target="_blank">system call overhead</a>, which this guide does not. For a tool that might give more accurate results, you might want to consider using <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Hdparm" target="_blank">hdparm</a>.</p>
<p>To factor out performance issues related to the file system, these examples show how to test the performance of your drives and arrays at the block level by reading and writing directly to/from their block devices. <strong>WARNING</strong>: The <em>write</em> tests will destroy any data on the block devices against which they are run. <strong>Do not run them against any device that contains data you want to keep!</strong></p>
<p> <span id="more-27499"></span> </p>
<h2>Four tests</h2>
<p>Below are four example dd commands that can be used to test the performance of a block device:</p>
<ol>
<li>One process reading from $MY_DISK:
<pre># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
</li>
<li>One process writing to $MY_DISK:
<pre># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
</li>
<li>Two processes reading concurrently from $MY_DISK:
<pre># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre>
</li>
<li>Two processes writing concurrently to $MY_DISK:
<pre># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct & (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre>
</li>
</ol>
<p>– The <em>iflag=nocache</em> and <em>oflag=direct</em> parameters are important when performing the read and write tests (respectively) because without them the dd command will sometimes show the resulting speed of transferring the data to/from <a href="https://en.wikipedia.org/wiki/Random-access_memory" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">RAM</a> rather than the hard drive.</p>
<p>– The values for the <em>bs</em> and <em>count</em> parameters are somewhat arbitrary and what I have chosen should be large enough to provide a decent average in most cases for current hardware.</p>
<p>– The <em>null</em> and <em>zero</em> devices are used for the destination and source (respectively) in the read and write tests because they are fast enough that they will not be the limiting factor in the performance tests.</p>
<p>– The <em>skip=200</em> parameter on the second dd command in the concurrent read and write tests is to ensure that the two copies of dd are operating on different areas of the hard drive.</p>
<h2>16 examples</h2>
<p>Below are demonstrations showing the results of running each of the above four tests against each of the following four block devices:</p>
<ol>
<li>MY_DISK=/dev/sda2 (used in examples 1-X)</li>
<li>MY_DISK=/dev/sdb2 (used in examples 2-X)</li>
<li>MY_DISK=/dev/md/stripped (used in examples 3-X)</li>
<li>MY_DISK=/dev/md/mirrored (used in examples 4-X)</li>
</ol>
<p>A video demonstration of the these tests being run on a PC is provided at the end of this guide.</p>
<p>Begin by putting your computer into <em>rescue</em> mode to reduce the chances that disk I/O from background services might randomly affect your test results. <strong>WARNING</strong>: This will shutdown all non-essential programs and services. Be sure to save your work before running these commands. You will need to know your <em>root</em> password to get into rescue mode. The <em>passwd</em> command, when run as the root user, will prompt you to (re)set your root account password.</p>
<pre class="wp-block-preformatted">$ sudo -i<br /># passwd<br /># setenforce 0<br /># systemctl rescue</pre>
<p>You might also want to temporarily disable logging to disk:</p>
<pre class="wp-block-preformatted"># sed -r -i.bak 's/^#?Storage=.*/Storage=none/' /etc/systemd/journald.conf<br /># systemctl restart systemd-journald.service</pre>
<p>If you have a swap device, it can be temporarily disabled and used to perform the following tests:</p>
<pre class="wp-block-preformatted"># swapoff -a<br /># MY_DEVS=$(mdadm --detail /dev/md/swap | grep active | grep -o "/dev/sd.*")<br /># mdadm --stop /dev/md/swap<br /># mdadm --zero-superblock $MY_DEVS</pre>
<h3>Example 1-1 (reading from sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.7003 s, <font color="red">123 MB/s</font></pre>
<h3>Example 1-2 (writing to sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67117 s, <font color="red">125 MB/s</font></pre>
<h3>Example 1-3 (reading concurrently from sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache & (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.42875 s, <font color="red">61.2 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.52614 s, <font color="red">59.5 MB/s</font></pre>
<h3>Example 1-4 (writing concurrently to sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct & (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.2435 s, <font color="red">64.7 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.60872 s, <font color="red">58.1 MB/s</font></pre>
<h3>Example 2-1 (reading from sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67285 s, <font color="red">125 MB/s</font></pre>
<h3>Example 2-2 (writing to sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67198 s, <font color="red">125 MB/s</font></pre>
<h3>Example 2-3 (reading concurrently from sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache & (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.52808 s, <font color="red">59.4 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.57736 s, <font color="red">58.6 MB/s</font></pre>
<h3>Example 2-4 (writing concurrently to sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct & (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.7841 s, <font color="red">55.4 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.81475 s, <font color="red">55.0 MB/s</font></pre>
<h3>Example 3-1 (reading from RAID0)</h3>
<pre class="wp-block-preformatted"># mdadm --create /dev/md/stripped --homehost=any --metadata=1.0 --level=0 --raid-devices=2 $MY_DEVS<br /># MY_DISK=/dev/md/stripped<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 0.837419 s, <font color="red">250 MB/s</font></pre>
<h3>Example 3-2 (writing to RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 0.823648 s, <font color="red">255 MB/s</font></pre>
<h3>Example 3-3 (reading concurrently from RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache & (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.31025 s, <font color="red">160 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.80016 s, <font color="red">116 MB/s</font></pre>
<h3>Example 3-4 (writing concurrently to RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct & (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.65026 s, <font color="red">127 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.81323 s, <font color="red">116 MB/s</font></pre>
<h3>Example 4-1 (reading from RAID1)</h3>
<pre class="wp-block-preformatted"># mdadm --stop /dev/md/stripped<br /># mdadm --create /dev/md/mirrored --homehost=any --metadata=1.0 --level=1 --raid-devices=2 --assume-clean $MY_DEVS<br /># MY_DISK=/dev/md/mirrored<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.74963 s, <font color="red">120 MB/s</font></pre>
<h3>Example 4-2 (writing to RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.74625 s, <font color="red">120 MB/s</font></pre>
<h3>Example 4-3 (reading concurrently from RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache & (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67171 s, <font color="red">125 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67685 s, <font color="red">125 MB/s</font></pre>
<h3>Example 4-4 (writing concurrently to RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct & (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 4.09666 s, <font color="red">51.2 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 4.1067 s, <font color="red">51.1 MB/s</font></pre>
<h3>Restore your swap device and journald configuration</h3>
<pre class="wp-block-preformatted"># mdadm --stop /dev/md/stripped /dev/md/mirrored<br /># mdadm --create /dev/md/swap --homehost=any --metadata=1.0 --level=1 --raid-devices=2 $MY_DEVS<br /># mkswap /dev/md/swap<br /># swapon -a<br /># mv /etc/systemd/journald.conf.bak /etc/systemd/journald.conf<br /># systemctl restart systemd-journald.service<br /># reboot</pre>
<h2>Interpreting the results</h2>
<p>Examples 1-1, 1-2, 2-1, and 2-2 show that each of my drives read and write at about 125 MB/s.</p>
<p>Examples 1-3, 1-4, 2-3, and 2-4 show that when two reads or two writes are done in parallel on the same drive, each process gets at about half the drive’s bandwidth (60 MB/s).</p>
<p>The 3-x examples show the performance benefit of putting the two drives together in a RAID0 (data stripping) array. The numbers, in all cases, show that the RAID0 array performs about twice as fast as either drive is able to perform on its own. The trade-off is that you are twice as likely to lose everything because each drive only contains half the data. A three-drive array would perform three times as fast as a single drive (all drives being equal) but it would be thrice as likely to suffer a <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://blog.elcomsoft.com/2019/01/why-ssds-die-a-sudden-death-and-how-to-deal-with-it/" target="_blank">catastrophic failure</a>.</p>
<p>The 4-x examples show that the performance of the RAID1 (data mirroring) array is similar to that of a single disk except for the case where multiple processes are concurrently reading (example 4-3). In the case of multiple processes reading, the performance of the RAID1 array is similar to that of the RAID0 array. This means that you will see a performance benefit with RAID1, but only when processes are reading concurrently. For example, if a process tries to access a large number of files in the background while you are trying to use a web browser or email client in the foreground. The main benefit of RAID1 is that your data is unlikely to be lost <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://www.computerworld.com/article/2484998/ssds-do-die--as-linus-torvalds-just-discovered.html" target="_blank">if a drive fails</a>.</p>
<h2>Video demo</h2>
<figure class="wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<iframe class='youtube-player' type='text/html' width='616' height='347' src='https://www.youtube.com/embed/wbLX239-ysQ?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent' allowfullscreen='true' style='border:0;'></iframe>
</div><figcaption>Testing storage throughput using dd</figcaption></figure>
<h2>Troubleshooting</h2>
<p>If the above tests aren’t performing as you expect, you might have a bad or failing drive. Most modern hard drives have built-in <u>S</u>elf-<u>M</u>onitoring, <u>A</u>nalysis and <u>R</u>eporting <u>T</u>echnology (<a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/S.M.A.R.T." target="_blank">SMART</a>). If your drive supports it, the <em>smartctl</em> command can be used to query your hard drive for its internal statistics:</p>
<pre class="wp-block-preformatted"># smartctl --health /dev/sda<br /># smartctl --log=error /dev/sda<br /># smartctl -x /dev/sda</pre>
<p>Another way that you might be able to tune your PC for better performance is by changing your <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/I/O_scheduling" target="_blank">I/O scheduler</a>. Linux systems support several I/O schedulers and the current default for Fedora systems is the <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://lwn.net/Articles/552904/" target="_blank">multiqueue</a> variant of the <a href="https://en.wikipedia.org/wiki/Deadline_scheduler" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">deadline</a> scheduler. The default performs very well overall and scales extremely well for large servers with many processors and large disk arrays. There are, however, a few more specialized schedulers that might perform better under certain conditions.</p>
<p>To view which I/O scheduler your drives are using, issue the following command:</p>
<pre class="wp-block-preformatted">$ for i in /sys/block/sd?/queue/scheduler; do echo "$i: $(<$i)"; done</pre>
<p>You can change the scheduler for a drive by writing the name of the desired scheduler to the /sys/block/<device name>/queue/scheduler file:</p>
<pre class="wp-block-preformatted"># echo bfq > /sys/block/sda/queue/scheduler</pre>
<p>You can make your changes permanent by creating a <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="http://www.reactivated.net/writing_udev_rules.html" target="_blank">udev rule</a> for your drive. The following example shows how to create a udev rule that will set all <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics" target="_blank">rotational drives</a> to use the <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="http://algo.ing.unimo.it/people/paolo/disk_sched/" target="_blank">BFQ</a> I/O scheduler:</p>
<pre class="wp-block-preformatted"># cat << END > /etc/udev/rules.d/60-ioscheduler-rotational.rules<br />ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"<br />END</pre>
<p>Here is another example that sets all <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Solid-state_drive" target="_blank">solid-state drives</a> to use the <a href="https://en.wikipedia.org/wiki/Noop_scheduler" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">NOOP</a> I/O scheduler:</p>
<pre class="wp-block-preformatted"># cat << END > /etc/udev/rules.d/60-ioscheduler-solid-state.rules<br />ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"<br />END</pre>
<p>Changing your I/O scheduler won’t affect the raw throughput of your devices, but it might make your PC seem more responsive by prioritizing the bandwidth for the foreground tasks over the background tasks or by eliminating unnecessary block reordering.</p>
<hr class="wp-block-separator" />
<p><em>Photo by </em><a href="https://unsplash.com/photos/0ZBRKEG_5no?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText"><em>James Donovan</em></a><em> on </em><a href="https://unsplash.com/search/photos/speed?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText"><em>Unsplash</em></a><em>.</em></p>
</div>
<div><p>This article includes some example commands to show you how to get a <em>rough</em> estimate of hard drive and RAID array performance using the <em>dd</em> command. Accurate measurements would have to take into account things like <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://www.ibm.com/developerworks/community/blogs/ibmnas/entry/misalignment_can_be_twice_the_cost1?lang=en" target="_blank">write amplification</a> and <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://eklitzke.org/efficient-file-copying-on-linux" target="_blank">system call overhead</a>, which this guide does not. For a tool that might give more accurate results, you might want to consider using <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Hdparm" target="_blank">hdparm</a>.</p>
<p>To factor out performance issues related to the file system, these examples show how to test the performance of your drives and arrays at the block level by reading and writing directly to/from their block devices. <strong>WARNING</strong>: The <em>write</em> tests will destroy any data on the block devices against which they are run. <strong>Do not run them against any device that contains data you want to keep!</strong></p>
<p> <span id="more-27499"></span> </p>
<h2>Four tests</h2>
<p>Below are four example dd commands that can be used to test the performance of a block device:</p>
<ol>
<li>One process reading from $MY_DISK:
<pre># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
</li>
<li>One process writing to $MY_DISK:
<pre># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
</li>
<li>Two processes reading concurrently from $MY_DISK:
<pre># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre></li>
<li>Two processes writing concurrently to $MY_DISK:
<pre># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct &
; (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre></li>
</ol>
<p>– The <em>iflag=nocache</em> and <em>oflag=direct</em> parameters are important when performing the read and write tests (respectively) because without them the dd command will sometimes show the resulting speed of transferring the data to/from <a href="https://en.wikipedia.org/wiki/Random-access_memory" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">RAM</a> rather than the hard drive.</p>
<p>– The values for the <em>bs</em> and <em>count</em> parameters are somewhat arbitrary and what I have chosen should be large enough to provide a decent average in most cases for current hardware.</p>
<p>– The <em>null</em> and <em>zero</em> devices are used for the destination and source (respectively) in the read and write tests because they are fast enough that they will not be the limiting factor in the performance tests.</p>
<p>– The <em>skip=200</em> parameter on the second dd command in the concurrent read and write tests is to ensure that the two copies of dd are operating on different areas of the hard drive.</p>
<h2>16 examples</h2>
<p>Below are demonstrations showing the results of running each of the above four tests against each of the following four block devices:</p>
<ol>
<li>MY_DISK=/dev/sda2 (used in examples 1-X)</li>
<li>MY_DISK=/dev/sdb2 (used in examples 2-X)</li>
<li>MY_DISK=/dev/md/stripped (used in examples 3-X)</li>
<li>MY_DISK=/dev/md/mirrored (used in examples 4-X)</li>
</ol>
<p>A video demonstration of the these tests being run on a PC is provided at the end of this guide.</p>
<p>Begin by putting your computer into <em>rescue</em> mode to reduce the chances that disk I/O from background services might randomly affect your test results. <strong>WARNING</strong>: This will shutdown all non-essential programs and services. Be sure to save your work before running these commands. You will need to know your <em>root</em> password to get into rescue mode. The <em>passwd</em> command, when run as the root user, will prompt you to (re)set your root account password.</p>
<pre class="wp-block-preformatted">$ sudo -i<br /># passwd<br /># setenforce 0<br /># systemctl rescue</pre>
<p>You might also want to temporarily disable logging to disk:</p>
<pre class="wp-block-preformatted"># sed -r -i.bak 's/^#?Storage=.*/Storage=none/' /etc/systemd/journald.conf<br /># systemctl restart systemd-journald.service</pre>
<p>If you have a swap device, it can be temporarily disabled and used to perform the following tests:</p>
<pre class="wp-block-preformatted"># swapoff -a<br /># MY_DEVS=$(mdadm --detail /dev/md/swap | grep active | grep -o "/dev/sd.*")<br /># mdadm --stop /dev/md/swap<br /># mdadm --zero-superblock $MY_DEVS</pre>
<h3>Example 1-1 (reading from sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.7003 s, <font color="red">123 MB/s</font></pre>
<h3>Example 1-2 (writing to sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67117 s, <font color="red">125 MB/s</font></pre>
<h3>Example 1-3 (reading concurrently from sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.42875 s, <font color="red">61.2 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.52614 s, <font color="red">59.5 MB/s</font></pre>
<h3>Example 1-4 (writing concurrently to sda)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 1)<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct &
; (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre><pre class="wp-block-preformatted">200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.2435 s, <font color="red">64.7 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.60872 s, <font color="red">58.1 MB/s</font></pre>
<h3>Example 2-1 (reading from sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67285 s, <font color="red">125 MB/s</font></pre>
<h3>Example 2-2 (writing to sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67198 s, <font color="red">125 MB/s</font></pre>
<h3>Example 2-3 (reading concurrently from sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.52808 s, <font color="red">59.4 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.57736 s, <font color="red">58.6 MB/s</font></pre>
<h3>Example 2-4 (writing concurrently to sdb)</h3>
<pre class="wp-block-preformatted"># MY_DISK=$(echo $MY_DEVS | cut -d ' ' -f 2)<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct &
; (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.7841 s, <font color="red">55.4 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 3.81475 s, <font color="red">55.0 MB/s</font></pre>
<h3>Example 3-1 (reading from RAID0)</h3>
<pre class="wp-block-preformatted"># mdadm --create /dev/md/stripped --homehost=any --metadata=1.0 --level=0 --raid-devices=2 $MY_DEVS<br /># MY_DISK=/dev/md/stripped<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 0.837419 s, <font color="red">250 MB/s</font></pre>
<h3>Example 3-2 (writing to RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 0.823648 s, <font color="red">255 MB/s</font></pre>
<h3>Example 3-3 (reading concurrently from RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.31025 s, <font color="red">160 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.80016 s, <font color="red">116 MB/s</font></pre>
<h3>Example 3-4 (writing concurrently to RAID0)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/stripped<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct &
; (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.65026 s, <font color="red">127 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.81323 s, <font color="red">116 MB/s</font></pre>
<h3>Example 4-1 (reading from RAID1)</h3>
<pre class="wp-block-preformatted"># mdadm --stop /dev/md/stripped<br /># mdadm --create /dev/md/mirrored --homehost=any --metadata=1.0 --level=1 --raid-devices=2 --assume-clean $MY_DEVS<br /># MY_DISK=/dev/md/mirrored<br /># dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.74963 s, <font color="red">120 MB/s</font></pre>
<h3>Example 4-2 (writing to RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct</pre>
<pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.74625 s, <font color="red">120 MB/s</font></pre>
<h3>Example 4-3 (reading concurrently from RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache &
; (dd if=$MY_DISK of=/dev/null bs=1MiB count=200 iflag=nocache skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67171 s, <font color="red">125 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 1.67685 s, <font color="red">125 MB/s</font></pre>
<h3>Example 4-4 (writing concurrently to RAID1)</h3>
<pre class="wp-block-preformatted"># MY_DISK=/dev/md/mirrored<br /># (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct &
; (dd if=/dev/zero of=$MY_DISK bs=1MiB count=200 oflag=direct skip=200 &</pre><pre class="wp-block-preformatted">200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 4.09666 s, <font color="red">51.2 MB/s</font><br />200+0 records in<br />200+0 records out<br />209715200 bytes (210 MB, 200 MiB) copied, 4.1067 s, <font color="red">51.1 MB/s</font></pre>
<h3>Restore your swap device and journald configuration</h3>
<pre class="wp-block-preformatted"># mdadm --stop /dev/md/stripped /dev/md/mirrored<br /># mdadm --create /dev/md/swap --homehost=any --metadata=1.0 --level=1 --raid-devices=2 $MY_DEVS<br /># mkswap /dev/md/swap<br /># swapon -a<br /># mv /etc/systemd/journald.conf.bak /etc/systemd/journald.conf<br /># systemctl restart systemd-journald.service<br /># reboot</pre>
<h2>Interpreting the results</h2>
<p>Examples 1-1, 1-2, 2-1, and 2-2 show that each of my drives read and write at about 125 MB/s.</p>
<p>Examples 1-3, 1-4, 2-3, and 2-4 show that when two reads or two writes are done in parallel on the same drive, each process gets at about half the drive’s bandwidth (60 MB/s).</p>
<p>The 3-x examples show the performance benefit of putting the two drives together in a RAID0 (data stripping) array. The numbers, in all cases, show that the RAID0 array performs about twice as fast as either drive is able to perform on its own. The trade-off is that you are twice as likely to lose everything because each drive only contains half the data. A three-drive array would perform three times as fast as a single drive (all drives being equal) but it would be thrice as likely to suffer a <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://blog.elcomsoft.com/2019/01/why-ssds-die-a-sudden-death-and-how-to-deal-with-it/" target="_blank">catastrophic failure</a>.</p>
<p>The 4-x examples show that the performance of the RAID1 (data mirroring) array is similar to that of a single disk except for the case where multiple processes are concurrently reading (example 4-3). In the case of multiple processes reading, the performance of the RAID1 array is similar to that of the RAID0 array. This means that you will see a performance benefit with RAID1, but only when processes are reading concurrently. For example, if a process tries to access a large number of files in the background while you are trying to use a web browser or email client in the foreground. The main benefit of RAID1 is that your data is unlikely to be lost <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://www.computerworld.com/article/2484998/ssds-do-die--as-linus-torvalds-just-discovered.html" target="_blank">if a drive fails</a>.</p>
<h2>Video demo</h2>
<figure class="wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<iframe class='youtube-player' type='text/html' width='616' height='347' src='https://www.youtube.com/embed/wbLX239-ysQ?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent' allowfullscreen='true' style='border:0;'></iframe>
</div><figcaption>Testing storage throughput using dd</figcaption></figure>
<h2>Troubleshooting</h2>
<p>If the above tests aren’t performing as you expect, you might have a bad or failing drive. Most modern hard drives have built-in <u>S</u>elf-<u>M</u>onitoring, <u>A</u>nalysis and <u>R</u>eporting <u>T</u>echnology (<a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/S.M.A.R.T." target="_blank">SMART</a>). If your drive supports it, the <em>smartctl</em> command can be used to query your hard drive for its internal statistics:</p>
<pre class="wp-block-preformatted"># smartctl --health /dev/sda<br /># smartctl --log=error /dev/sda<br /># smartctl -x /dev/sda</pre>
<p>Another way that you might be able to tune your PC for better performance is by changing your <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/I/O_scheduling" target="_blank">I/O scheduler</a>. Linux systems support several I/O schedulers and the current default for Fedora systems is the <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://lwn.net/Articles/552904/" target="_blank">multiqueue</a> variant of the <a href="https://en.wikipedia.org/wiki/Deadline_scheduler" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">deadline</a> scheduler. The default performs very well overall and scales extremely well for large servers with many processors and large disk arrays. There are, however, a few more specialized schedulers that might perform better under certain conditions.</p>
<p>To view which I/O scheduler your drives are using, issue the following command:</p>
<pre class="wp-block-preformatted">$ for i in /sys/block/sd?/queue/scheduler; do echo "$i: $(<$i)"; done</pre>
<p>You can change the scheduler for a drive by writing the name of the desired scheduler to the /sys/block/<device name>/queue/scheduler file:</p>
<pre class="wp-block-preformatted"># echo bfq > /sys/block/sda/queue/scheduler</pre>
<p>You can make your changes permanent by creating a <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="http://www.reactivated.net/writing_udev_rules.html" target="_blank">udev rule</a> for your drive. The following example shows how to create a udev rule that will set all <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics" target="_blank">rotational drives</a> to use the <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="http://algo.ing.unimo.it/people/paolo/disk_sched/" target="_blank">BFQ</a> I/O scheduler:</p>
<pre class="wp-block-preformatted"># cat << END > /etc/udev/rules.d/60-ioscheduler-rotational.rules<br />ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"<br />END</pre>
<p>Here is another example that sets all <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://en.wikipedia.org/wiki/Solid-state_drive" target="_blank">solid-state drives</a> to use the <a href="https://en.wikipedia.org/wiki/Noop_scheduler" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">NOOP</a> I/O scheduler:</p>
<pre class="wp-block-preformatted"># cat << END > /etc/udev/rules.d/60-ioscheduler-solid-state.rules<br />ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"<br />END</pre>
<p>Changing your I/O scheduler won’t affect the raw throughput of your devices, but it might make your PC seem more responsive by prioritizing the bandwidth for the foreground tasks over the background tasks or by eliminating unnecessary block reordering.</p>
<hr class="wp-block-separator" />
<p><em>Photo by </em><a href="https://unsplash.com/photos/0ZBRKEG_5no?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText"><em>James Donovan</em></a><em> on </em><a href="https://unsplash.com/search/photos/speed?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText"><em>Unsplash</em></a><em>.</em></p>
</div>