Self-Test Diagnostics

We added these commands in response to inefficiencies (and in some case firmware bugs), associated with the built-in self-test functions found in most SCSI and Fibre Channel disk drives. We wanted to provide a tool that would scan the entire disk and produce a report of all errors (or warnings/retries) by block number. The administrator and storage vendor could analyze and correct the most common errors such as unrecoverable read/write errors due to a failed sector without having to re-run the self-test after repairing the next bad block. (Self-tests only report one error, then they stop).

Like the self-tests described in the Self-Test Diagnostics ANSI section, all of these tests are safe to run in a live environment with user I/O running in the background. As the scrubbing self tests described in this section are controlled by the host, there is additional overhead. This overhead is one I/O per 512, 520 or whatever block-size you have times the number of blocks there are on the disk drive. As only one block is read at a time (with -scrub) or only 32 blocks are read at a time with (-scrubq), the test would generally take 30 minutes to several hours to run, even on a system with little overhead.

If you have to test multiple drives, it is best to run multiple instances of the program concurrently. CPU overhead is almost zero. The bottleneck is your disk I/O channel.

-scrubq	Initiates full media read test, with 32-block chunk size
-scrub	Initiates full media read test, with 1-block chunk size.
-scrubr	Pseudo-random read test using SEEK(10 SCSI Command)
-scrubs	Sequential read fitness test using SEEK(10 SCSI Command)
-scrubv	May be combined with either option above to set verbose mode so that errors, percentage complete, and remaining time appear as they are discovered.
-scrubt	This terminates any fitness test on the first error and causes the program to return error code #11 (SCRUB_T_ERR). The -scrubt must be combined with the -scrub, or -scrubq command.
-16	May be combined with any of the above options to utilize 16-byte SCSI commands READ(16) and WRITE(16)
-12	May be combined with any of the above options to utilize 12-byte SCSI commands READ(12) and WRITE(12)

Test Option

Description

Type of Test / Methodology

Strengths

Weaknesses

-stsb

short background (ANSI-defined test, built into the device's firmware)

•	Single command sent, disk runs test for up to 2 minutes, saves result in log page.

•	Once command is launched by SMARTMonUX, no further interaction required. Unlimited number of disks can be tested concurrently without adversely affecting host system or I/O bandwidth.

•	Full test of all except media, but media does have light test.

•	Completes in less than 2 minutes regardless of host I/O load.

•	Unlimited instances can be run concurrently w/o adverse affect on host.

•	Not good for certifying media, but can be combined with -scrub for a thorough test (but best to combine -steb with -scrub for most complete test).

•	Useless for testing DVD and CDROM media.

-steb

extended background

(ANSI-defined test, built into the device's firmware)

•	Disk vendors use this as a pass/fail criteria to authorize warranty returns.

•	Results viewable with -C and -str commands.

•	Tests 100% of disk, including random I/O.

•	Like the -stsb, this test also has no host overhead once it is accepted by the disk.

•	It only returns first error then terminates.

•	Only way to get a full disk test if you have any errors is to correct problem and start again. This could take days of operator time if you have multiple errors towards end of a large disk.

-scrub

scrub test

•	Reads all blocks on disk and reports sense information resulting from every I/O.

•	Automatic retries as necessary depending on the errors.

•	Full report of non-zero sense information and errors/retries

•	Single pass-read-everything, returns all errors in report by block number.

•	Use it to then manually reassign sectors in single pass or to send to storage vendor for analysis for drive replacement.

•	No random I/O test.

•	No non-media tests.

•	You should combine this test with the -steb to guarantee 100% testing. Run -scrub first and reassign all sectors first so the -steb will not stop when it finds first error.

-scrubq

quick scrub test

•	Same as above, but it does 32 blocks at a time to finish test much earlier.

•	Does full read, but finishes much faster than -scrub.

•	Use it to quickly find out if there is any sense data indicating drive needs to be replaced or if further action required to repair it.

•	Blocks are read in chunks of 32, so sense errors are tied to range of blocks.

•	You will have to run the -scrub or -steb options determine exactly what block(s) you need to remap.

-scrubr

random seek test

•	Repositions the head in a pseudo-random sequence until one seek has been done for every 16 blocks of data.on the disk. This invokes the SEEK(10) SCSI CDB.

•	This is an important test and successful sequential reads or write tests will not stress the drive arm assembly sufficiently.

•	The -scrubr & -scrubs commands are mutually exclusive. You must perform each test separately.

-scrubs

sequential seek test

•	Repositions the head from beginning to end of disk using the SEEK(10) SCSI CDB

•	Arguably not as useful or stressful on a disk then performing random seeks.

•	The -scrubr & -scrubs commands are mutually exclusive. You must perform each test separately.

-scrubt

terminate on first error

•	Terminates any of these self-test diagnostics upon first error

•	Self test aborts if problem found, dramatically speeding up process of testing multiple devices.

•	Test does not report all errors found and/or repaired.

-scrubv

Verbose scrub

Combine with -scrub or -scrubq to show results in foreground.

•	It shows percentage complete and remaining time.

•	Do not redirect output to a file as the file will contain large amount of formatted text and backspace chars.

Discovered SEAGATE ST373405FC S/N "3EK0V6SG" on /dev/sg9 [SES] (Not Enabling SMART)(70007 MB)

(Note percentage complete information and time remaining will appear and automatically update as this procedure progresses. This is not shown below)

Beginning SANtools fitness test for SEAGATE ST373405FC at /dev/sg9 (143374740 blocks, blocksize=512)

Block 145614 Sense: 4/32/00 [Controller/drive hardware failed] No defect spare location available

Block 145618 Sense: 4/32/00 [Controller/drive hardware failed] No defect spare location available

Block 145619 Sense: 4/32/00 [Controller/drive hardware failed] No defect spare location available

Block 145614 4/32/00 Count=1 [Controller/drive hardware failed] No defect spare location available

Block 145618 4/32/00 Count=3 [Controller/drive hardware failed] No defect spare location available

Block 145619 4/32/00 Count=2 [Controller/drive hardware failed] No defect spare location available

The -scrub command reports errors at the block level, by reading each block individually. As such, it sacrifices speed for granularity. Our 146GB 15000RPM SAS disk takes 10 hours to complete using these options. If you don't care about individual block numbers, but still want a count of the bad blocks, then use the -scrubq which reads 32 blocks at a time. The same disk that took 10 hours to test with the -scrubq command takes 32 minutes to complete.

If you just need a pass-fail test too see if a particular disk has any read problems, then be sure to add the -scrubt option so that it terminates on the first error. The results below were run on the same disk which has bad blocks which we created with this software on blocks 123 and 456.

Discovered SEAGATE ST3146855SS S/N "3LN29ZZ5" on /dev/rdsk/c4t15d0s0 (Not Enabling SMART)(140014 MB)

Block 123 4/09/00 Count=3 [Controller/drive hardware failed] Track following error

Block 456 4/09/00 Count=3 [Controller/drive hardware failed] Track following error

Discovered SEAGATE ST3146855SS S/N "3LN29ZZ5" on /dev/rdsk/c4t15d0s0 (Not Enabling SMART)(140014 MB)

Blocks 96 - 112 4/09/00 Count=3 [Controller/drive hardware failed] Track following error

Blocks 448 - 464 4/09/00 Count=3 [Controller/drive hardware failed] Track following error

Discovered SEAGATE ST3146855SS S/N "3LN29ZZ5" on /dev/rdsk/c4t15d0s0 (Not Enabling SMART)(140014 MB)

Blocks 96 - 128 4/09/00 Count=1 [Controller/drive hardware failed] Track following error

If your disks support background media scanning, then you can just ask the disk if it has any problems via the -bmsr command (assuming scanning is enabled). This will generate a report based on the last background scan the selected disk ran, and any subsequent activity since that scan. It will take less than a second to report all bad blocks on the disk, regardless of how many you have and where they are located. The disk retains this information through power-cycles.

Discovered SEAGATE ST3146855SS S/N "3LN29ZZ5" on /dev/rdsk/c4t15d0s0 (Not Enabling SMART)(140014 MB)

Background scanning status: medium scan halted, waiting for interval timer expiration

0 133 37fc7 OK recovered via in-place rewrite Recovered error Recovered data with retries

1 117114 2bf620f OK recovered via in-place rewrite Recovered error Recovered data with retries

2 130954 7b ERR waiting for WRITE Controller/drive hardware failed Track following error

3 130954 1c8 ERR waiting for WRITE Controller/drive hardware failed Track following error

4 130954 37fc7 OK recovered via in-place rewrite Recovered error Recovered data with retries

5 131392 37fc8 OK recovered via in-place rewrite Recovered error Recovered data with retries

6 133380 38039 OK recovered via in-place rewrite Recovered error Recovered data with retries

7 133792 d699104 OK recovered via in-place rewrite Recovered error Recovered data with retries

8 134753 dccde66 OK recovered via in-place rewrite Recovered error Recovered data with retries

9 134755 e2bede7 OK recovered via in-place rewrite Recovered error Recovered data with retries