Asus C60M1-I - FreeNAS
Hardware info on main page.
FreeNAS build: FreeNAS-8.3.0-RELEASE-x64 (r12701M)
Hints: web gui user is 'admin' (no, not with a default password).
Work log
2021-04-05: 'smartctl -H' check on the drives (ada3 is gone), all report "passed", ada4 reports this
tingo@kg-f5$ sudo smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 066 045 045 Old_age Always In_the_past 34 (Min/Max 31/38)
ok
2021-04-05: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 132K in 28h50m with 0 errors on Mon Apr 5 04:50:42 2021 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/9674e5d0-899c-11eb-8389-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 41 583K 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
do a zpool clear
tingo@kg-f5$ sudo zpool clear z5 tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scan: scrub repaired 132K in 28h50m with 0 errors on Mon Apr 5 04:50:42 2021 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/9674e5d0-899c-11eb-8389-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 15980096276240037494 UNAVAIL 0 0 0 was /dev/gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
that's interesting. Check which device
tingo@kg-f5$ gptid-to-device /dev/gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 tingo@kg-f5$ gptid-to-device gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ada0p2 tingo@kg-f5$ gptid-to-device gptid/9674e5d0-899c-11eb-8389-3085a9ebf2a2 ada1p2 tingo@kg-f5$ gptid-to-device gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ada2p2 tingo@kg-f5$ gptid-to-device gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ada4p2 tingo@kg-f5$ gptid-to-device gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ada5p2 tingo@kg-f5$ gptid-to-device gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2
ok, so it is ada3 then.
2021-03-21: latest zpool status
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 2.66T in 23h22m with 0 errors on Sun Mar 21 17:14:48 2021 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/9674e5d0-899c-11eb-8389-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
good.
2021-03-20: replacing drive ada1. After physically replacing the drive and powering up the machine again,In the gui, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "16885783182919210731". I select "Replace", pick the new ada1, and click "Replace Disk". It chews on that for a few seconds, then reloads the volume status screen. "16885783182919210731" is still there, so I do a Detach on it. And the resilver is in progress:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Mar 20 17:52:03 2021 41.7G scanned out of 16.0T at 145M/s, 31h58m to go 6.93G resilvered, 0.25% done config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/9674e5d0-899c-11eb-8389-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2021-03-20: after rebooting the machine, I had to fix sudo again. I fixed sudo access by doing 'su -' and then running 'visudo', adding the line for the wheel group, like this:
tingo@kg-f5$ cat /usr/local/etc/sudoers Defaults syslog_goodpri = debug # Let find_alias_for_smtplib.py runs as root (it needs database access) ALL ALL=(ALL) NOPASSWD: /etc/find_alias_for_smtplib.py %wheel ALL=(ALL) NOPASSWD: ALL
ok.
2021-02-22: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 25h41m with 0 errors on Mon Feb 22 01:41:59 2021 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 FAULTED 0 1.19M 0 too many errors gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok. I should really replace that drive soon.
2021-01-11: latest scrub result
tingo@kg-f5$ date;zpool status z5 Mon Jan 11 12:00:58 CET 2021 pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 26h0m with 0 errors on Mon Jan 11 02:00:35 2021 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 FAULTED 0 1.19M 0 too many errors gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
which disk
tingo@kg-f5$ gptid-to-device gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ada1p2
ok
2020-11-30: latest scrub result
tingo@kg-f5$ date;zpool status z5 Mon Nov 30 20:35:38 CET 2020 pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 26h19m with 0 errors on Mon Nov 30 02:19:06 2020 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 FAULTED 0 1.19M 0 too many errors gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2020-11-29: latest status for pool z5
tingo@kg-f5$ zpool status z5 pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub in progress since Sun Nov 29 00:00:05 2020 9.88T scanned out of 16.0T at 178M/s, 9h57m to go 0 repaired, 61.88% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 FAULTED 0 1.19M 0 too many errors gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
check which drive
tingo@kg-f5$ gptid-to-device gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ada1p2
ok, so ada1 needs to be replaced.
tingo@kg-f5$ sudo smartctl -H /dev/ada1 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net (pass1:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 01 00 (pass1:ahcich1:0:0:0): CAM status: ATA Status Error (pass1:ahcich1:0:0:0): ATA status: 71 (DRDY DF SERV ERR), error: 04 (ABRT ) (pass1:ahcich1:0:0:0): RES: 71 04 00 00 00 40 00 00 00 01 00 (pass1:ahcich1:0:0:0): ATAPI_IDENTIFY. ACB: a1 00 00 00 00 40 00 00 00 00 01 00 (pass1:ahcich1:0:0:0): CAM status: ATA Status Error (pass1:ahcich1:0:0:0): ATA status: 71 (DRDY DF SERV ERR), error: 04 (ABRT ) (pass1:ahcich1:0:0:0): RES: 71 04 00 00 00 40 00 00 00 01 00 Smartctl: Device Read Identity Failed: Unknown error A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
smartctl agrees.
2020-10-26: latest scrub result
tingo@kg-f5$ date;zpool status Mon Oct 26 12:16:28 CET 2020 pool: z5 state: ONLINE scan: scrub repaired 64K in 26h49m with 0 errors on Mon Oct 26 01:49:22 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2020-09-28: zpool status
tingo@kg-f5$ date;zpool status Mon Sep 28 19:45:29 CEST 2020 pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: resilvered 2.66T in 23h43m with 0 errors on Mon Sep 28 16:27:09 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 1 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
hmm, which device? nothing in /var/log/messages (only error count increases) ada1 has one marginal value
tingo@kg-f5$ sudo smartctl -H /dev/ada1 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 062 043 045 Old_age Always In_the_past 38 (1 201 40 34 0) ada3 too tingo@kg-f5$ sudo smartctl -H /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 060 043 045 Old_age Always In_the_past 40 (2 60 43 34 0) ada4 tingo@kg-f5$ sudo smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 063 045 045 Old_age Always In_the_past 37 (Min/Max 36/39)
but that's it. Clear errors
tingo@kg-f5$ sudo zpool clear z5
verify
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 2.66T in 23h43m with 0 errors on Mon Sep 28 16:27:09 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
good.
2020-09-27: sudo - I fixed sudo access by doing 'su -' and then running 'visudo', adding the line for the wheel group, like this:
tingo@kg-f5$ more /usr/local/etc/sudoers Defaults syslog_goodpri = debug # Let find_alias_for_smtplib.py runs as root (it needs database access) ALL ALL=(ALL) NOPASSWD: /etc/find_alias_for_smtplib.py %wheel ALL=(ALL) NOPASSWD: ALL
ok
2020-09-27: the pool is resilvering
tingo@kg-f5$ date; zpool status Sun Sep 27 16:47:56 CEST 2020 pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Sep 27 16:44:01 2020 28.2G scanned out of 16.0T at 123M/s, 37h52m to go 4.69G resilvered, 0.17% done config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/deb5c996-00cf-11eb-9ff9-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
good so far.
2020-09-27: replacing ada2. I did the physical replacement of ada2, let's see if I can replace it in the gui. into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "1471878445799208237". I select "Replace", pick the new ada2, and click "Replace Disk". It chews on that for a few seconds, then reloads the volume status screen. "1471878445799208237" is still there, so I do a Detach on it.
2020-09-14: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 448K in 27h31m with 0 errors on Mon Sep 14 03:31:30 2020 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 FAULTED 0 1.34K 0 too many errors gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
yeah, I should replace ada2 now.
2020-09-13: current zpool status
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub in progress since Sun Sep 13 00:00:05 2020 6.13T scanned out of 16.0T at 170M/s, 16h55m to go 128K repaired, 38.31% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 (repairing) gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 FAULTED 0 1.34K 0 too many errors gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
this is
tingo@kg-f5$ gptid-to-device gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ada2p2
so ada2 needs to be replaced.
2020-08-03: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.72M in 31h54m with 0 errors on Mon Aug 3 07:54:37 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2020-06-22: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.94M in 28h50m with 0 errors on Mon Jun 22 04:50:47 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2020-06-21: some of the drives have marginal parameters
tingo@kg-f5$ sudo smartctl -H /dev/ada1 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 045 043 045 Old_age Always FAILING_NOW 55 (1 57 57 21 0) tingo@kg-f5$ sudo smartctl -H /dev/ada2 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 043 041 045 Old_age Always FAILING_NOW 57 (2 148 59 21 0) tingo@kg-f5$ sudo smartctl -H /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 044 043 045 Old_age Always FAILING_NOW 56 (0 159 57 21 0) ada4 had one in the past tingo@kg-f5$ sudo smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 047 045 045 Old_age Always In_the_past 53 (Min/Max 21/55) ada0 as ada5 seem ok for now tingo@kg-f5$ sudo smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED tingo@kg-f5$ sudo smartctl -H /dev/ada5 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
ok
2020-05-11: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 672K in 27h50m with 0 errors on Mon May 11 03:50:43 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok.
2020-03-30: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 26h47m with 0 errors on Mon Mar 30 03:47:41 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok
2020-02-17: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 224K in 26h35m with 0 errors on Mon Feb 17 02:35:39 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok.
2020-01-06: latest scrub result
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 128K in 26h22m with 0 errors on Mon Jan 6 02:22:40 2020 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
ok.
2019-11-25: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 640K in 27h11m with 0 errors on Mon Nov 25 03:11:33 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-10-21: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 512K in 26h21m with 0 errors on Mon Oct 21 02:21:42 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-09-09: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 768K in 26h5m with 0 errors on Mon Sep 9 02:05:18 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-07-29: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 7.19M in 27h52m with 0 errors on Mon Jul 29 03:52:12 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-06-17: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.34M in 27h14m with 0 errors on Mon Jun 17 03:14:22 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-05-06: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 4.91M in 27h7m with 0 errors on Mon May 6 03:07:21 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2019-03-25: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 832K in 26h9m with 0 errors on Mon Mar 25 02:09:42 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok
2019-02-11: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 128K in 29h8m with 0 errors on Mon Feb 11 05:08:08 2019 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-12-31: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 1.80M in 29h7m with 0 errors on Mon Dec 31 05:07:59 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-11-27: nfs - enabling a nfs share is easy from the gui Sharing, Unix (NFS) Shares, Add Unix (NFS) Share, fill in info (all directories + path), enable. Then it shows up in showmount like so tingo@kg-f5$ showmount -e Exports list on localhost: /mnt/z5/
2018-11-19: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 4K in 30h13m with 0 errors on Mon Nov 19 06:13:15 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-10-15: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 384K in 26h33m with 0 errors on Mon Oct 15 02:33:09 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-09-03: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 128K in 24h37m with 0 errors on Mon Sep 3 00:37:27 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-07-23: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 4.44M in 24h48m with 0 errors on Mon Jul 23 00:48:04 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-06-10: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 23h40m with 0 errors on Sun Jun 10 23:40:58 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-05-04: zpool - resilver complete tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 2.00T in 16h30m with 0 errors on Fri May 4 18:13:25 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok 2018-05-04: zpool is resilvering tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri May 4 01:42:34 2018 27.3G scanned out of 12.0T at 168M/s, 20h41m to go 4.53G resilvered, 0.22% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/a2ac8dc0-4f2b-11e8-b456-3085a9ebf2a2 ONLINE 0 0 0 (resilvering)
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors good. 2018-05-04: replacing ada2.Let's see if I can replace it now. into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "17249042697807205164". I select "Replace", pick the new ada2, and click "Replace Disk". It chews on that for a few seconds, then reloads the volume status screen. "17249042697807205164" is still there, so I do a Detach on it. 2018-05-04: gui - ok, copying enough of /usr/local/etc/nginx/nginx.conf from f3 so that I get a working nginx config allows me to login to the gui again. Temporary solution, but anyway. 2018-05-04: gui - checking for errors [root@kg-f5] ~# sqlite3 /data/freenas-v1.db SQLite version 3.7.13 2012-06-11 02:05:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> SELECT stg_guiaddress FROM system_settings; 0.0.0.0 sqlite> .quit [root@kg-f5] ~# ok, so the gui isn't bound to wrong address. 2018-05-04: replacing ada2. After physically replacing ada2, I try to log into the gui again, but all I get is the "Welcome to nginx!" screen. what now? 2018-05-04: pool z5 - scrub is slow, very slow. tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub in progress since Sun Apr 29 00:00:05 2018 1.45T scanned out of 12.0T at 3.51M/s, (scan is slow, no estimated time) 0 repaired, 12.09% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors from /var/log/messages, the likely culprit is ada2 May 3 23:03:32 kg-f5 smartd[2382]: Device: /dev/ada2, 63816 Currently unreadable (pending) sectors (changed +24) May 3 23:03:32 kg-f5 smartd[2382]: Device: /dev/ada2, 63816 Offline uncorrectable sectors (changed +24) May 3 23:33:31 kg-f5 smartd[2382]: Device: /dev/ada2, 63840 Currently unreadable (pending) sectors (changed +24) May 3 23:33:31 kg-f5 smartd[2382]: Device: /dev/ada2, 63840 Offline uncorrectable sectors (changed +24) May 4 00:03:35 kg-f5 smartd[2382]: Device: /dev/ada2, 63856 Currently unreadable (pending) sectors (changed +16) May 4 00:03:35 kg-f5 smartd[2382]: Device: /dev/ada2, 63856 Offline uncorrectable sectors (changed +16) it should be replaced now. Info tingo@kg-f5$ gptid-to-device gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ada2p2 ok 2018-04-23: pool z5 - resilver complete, all ok tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 1.99T in 16h43m with 0 errors on Mon Apr 23 12:23:38 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-04-22: replacing ada3. After physically replacing ada3, I log into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "14233525495841027009". I select "Replace", pick the new ada3, and click "Replace Disk". It chews on that for a few seconds, then reloads the volume status screen. "14233525495841027009" is still l there, so I do a Detach on it. and the pool is resilvering: tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Apr 22 19:39:48 2018 16.7G scanned out of 12.0T at 157M/s, 22h10m to go 2.78G resilvered, 0.14% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/262c7786-4654-11e8-b2e4-3085a9ebf2a2 ONLINE 0 0 0 (resilvering)
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2018-04-17: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 1.72M in 719h7m with 0 errors on Tue Apr 17 00:07:37 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 39 196K 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors Using gptid-to-device from freebsd-tools by Pavel Merdin to map gptid's to device tingo@kg-f5$ gptid-to-device gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ada3p2 I should replace ada3. Also, from /var/log/messages, ada2 doesn't look error-free either.
2018-02-04: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 128K in 20h21m with 0 errors on Sun Feb 4 20:21:51 2018 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-11-12: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 17h6m with 0 errors on Sun Nov 12 17:06:28 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-10-08: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 17h44m with 0 errors on Sun Oct 8 17:44:34 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-08-28: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 1.34M in 18h24m with 0 errors on Sun Aug 27 18:24:56 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-07-16: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 17h2m with 0 errors on Sun Jul 16 17:02:36 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-06-04: pool status tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 96K in 15h30m with 0 errors on Sun Jun 4 15:30:56 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-05-29: pool - resilver complete tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 1.61T in 13h29m with 0 errors on Mon May 29 04:48:39 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-05-28: replacing ada4. After physically replacing ada4, I log into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "7759962039369557683". I select "Replace", pick the new ada4, and click "Replace Disk". It chews on that for a few seconds, then reloads the volume status screen. "7759962039369557683" is still l there, so I do a Detach on it. and now the pool is resilvering tingo@kg-f5$ date;zpool status Sun May 28 15:21:37 CEST 2017 pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun May 28 15:18:43 2017 31.5G scanned out of 9.65T at 185M/s, 15h6m to go 5.25G resilvered, 0.32% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/2aeaa816-43a8-11e7-b637-3085a9ebf2a2 ONLINE 0 0 0 (resilvering)
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok. 2017-05-28: pool status - it took a looong time; 35 days 14 hours and 1 minute. tingo@kg-f5$ date;zpool status Sun May 28 14:03:27 CEST 2017 pool: z5 state: ONLINE scan: scrub repaired 115M in 854h1m with 0 errors on Sun May 28 14:02:55 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors I'd better change out the bad drive soon. 2017-05-28: ada4 error changes: May 28 05:40:47 kg-f5 smartd[2382]: Device: /dev/ada4, 58288 Currently unreadable (pending) sectors (changed -6800) May 28 05:40:47 kg-f5 smartd[2382]: Device: /dev/ada4, 58288 Offline uncorrectable sectors (changed -6800) May 28 05:40:48 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 0 to 286 May 28 06:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 50048 Currently unreadable (pending) sectors (changed -8240) May 28 06:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 50048 Offline uncorrectable sectors (changed -8240) May 28 06:10:58 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 286 to 600 May 28 06:40:42 kg-f5 smartd[2382]: Device: /dev/ada4, 40744 Currently unreadable (pending) sectors (changed -9304) May 28 06:40:42 kg-f5 smartd[2382]: Device: /dev/ada4, 40744 Offline uncorrectable sectors (changed -9304) May 28 06:40:53 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 600 to 911 May 28 07:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, failed to read SMART Attribute Data May 28 07:10:40 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 911 to 1192 May 28 07:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 23760 Currently unreadable (pending) sectors (changed -16984) May 28 07:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 23760 Offline uncorrectable sectors (changed -16984) May 28 07:40:37 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 1192 to 1526
May 28 08:11:00 kg-f5 smartd[2382]: Device: /dev/ada4, 13008 Currently unreadable (pending) sectors (changed -10752) May 28 08:11:00 kg-f5 smartd[2382]: Device: /dev/ada4, 13008 Offline uncorrectable sectors (changed -10752) May 28 08:11:00 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 1526 to 1843 May 28 08:40:39 kg-f5 smartd[2382]: Device: /dev/ada4, 4472 Currently unreadable (pending) sectors (changed -8536) May 28 08:40:39 kg-f5 smartd[2382]: Device: /dev/ada4, 4472 Offline uncorrectable sectors (changed -8536) May 28 08:40:51 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 1843 to 2163 May 28 09:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 59640 Currently unreadable (pending) sectors (changed +55168) May 28 09:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 59640 Offline uncorrectable sectors (changed +55168) May 28 09:10:39 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 2163 to 2465 May 28 09:40:36 kg-f5 smartd[2382]: Device: /dev/ada4, 48368 Currently unreadable (pending) sectors (changed -11272) May 28 09:40:36 kg-f5 smartd[2382]: Device: /dev/ada4, 48368 Offline uncorrectable sectors (changed -11272) May 28 09:40:51 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 2465 to 2766 May 28 10:10:37 kg-f5 smartd[2382]: Device: /dev/ada4, 37104 Currently unreadable (pending) sectors (changed -11264) May 28 10:10:37 kg-f5 smartd[2382]: Device: /dev/ada4, 37104 Offline uncorrectable sectors (changed -11264) May 28 10:10:44 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 2766 to 3080 May 28 10:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 28744 Currently unreadable (pending) sectors (changed -8360) May 28 10:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 28744 Offline uncorrectable sectors (changed -8360) May 28 10:40:39 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 3080 to 3368 May 28 11:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 19584 Currently unreadable (pending) sectors (changed -9160) May 28 11:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 19584 Offline uncorrectable sectors (changed -9160) May 28 11:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 3368 to 3680 May 28 11:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 11248 Currently unreadable (pending) sectors (changed -8336) May 28 11:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 11248 Offline uncorrectable sectors (changed -8336) May 28 11:40:38 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 3680 to 4008
May 28 12:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 600 Currently unreadable (pending) sectors (changed -10648) May 28 12:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 600 Offline uncorrectable sectors (changed -10648) May 28 12:10:32 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 4008 to 4285 this drive doesn't last much longer. 2017-05-28: ada2 error changes May 28 12:40:26 kg-f5 smartd[2382]: Device: /dev/ada2, 344 Currently unreadable (pending) sectors (changed +152) May 28 12:40:26 kg-f5 smartd[2382]: Device: /dev/ada2, 344 Offline uncorrectable sectors (changed +152) May 28 12:40:29 kg-f5 smartd[2382]: Device: /dev/ada2, ATA error count increased from 0 to 185 May 28 13:10:24 kg-f5 smartd[2382]: Device: /dev/ada2, 608 Currently unreadable (pending) sectors (changed +264) May 28 13:10:24 kg-f5 smartd[2382]: Device: /dev/ada2, 608 Offline uncorrectable sectors (changed +264) May 28 13:10:24 kg-f5 smartd[2382]: Device: /dev/ada2, ATA error count increased from 185 to 387 ok.
2017-05-07: pool status - after a week, the scrub has done less than 32%. tingo@kg-f5$ date;zpool status Sun May 7 12:45:06 CEST 2017 pool: z5 state: ONLINE scan: scrub in progress since Sun Apr 23 00:01:24 2017 3.07T scanned out of 9.63T at 2.56M/s, (scan is slow, no estimated time) 0 repaired, 31.85% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-04-23: pool status - at 18:11 local time, the scrub is still ongoing: tingo@kg-f5$ date;zpool status Sun Apr 23 18:11:26 CEST 2017 pool: z5 state: ONLINE scan: scrub in progress since Sun Apr 23 00:01:24 2017 136G scanned out of 9.63T at 2.14M/s, (scan is slow, no estimated time) 0 repaired, 1.38% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok. 2017-04-23: ada4 error changes Apr 23 08:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 65088 Currently unreadable (pending) sectors (changed +8) Apr 23 08:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 65088 Offline uncorrectable sectors (changed +8) ok.
2017-04-21: ada4 error changes Apr 21 06:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 65080 Currently unreadable (pending) sectors (changed +8) Apr 21 06:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 65080 Offline uncorrectable sectors (changed +8) ok
2017-04-20: ada4 error changes Apr 20 17:40:26 kg-f5 smartd[2382]: Device: /dev/ada4, 65072 Currently unreadable (pending) sectors (changed +8) Apr 20 17:40:26 kg-f5 smartd[2382]: Device: /dev/ada4, 65072 Offline uncorrectable sectors (changed +8) ok
2017-04-19: ada2 health [root@kg-f5] ~# smartctl -H /dev/ada2 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2017-04-19: ada4 health [root@kg-f5] ~# smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2017-04-19: ada2 errors and changes Apr 19 21:10:24 kg-f5 smartd[2382]: Device: /dev/ada2, 136 Currently unreadable (pending) sectors Apr 19 21:10:24 kg-f5 smartd[2382]: Device: /dev/ada2, 136 Offline uncorrectable sectors Apr 19 23:40:24 kg-f5 smartd[2382]: Device: /dev/ada2, 144 Currently unreadable (pending) sectors (changed +8) Apr 19 23:40:24 kg-f5 smartd[2382]: Device: /dev/ada2, 144 Offline uncorrectable sectors (changed +8) ok 2017-04-19: ada4 error changes Apr 19 20:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 58496 Currently unreadable (pending) sectors (changed -6616) Apr 19 20:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 58496 Offline uncorrectable sectors (changed -6616) Apr 19 21:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 61440 Currently unreadable (pending) sectors (changed +2944) Apr 19 21:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 61440 Offline uncorrectable sectors (changed +2944) Apr 19 21:40:34 kg-f5 smartd[2382]: Device: /dev/ada4, 62504 Currently unreadable (pending) sectors (changed +1064) Apr 19 21:40:34 kg-f5 smartd[2382]: Device: /dev/ada4, 62504 Offline uncorrectable sectors (changed +1064) Apr 19 22:10:34 kg-f5 smartd[2382]: Device: /dev/ada4, 64816 Currently unreadable (pending) sectors (changed +2312) Apr 19 22:10:34 kg-f5 smartd[2382]: Device: /dev/ada4, 64816 Offline uncorrectable sectors (changed +2312) Apr 19 22:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 65064 Currently unreadable (pending) sectors (changed +248) Apr 19 22:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 65064 Offline uncorrectable sectors (changed +248) ok 2017-04-19: from /var/log/messages: Apr 19 20:31:30 kg-f5 kernel: ahcich4: Timeout on slot 14 port 0 Apr 19 20:31:30 kg-f5 kernel: ahcich4: is 00000000 cs 0000c000 ss 00000000 rs 0000c000 tfd c0 serr 00000000 cmd 0000ee17 Apr 19 20:32:02 kg-f5 kernel: ahcich4: AHCI reset: device not ready after 31000ms (tfd = 00000080) Apr 19 20:32:32 kg-f5 kernel: ahcich4: Timeout on slot 15 port 0 Apr 19 20:32:32 kg-f5 kernel: ahcich4: is 00000000 cs 00008000 ss 00000000 rs 00008000 tfd 00 serr 00000000 cmd 0000ef17 Apr 19 20:33:09 kg-f5 kernel: ahcich4: Timeout on slot 3 port 0 Apr 19 20:33:09 kg-f5 kernel: ahcich4: is 00000000 cs 00000018 ss 00000000 rs 00000018 tfd c0 serr 00000000 cmd 0000e317 Apr 19 20:34:09 kg-f5 kernel: ahcich4: Timeout on slot 7 port 0 Apr 19 20:34:09 kg-f5 kernel: ahcich4: is 00000000 cs 00000180 ss 00000000 rs 00000180 tfd c0 serr 00000000 cmd 0000e717 Apr 19 20:35:02 kg-f5 kernel: ahcich4: Timeout on slot 22 port 0 Apr 19 20:35:02 kg-f5 kernel: ahcich4: is 00000000 cs 00000000 ss 0fc3c000 rs 0fc3c000 tfd 40 serr 00000000 cmd 0000f117 Apr 19 20:35:33 kg-f5 kernel: ahcich4: AHCI reset: device not ready after 31000ms (tfd = 00000080) ok.
2017-04-06: pool status tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 15h11m with 0 errors on Sun Mar 12 15:11:51 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok. 2017-04-06: ada4 health: [root@kg-f5] ~# smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2017-04-05: messages about ada4 in /var/log/messages: Apr 5 00:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 440 Currently unreadable (pending) sectors Apr 5 00:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 440 Offline uncorrectable sectors Apr 5 00:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 3800 Currently unreadable (pending) sectors (changed +3360) Apr 5 00:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 3800 Offline uncorrectable sectors (changed +3360) Apr 5 01:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 7184 Currently unreadable (pending) sectors (changed +3384) Apr 5 01:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 7184 Offline uncorrectable sectors (changed +3384) Apr 5 01:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 10592 Currently unreadable (pending) sectors (changed +3408) Apr 5 01:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 10592 Offline uncorrectable sectors (changed +3408) Apr 5 02:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 13904 Currently unreadable (pending) sectors (changed +3312) Apr 5 02:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 13904 Offline uncorrectable sectors (changed +3312) Apr 5 02:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 17240 Currently unreadable (pending) sectors (changed +3336) Apr 5 02:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 17240 Offline uncorrectable sectors (changed +3336) Apr 5 03:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 20680 Currently unreadable (pending) sectors (changed +3440) Apr 5 03:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 20680 Offline uncorrectable sectors (changed +3440) Apr 5 03:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 23672 Currently unreadable (pending) sectors (changed +2992) Apr 5 03:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 23672 Offline uncorrectable sectors (changed +2992) Apr 5 04:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 26760 Currently unreadable (pending) sectors (changed +3088) Apr 5 04:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 26760 Offline uncorrectable sectors (changed +3088) Apr 5 04:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 30168 Currently unreadable (pending) sectors (changed +3408) Apr 5 04:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 30168 Offline uncorrectable sectors (changed +3408) Apr 5 05:10:32 kg-f5 smartd[2382]: Device: /dev/ada4, 33728 Currently unreadable (pending) sectors (changed +3560) Apr 5 05:10:32 kg-f5 smartd[2382]: Device: /dev/ada4, 33728 Offline uncorrectable sectors (changed +3560) Apr 5 05:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 36904 Currently unreadable (pending) sectors (changed +3176) Apr 5 05:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 36904 Offline uncorrectable sectors (changed +3176) Apr 5 06:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 40016 Currently unreadable (pending) sectors (changed +3112) Apr 5 06:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 40016 Offline uncorrectable sectors (changed +3112) Apr 5 06:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 43496 Currently unreadable (pending) sectors (changed +3480) Apr 5 06:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 43496 Offline uncorrectable sectors (changed +3480) Apr 5 07:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 46536 Currently unreadable (pending) sectors (changed +3040) Apr 5 07:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 46536 Offline uncorrectable sectors (changed +3040) Apr 5 07:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 49664 Currently unreadable (pending) sectors (changed +3128) Apr 5 07:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 49664 Offline uncorrectable sectors (changed +3128) Apr 5 08:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 52576 Currently unreadable (pending) sectors (changed +2912) Apr 5 08:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 52576 Offline uncorrectable sectors (changed +2912) Apr 5 08:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 55624 Currently unreadable (pending) sectors (changed +3048) Apr 5 08:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 55624 Offline uncorrectable sectors (changed +3048) Apr 5 09:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 58872 Currently unreadable (pending) sectors (changed +3248) Apr 5 09:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 58872 Offline uncorrectable sectors (changed +3248) Apr 5 09:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 62168 Currently unreadable (pending) sectors (changed +3296) Apr 5 09:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 62168 Offline uncorrectable sectors (changed +3296) Apr 5 10:10:26 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Currently unreadable (pending) sectors (changed -62144) Apr 5 10:10:26 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Offline uncorrectable sectors (changed -62144) Apr 5 10:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 2264 Currently unreadable (pending) sectors (changed +2240) Apr 5 10:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 2264 Offline uncorrectable sectors (changed +2240) Apr 5 11:10:24 kg-f5 smartd[2382]: Device: /dev/ada4, 5328 Currently unreadable (pending) sectors (changed +3064) Apr 5 11:10:24 kg-f5 smartd[2382]: Device: /dev/ada4, 5328 Offline uncorrectable sectors (changed +3064) Apr 5 11:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 8584 Currently unreadable (pending) sectors (changed +3256) Apr 5 11:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 8584 Offline uncorrectable sectors (changed +3256) Apr 5 12:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 12024 Currently unreadable (pending) sectors (changed +3440) Apr 5 12:10:29 kg-f5 smartd[2382]: Device: /dev/ada4, 12024 Offline uncorrectable sectors (changed +3440) Apr 5 12:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 14392 Currently unreadable (pending) sectors (changed +2368) Apr 5 12:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 14392 Offline uncorrectable sectors (changed +2368) Apr 5 13:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 17384 Currently unreadable (pending) sectors (changed +2992) Apr 5 13:10:25 kg-f5 smartd[2382]: Device: /dev/ada4, 17384 Offline uncorrectable sectors (changed +2992) Apr 5 13:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 19936 Currently unreadable (pending) sectors (changed +2552) Apr 5 13:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 19936 Offline uncorrectable sectors (changed +2552) Apr 5 14:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 23088 Currently unreadable (pending) sectors (changed +3152) Apr 5 14:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 23088 Offline uncorrectable sectors (changed +3152) Apr 5 14:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 26112 Currently unreadable (pending) sectors (changed +3024) Apr 5 14:40:25 kg-f5 smartd[2382]: Device: /dev/ada4, 26112 Offline uncorrectable sectors (changed +3024) Apr 5 15:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 28336 Currently unreadable (pending) sectors (changed +2224) Apr 5 15:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 28336 Offline uncorrectable sectors (changed +2224) Apr 5 15:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 31568 Currently unreadable (pending) sectors (changed +3232) Apr 5 15:40:30 kg-f5 smartd[2382]: Device: /dev/ada4, 31568 Offline uncorrectable sectors (changed +3232) Apr 5 16:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 34720 Currently unreadable (pending) sectors (changed +3152) Apr 5 16:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 34720 Offline uncorrectable sectors (changed +3152) Apr 5 16:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 37056 Currently unreadable (pending) sectors (changed +2336) Apr 5 16:40:31 kg-f5 smartd[2382]: Device: /dev/ada4, 37056 Offline uncorrectable sectors (changed +2336) Apr 5 17:10:27 kg-f5 smartd[2382]: Device: /dev/ada4, 40400 Currently unreadable (pending) sectors (changed +3344) Apr 5 17:10:27 kg-f5 smartd[2382]: Device: /dev/ada4, 40400 Offline uncorrectable sectors (changed +3344) Apr 5 17:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 43608 Currently unreadable (pending) sectors (changed +3208) Apr 5 17:40:28 kg-f5 smartd[2382]: Device: /dev/ada4, 43608 Offline uncorrectable sectors (changed +3208) Apr 5 18:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 46616 Currently unreadable (pending) sectors (changed +3008) Apr 5 18:10:28 kg-f5 smartd[2382]: Device: /dev/ada4, 46616 Offline uncorrectable sectors (changed +3008) Apr 5 18:40:32 kg-f5 smartd[2382]: Device: /dev/ada4, 48336 Currently unreadable (pending) sectors (changed +1720) Apr 5 18:40:32 kg-f5 smartd[2382]: Device: /dev/ada4, 48336 Offline uncorrectable sectors (changed +1720) Apr 5 19:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 51280 Currently unreadable (pending) sectors (changed +2944) Apr 5 19:10:31 kg-f5 smartd[2382]: Device: /dev/ada4, 51280 Offline uncorrectable sectors (changed +2944) Apr 5 19:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 53976 Currently unreadable (pending) sectors (changed +2696) Apr 5 19:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 53976 Offline uncorrectable sectors (changed +2696) and it continues Apr 5 22:10:26 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors (changed +1048) Apr 5 22:40:26 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Currently unreadable (pending) sectors Apr 5 22:40:26 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors Apr 5 23:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Currently unreadable (pending) sectors Apr 5 23:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors Apr 5 23:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Currently unreadable (pending) sectors Apr 5 23:40:29 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors Apr 6 00:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Currently unreadable (pending) sectors Apr 6 00:10:30 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors Apr 6 00:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Currently unreadable (pending) sectors Apr 6 00:40:27 kg-f5 smartd[2382]: Device: /dev/ada4, 65112 Offline uncorrectable sectors better check the health. 2017-03-12: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 15h11m with 0 errors on Sun Mar 12 15:11:51 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2017-01-29: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 14h57m with 0 errors on Sun Jan 29 14:57:24 2017 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-12-18: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 14h56m with 0 errors on Sun Dec 18 14:56:47 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-11-06: latest scrub result tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 14h56m with 0 errors on Sun Nov 6 14:56:57 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-10-02: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 14h19m with 0 errors on Sun Oct 2 14:19:46 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-08-25: uptime status: tingo@kg-f5$ date;uptime Thu Aug 25 22:19:30 CEST 2016 10:19PM up 245 days, 4:19, 3 users, load averages: 0.02, 0.02, 0.00 ok.
2016-08-21: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 12h30m with 0 errors on Sun Aug 21 12:30:13 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-07-10: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 11h54m with 0 errors on Sun Jul 10 11:54:32 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-05-29: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 8h42m with 0 errors on Sun May 29 08:42:53 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-04-17: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 7h32m with 0 errors on Sun Apr 17 07:32:42 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2016-03-06: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 7h31m with 0 errors on Sun Mar 6 07:31:49 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors good.
2016-01-24: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 6h27m with 0 errors on Sun Jan 24 06:27:45 2016 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2015-12-24: replacing ada0. After physical replacing ada0, I log into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5), I see the missing drive, it is labeled "6048661200277659165" (copied from 'zpool status' in a shell). I select "Replace", pick the new ada0 and click "Replace Disk". It chews on that for a few seconds, the reloads the volume status screen. "6048661200277659165" is still there, so I do a Detach on it. zpool status now: tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu Dec 24 17:12:34 2015 30.0G scanned out of 3.49T at 200M/s, 5h2m to go 5.00G resilvered, 0.84% done config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0 (resilvering)
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors good. A few hours later it is finished: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 595G in 4h44m with 0 errors on Thu Dec 24 21:57:15 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/234c4d71-aa59-11e5-b934-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors great. 2015-12-24: zpool status now: tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 2.78M in 5h21m with 0 errors on Sun Dec 13 05:21:07 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 3 1.58K 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok 2015-12-24: more errors, and the ada0 device gets lost: Dec 24 13:52:23 kg-f5 kernel: ahcich0: Timeout on slot 7 port 0 Dec 24 13:52:23 kg-f5 kernel: ahcich0: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd c0 serr 00000000 cmd 0000e717 Dec 24 13:52:54 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:53:24 kg-f5 kernel: ahcich0: Timeout on slot 7 port 0 Dec 24 13:53:24 kg-f5 kernel: ahcich0: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd 00 serr 00000000 cmd 0000e717 Dec 24 13:53:55 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:54:41 kg-f5 kernel: ahcich0: Timeout on slot 21 port 0 Dec 24 13:54:41 kg-f5 kernel: ahcich0: is 00000000 cs 00600000 ss 00000000 rs 00600000 tfd c0 serr 00000000 cmd 0000f517 Dec 24 13:55:12 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:55:42 kg-f5 kernel: ahcich0: Timeout on slot 22 port 0 Dec 24 13:55:42 kg-f5 kernel: ahcich0: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd 00 serr 00000000 cmd 0000f617 Dec 24 13:56:41 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 24 13:56:41 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd c0 serr 00000000 cmd 0000e417 Dec 24 13:57:12 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:57:42 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 24 13:57:42 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 00 serr 00000000 cmd 0000e417 Dec 24 13:58:14 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:58:44 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 24 13:58:44 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 00 serr 00000000 cmd 0000e417 Dec 24 13:58:44 kg-f5 kernel: (ada0:ahcich0:0:0:0): lost device Dec 24 13:59:15 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 13:59:45 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 24 13:59:45 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 00 serr 00000000 cmd 0000e417 Dec 24 14:00:35 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 14:00:35 kg-f5 kernel: ahcich0: Poll timeout on slot 4 port 0 Dec 24 14:00:35 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 00 serr 00000000 cmd 0000e417 Dec 24 14:01:05 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 24 14:01:05 kg-f5 kernel: ahcich0: is 00000000 cs 00000070 ss 00000070 rs 00000070 tfd 00 serr 00000000 cmd 0000e417 Dec 24 14:01:54 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 24 14:01:54 kg-f5 kernel: ahcich0: Poll timeout on slot 6 port 0 Dec 24 14:01:54 kg-f5 kernel: ahcich0: is 00000000 cs 00000040 ss 00000000 rs 00000040 tfd 00 serr 00000000 cmd 0000e617 verify tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 98 Apr 25 2015 /dev/ada1 crw-r----- 1 root operator 0, 106 Apr 25 2015 /dev/ada1p1 crw-r----- 1 root operator 0, 107 Apr 25 2015 /dev/ada1p2 crw-r----- 1 root operator 0, 99 Apr 25 2015 /dev/ada2 crw-r----- 1 root operator 0, 108 Apr 25 2015 /dev/ada2p1 crw-r----- 1 root operator 0, 109 Apr 25 2015 /dev/ada2p2 crw-r----- 1 root operator 0, 100 Apr 25 2015 /dev/ada3 crw-r----- 1 root operator 0, 110 Apr 25 2015 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Apr 25 2015 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Apr 25 2015 /dev/ada4 crw-r----- 1 root operator 0, 112 Apr 25 2015 /dev/ada4p1 crw-r----- 1 root operator 0, 113 Apr 25 2015 /dev/ada4p2 crw-r----- 1 root operator 0, 102 Apr 25 2015 /dev/ada5 crw-r----- 1 root operator 0, 114 Apr 25 2015 /dev/ada5p1 crw-r----- 1 root operator 0, 115 Apr 25 2015 /dev/ada5p2 tingo@kg-f5$ swapinfo -h Device 1K-blocks Used Avail Capacity /dev/#C:0:0x60 2097152 0B 2.0G 0% /dev/ada1p1 2097152 0B 2.0G 0% /dev/ada2p1 2097152 0B 2.0G 0% /dev/ada3p1 2097152 0B 2.0G 0% /dev/ada4p1 2097152 0B 2.0G 0% /dev/ada5p1 2097152 0B 2.0G 0% Total 12582912 0B 12G 0% Yes, ada0 is gone. Time to replace it.
2015-12-24: hard drive ada0 error changes: Dec 24 02:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 1816 Currently unreadable (pending) sectors (changed +8) Dec 24 02:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 1816 Offline uncorrectable sectors (changed +8) ok. More changes: Dec 24 13:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1808 Currently unreadable (pending) sectors (changed -8) Dec 24 13:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1808 Offline uncorrectable sectors (changed -8) ok. 2015-12-24: timeout messages: Dec 24 01:46:34 kg-f5 kernel: ahcich0: Timeout on slot 31 port 0 Dec 24 01:46:34 kg-f5 kernel: ahcich0: is 00000000 cs 80000000 ss 00000000 rs 80000000 tfd d0 serr 00000000 cmd 0000ff17 that's all.
2015-12-22: hard drive ada0 health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2015-12-22: hard drive ada0 error changes: Dec 22 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1808 Currently unreadable (pending) sectors (changed +184) Dec 22 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1808 Offline uncorrectable sectors (changed +184) ok.
2015-12-21: hard drive ada0 error changes: Dec 21 01:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 1624 Currently unreadable (pending) sectors (changed +512) Dec 21 01:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 1624 Offline uncorrectable sectors (changed +512) nothing more. 2015-12-21: hard drive ada0 health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2015-12-21: hard drive ada0 error changes: Dec 21 00:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1112 Currently unreadable (pending) sectors (changed +264) Dec 21 00:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 1112 Offline uncorrectable sectors (changed +264) and (possibly related) timeout messages: Dec 21 00:41:49 kg-f5 kernel: ahcich0: Timeout on slot 3 port 0 Dec 21 00:41:49 kg-f5 kernel: ahcich0: is 00000000 cs 00000018 ss 00000000 rs 00000018 tfd c0 serr 00000000 cmd 0000e317 Dec 21 00:42:23 kg-f5 kernel: ahcich0: Timeout on slot 4 port 0 Dec 21 00:42:23 kg-f5 kernel: ahcich0: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 1d0 serr 00000000 cmd 0000e417
Dec 21 00:55:18 kg-f5 kernel: ahcich0: Timeout on slot 27 port 0 Dec 21 00:55:18 kg-f5 kernel: ahcich0: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd c0 serr 00000000 cmd 0000fb17 Dec 21 00:56:25 kg-f5 kernel: ahcich0: Timeout on slot 27 port 0 Dec 21 00:56:25 kg-f5 kernel: ahcich0: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd d0 serr 00000000 cmd 0000fb17 Dec 21 00:56:56 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Dec 21 00:58:33 kg-f5 kernel: ahcich0: Timeout on slot 18 port 0 Dec 21 00:58:33 kg-f5 kernel: ahcich0: is 00000000 cs 000c0000 ss 00000000 rs 000c0000 tfd c0 serr 00000000 cmd 0000f217 Dec 21 00:59:04 kg-f5 kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) is the device there? tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Apr 25 2015 /dev/ada0 crw-r----- 1 root operator 0, 96 Apr 25 2015 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Apr 25 2015 /dev/ada0p2 crw-r----- 1 root operator 0, 98 Apr 25 2015 /dev/ada1 crw-r----- 1 root operator 0, 106 Apr 25 2015 /dev/ada1p1 crw-r----- 1 root operator 0, 107 Apr 25 2015 /dev/ada1p2 crw-r----- 1 root operator 0, 99 Apr 25 2015 /dev/ada2 crw-r----- 1 root operator 0, 108 Apr 25 2015 /dev/ada2p1 crw-r----- 1 root operator 0, 109 Apr 25 2015 /dev/ada2p2 crw-r----- 1 root operator 0, 100 Apr 25 2015 /dev/ada3 crw-r----- 1 root operator 0, 110 Apr 25 2015 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Apr 25 2015 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Apr 25 2015 /dev/ada4 crw-r----- 1 root operator 0, 112 Apr 25 2015 /dev/ada4p1 crw-r----- 1 root operator 0, 113 Apr 25 2015 /dev/ada4p2 crw-r----- 1 root operator 0, 102 Apr 25 2015 /dev/ada5 crw-r----- 1 root operator 0, 114 Apr 25 2015 /dev/ada5p1 crw-r----- 1 root operator 0, 115 Apr 25 2015 /dev/ada5p2 all devices present.
2015-12-13: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.78M in 5h21m with 0 errors on Sun Dec 13 05:21:07 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2015-11-14: change to new firewall noted in /var/log/messages: Nov 14 13:48:26 kg-f5 kernel: arp: 10.1.10.1 moved from 00:10:4b:e3:38:8b to 80:ee:73:60:61:0c on re0 nothing more. 2015-11-14: hard drive ada0 health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok.
2015-11-11: hard drive ada0 error changes: Nov 11 19:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 848 Currently unreadable (pending) sectors (changed +8) Nov 11 19:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 848 Offline uncorrectable sectors (changed +8) ok.
2015-11-01: hard drive ada0 health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok 2015-11-01: hard drive ada0 error changes: Nov 1 00:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 648 Currently unreadable (pending) sectors (changed -272) Nov 1 00:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 648 Offline uncorrectable sectors (changed -272)
Nov 1 01:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 784 Currently unreadable (pending) sectors (changed +136) Nov 1 01:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 784 Offline uncorrectable sectors (changed +136)
Nov 1 04:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 760 Currently unreadable (pending) sectors (changed -24) Nov 1 04:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 760 Offline uncorrectable sectors (changed -24) Nov 1 04:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, ATA error count increased from 0 to 148
Nov 1 05:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 840 Currently unreadable (pending) sectors (changed +80) Nov 1 05:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 840 Offline uncorrectable sectors (changed +80) ok. 2015-11-01: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.66M in 5h24m with 0 errors on Sun Nov 1 05:24:48 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2015-10-20: hard drive ada0 error changes: Oct 20 02:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 920 Currently unreadable (pending) sectors (changed +56) Oct 20 02:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 920 Offline uncorrectable sectors (changed +56) ada0 health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED noted.
2015-10-17: hard drive ada0 error changes: Oct 17 19:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 864 Currently unreadable (pending) sectors (changed +32) Oct 17 19:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 864 Offline uncorrectable sectors (changed +32) noted. 2015-10-17: hard drive health, ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok.
2015-10-16: hard drive ada0 error changes: Oct 16 20:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 832 Currently unreadable (pending) sectors (changed +88) Oct 16 20:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 832 Offline uncorrectable sectors (changed +88) noted.
2015-10-05:hard drive ada0 error changes: Oct 5 01:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 744 Currently unreadable (pending) sectors (changed -24) Oct 5 01:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 744 Offline uncorrectable sectors (changed -24) it goes up and down. drive health, ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok.
2015-10-04:hard drive ada0 error changes: Oct 4 15:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 768 Currently unreadable (pending) sectors (changed +24) Oct 4 15:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 768 Offline uncorrectable sectors (changed +24) ok.
2015-10-01: ada0 drive health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED nothing more.
2015-09-30: hard drive ada0 error changes: Sep 30 19:46:17 kg-f5 smartd[2382]: Device: /dev/ada0, 744 Currently unreadable (pending) sectors (changed +16) Sep 30 19:46:17 kg-f5 smartd[2382]: Device: /dev/ada0, 744 Offline uncorrectable sectors (changed +16) that's it.
2015-09-29: hard drive ada0 error changes: Sep 29 23:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 336 Currently unreadable (pending) sectors (changed +72) Sep 29 23:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 336 Offline uncorrectable sectors (changed +72) Sep 29 23:46:20 kg-f5 smartd[2382]: Device: /dev/ada0, 728 Currently unreadable (pending) sectors (changed +392) Sep 29 23:46:20 kg-f5 smartd[2382]: Device: /dev/ada0, 728 Offline uncorrectable sectors (changed +392) large increase there.
2015-09-28: hard drive ada0 error changes: Sep 28 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 264 Currently unreadable (pending) sectors (changed +16) Sep 28 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 264 Offline uncorrectable sectors (changed +16) smartctl health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok.
2015-09-27: hard drive ada0 error changes: Sep 27 11:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 248 Currently unreadable (pending) sectors (changed +40) Sep 27 11:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 248 Offline uncorrectable sectors (changed +40) smartctl health: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ok. 2015-09-27: latest scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 5h3m with 0 errors on Sun Sep 27 05:03:52 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors automatic scrub (via cron) works now. 2015-09-27: from /var/log/messages: Sep 27 01:47:43 kg-f5 kernel: re0: watchdog timeout Sep 27 01:47:43 kg-f5 kernel: re0: link state changed to DOWN Sep 27 01:47:47 kg-f5 kernel: re0: link state changed to UP nothing more.
2015-09-20: the entry in /etc/crontab for scrub looks like this:
minute hour mday month wday who command
00 00 * 1,2,3,4,5,6,7,8,9,a,b,c 7 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 z5 it doesn't look correct to me. The month specification should be just a star. Fixed from the GUI; Storage, ZFS Scrubs, find the one for z5 and save it by pressing the "ok" button. Afterwards the crontab entry looks much better:
minute hour mday month wday who command
00 00 * * 7 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 z5 now I will see if the automatic scrub runs.
2015-09-06: pool status: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 2h48m with 0 errors on Wed Aug 5 11:38:20 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok.
2015-09-05: hard drive ada0 error changes: Sep 5 10:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 208 Currently unreadable (pending) sectors (changed +152) Sep 5 10:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 208 Offline uncorrectable sectors (changed +152) smartctl status for ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED nothing more.
2015-09-01: check last scrub time: tingo@kg-f5$ su - Password: [root@kg-f5] ~# date Tue Sep 1 16:13:24 CEST 2015 [root@kg-f5] ~# scrub -t 35 z5 skipping scrubbing of pool 'z5': last scrubbing is 27 days ago, threshold is set to 35 days ok.
2015-08-07: pool status: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 2h48m with 0 errors on Wed Aug 5 11:38:20 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
nothing more. 2015-08-07: smartctl status for ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED good.
2015-08-05: hard drive ada0 error changes: Aug 5 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 56 Currently unreadable (pending) sectors (changed +16) Aug 5 22:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 56 Offline uncorrectable sectors (changed +16) that's it. 2015-08-05: latest (manual) scrub result: tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 2h48m with 0 errors on Wed Aug 5 11:38:20 2015 config:
NAME STATE READ WRITE CKSUM
z5 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0
gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0
errors: No known data errors ok. check root@kg-f5] ~# sh
zpool history z5 | egrep "^[0-9.\:-]{19} zpool scrub z5\$" | tail -1 |cut -d ' ' -f 1
2015-08-05.08:50:01 ok. 2015-08-05: for some reason, it seems that automatic scrubs doesn't run.
zpool history z5 | egrep "^[0-9.\:-]{19} zpool scrub z5\$" | tail -1 |cut -d ' ' -f 1
2015-06-02.16:25:40
scrub
usage: /usr/local/sbin/scrub [
scrub z5
starting scrub of pool 'z5' ok.
2015-07-01: smartctl status for ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED that's it.
2015-06-28: hard drive ada0 error changes: Jun 28 18:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 40 Currently unreadable (pending) sectors (changed +8) Jun 28 18:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 40 Offline uncorrectable sectors (changed +8) Jun 28 19:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 40 Currently unreadable (pending) sectors nothing more.
2015-05-29: hard drive ada0 error changes: May 29 17:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 32 Currently unreadable (pending) sectors (changed +8) May 29 17:46:14 kg-f5 smartd[2382]: Device: /dev/ada0, 32 Offline uncorrectable sectors (changed +8) May 29 18:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 32 Currently unreadable (pending) sectors May 29 18:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 32 Offline uncorrectable sectors smartctl status of ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED nothing more.
2015-05-12: smartctl status of ada0: [root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED that was all.
2015-05-10: hard drive ada0 error changes: May 10 22:16:15 kg-f5 smartd[2382]: Device: /dev/ada0, 8 Offline uncorrectable sectors May 10 22:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 24 Currently unreadable (pending) sectors (changed +16) May 10 22:46:15 kg-f5 smartd[2382]: Device: /dev/ada0, 24 Offline uncorrectable sectors (changed +16) May 10 23:16:14 kg-f5 smartd[2382]: Device: /dev/ada0, 24 Currently unreadable (pending) sectors nothing more.
2015-04-06: resilver - almost finished:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 6 12:33:04 2015 678G scanned out of 695G at 211M/s, 0h1m to go 113G resilvered, 97.57% done config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
and finished
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: resilvered 116G in 0h56m with 0 errors on Mon Apr 6 13:29:11 2015 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0 gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
good.
2015-04-06: after physically replacing ada3, I log into the gui again, go to Storage, Volumes, View Volumes, Volume Status (for z5). I see the missing drive, it is labeled "12213007410532765536" (instead of ada3), I select "Replace", pick the new ada3 and click "Replace Disk". After that is done, "12213007410532765536" is still hanging around, so I do a "Detach" on it. Now the status is:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Apr 6 12:33:04 2015 29.6G scanned out of 695G at 204M/s, 0h55m to go 4.93G resilvered, 4.27% done config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/4d88b8f2-dc48-11e4-911f-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
good.
2015-04-02: ada3 lost:
Apr 2 00:22:17 kg-f5 kernel: ahcich3: Timeout on slot 2 port 0 Apr 2 00:22:17 kg-f5 kernel: ahcich3: is 00000000 cs 00000004 ss 00000000 rs 00000004 tfd c0 serr 00000000 cmd 0000e217 Apr 2 00:22:49 kg-f5 kernel: ahcich3: AHCI reset: device not ready after 31000ms (tfd = 00000080) Apr 2 00:23:19 kg-f5 kernel: ahcich3: Timeout on slot 2 port 0 Apr 2 00:23:19 kg-f5 kernel: ahcich3: is 00000000 cs 00000004 ss 00000000 rs 00000004 tfd 00 serr 00000000 cmd 0000e217 Apr 2 00:23:50 kg-f5 kernel: ahcich3: AHCI reset: device not ready after 31000ms (tfd = 00000080) Apr 2 00:24:20 kg-f5 kernel: ahcich3: Timeout on slot 2 port 0 Apr 2 00:24:20 kg-f5 kernel: ahcich3: is 00000000 cs 00000004 ss 00000000 rs 00000004 tfd 00 serr 00000000 cmd 0000e217 Apr 2 00:24:20 kg-f5 kernel: (ada3:ahcich3:0:0:0): lost device
zpool status
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 0 in 0h14m with 0 errors on Fri Feb 13 17:02:50 2015 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/0171e4aa-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 4 78.4K 0 gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
now I know.
2015-02-17: hard drive ada3 error changes:
Feb 17 04:41:38 kg-f5 smartd[2382]: Device: /dev/ada3, 176 Currently unreadable (pending) sectors (changed +152) Feb 17 04:41:38 kg-f5 smartd[2382]: Device: /dev/ada3, 176 Offline uncorrectable sectors (changed +152)
smartctl status:
[root@kg-f5] ~# smartctl -H /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 058 044 045 Old_age Always In_the_past 42 (1 46 43 30 0)
ok.
2015-02-13: hard drive ada0 errors:
Feb 13 19:28:16 kg-f5 smartd[2383]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 13 19:28:16 kg-f5 smartd[2383]: Device: /dev/ada0, 8 Offline uncorrectable sectors
smartctl status:
[root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
ok.
2015-02-13: result of a manual scrub:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 0h14m with 0 errors on Fri Feb 13 17:02:50 2015 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/0171e4aa-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
that's all.
2015-02-13: hard drive ada3 error changes:
Feb 13 16:28:16 kg-f5 smartd[2383]: Device: /dev/ada3, 24 Currently unreadable (pending) sectors Feb 13 16:28:16 kg-f5 smartd[2383]: Device: /dev/ada3, 24 Offline uncorrectable sectors Feb 13 16:58:16 kg-f5 smartd[2383]: Device: /dev/ada3, 65376 Currently unreadable (pending) sectors (changed +65352) Feb 13 16:58:16 kg-f5 smartd[2383]: Device: /dev/ada3, 65376 Offline uncorrectable sectors (changed +65352) Feb 13 17:28:16 kg-f5 smartd[2383]: Device: /dev/ada3, 24 Currently unreadable (pending) sectors (changed -65352) Feb 13 17:28:16 kg-f5 smartd[2383]: Device: /dev/ada3, 24 Offline uncorrectable sectors (changed -65352)
what does smartctl say?
[root@kg-f5] ~# smartctl -H /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 057 044 045 Old_age Always In_the_past 43 (1 46 48 31 0)
hmm, passed.
2015-01-25: create dataset - I created two more datasets: media, xxx, with tingo:users for ownership, and rwxr-x--- for permissions. It looks like this:
tingo@kg-f5$ zfs list NAME USED AVAIL REFER MOUNTPOINT z5 2.15M 10.7T 336K /mnt/z5 z5/h-tingo 336K 10.7T 336K /mnt/z5/h-tingo z5/media 288K 10.7T 288K /mnt/z5/media z5/xxx 288K 10.7T 288K /mnt/z5/xxx
mounted filesystems:
tingo@kg-f5$ df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/FreeNASs1a 926M 383M 469M 45% / devfs 1.0k 1.0k 0B 100% /dev /dev/md0 4.6M 3.2M 979k 77% /etc /dev/md1 823k 2.0k 756k 0% /mnt /dev/md2 149M 15M 122M 11% /var /dev/ufs/FreeNASs4 19M 1.3M 17M 7% /data z5 10T 335k 10T 0% /mnt/z5 z5/h-tingo 10T 335k 10T 0% /mnt/z5/h-tingo z5/media 10T 287k 10T 0% /mnt/z5/media z5/xxx 10T 287k 10T 0% /mnt/z5/xxx
that's all.
2015-01-25: create dataset - from Volumes, /mnt/z5, I select Create ZFS Dataset, and named it h-tingo. (It is still set as the home directory for the user tingo). I the changed ownership (tingo:users) and permissions rwxr-x--- (unix type ACL) on it.
2015-01-24: the new pool looks like this:
[root@kg-f5] ~# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT z5 16.2T 1.41M 16.2T 0% 1.00x ONLINE /mnt [root@kg-f5] ~# zpool status pool: z5 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/fd21b55b-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fdcef5f4-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/fe7dada2-a41b-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/0171e4aa-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/0215485c-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 gptid/02cae973-a41c-11e4-a84c-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
and is mounted
[root@kg-f5] ~# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/FreeNASs1a 926M 383M 469M 45% / devfs 1.0k 1.0k 0B 100% /dev /dev/md0 4.6M 3.2M 979k 77% /etc /dev/md1 823k 2.0k 756k 0% /mnt /dev/md2 149M 14M 123M 10% /var /dev/ufs/FreeNASs4 19M 1.3M 17M 7% /data z5 10T 288k 10T 0% /mnt/z5
how about datasets?
[root@kg-f5] ~# zfs list NAME USED AVAIL REFER MOUNTPOINT z5 959K 10.7T 288K /mnt/z5
only one.
2015-01-24: create volume. From Volume Manager, I name my new volume "z5", select ada0 - ada5 (all disks), zfs, force 4096 bytes sector size, leave dedup "off" and select RAID-Z2 as "Group type". I click on "Add volume", it churns a while and is ready. From /var/log/messages:
Jan 24 23:54:43 kg-f5 notifier: swapoff: removing /dev/ada0p1 as swap device Jan 24 23:54:43 kg-f5 notifier: swapoff: removing /dev/ada1p1 as swap device Jan 24 23:54:43 kg-f5 notifier: swapoff: removing /dev/ada3p1 as swap device Jan 24 23:54:43 kg-f5 notifier: swapoff: removing /dev/ada4p1 as swap device Jan 24 23:54:43 kg-f5 notifier: 1+0 records in Jan 24 23:54:43 kg-f5 notifier: 1+0 records out Jan 24 23:54:43 kg-f5 notifier: 1048576 bytes transferred in 0.006900 secs (151971199 bytes/sec) Jan 24 23:54:43 kg-f5 notifier: dd: /dev/ada0: short write on character device Jan 24 23:54:43 kg-f5 notifier: dd: /dev/ada0: end of device Jan 24 23:54:43 kg-f5 notifier: 5+0 records in Jan 24 23:54:43 kg-f5 notifier: 4+1 records out Jan 24 23:54:43 kg-f5 notifier: 4677632 bytes transferred in 0.084738 secs (55201259 bytes/sec) Jan 24 23:54:45 kg-f5 notifier: 1+0 records in Jan 24 23:54:45 kg-f5 notifier: 1+0 records out Jan 24 23:54:45 kg-f5 notifier: 1048576 bytes transferred in 0.006399 secs (163861643 bytes/sec) Jan 24 23:54:45 kg-f5 notifier: dd: /dev/ada1: short write on character device Jan 24 23:54:45 kg-f5 notifier: dd: /dev/ada1: end of device Jan 24 23:54:45 kg-f5 notifier: 5+0 records in Jan 24 23:54:45 kg-f5 notifier: 4+1 records out Jan 24 23:54:45 kg-f5 notifier: 4677632 bytes transferred in 0.024270 secs (192732628 bytes/sec) Jan 24 23:54:46 kg-f5 notifier: 1+0 records in Jan 24 23:54:46 kg-f5 notifier: 1+0 records out Jan 24 23:54:46 kg-f5 notifier: 1048576 bytes transferred in 0.006383 secs (164271711 bytes/sec) Jan 24 23:54:46 kg-f5 notifier: dd: /dev/ada2: short write on character device Jan 24 23:54:46 kg-f5 notifier: dd: /dev/ada2: end of device Jan 24 23:54:46 kg-f5 notifier: 5+0 records in Jan 24 23:54:46 kg-f5 notifier: 4+1 records out Jan 24 23:54:46 kg-f5 notifier: 4677632 bytes transferred in 0.024309 secs (192424510 bytes/sec) Jan 24 23:54:47 kg-f5 notifier: 1+0 records in Jan 24 23:54:47 kg-f5 notifier: 1+0 records out Jan 24 23:54:47 kg-f5 notifier: 1048576 bytes transferred in 0.006090 secs (172175325 bytes/sec) Jan 24 23:54:47 kg-f5 notifier: dd: /dev/ada3: short write on character device Jan 24 23:54:47 kg-f5 notifier: dd: /dev/ada3: end of device Jan 24 23:54:47 kg-f5 notifier: 5+0 records in Jan 24 23:54:47 kg-f5 notifier: 4+1 records out Jan 24 23:54:47 kg-f5 notifier: 4677632 bytes transferred in 0.098756 secs (47365511 bytes/sec) Jan 24 23:54:52 kg-f5 notifier: 1+0 records in Jan 24 23:54:52 kg-f5 notifier: 1+0 records out Jan 24 23:54:52 kg-f5 notifier: 1048576 bytes transferred in 0.006370 secs (164609870 bytes/sec) Jan 24 23:54:52 kg-f5 notifier: dd: /dev/ada4: short write on character device Jan 24 23:54:52 kg-f5 notifier: dd: /dev/ada4: end of device Jan 24 23:54:52 kg-f5 notifier: 5+0 records in Jan 24 23:54:52 kg-f5 notifier: 4+1 records out Jan 24 23:54:52 kg-f5 notifier: 4677632 bytes transferred in 0.024653 secs (189739179 bytes/sec) Jan 24 23:54:53 kg-f5 notifier: 1+0 records in Jan 24 23:54:53 kg-f5 notifier: 1+0 records out Jan 24 23:54:53 kg-f5 notifier: 1048576 bytes transferred in 0.006284 secs (166864458 bytes/sec) Jan 24 23:54:53 kg-f5 notifier: dd: /dev/ada5: short write on character device Jan 24 23:54:53 kg-f5 notifier: dd: /dev/ada5: end of device Jan 24 23:54:53 kg-f5 notifier: 5+0 records in Jan 24 23:54:53 kg-f5 notifier: 4+1 records out Jan 24 23:54:53 kg-f5 notifier: 4677632 bytes transferred in 0.024366 secs (191972628 bytes/sec) Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada0p2.nop created. Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada1p2.nop created. Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada2p2.nop created. Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada3p2.nop created. Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada4p2.nop created. Jan 24 23:54:54 kg-f5 kernel: GEOM_NOP: Device ada5p2.nop created. Jan 24 23:54:58 kg-f5 kernel: GEOM_NOP: Device ada0p2.nop removed. Jan 24 23:54:58 kg-f5 kernel: GEOM_NOP: Device ada1p2.nop removed. Jan 24 23:54:59 kg-f5 kernel: GEOM_NOP: Device ada2p2.nop removed. Jan 24 23:54:59 kg-f5 kernel: GEOM_NOP: Device ada3p2.nop removed. Jan 24 23:54:59 kg-f5 kernel: GEOM_NOP: Device ada4p2.nop removed. Jan 24 23:54:59 kg-f5 kernel: GEOM_NOP: Device ada5p2.nop removed.
nothing more here.
2015-01-24: destroying the volume. From the gui, Storage, Volumes, View Volumes, I select the Detach Volume button, and check the "Mark the disks as new (destroy data)" checkbox. I confirm, and a while later the volume is gone.
2015-01-24: after replacing both ada2 and ad5 ad rebooting the FreeNAS box, I can see this:
[root@kg-f5] ~# zpool status no pools available [root@kg-f5] ~# zpool list no pools available [root@kg-f5] ~# zfs list no datasets available
which is expected.
[root@kg-f5] ~# zpool import pool: z5 id: 5295825421709762899 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-3C config: z5 UNAVAIL insufficient replicas raidz1-0 UNAVAIL insufficient replicas gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE replacing-1 DEGRADED 4866735489841404330 OFFLINE gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 13537984888612930332 UNAVAIL cannot open gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE replacing-4 DEGRADED 13212241896674284510 UNAVAIL cannot open gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 15720556259919554590 UNAVAIL cannot open
so I can see the broken pool with zpool import. Well, doesn't matter I'm going to destroy it anyway.
2015-01-24: status of the scrub now:
tingo@kg-f5$ date Sat Jan 24 13:57:17 CET 2015 tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scan: scrub in progress since Sun Jan 18 00:00:25 2015 144G scanned out of 9.00T at 266K/s, (scan is slow, no estimated time) 0 repaired, 1.57% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 33 11 235M raidz1-0 DEGRADED 65 24 470M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 13537984888612930332 UNAVAIL 3 87.6M 0 was /dev/gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 77 1.58K 0 errors: 48334750 data errors, use '-v' for a list
Perhaps the best course of action would be to replace the faulted drives, let the scrub finish and then rebuild / recreate the pool.
tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Sep 17 18:21 /dev/ada0 crw-r----- 1 root operator 0, 96 Sep 17 18:21 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Sep 17 18:21 /dev/ada0p2 crw-r----- 1 root operator 0, 98 Sep 17 18:21 /dev/ada1 crw-r----- 1 root operator 0, 106 Sep 17 18:21 /dev/ada1p1 crw-r----- 1 root operator 0, 107 Sep 17 18:21 /dev/ada1p2 crw-r----- 1 root operator 0, 100 Sep 17 18:21 /dev/ada3 crw-r----- 1 root operator 0, 110 Sep 17 18:21 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Sep 17 18:21 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Sep 17 18:28 /dev/ada4 crw-r----- 1 root operator 0, 122 Sep 17 18:28 /dev/ada4p1 crw-r----- 1 root operator 0, 127 Sep 17 18:28 /dev/ada4p2
still ada2 and ada5.
2015-01-18: another automatic scrub started:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scan: scrub in progress since Sun Jan 18 00:00:25 2015 144G scanned out of 9.00T at 3.38M/s, (scan is slow, no estimated time) 0 repaired, 1.57% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 33 11 235M raidz1-0 DEGRADED 65 24 470M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 13537984888612930332 UNAVAIL 3 87.6M 0 was /dev/gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 77 1.58K 0 errors: 48334750 data errors, use '-v' for a list
listing the disks:
tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Sep 17 18:21 /dev/ada0 crw-r----- 1 root operator 0, 96 Sep 17 18:21 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Sep 17 18:21 /dev/ada0p2 crw-r----- 1 root operator 0, 98 Sep 17 18:21 /dev/ada1 crw-r----- 1 root operator 0, 106 Sep 17 18:21 /dev/ada1p1 crw-r----- 1 root operator 0, 107 Sep 17 18:21 /dev/ada1p2 crw-r----- 1 root operator 0, 100 Sep 17 18:21 /dev/ada3 crw-r----- 1 root operator 0, 110 Sep 17 18:21 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Sep 17 18:21 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Sep 17 18:28 /dev/ada4 crw-r----- 1 root operator 0, 122 Sep 17 18:28 /dev/ada4p1 crw-r----- 1 root operator 0, 127 Sep 17 18:28 /dev/ada4p2
ada2 and ada5 are missing. swapinfo
tingo@kg-f5$ swapinfo -h Device 1K-blocks Used Avail Capacity /dev/ada0p1 2097152 0B 2.0G 0% /dev/ada1p1 2097152 0B 2.0G 0% /dev/#C:0:0x6c 2097152 0B 2.0G 0% /dev/ada3p1 2097152 0B 2.0G 0% /dev/ada4p1 2097152 0B 2.0G 0% /dev/#C:0:0x70 2097152 0B 2.0G 0% Total 12582912 0B 12G 0%
I should find time to rebuild this soon.
2014-12-22: it looks like ada2 has gone too:
tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Sep 17 18:21 /dev/ada0 crw-r----- 1 root operator 0, 96 Sep 17 18:21 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Sep 17 18:21 /dev/ada0p2 crw-r----- 1 root operator 0, 98 Sep 17 18:21 /dev/ada1 crw-r----- 1 root operator 0, 106 Sep 17 18:21 /dev/ada1p1 crw-r----- 1 root operator 0, 107 Sep 17 18:21 /dev/ada1p2 crw-r----- 1 root operator 0, 100 Sep 17 18:21 /dev/ada3 crw-r----- 1 root operator 0, 110 Sep 17 18:21 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Sep 17 18:21 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Sep 17 18:28 /dev/ada4 crw-r----- 1 root operator 0, 122 Sep 17 18:28 /dev/ada4p1 crw-r----- 1 root operator 0, 127 Sep 17 18:28 /dev/ada4p2 crw-r----- 1 root operator 0, 102 Sep 17 18:21 /dev/ada5 crw-r----- 1 root operator 0, 112 Sep 17 18:21 /dev/ada5p1 crw-r----- 1 root operator 0, 113 Sep 17 18:21 /dev/ada5p2
swapinfo confirms:
tingo@kg-f5$ swapinfo -h Device 1K-blocks Used Avail Capacity /dev/ada0p1 2097152 0B 2.0G 0% /dev/ada1p1 2097152 0B 2.0G 0% /dev/#C:0:0x6c 2097152 0B 2.0G 0% /dev/ada3p1 2097152 0B 2.0G 0% /dev/ada4p1 2097152 0B 2.0G 0% Total 10485760 0B 10G 0%
bummer. For good measure, check with smartctl too:
[root@kg-f5] ~# smartctl -H /dev/ada0 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED [root@kg-f5] ~# smartctl -H /dev/ada1 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
but ada2 is missing:
[root@kg-f5] ~# smartctl -H /dev/ada2 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net /dev/ada2: No such file or directory Smartctl: please specify device type with the -d option. Use smartctl -h to get a usage summary
ada3 has a marginal parameter:
[root@kg-f5] ~# smartctl -H /dev/ada3 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
the rest looks better:
[root@kg-f5] ~# smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED [root@kg-f5] ~# smartctl -H /dev/ada5 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
cool.
2014-12-16: automatic scrub finished:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 80K in 219h38m with 47421777 errors on Tue Dec 16 03:38:18 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 234M raidz1-0 DEGRADED 0 0 468M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 3 87.6M 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47421777 data errors, use '-v' for a list
didn't help.
2014-12-07: automatic scrub in progress:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Sun Dec 7 00:00:11 2014 431G scanned out of 9.00T at 12.1M/s, 205h54m to go 0 repaired, 4.68% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 192M raidz1-0 DEGRADED 0 0 383M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 50209637 data errors, use '-v' for a list
we will see how long this takes.
2014-11-04: scrub finished - it did not help:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 190h24m with 47421777 errors on Tue Nov 4 21:10:14 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 189M raidz1-0 DEGRADED 0 0 378M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47421777 data errors, use '-v' for a list
the number of errors is the same number as after the previous scrub.
2014-10-27: starting scrub again:
[root@kg-f5] ~# zpool scrub z5 [root@kg-f5] ~# zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Mon Oct 27 22:45:36 2014 722M scanned out of 9.00T at 80.3M/s, 32h38m to go 0 repaired, 0.01% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 144M raidz1-0 DEGRADED 0 0 287M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47421777 data errors, use '-v' for a list
in a week or so we will see if it helped at all. If not, I will have to do something else.
2014-10-27: scrub finished:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 195h33m with 47421777 errors on Mon Oct 27 00:56:16 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 144M raidz1-0 DEGRADED 0 0 287M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47421777 data errors, use '-v' for a list
still problems, but it did reduce the errors a bit. Do another one?
2014-10-18: ok, let's start another scrub.
[root@kg-f5] ~# zpool scrub z5 [root@kg-f5] ~# zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Sat Oct 18 22:22:47 2014 1.53G scanned out of 9.00T at 71.1M/s, 36h52m to go 0 repaired, 0.02% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 98.4M raidz1-0 DEGRADED 0 0 197M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47446284 data errors, use '-v' for a list
let's see how this one plays out.
2014-10-18: trying to replace hard drive ada5. From FreeNAS' gui, I go to Storage, Volumes, View Volumes, Volume Status, then I offline the drive (ada5p2), except that I get the message:
Oct 18 22:08:28 kg-f5 manage.py: [middleware.exceptions:38] [MiddlewareError: Disk offline failed: "cannot offline gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2: no valid replicas, "]
odd, I thought I had enough replicas:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 56.9M in 193h55m with 47445682 errors on Thu Oct 16 18:38:34 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 98.4M raidz1-0 DEGRADED 0 0 197M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47445682 data errors, use '-v' for a list
ok, lets see if I can detach the old ones then:
first the old ada1 (4866735489841404330) the command succeeded, but after a reload the situation is the same Next the old ada4 (13212241896674284510) the command succeeded, but after a reload the situation is the same
a new zpool status:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 56.9M in 193h55m with 47445682 errors on Thu Oct 16 18:38:34 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 98.4M raidz1-0 DEGRADED 0 0 197M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47445682 data errors, use '-v' for a list
Hmm, what now. Trying the commands directly:
[root@kg-f5] ~# zpool detach z5 13212241896674284510 cannot detach 13212241896674284510: no valid replicas [root@kg-f5] ~# zpool detach z5 4866735489841404330 cannot detach 4866735489841404330: no valid replicas
Well, that's an answer I guess.
2014-10-17: smartctl says that the ada5 drive is healthy:
[root@kg-f5] ~# smartctl -H /dev/ada5 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
but still...
2014-10-17: hard drive ada5 error changes:
Oct 17 01:23:25 kg-f5 smartd[2382]: Device: /dev/ada5, 264 Currently unreadable (pending) sectors (changed -264) Oct 17 01:23:25 kg-f5 smartd[2382]: Device: /dev/ada5, 264 Offline uncorrectable sectors (changed -264) Oct 17 01:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 272 Currently unreadable (pending) sectors (changed +8) Oct 17 01:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 272 Offline uncorrectable sectors (changed +8) Oct 17 02:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 320 Currently unreadable (pending) sectors (changed +48) Oct 17 02:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 320 Offline uncorrectable sectors (changed +48) Oct 17 02:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 528 Currently unreadable (pending) sectors (changed +208) Oct 17 02:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 528 Offline uncorrectable sectors (changed +208)
all for now.
2014-10-16: scrub finished:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 56.9M in 193h55m with 47445682 errors on Thu Oct 16 18:38:34 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 98.4M raidz1-0 DEGRADED 0 0 197M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 2.06K 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 47445682 data errors, use '-v' for a list
well, it finished at least.
2014-10-08: I started a scrub - we will see how long it takes to complete.
scan: scrub in progress since Wed Oct 8 16:42:57 2014
nothing more.
2014-10-05: hard drive ada5 error changes:
Oct 5 02:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 336 Currently unreadable (pending) sectors (changed +320) Oct 5 02:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 336 Offline uncorrectable sectors (changed +320) Oct 5 02:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 360 Currently unreadable (pending) sectors (changed +24) Oct 5 02:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 360 Offline uncorrectable sectors (changed +24) Oct 5 03:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 376 Currently unreadable (pending) sectors (changed +16) Oct 5 03:23:24 kg-f5 smartd[2382]: Device: /dev/ada5, 376 Offline uncorrectable sectors (changed +16) Oct 5 06:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 824 Currently unreadable (pending) sectors (changed +448) Oct 5 06:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 824 Offline uncorrectable sectors (changed +448)
all for now.
2014-10-05: ahci messages:
Oct 5 02:07:20 kg-f5 kernel: ahcich5: Timeout on slot 12 port 0 Oct 5 02:07:20 kg-f5 kernel: ahcich5: is 00000000 cs 00003000 ss 00000000 rs 00003000 tfd c0 serr 00000000 cmd 0000ec17 Oct 5 02:07:51 kg-f5 kernel: ahcich5: AHCI reset: device not ready after 31000ms (tfd = 00000080) Oct 5 02:08:48 kg-f5 kernel: ahcich5: Timeout on slot 3 port 0 Oct 5 02:08:48 kg-f5 kernel: ahcich5: is 00000000 cs 00000000 ss 00001ff8 rs 00001ff8 tfd 40 serr 00000000 cmd 0000ec17 Oct 5 02:09:19 kg-f5 kernel: ahcich5: AHCI reset: device not ready after 31000ms (tfd = 00000080) Oct 5 02:09:49 kg-f5 kernel: ahcich5: Timeout on slot 12 port 0 Oct 5 02:09:49 kg-f5 kernel: ahcich5: is 00000000 cs 00001000 ss 00000000 rs 00001000 tfd 00 serr 00000000 cmd 0000ec17 Oct 5 02:17:58 kg-f5 kernel: ahcich5: Timeout on slot 9 port 0 Oct 5 02:17:58 kg-f5 kernel: ahcich5: is 00000000 cs 00000000 ss 00001e00 rs 00001e00 tfd 40 serr 00000000 cmd 0000ec17 Oct 5 02:18:30 kg-f5 kernel: ahcich5: AHCI reset: device not ready after 31000ms (tfd = 00000080) Oct 5 02:19:00 kg-f5 kernel: ahcich5: Timeout on slot 12 port 0 Oct 5 02:19:00 kg-f5 kernel: ahcich5: is 00000000 cs 00001000 ss 00000000 rs 00001000 tfd 00 serr 00000000 cmd 0000ec17
looks like trouble to me.
2014-10-05: hard drive ada5 errors:
Oct 5 01:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Currently unreadable (pending) sectors Oct 5 01:53:24 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Offline uncorrectable sectors
that is it.
2014-10-01: I try to run zdb to find out more about damaged objects in my pool - running it takes a looong time -
[root@kg-f5] ~# zdb -ccv z5 Traversing all blocks to verify checksums and verify nothing leaked ...
we'll see what happens. Well, it took days, and then just spat out error messages.
2014-09-20: after doing "detach" on the two replaced disks in the gui, zpool status still shows:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: resilvered 38.0G in 55h41m with 55621108 errors on Sat Sep 20 02:10:02 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 53.0M raidz1-0 DEGRADED 0 0 106M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 UNAVAIL 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 55621108 data errors, use '-v' for a list
and I have made a list of affected files from 'zpool status -v'.
2014-09-20: resilvering done:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: resilvered 38.0G in 55h41m with 55621108 errors on Sat Sep 20 02:10:02 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 53.0M raidz1-0 DEGRADED 0 0 106M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 UNAVAIL 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 55621108 data errors, use '-v' for a list
not sure if that is the best result, but...
2014-09-17: replacing hard drive ada4. It was not possible to offline the drive ada4 (Storage, Volumes, View Volumes, Volume Status, then offline on the correct drive) from the FreeNAS gui - the "offline" button was missing - the drive is probable totally dead. smartctl says:
[root@kg-f5] ~# smartctl -i /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net (pass4:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 01 00 (pass4:ahcich4:0:0:0): CAM status: ATA Status Error (pass4:ahcich4:0:0:0): ATA status: 71 (DRDY DF SERV ERR), error: 04 (ABRT ) (pass4:ahcich4:0:0:0): RES: 71 04 00 00 00 40 00 00 00 01 00 (pass4:ahcich4:0:0:0): ATAPI_IDENTIFY. ACB: a1 00 00 00 00 40 00 00 00 00 01 00 (pass4:ahcich4:0:0:0): CAM status: ATA Status Error (pass4:ahcich4:0:0:0): ATA status: 71 (DRDY DF SERV ERR), error: 04 (ABRT ) (pass4:ahcich4:0:0:0): RES: 71 04 00 00 00 40 00 00 00 01 00 Smartctl: Device Read Identity Failed: Unknown error A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
so - yes - dead. So I powered off the machine from the gui. Next I removed the four thumbscrews from the back of the case, and took the lid off. Removing bracket with drives I removed the security screw (Philips, medium size screwdriver) and loosened the two thumbscrews which held the rightmost hard drive bracket (seen from the front of the case) and disconnected sata (locked cables) and sata power cables from the two drives. Next I unscrewed the left side drive from the bracket and replaced it with a new one. Hopefully this will be ada1 (drives are numbered from left to right, 0 - 5).
old drive: serial number: W1F159GV, firmware: CC4B, WWN: 5000C5005367C975 new drive: serial number: Z50060XB, firmware: CC43, WWN: not written on drive
replacing bracket with drives. I replaced the bracket in the case and secured it with two thumbscrews and the security screw, the I connected sata and sata power cables to the two drives. then I replaced the lid on the case. This completes the hardware replacement. Replacing the drive in the pool In the GUI, I go back to Storage, Volumes, View Volumes, Volume Status and click on the "Replace" button next to the old (offlined) ada4 (it is shown as a many digits number now) I select ada4 from the drop down menu (there are no other drives to select) and click on "Replace Disk". After a short while it completes. From the shell ada4 now looks like this:
tingo@kg-f5$ ls -l /dev/ada4* crw-r----- 1 root operator 0, 101 Sep 17 18:28 /dev/ada4 crw-r----- 1 root operator 0, 122 Sep 17 18:28 /dev/ada4p1 crw-r----- 1 root operator 0, 127 Sep 17 18:28 /dev/ada4p2
and zpool status shows:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Wed Sep 17 18:28:10 2014 263M scanned out of 8.50T at 2.35M/s, (scan is slow, no estimated time) 46.1M resilvered, 0.00% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-4 UNAVAIL 0 0 0 13212241896674284510 UNAVAIL 0 0 0 was /dev/gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 gptid/9bb3d90d-3e87-11e4-80a2-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 55621108 data errors, use '-v' for a list
better speed this time.
2014-09-17: pool status (after a 2 weeks+ holiday trip):
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 55h48m with 55621108 errors on Tue Sep 16 07:49:00 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 3.25K 0 106M raidz1-0 DEGRADED 3.25K 0 212M gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 DEGRADED 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 FAULTED 25.1K 131M 0 too many errors gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 55621108 data errors, use '-v' for a list
looks like I timed that trip well.
2014-08-30: resilvering progress:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Aug 29 22:14:12 2014 51.4G scanned out of 8.48T at 773K/s, (scan is slow, no estimated time) 8.07G resilvered, 0.59% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 2.53K 0 0 raidz1-0 DEGRADED 2.53K 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 OFFLINE 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 2.53K 0 0 (resilvering) gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: 2592 data errors, use '-v' for a list
nothing more to say.
2014-08-30: hard drive ada4 error count changes: (as the rebuild of the pool progresses, the errors on ada4 increase)
Aug 30 00:28:30 kg-f5 smartd[2382]: Device: /dev/ada4, 6256 Currently unreadable (pending) sectors (changed +440) Aug 30 00:28:30 kg-f5 smartd[2382]: Device: /dev/ada4, 6256 Offline uncorrectable sectors (changed +440) Aug 30 00:28:36 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 1614 to 2078 Aug 30 00:58:32 kg-f5 smartd[2382]: Device: /dev/ada4, 6496 Currently unreadable (pending) sectors (changed +240) Aug 30 00:58:32 kg-f5 smartd[2382]: Device: /dev/ada4, 6496 Offline uncorrectable sectors (changed +240) Aug 30 00:58:42 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 2078 to 2528 Aug 30 01:28:41 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 2528 to 3032 Aug 30 01:58:42 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 3032 to 3534 Aug 30 02:28:37 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 3534 to 4014 Aug 30 02:58:36 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 4014 to 4489 Aug 30 03:28:37 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 4489 to 5015 Aug 30 03:58:38 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 5015 to 5509 Aug 30 04:28:36 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 5509 to 6074 Aug 30 04:58:37 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 6074 to 6654 Aug 30 05:28:38 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 6654 to 7265 Aug 30 05:58:36 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 7265 to 7886 [...] Aug 30 16:58:46 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 18858 to 19316 Aug 30 17:28:31 kg-f5 smartd[2382]: Device: /dev/ada4, failed to read SMART Attribute Data Aug 30 17:28:39 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 19316 to 19792
Urgh.
2014-08-29: hard drive ada4 error count changes:
Aug 29 22:28:31 kg-f5 smartd[2382]: Device: /dev/ada4, 4312 Currently unreadable (pending) sectors (changed +656) Aug 29 22:28:31 kg-f5 smartd[2382]: Device: /dev/ada4, 4312 Offline uncorrectable sectors (changed +656) Aug 29 22:28:39 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 25 to 168 Aug 29 22:58:34 kg-f5 smartd[2382]: Device: /dev/ada4, 4880 Currently unreadable (pending) sectors (changed +568) Aug 29 22:58:34 kg-f5 smartd[2382]: Device: /dev/ada4, 4880 Offline uncorrectable sectors (changed +568) Aug 29 22:58:38 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 168 to 656 Aug 29 23:28:30 kg-f5 smartd[2382]: Device: /dev/ada4, failed to read SMART Attribute Data Aug 29 23:58:32 kg-f5 smartd[2382]: Device: /dev/ada4, 5816 Currently unreadable (pending) sectors (changed +936) Aug 29 23:58:32 kg-f5 smartd[2382]: Device: /dev/ada4, 5816 Offline uncorrectable sectors (changed +936) Aug 29 23:58:43 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 656 to 1614
Let's hope it doesn't die before resilvering is complete.
2014-08-29: resilvering progress:
tingo@kg-f5$ date;zpool status Fri Aug 29 22:24:49 CEST 2014 pool: z5 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Aug 29 22:14:12 2014 40.1G scanned out of 8.48T at 64.4M/s, 38h9m to go 6.35G resilvered, 0.46% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 OFFLINE 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
This will be interesting.
2014-08-29: daily status:
tingo@kg-f5$ date;swapinfo -h;tvlm;df -h;uptime Fri Aug 29 22:22:46 CEST 2014 Device 1K-blocks Used Avail Capacity /dev/ada0p1 2097152 0B 2.0G 0% /dev/ada2p1 2097152 0B 2.0G 0% /dev/ada3p1 2097152 0B 2.0G 0% /dev/ada4p1 2097152 0B 2.0G 0% /dev/ada5p1 2097152 0B 2.0G 0% /dev/ada1p1 2097152 0B 2.0G 0% Total 12582912 0B 12G 0% Aug 29 22:14:09 kg-f5 notifier: 1+0 records in Aug 29 22:14:09 kg-f5 notifier: 1+0 records out Aug 29 22:14:09 kg-f5 notifier: 1048576 bytes transferred in 0.006186 secs (169507689 bytes/sec) Aug 29 22:14:09 kg-f5 notifier: dd: /dev/ada1: short write on character device Aug 29 22:14:09 kg-f5 notifier: dd: /dev/ada1: end of device Aug 29 22:14:09 kg-f5 notifier: 5+0 records in Aug 29 22:14:09 kg-f5 notifier: 4+1 records out Aug 29 22:14:09 kg-f5 notifier: 4677632 bytes transferred in 0.025344 secs (184564685 bytes/sec) Aug 29 22:14:24 kg-f5 notifier: swapon: /dev/ada1p1: device already in use Aug 29 22:14:31 kg-f5 ntpd[2214]: kernel time sync status change 2001 Filesystem Size Used Avail Capacity Mounted on /dev/ufs/FreeNASs1a 926M 383M 469M 45% / devfs 1.0k 1.0k 0B 100% /dev /dev/md0 4.6M 3.2M 979k 77% /etc /dev/md1 823k 2.0k 756k 0% /mnt /dev/md2 149M 15M 121M 11% /var /dev/ufs/FreeNASs4 19M 1.4M 16M 8% /data z5 6.0T 268k 6.0T 0% /mnt/z5 z5/h-tingo 6.0T 42G 6.0T 1% /mnt/z5/h-tingo z5/media 6.1T 84G 6.0T 1% /mnt/z5/media z5/xxx 12T 6.7T 6.0T 53% /mnt/z5/xxx 10:22PM up 26 mins, 1 user, load averages: 1.31, 0.94, 0.51
nothing more.
2014-08-29: replacing hard drive ada1. First I used the FreeNAS gui to offline the drive ada1 (Storage, Volumes, View Volumes, Volume Status, then offline on the correct drive) after a while the Volume status view reloaded. From the shell it now looks like this:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 4.03M in 11h29m with 0 errors on Sun Jul 27 11:29:50 2014 config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 4866735489841404330 OFFLINE 3 290 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
Good (I think). Opening the case Next I removed the four thumbscrews from the back of the case, and took the lid off. Looking inside the case, it is hard to remove a couple of drives without touching other stuff, so I'll shutdown the machine. Removing bracket with drives After shutting down the machine (also from the gui, it powered off in a couple of minutes) I removed the security screw (Philips, medium size screwdriver) and the two thumbscrews which held the leftmost hard drive bracket (seen from the front of the case) and disconnected sata (locked cables) and sata power cables from the two drives. Next I unscrewed the right side drive from the bracket and replaced it with a new one. Hopefully this will be ada1 (drives are numbered from left to right, 0 - 5).
old drive: serial number: W1F14KNV, firmware: CC4B, WWN: 5000C500536274BC new drive: serial number: Z50063XD, firmware: CC43, WWN: not written on drive
replacing bracket with drives. I replaced the bracket in the case (in retrospect, it is enough to loosen the two thumbscrews) and secured it with two thumbscrews and the security screw, then I connected sata and sata power cables to the two drives. Checking if it is the correct drive. Next, I powered up the machine and after a while I could login. From the shell it looks like this:
tingo@kg-f5$ ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Aug 29 21:56 /dev/ada0 crw-r----- 1 root operator 0, 96 Aug 29 21:56 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Aug 29 21:56 /dev/ada0p2 crw-r----- 1 root operator 0, 98 Aug 29 21:56 /dev/ada1 crw-r----- 1 root operator 0, 99 Aug 29 21:56 /dev/ada2 crw-r----- 1 root operator 0, 106 Aug 29 21:56 /dev/ada2p1 crw-r----- 1 root operator 0, 107 Aug 29 21:56 /dev/ada2p2 crw-r----- 1 root operator 0, 100 Aug 29 21:56 /dev/ada3 crw-r----- 1 root operator 0, 108 Aug 29 21:56 /dev/ada3p1 crw-r----- 1 root operator 0, 109 Aug 29 21:56 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Aug 29 21:56 /dev/ada4 crw-r----- 1 root operator 0, 110 Aug 29 21:56 /dev/ada4p1 crw-r----- 1 root operator 0, 111 Aug 29 21:56 /dev/ada4p2 crw-r----- 1 root operator 0, 102 Aug 29 21:56 /dev/ada5 crw-r----- 1 root operator 0, 112 Aug 29 21:56 /dev/ada5p1 crw-r----- 1 root operator 0, 113 Aug 29 21:56 /dev/ada5p2
Ok, so it looks like I replaced the correct drive. From the GUI (Storage, Volumes, View Disks) you can get the serial numbers of each drive. Currently (after replacing ada1) it looks like this:
drive serial number ada0 W1F15YWB ada1 Z50063XD ada2 W1F15V48 ada3 W1F15MRX ada4 W1F159GV ada5 W1F15YED
Ok, go ahead with the replacement. In the GUI, I go back to Storage, Volumes, View Volumes, Volume Status and click on the "Replace" button next to the old (offlined) ada1 (it is shown as a many digits number now) I select ada1 from the drop down menu (there are no other drives to select) and click on "Replace Disk". After a short while it completes. From the shell ada1 now looks like this:
tingo@kg-f5$ ls -l /dev/ada1* crw-r----- 1 root operator 0, 98 Aug 29 22:14 /dev/ada1 crw-r----- 1 root operator 0, 122 Aug 29 22:14 /dev/ada1p1 crw-r----- 1 root operator 0, 127 Aug 29 22:14 /dev/ada1p2
and zpool status:
tingo@kg-f5$ zpool status pool: z5 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Aug 29 22:14:12 2014 13.4M scanned out of 8.48T at 74.1K/s, (scan is slow, no estimated time) 2.23M resilvered, 0.00% done config: NAME STATE READ WRITE CKSUM z5 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 replacing-1 OFFLINE 0 0 0 4866735489841404330 OFFLINE 0 0 0 was /dev/gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 gptid/09858204-2fb9-11e4-8f91-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 (resilvering) gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
Gulp - this looks dangerous. We shall see if it works.
2014-08-29: hard drive ada1 - it doesn't look good in the swapinfo command either:
tingo@kg-f5$ date;swapinfo -h;tvlm;df -h;uptime Fri Aug 29 20:09:18 CEST 2014 Device 1K-blocks Used Avail Capacity /dev/ada0p1 2097152 0B 2.0G 0% /dev/#C:0:0x6a 2097152 0B 2.0G 0% /dev/ada2p1 2097152 0B 2.0G 0% /dev/ada3p1 2097152 0B 2.0G 0% /dev/ada4p1 2097152 0B 2.0G 0% /dev/ada5p1 2097152 0B 2.0G 0% Total 12582912 0B 12G 0% Aug 29 18:42:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Currently unreadable (pending) sectors Aug 29 18:42:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Offline uncorrectable sectors Aug 29 19:12:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Currently unreadable (pending) sectors Aug 29 19:12:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Offline uncorrectable sectors Aug 29 19:12:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Currently unreadable (pending) sectors Aug 29 19:12:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Offline uncorrectable sectors Aug 29 19:42:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Currently unreadable (pending) sectors Aug 29 19:42:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Offline uncorrectable sectors Aug 29 19:42:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Currently unreadable (pending) sectors Aug 29 19:42:24 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Offline uncorrectable sectors Filesystem Size Used Avail Capacity Mounted on /dev/ufs/FreeNASs1a 926M 383M 469M 45% / devfs 1.0k 1.0k 0B 100% /dev /dev/md0 4.6M 3.2M 979k 77% /etc /dev/md1 823k 2.0k 756k 0% /mnt /dev/md2 149M 15M 121M 11% /var /dev/ufs/FreeNASs4 19M 1.4M 16M 7% /data z5 6.0T 268k 6.0T 0% /mnt/z5 z5/h-tingo 6.0T 42G 6.0T 1% /mnt/z5/h-tingo z5/media 6.1T 84G 6.0T 1% /mnt/z5/media z5/xxx 12T 6.7T 6.0T 53% /mnt/z5/xxx 8:09PM up 49 days, 2:28, 2 users, load averages: 1.25, 1.25, 1.19
It is time to replace it, perhaps.
2014-08-28: zpool status:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 4.03M in 11h29m with 0 errors on Sun Jul 27 11:29:50 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 3 222 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
should I run a scrub now, or wait until it does automatically? smartctl status for ada4:
[root@kg-f5] ~# smartctl -H /dev/ada4 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
smartctl status for ada1
[root@kg-f5] ~# smartctl -H /dev/ada1 smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net /dev/ada1: No such file or directory Smartctl: please specify device type with the -d option. Use smartctl -h to get a usage summary
Hmm?
[root@kg-f5] ~# ls -l /dev/ada* crw-r----- 1 root operator 0, 95 Jul 11 17:40 /dev/ada0 crw-r----- 1 root operator 0, 96 Jul 11 17:40 /dev/ada0p1 crw-r----- 1 root operator 0, 97 Jul 11 17:40 /dev/ada0p2 crw-r----- 1 root operator 0, 99 Jul 11 17:40 /dev/ada2 crw-r----- 1 root operator 0, 108 Jul 11 17:40 /dev/ada2p1 crw-r----- 1 root operator 0, 109 Jul 11 17:40 /dev/ada2p2 crw-r----- 1 root operator 0, 100 Jul 11 17:40 /dev/ada3 crw-r----- 1 root operator 0, 110 Jul 11 17:40 /dev/ada3p1 crw-r----- 1 root operator 0, 111 Jul 11 17:40 /dev/ada3p2 crw-r----- 1 root operator 0, 101 Jul 11 17:40 /dev/ada4 crw-r----- 1 root operator 0, 112 Jul 11 17:40 /dev/ada4p1 crw-r----- 1 root operator 0, 113 Jul 11 17:40 /dev/ada4p2 crw-r----- 1 root operator 0, 102 Jul 11 17:40 /dev/ada5 crw-r----- 1 root operator 0, 114 Jul 11 17:40 /dev/ada5p1 crw-r----- 1 root operator 0, 115 Jul 11 17:40 /dev/ada5p2
Ok, it looks like we really lost ada1.
2014-08-27: errors on ahci - ada1:
Aug 27 20:34:35 kg-f5 kernel: ahcich1: Timeout on slot 4 port 0 Aug 27 20:34:35 kg-f5 kernel: ahcich1: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd c0 serr 00000000 cmd 0000e417 Aug 27 20:35:21 kg-f5 kernel: ahcich1: Timeout on slot 4 port 0 Aug 27 20:35:21 kg-f5 kernel: ahcich1: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 1d0 serr 00000000 cmd 0000e417 Aug 27 20:35:52 kg-f5 kernel: ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000180) Aug 27 20:36:22 kg-f5 kernel: ahcich1: Timeout on slot 4 port 0 Aug 27 20:36:22 kg-f5 kernel: ahcich1: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 100 serr 00000000 cmd 0000e417 Aug 27 20:36:22 kg-f5 kernel: (ada1:ahcich1:0:0:0): lost device
Hmm, not good for ada1.
hard drive ada4 errors:
Aug 27 22:12:24 kg-f5 smartd[2382]: Device: /dev/ada4, 32 Currently unreadable (pending) sectors Aug 27 22:12:24 kg-f5 smartd[2382]: Device: /dev/ada4, 32 Offline uncorrectable sectors
hard drive ada4 error count changes:
Aug 27 22:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 400 Currently unreadable (pending) sectors (changed +368) Aug 27 22:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 400 Offline uncorrectable sectors (changed +368) Aug 27 23:12:23 kg-f5 smartd[2382]: Device: /dev/ada4, 528 Currently unreadable (pending) sectors (changed +128) Aug 27 23:12:23 kg-f5 smartd[2382]: Device: /dev/ada4, 528 Offline uncorrectable sectors (changed +128) Aug 27 23:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 920 Currently unreadable (pending) sectors (changed +392) Aug 27 23:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 920 Offline uncorrectable sectors (changed +392) Aug 28 00:12:23 kg-f5 smartd[2382]: Device: /dev/ada4, 1400 Currently unreadable (pending) sectors (changed +480) Aug 28 00:12:23 kg-f5 smartd[2382]: Device: /dev/ada4, 1400 Offline uncorrectable sectors (changed +480) Aug 28 00:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Currently unreadable (pending) sectors (changed +2208) Aug 28 00:42:23 kg-f5 smartd[2382]: Device: /dev/ada4, 3608 Offline uncorrectable sectors (changed +2208)
looks like ada4 is dying.
2014-08-17: hard drive ada0 error count:
Aug 17 09:12:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Currently unreadable (pending) sectors Aug 17 09:12:23 kg-f5 smartd[2382]: Device: /dev/ada0, 16 Offline uncorrectable sectors
that is all.
2014-07-28: latest scrub result:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 4.03M in 11h29m with 0 errors on Sun Jul 27 11:29:50 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 16 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
Ok, so clear the error:
tingo@kg-f5$ su - Password: [root@kg-f5] ~# zpool clear z5
verify:
[root@kg-f5] ~# zpool status pool: z5 state: ONLINE scan: scrub repaired 4.03M in 11h29m with 0 errors on Sun Jul 27 11:29:50 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
done.
2014-07-27: hard drive ada1 error count changes:
Jul 27 10:12:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1344 Currently unreadable (pending) sectors (changed -8) Jul 27 10:12:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1344 Offline uncorrectable sectors (changed -8) Jul 27 10:12:23 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 565 to 571 Jul 27 10:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1064 Currently unreadable (pending) sectors (changed -280) Jul 27 10:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1064 Offline uncorrectable sectors (changed -280) Jul 27 10:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 571 to 765 Jul 27 11:12:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1064 Currently unreadable (pending) sectors Jul 27 11:12:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1064 Offline uncorrectable sectors
and SMART attribute failures:
Jul 27 11:12:24 kg-f5 smartd[2382]: Device: /dev/ada1, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jul 27 11:12:24 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jul 27 11:42:24 kg-f5 smartd[2382]: Device: /dev/ada1, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel.
2014-07-12: latest zpool status:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 6.54M in 11h57m with 0 errors on Sun Jun 15 11:57:20 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
nothing more.
2014-07-11: hard drive ada1 error count changes:
Jul 11 18:12:24 kg-f5 smartd[2382]: Device: /dev/ada1, 408 Currently unreadable (pending) sectors Jul 11 18:12:24 kg-f5 smartd[2382]: Device: /dev/ada1, 408 Offline uncorrectable sectors Jul 11 18:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1024 Currently unreadable (pending) sectors (changed +616) Jul 11 18:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1024 Offline uncorrectable sectors (changed +616) Jul 12 02:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1352 Currently unreadable (pending) sectors (changed +328) Jul 12 02:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, 1352 Offline uncorrectable sectors (changed +328)
and SMART attribute failures:
Jul 11 20:42:23 kg-f5 smartd[2382]: Device: /dev/ada1, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jul 11 20:42:23 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jul 11 20:42:23 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel.
2014-07-11: I booted the machine after a power outage earlier in the day (while I was away).
2014-07-06: uptime has passed 400 days.
tingo@kg-f5$ date;uptime Sun Jul 6 12:56:13 CEST 2014 12:56PM up 401 days, 5:32, 2 users, load averages: 0.11, 0.08, 0.02
Nothing more.
2014-06-28: clearing pool errors:
tingo@kg-f5$ su - Password: [root@kg-f5] ~# zpool clear z5
check
[root@kg-f5] ~# zpool status pool: z5 state: ONLINE scan: scrub repaired 6.54M in 11h57m with 0 errors on Sun Jun 15 11:57:20 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
that's all.
2014-06-15: latest scrub result:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 6.54M in 11h57m with 0 errors on Sun Jun 15 11:57:20 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 8 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 1 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
nothing more.
2014-06-15: hard drive ada5 error count changes:
Jun 15 11:55:26 kg-f5 smartd[2382]: Device: /dev/ada5, ATA error count increased from 0 to 5
nothing more.
2014-06-15: hard drive ada1 error count changes:
Jun 15 10:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 130 to 145 Jun 15 11:25:35 kg-f5 smartd[2382]: Device: /dev/ada1, 472 Currently unreadable (pending) sectors (changed +48) Jun 15 11:25:35 kg-f5 smartd[2382]: Device: /dev/ada1, 472 Offline uncorrectable sectors (changed +48) Jun 15 11:25:43 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 145 to 440 Jun 15 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 376 Currently unreadable (pending) sectors (changed -96) Jun 15 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 376 Offline uncorrectable sectors (changed -96) Jun 15 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 440 to 565 Jun 15 12:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 392 Currently unreadable (pending) sectors (changed +16) Jun 15 12:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 392 Offline uncorrectable sectors (changed +16)
that's it.
2014-06-15: timeout messages:
Jun 15 11:21:26 kg-f5 kernel: ahcich1: Timeout on slot 7 port 0 Jun 15 11:21:26 kg-f5 kernel: ahcich1: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd c0 serr 00000000 cmd 0000e717 Jun 15 11:21:26 kg-f5 kernel: ahcich1: Error while READ LOG EXT Jun 15 11:30:49 kg-f5 kernel: ahcich1: Timeout on slot 19 port 0 Jun 15 11:30:49 kg-f5 kernel: ahcich1: is 00000000 cs 00080000 ss 00000000 rs 00080000 tfd c0 serr 00000000 cmd 0000f317 Jun 15 11:30:49 kg-f5 kernel: ahcich1: Error while READ LOG EXT
that's all of them for now.
2014-06-15: hard drive ada2 and ada3 SMART changes:
Jun 15 00:25:24 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 00:55:25 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 01:25:25 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 01:55:24 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 02:25:25 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada2, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel.
and ada3 changes:
Jun 15 00:55:25 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 01:25:25 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 01:55:25 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 02:25:25 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel. Jun 15 04:55:25 kg-f5 smartd[2382]: Device: /dev/ada3, Failed SMART usage Attribute: 190 Airflow_Temperature_Cel.
that's it.
2014-06-07: hard drive ada5 error count:
tingo@kg-f5$ grep ada5 /var/log/messages Jun 7 18:25:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Currently unreadable (pending) sectors Jun 7 18:25:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Offline uncorrectable sectors Jun 7 18:55:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Currently unreadable (pending) sectors Jun 7 18:55:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Offline uncorrectable sectors Jun 7 19:25:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Currently unreadable (pending) sectors Jun 7 19:25:25 kg-f5 smartd[2382]: Device: /dev/ada5, 16 Offline uncorrectable sectors
that's all.
2014-05-25: hard drive ada1 error count change:
May 25 02:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 424 Currently unreadable (pending) sectors (changed -112) May 25 02:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 424 Offline uncorrectable sectors (changed -112)
nothing more.
2014-05-20: hard drive ada1 error count increase:
May 20 03:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 536 Currently unreadable (pending) sectors (changed +224) May 20 03:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 536 Offline uncorrectable sectors (changed +224)
nothing more.
2014-05-11: The newest version of FreeNAS is 9.2.1.5, released on April 24th, 2014. This machine still runs FreeNAS-8.3.0-RELEASE-x64 (r12701M). Looking at the FreeNAS download site, version 8.3.1 and 8.3.2 is available.
2014-05-07: hard drive ada1 error count increase:
May 7 15:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 312 Currently unreadable (pending) sectors (changed +8) May 7 15:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 312 Offline uncorrectable sectors (changed +8)
nothing more.
2014-05-04: hard drives new error counts and changes: ada1
May 4 10:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1024 Currently unreadable (pending) sectors (changed -480) May 4 10:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1024 Offline uncorrectable sectors (changed -480) May 4 10:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 304 Currently unreadable (pending) sectors (changed -720) May 4 10:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 304 Offline uncorrectable sectors (changed -720) May 4 10:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, ATA error count increased from 73 to 130 May 4 11:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 304 Currently unreadable (pending) sectors May 4 11:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 304 Offline uncorrectable sectors
ada4
May 4 10:25:26 kg-f5 smartd[2382]: Device: /dev/ada4, 16 Currently unreadable (pending) sectors (changed -8) May 4 10:25:26 kg-f5 smartd[2382]: Device: /dev/ada4, 16 Offline uncorrectable sectors (changed -8) May 4 10:25:26 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 0 to 5 May 4 10:55:25 kg-f5 smartd[2382]: Device: /dev/ada4, ATA error count increased from 5 to 15
that's it.
2014-05-04: latest scrub result:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 2.37M in 11h1m with 0 errors on Sun May 4 11:01:37 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
2014-04-23: hard drive ada1 error count increase:
Apr 23 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1504 Currently unreadable (pending) sectors (changed +16) Apr 23 11:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1504 Offline uncorrectable sectors (changed +16)
2014-04-17: latest scrub result:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 10h19m with 0 errors on Sun Mar 23 10:19:42 2014 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
2014-04-17: hard drives ada1 ada4 error counts remain stable:
Apr 17 09:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1480 Currently unreadable (pending) sectors Apr 17 09:25:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1480 Offline uncorrectable sectors Apr 17 09:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Currently unreadable (pending) sectors Apr 17 09:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Offline uncorrectable sectors
2014-04-16: hard drives - ada1 errors increasing:
Apr 16 08:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1480 Currently unreadable (pending) sectors (changed +312) Apr 16 08:55:25 kg-f5 smartd[2382]: Device: /dev/ada1, 1480 Offline uncorrectable sectors (changed +312)
2014-04-15: hard drives - ada4 got a few more errors:
Apr 15 03:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Currently unreadable (pending) sectors (changed +16) Apr 15 03:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 24 Offline uncorrectable sectors (changed +16)
2014-04-14: hard drives - ada1 some errors gets fixed:
Apr 14 01:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1168 Currently unreadable (pending) sectors (changed -8) Apr 14 01:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1168 Offline uncorrectable sectors (changed -8)
2014-04-09: hard drives - ada1 gets a lot of errors:
Apr 9 16:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 408 Currently unreadable (pending) sectors Apr 9 16:25:25 kg-f5 smartd[2382]: Device: /dev/ada1, 408 Offline uncorrectable sectors
which increases quickly
Apr 9 16:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1176 Currently unreadable (pending) sectors (changed +768) Apr 9 16:55:24 kg-f5 smartd[2382]: Device: /dev/ada1, 1176 Offline uncorrectable sectors (changed +768)
2014-04-01: hard drives - ada4 got a few errors:
Apr 1 19:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 8 Currently unreadable (pending) sectors Apr 1 19:25:25 kg-f5 smartd[2382]: Device: /dev/ada4, 8 Offline uncorrectable sectors
2013-07-21: latest scrub result:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 8h5m with 0 errors on Sun Jul 21 08:05:17 2013 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
That was all.
2013-03-17: Ok, the automatic scrub job works now:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 7h35m with 0 errors on Sun Mar 17 07:35:12 2013 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
Output from cron logfiles:
[root@kg-f5] ~# grep scrub /var/log/cron [root@kg-f5] ~# ll /var/log/cron* -rw------- 1 root wheel 3219 Mar 17 18:15 /var/log/cron -rw------- 1 root wheel 3539 Mar 17 18:00 /var/log/cron.0.bz2 -rw------- 1 root wheel 3802 Mar 17 10:00 /var/log/cron.1.bz2 -rw------- 1 root wheel 3669 Mar 17 02:00 /var/log/cron.2.bz2 -rw------- 1 root wheel 3484 Mar 16 18:00 /var/log/cron.3.bz2 [root@kg-f5] ~# sh # for i in 0 1 2 3 > { > bzcat /var/log/cron.${i}.bz2 | grep scrub > } Mar 17 00:00:00 kg-f5 /usr/sbin/cron[8752]: (root) CMD (PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 z5) # exit
That's all.
2013-03-12: automatic scrub doesn't work, probably because of an error in the crontab entry:
[root@kg-f5] ~# cat /etc/crontab | grep scrub 00 00 * 1,2,3,4,5,6,7,8,9,a,b,c 7 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 z5
reported in the FreeNAS forum. After just saving the scrub job from the FreeNAS gui, the crontab entry looks much better:
tingo@kg-f5$ cat /etc/crontab | grep scrub 00 00 * * 7 root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/local/sbin/scrub -t 35 z5
Hopefully it will work now.
2013-01-29: I did a manual scrub yesterday. Results:
tingo@kg-f5$ zpool status pool: z5 state: ONLINE scan: scrub repaired 0 in 7h39m with 0 errors on Tue Jan 29 01:20:21 2013 config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
Nothing more.
2012-12-31: I just noticed these lines in /var/log/messages:
Dec 31 17:31:35 kg-f5 kernel: re0: watchdog timeout Dec 31 17:31:35 kg-f5 kernel: re0: link state changed to DOWN Dec 31 17:31:38 kg-f5 kernel: re0: link state changed to UP
That's all.
2012-12-15: running a disk test:
[root@kg-f5] ~# cd /mnt/z5/h-tingo [root@kg-f5] /mnt/z5/h-tingo# mkdir test [root@kg-f5] /mnt/z5/h-tingo# cd test [root@kg-f5] /mnt/z5/h-tingo/test#
what parameters:
[root@kg-f5] /mnt/z5/h-tingo/test# zdb -C z5 | grep ashift ashift: 12 [root@kg-f5] /mnt/z5/h-tingo/test# sysctl hw.physmem hw.physmem: 16742060032 [root@kg-f5] /mnt/z5/h-tingo/test# sysctl kstat.zfs.misc.arcstats.c kstat.zfs.misc.arcstats.c: 11838246179
the test
[root@kg-f5] /mnt/z5/h-tingo/test# iozone -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -s 6g -i 0 -i 1 -i 2 Command line used: iozone -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -s 6g -i 0 -i 1 -i 2 Output is in Kbytes/sec Time Resolution = 0.000002 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 4 63605 56935 230524 148833 173698 5654 6291456 8 92586 104812 397195 403990 307068 10628 6291456 16 115676 141576 570036 578388 475816 20038 6291456 32 128819 158171 746870 796512 696080 49353 6291456 64 143069 149511 956121 972336 908076 100482 6291456 128 188834 191536 1048040 1095608 1062198 197302 iozone test complete.
parts of output cut for brevity.
2012-12-08: a simplistic disk speed test:
$ dd if=/dev/zero of=./testfile.test bs=8192k count=1000 1000+0 records in 1000+0 records out 8388608000 bytes transferred in 46.211054 secs (181528168 bytes/sec)
which is about 173 Mbytes per second. Another test:
$ dd if=/dev/zero of=./testfile2.test bs=1024k count=20000 20000+0 records in 20000+0 records out 20971520000 bytes transferred in 117.820681 secs (177995237 bytes/sec)
so only about 170 Mbytes per second.
2012-12-08: creating datasets. I created a dataset called h-tingo, it is mounted on /mnt/z5/h-tingo. I set it up as the home directory for user tingo, and gave the user permissions on the dataset.
2012-12-08: creating a volume. From the volume manager in the web gui, I named my volume z5, selected all six drives (ada0 - ada5), selected zfs, checked force 4096 bytes sector size, set group type to RAID-Z and hit "add volume". Deduplication is off (I don't need it, and don't have the RAM for it), After a while /mnt/z5 was ready. Verify that the pool uses 4k sectors:
[root@kg-f5] ~# zdb -C z5 | grep ashift ashift: 12
(ashift = 9 is 512 byte sectors) What it (the pool) looks like from ssh:
[root@kg-f5] ~# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/FreeNASs1a 926M 383M 469M 45% / devfs 1.0k 1.0k 0B 100% /dev /dev/md0 4.6M 3.2M 979k 77% /etc /dev/md1 823k 2.0k 756k 0% /mnt /dev/md2 149M 14M 123M 10% /var /dev/ufs/FreeNASs4 19M 470k 17M 3% /data z5 12T 230k 12T 0% /mnt/z5 [root@kg-f5] ~# zpool status pool: z5 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM z5 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/202e9138-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/20e9bc37-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/21a0e079-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/225d33a6-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/2319281f-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 gptid/23d54afc-4124-11e2-a433-3085a9ebf2a2 ONLINE 0 0 0 errors: No known data errors
and gpart info:
[root@kg-f5] ~# gpart show => 63 7821249 da0 MBR (3.7G) 63 1930257 1 freebsd [active] (942M) 1930320 63 - free - (31k) 1930383 1930257 2 freebsd (942M) 3860640 3024 3 freebsd (1.5M) 3863664 41328 4 freebsd (20M) 3904992 3916320 - free - (1.9G) => 0 1930257 da0s1 BSD (942M) 0 16 - free - (8.0k) 16 1930241 1 !0 (942M) => 34 5860533101 ada0 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 5860533101 ada1 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 5860533101 ada2 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 5860533101 ada3 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 5860533101 ada4 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 5860533101 ada5 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k)
Cool.
2012-12-08: FreeNAS information. From web gui:
Hostname kg-f5.kg4.no Build FreeNAS-8.3.0-RELEASE-x64 (r12701M) Platform AMD C-60 APU with Radeon(tm) HD Graphics Memory 15966MB System Time Sat Dec 08 11:02:43 CET 2012 Uptime 11:02AM up 27 mins, 0 users Load Average 0.10, 0.23, 0.20 Connected through 10.1.161.6
(from the "System Information" screen) From the command line, it looks like this:
tingo@kg-v2$ ssh f5 tingo@kg-f5.kg4.no's password: Last login: Sat Dec 8 11:22:00 2012 from 10.1.150.17 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 8.3-RELEASE-p4 (FREENAS.amd64) #0 r241984M: Wed Oct 24 00:57:10 PDT 2012 FreeNAS (c) 2009-2012, The FreeNAS Development Team All rights reserved. FreeNAS is released under the modified BSD license. For more information, documentation, help or support, go here: http://freenas.org Welcome to FreeNAS Could not chdir to home directory /nonexistent: No such file or directory $ $ uname -a FreeBSD kg-f5.kg4.no 8.3-RELEASE-p4 FreeBSD 8.3-RELEASE-p4 #0 r241984M: Wed Oct 24 00:57:10 PDT 2012 root@build.ixsystems.com:/usr/home/jpaetzel/8.3.0-RELEASE/os-base/amd64/usr/home/jpaetzel/8.3.0-RELEASE/FreeBSD/src/sys/FREENAS.amd64 amd64
pciconf -lv output
$ pciconf -lv hostb0@pci0:0:0:0: class=0x060000 card=0x853a1043 chip=0x15101022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI vgapci0@pci0:0:1:0: class=0x030000 card=0x853a1043 chip=0x98071002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' class = display subclass = VGA pcib1@pci0:0:4:0: class=0x060400 card=0x12341022 chip=0x15121022 rev=0x00 hdr=0x01 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = PCI-PCI ahci0@pci0:0:17:0: class=0x010601 card=0x84961043 chip=0x43911002 rev=0x40 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 SATA Controller [AHCI mode]' class = mass storage subclass = SATA ohci0@pci0:0:18:0: class=0x0c0310 card=0x84961043 chip=0x43971002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB OHCI0 Controller' class = serial bus subclass = USB ehci0@pci0:0:18:2: class=0x0c0320 card=0x84961043 chip=0x43961002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB EHCI Controller' class = serial bus subclass = USB ohci1@pci0:0:19:0: class=0x0c0310 card=0x84961043 chip=0x43971002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB OHCI0 Controller' class = serial bus subclass = USB ehci1@pci0:0:19:2: class=0x0c0320 card=0x84961043 chip=0x43961002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB EHCI Controller' class = serial bus subclass = USB none0@pci0:0:20:0: class=0x0c0500 card=0x84961043 chip=0x43851002 rev=0x42 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'ATI SMBus (ATI RD600/RS600)' class = serial bus subclass = SMBus none1@pci0:0:20:2: class=0x040300 card=0x84151043 chip=0x43831002 rev=0x40 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'IXP SB600 High Definition Audio Controller' class = multimedia subclass = HDA isab0@pci0:0:20:3: class=0x060100 card=0x84961043 chip=0x439d1002 rev=0x40 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 LPC host controller' class = bridge subclass = PCI-ISA pcib2@pci0:0:20:4: class=0x060401 card=0x00000000 chip=0x43841002 rev=0x40 hdr=0x01 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'IXP SB600 PCI to PCI Bridge' class = bridge subclass = PCI-PCI ohci2@pci0:0:20:5: class=0x0c0310 card=0x84961043 chip=0x43991002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB OHCI2 Controller' class = serial bus subclass = USB pcib3@pci0:0:21:0: class=0x060400 card=0x00001002 chip=0x43a01002 rev=0x00 hdr=0x01 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' class = bridge subclass = PCI-PCI pcib4@pci0:0:21:1: class=0x060400 card=0x00001002 chip=0x43a11002 rev=0x00 hdr=0x01 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' class = bridge subclass = PCI-PCI ohci3@pci0:0:22:0: class=0x0c0310 card=0x84961043 chip=0x43971002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB OHCI0 Controller' class = serial bus subclass = USB ehci2@pci0:0:22:2: class=0x0c0320 card=0x84961043 chip=0x43961002 rev=0x00 hdr=0x00 vendor = 'ATI Technologies Inc. / Advanced Micro Devices, Inc.' device = 'SB700 USB EHCI Controller' class = serial bus subclass = USB hostb1@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x17001022 rev=0x43 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb2@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x17011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb3@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x17021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb4@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x17031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb5@pci0:0:24:4: class=0x060000 card=0x00000000 chip=0x17041022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb6@pci0:0:24:5: class=0x060000 card=0x00000000 chip=0x17181022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb7@pci0:0:24:6: class=0x060000 card=0x00000000 chip=0x17161022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI hostb8@pci0:0:24:7: class=0x060000 card=0x00000000 chip=0x17191022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' class = bridge subclass = HOST-PCI re0@pci0:4:0:0: class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)' class = network subclass = ethernet
kenv output (some of it):
[root@kg-f5] ~# kenv | grep smbios smbios.bios.reldate="08/07/2012" smbios.bios.vendor="American Megatrends Inc." smbios.bios.version="0305" smbios.chassis.maker="Chassis Manufacture" smbios.chassis.serial="Chassis Serial Number" smbios.chassis.tag="Asset-1234567890" smbios.chassis.version="Chassis Version" smbios.memory.enabled="16777216" smbios.planar.maker="ASUSTeK COMPUTER INC." smbios.planar.product="C60M1-I" smbios.planar.serial="120701021200852" smbios.planar.version="Rev X.0x" smbios.socket.enabled="1" smbios.socket.populated="1" smbios.system.maker="System manufacturer" smbios.system.product="System Product Name" smbios.system.serial="System Serial Number" smbios.system.uuid="f33f4a20-b8c1-11dc-9658-3085a9ebf2a2" smbios.system.version="System Version" smbios.version="2.6"
that's it.
2012-12-08: fixing the system so that it will have correct ip address. I log in via ssh:
tingo@kg-v2$ ssg 10.1.161.6 Warning: Permanently added '10.1.161.6' (RSA) to the list of known hosts. tingo@10.1.161.6's password: Last login: Sat Dec 8 11:09:02 2012 from 10.1.150.17 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 8.3-RELEASE-p4 (FREENAS.amd64) #0 r241984M: Wed Oct 24 00:57:10 PDT 2012 FreeNAS (c) 2009-2012, The FreeNAS Development Team All rights reserved. FreeNAS is released under the modified BSD license. For more information, documentation, help or support, go here: http://freenas.org Welcome to FreeNAS Could not chdir to home directory /nonexistent: No such file or directory
su
$ su - Password: [root@kg-f5] ~#
mount root fs r/w:
[root@kg-f5] ~# mount -w /
edit /conf/base/etc/dhclient.conf to include the line
send dhcp-client-identifier "kg-f5.kg4.no";
mount root fs read-only again:
[root@kg-f5] ~# mount -r /
then reboot the machine.
2012-12-08: The first FreeNAS boot took some time, but came up ok, and had an ip address assigned, and the web interface works. Nice! FreeNAS setup:
- changed password for admin user - added a group called users (gid: 1001 - default) - created a new user (tingo), set 'users' as primary group, 'sh' as shell, added it to the 'wheel' group. Currently this user doesn't have a home directory (will be fixed later). In Settings / General, changed timezone to Europe/Oslo. In Network / Global Configuration, changed hostname to kg-f5 and domain to kg4.no. in Services / Control Services, I turned on ssh.
2012-12-08: I downloaded FreeNAS 8.3.0 (FreeNAS-8.3.0-RELEASE-x64.img.xz) and verified that the sha256 checksum was good. Then I wrote the image to a 4 GB usb flash drive (Sandisk Cruzer Blade) with the commands:
tingo@kg-v2$ su Password: root@kg-v2# ls -l total 81620 -rw-r--r-- 1 tingo users 83474332 Dec 8 09:45 FreeNAS-8.3.0-RELEASE-x64.img.xz -rw-r--r-- 1 tingo users 65 Dec 8 09:43 FreeNAS-8.3.0-RELEASE-x64.img.xz.sha256.txt -rw-r--r-- 1 tingo users 34020 Dec 8 09:42 README root@kg-v2# ls /dev/da3* /dev/da3 /dev/da3s1 root@kg-v2# xzcat FreeNAS-8.3.0-RELEASE-x64.img.xz | dd of=/dev/da3 bs=64k 0+244141 records in 0+244141 records out 2000000000 bytes transferred in 985.313490 secs (2029811 bytes/sec)
which took about 16 minutes. Wow, that was long. gpart shows this about the flash drive afterwards:
root@kg-v2# gpart show da3 => 63 7821249 da3 MBR (3.7G) 63 1930257 1 freebsd [active] (942M) 1930320 63 - free - (31k) 1930383 1930257 2 freebsd (942M) 3860640 3024 3 freebsd (1.5M) 3863664 41328 4 freebsd (20M) 3904992 3916320 - free - (1.9G)
ok.