Author: Bostjan Language: text
Description: ZFS issues Timestamp: 2015-04-29 15:11:35 -0400
View raw paste Reply
  1. I had (perhaps still have) a FreeNAS ZFS problem. I searched online but no useful help. You are the best ZFS guru I know. I hope you can help me with my UNAVAIL FreeNAS ZFS problem.
  2.  
  3. I was copying some files from FreeNAS to my computer when suddenly the transfer stopped. I was the only one using FreeNAS.
  4.  
  5. I didn’t do any upgrades, I didn’t replace any disks. The scrub of pool happened 15 days ago, scrub of pool 'freenas-boot' occurred one day ago.
  6.  
  7. At the time when the transfer stopped I’ve got emails
  8.  
  9. ------Critical Alerts---------
  10. The volume tank (ZFS) state is UNAVAIL: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
  11. ---------------
  12.  
  13. ------Critical Alerts---------
  14. The volume tank (ZFS) state is UNAVAIL: One or more devices are faulted in response to IO failures.
  15. ---------------
  16.  
  17. I run FreeNAS-9.3-STABLE. Six disks in mirror. ECC 8GB RAM.
  18. I couldn’t log in through browser - I got an error ERR_EMPTY_RESPONSE
  19. I couldn’t access FreeNAS through samba - I tried it on windows and on android (where it usually worked).
  20. I tried SSH. It asked me for username and password but then it stopped. No command worked. There was nothing written in front of the cursor.
  21. Ping worked.
  22.  
  23. I’ve connected screen to the box. The screen showed this:
  24. vm_fault: pager read error, pid 87337 (nginx)
  25. swap_pager: I/O error - pagein failed; blkno 2622307, size 8192, error 6
  26.  
  27. Those two lines were repeating through the whole screen.
  28. Second one (swap_pager: I/O error - pagein failed; blkno 2622307, size 8192, error 6) was the same the whole time.
  29. First one (vm_fault: pager read error, pid 87337 (nginx)), the number incremented - sometimes by one, sometimes by 300.
  30.  
  31. I plugget keyboard and mouse to the FreeNAS. I just pressed Enter key, just to see if keyboard worked and if FreeNAS responded. After I pressed the Enter key, for some time a lot of stuff ran on the screen (to fast to read). When it stopped I noticed that it was (re)booting.
  32.  
  33. After that I could log in and everything worked.
  34.  
  35. Then I ran #zpool status -v
  36.  
  37.   pool: tank
  38.  state: ONLINE
  39. status: One or more devices has experienced an error resulting in data
  40.         corruption.  Applications may be affected.
  41. action: Restore the file in question if possible.  Otherwise restore the
  42.         entire pool from backup.
  43.    see: http://illumos.org/msg/ZFS-8000-8A
  44.   scan: scrub repaired 0 in 6h27m with 0 errors on Sun Apr 12 06:27:32 2015
  45. config:
  46.  
  47.         NAME                                            STATE     READ WRITE CKS  UM
  48.         fn                                              ONLINE       0     0     0
  49.           mirror-0                                      ONLINE       0     0     0
  50.             gptid/5635b592-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  51.             gptid/571c8555-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     1
  52.           mirror-1                                      ONLINE       0     0     0
  53.             gptid/57c8d0a5-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  54.             gptid/589247dc-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  55.           mirror-2                                      ONLINE       0     0     0
  56.             gptid/a0de1680-d712-11e4-88b5-d050991b6427  ONLINE       0     0     0
  57.             gptid/a1c305d7-d712-11e4-88b5-d050991b6427  ONLINE       0     0     0
  58.  
  59. errors: Permanent errors have been detected in the following files:
  60.  
  61.         /var/db/system/update/MANIFEST
  62.         /mnt/tank/jails/plexmediaserver_1/var/db/plexdata/Plex Media Server/Plug-ins/WebClient.bundle/Contents/Resources/js/plex.js
  63.  
  64.   pool: freenas-boot
  65.  state: ONLINE
  66.   scan: scrub repaired 0 in 0h0m with 0 errors on Mon Apr 27 03:45:42 2015
  67. config:
  68.  
  69.         NAME        STATE     READ WRITE CKSUM
  70.         freenas-boot  ONLINE       0     0     0
  71.           da0p2     ONLINE       0     0     0
  72.  
  73. errors: No known data errors
  74.  
  75.  
  76. I have stopped all jails. I deleted most of them. I have updated FreeNAS.
  77.  
  78. Today I also ran #zpool status -v
  79. It’s a little bit different than day before. UM column is different and errors at the end are different.
  80.  
  81.   pool: tank
  82.  state: ONLINE
  83. status: One or more devices has experienced an error resulting in data
  84.         corruption.  Applications may be affected.
  85. action: Restore the file in question if possible.  Otherwise restore the
  86.         entire pool from backup.
  87.    see: http://illumos.org/msg/ZFS-8000-8A
  88.   scan: scrub repaired 0 in 6h27m with 0 errors on Sun Apr 12 06:27:32 2015
  89. config:
  90.  
  91.         NAME                                            STATE     READ WRITE CKS     UM
  92.         fn                                              ONLINE       0     0     0
  93.           mirror-0                                      ONLINE       0     0     0
  94.             gptid/5635b592-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  95.             gptid/571c8555-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  96.           mirror-1                                      ONLINE       0     0     0
  97.             gptid/57c8d0a5-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  98.             gptid/589247dc-2b8b-11e4-8622-d050991b6427  ONLINE       0     0     0
  99.           mirror-2                                      ONLINE       0     0     0
  100.             gptid/a0de1680-d712-11e4-88b5-d050991b6427  ONLINE       0     0     0
  101.             gptid/a1c305d7-d712-11e4-88b5-d050991b6427  ONLINE       0     0     0
  102.  
  103. errors: Permanent errors have been detected in the following files:
  104.  
  105.         tank/.system:<0x9a>
  106.         /mnt/tank/jails/plexmediaserver_1/var/db/plexdata/Plex Media Server/Plug-ins/WebClient.bundle/Contents/Resources/js/plex.js
  107.  
  108.   pool: freenas-boot
  109.  state: ONLINE
  110.   scan: scrub repaired 0 in 0h0m with 0 errors on Mon Apr 27 03:45:42 2015
  111. config:
  112.  
  113.         NAME        STATE     READ WRITE CKSUM
  114.         freenas-boot  ONLINE       0     0     0
  115.           da0p2     ONLINE       0     0     0
  116.  
  117. errors: No known data errors
  118.  
  119.  
  120.  
  121. Next day after the initial error I received an email from FreeNAS
  122. ------Critical Alerts---------
  123. The volume tank (ZFS) state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
  124. ---------------
  125.  
  126.  
  127. I don’t know what to do anymore. What all error mean which I’ve got to my email? Why the error occurred? What can I do so that the error won’t occur anymore? Is it normal that my system just went offline like that?
  128.  
  129. Thanks.
View raw paste Reply