Author: Frank Language: text
Description: Ports / zvols Timestamp: 2014-08-18 15:45:34 -0400
View raw paste Reply
  1. Why should we not mix ports and packages?
  2.  
  3. I sometimes hear that. However, my understanding is that FreeBSD builds
  4. a package from a port and then installs that package? Just like Arch and
  5. Gentoo do, the way I like it. When installing newer software from source
  6. on CentOS or Debian I mostly use Checkinstall, a piece of software that,
  7. suprisingly, creates a package from source and then lets the package
  8. manager install that. Keeps the system clean, makes upgrading easy,
  9. removal even easier. Checkinstall however does not create nice/clean
  10. packages, but for local stuff that is OK.
  11.  
  12. -----------------------------------------------------
  13.  
  14. I've recently been hired for a rather large KVM project. A company that
  15. does stuff with molecular biology did some tests with virtualising their
  16. nodes that do the heavy math. They used to run on big hardware but since
  17. KVM supports certain offloading features and passthrough they evaluated
  18. it again. Turns out that 5000 VM's is way easier and more scalable than
  19. 700 pizza boxes in a rack every time with IPMI/PXE boot.
  20.  
  21. They've written some custom software to deploy new vm's with a nice
  22. Django GUI and I was hired for some scaling problems. Some include the
  23. HA setup of the KVM hypervisors and the Django app managing it, some
  24. included a HA postgres cluster for their calculation stuff, some was the
  25. upscaling of their gigabit network to 10 gbit over fiber (damn sexy
  26. expensive stuff that is. More than a yearly salary in one fiber switch,
  27. don't drop that... Since my hourly rate there is about EUR 200 for 40 to
  28. 60 hours for 6 months) and some other HA and scaling related issues. You
  29. might guess what my speciality seems to be, HA HA HA (no pun intended).
  30.  
  31. The main issue however was random zfs storage backend hangs. They have
  32. the root disks as sparse files on a zfs server (all over 10Gbit
  33. ethernet, plus a 10 Gbit fiber between the data centers, so link is not
  34. the problem here) exported over NFS. They keep the root disks small, and
  35. mount extra disks for larger/persistent storage/data. Those are created
  36. as zvols and exported over iscsi. However, the zfs storage nodes hang
  37. rather often. The volumes get mounted as read only (because linux has
  38. those mount options), the zfs server itself runs fine otherwise. The
  39. only thing that helps is rebooting. It's all FreeBSD 10-RELEASE, however
  40. a test upgrade to 10-STABLE (one month ago) did not fix the problem.
  41. Same for 9.2-RELEASE. The cause is not clear. Restarting istgt or
  42. anything does not help, remounting the volume on the kvm boxes also not,
  43. nothing except rebooting.
  44.  
  45. Mounting the volumes the same way as root disks (over nfs) works fine,
  46. no hanging problems there. When the zvol's all hang (they all hang at
  47. the same time) the nfs just works fine without problems.
  48.  
  49. They've done it this way because the performance of zvols over iscsi is
  50. almost 6 times better than the iscsi one, however now reliability
  51. becomes a factor more than speed. The KVM nodes are a mix of Ubuntu
  52. 12.04, Ubuntu 14.04, CentOS 6, OpenSUSE and CentOS 7. They all have the
  53. same issue, even with different kernel versions or user land and such.
  54.  
  55. We are talking about 60 FreeBSD nodes here, all with UFS as root file
  56. system and a 24 to 36 512GB, 1 TB and 2 TB 15k SAS drives on a mix of
  57. LSI and 3ware controllers. Setups with different controllers or just 4
  58. disks directly on the mother board also had the issue after a while. The
  59. only thing we can do to trigger it is generate heavy IO on multiple KVM
  60. machines. The pizza boxes themselves are a mix of HP, Dell and Supermicro.
  61.  
  62. There are about 80 to 500 zvols per server, size from 40 GB to 1000 GB.
  63.  
  64. The issue is now solved by changing their software to use NFS with
  65. sparse files, but I'm still wondering if you guys know anything more.
  66.  
  67. Still love the show, keep it up!
View raw paste Reply