NIC change, double performance

lohelle · Mon Jul 21, 2014 9:29 am

I would just like to share my experience after changing nics on some of my servers (SAN and some ESXi nodes)

I had mostly Intel (520) and Mellanox (ConnectX) 10Gig cards. After changing to Emulex cards (rather old OCE10102 cards), we went from 3-400 MB/s reads in VMs to 6-700MB/s.
We could (almost) saturate the links using iperf etc before, but we needed a lot of threads. It looks like the Emulex-cards will saturate the links with less threads.

I bought a few Emulex'es for my home lab, and ended up with buying 15 cards for some of the production servers..

Will do some more testing, and I have only upgraded one of my HA-nodes. I hope upgrading the last node will improve writes as well (as sync channel will use a better NIC on both nodes).

The Mellanox-cards should be pretty decent cards, but these Emulex-cards are far better.

Just shows that choosing the right HW is imprtant.

We are using Intel S3700 800GB SSD's (RAID) on both nodes, with some spindles of HDDs for low-IO storage (and some Samsung SSD's still doing OK)

rriiicchh · Mon Jul 21, 2014 3:30 pm

Good to know as I am about to purchase some NICs so I might focus my attention on Emulex cards rather than Mellanox

lohelle · Mon Jul 21, 2014 6:14 pm

The Mellanox-cards I have a lot of from before is MNPH29B-XTC, so they are quite old. The Emulex cards is model OCE10102. Both companies have newer models, so things might have changed. The Emulex-cards can be bought as NIC only, FCOE or ISCSI options (much cheaper as pure NIC, but can be upgraded with license as the HW is the same I think)

Wed Jul 23, 2014 1:22 pm

Did you try the most recent Mellanox cards? We have the best experience with them. Did not try Emulex ones for a long time however...

lohelle · Wed Jul 23, 2014 1:29 pm

As i said (or tried to), this was rather old cards. The point was that not all nics have the same performance, and the specs (pci-e 8x etc) is not all.

I would not be surprised if the ConnectX-3 cards is even better than these Emulex'es.

Wed Jul 23, 2014 2:38 pm

That would be my guess as well

PS We;; be soon getting new Mellanox 56 Gbps pre-production ones.

lohelle wrote:As i said (or tried to), this was rather old cards. The point was that not all nics have the same performance, and the specs (pci-e 8x etc) is not all.

I would not be surprised if the ConnectX-3 cards is even better than these Emulex'es.

Wed Jul 23, 2014 5:32 pm

Thanks for sharing that! Pretty useful!