SuperMicro vs Intel NUC

A couple of weeks ago I was talking to William Lam (http://www.virtuallyghetto.com/) and Alan Renouf (http://www.virtu-al.net/) about their exciting USB to SDDC demonstration, they were using an Intel NUC to deploy a VMware SDDC environment to a single node using VSAN. I offered them the opportunity to test out the same capability with one of my SuperMicro E200-8D servers and they took me up on the opportunity. Since then I have been approached by a number of people with requests for information about why I chose to go with the SuperMicro E200 for my home lab over the Intel NUC. I’ve never written a blog before but I thought this might be a good way to “cut out the middle man” so to speak. So here it goes, my reasons for why I chose the SuperMicro over the Intel NUC.

My previous home labs have generally been made up of used enterprise servers that can be picked up cheaply. These used servers are loud, power hungry and heavy. My goal was to firstly consume less power and secondly make my lab somewhat portable. These requirements appear to be popular amongst the community at the moment and there are a lot of stories about people using the Intel NUC’s to achieve these outcomes. I started to look around and it was fairly obvious that there were two stand-out options, the Intel NUC and the SuperMicro E200. I was left with a decision to be made and had to ask myself some additional questions about what I really wanted in my home lab. I came up with the following requirements.

  1. Minimal power consumption.
  2. Small and lightweight.
  3. Capable of a good consolidation ratio of VMs to Host.
  4. Capable of using all flash VSAN (albeit in an unsupported configuration).
  5. Enough scalability to expand the lab in the future.
  6. Good availability. These need to be up when I am doing demonstrations to customers.
  7. Ability to set up an enterprise-like environment for comparisons with customer environments.

The next step was to compare my options. The following table takes information from the respective vendor sites and addresses my specific requirements plus a couple of additional considerations. This table made my decision easy for me. It became quite obvious to me that the SuperMicro was the more superior option for my home lab, in fact the SuperMicro is an enterprise ready solution.

SuperMicro E200-8D Intel NUC 7th Gen
ESXi 6.5 Compatible

Native install works

Requires NIC drivers

CPU Type

XEON D-1528

Intel i7-7567U

CPU Capacity

6 core 12 threads

1.9 GHz – 2.2 GHz

Dual Core

3.5 GHz – 4.0 GHz

RAM Type

4x DDR4

2x DDR4 SODIMM

RAM Capacity

128GB ECC RDIMM

64 GB Non-ECC UDIMM

32GB Non-ECC

Intel Optane Ready

Yes

Yes

HDD Capacity

1x 2.5”

1x 2.5”

NVMe / M.2 SATA

YES

YES

SATADOM

YES

NO

Micro SDXC

NO

YES

1Gbe Networking

2x 1Gbe

1x 1Gbe

10Gbe Networking

2x 10Gbe

1x Thunderbolt 40Gbps

Wireless

NO

802.11ac

IPMI

YES

NO

Power Consumption

60W

64W

Rack Mounting

YES

 NO

SR-IOV Support

YES

NO

Video Port

VGA only

HDMI 4k

Noise Comparison

2x Fans

Fan-less

USB Capacity

2 Ports

4 Ports

Price Comparison

US$799

US$630

Hopefully the above table has also helped you with your decision. The reason I opted for the SuperMicro E200 is that although it does cost  little bit more, it is an enterprise ready solution that accepts ECC memory, uses a XEON CPU, has a larger RAM capacity and a larger CPU capacity.

To provide more information, here are the more detailed comparisons between the SuperMicro E200 and the Intel NUC.

ESXi 6.5 Compatible

Both the NUC and the SuperMicro require additional drivers and configurations to the native ESXi installation in order to work properly, so this point is more for information rather than defining the obvious choice. The 1Gbe NICs on the SuperMicro work with the native ESXi drivers and will work out-of-the-box, you need to install additional drivers to get the 10Gbe NICs to work. The 10Gbe drivers are supported with ESXi 6.0 and can be found on the VMware Downloads page here: https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI60-INTEL-IXGBE-451&productId=491

The NUC requires additional drivers in order to get the 1Gbe NIC to work. This means that you will need to either build a custom image or install the drivers locally after ESXi is installed. William Lam has detailed the options and procedures here: http://www.virtuallyghetto.com/2017/02/update-on-intel-nuc-7th-gen-kaby-lake-esxi-6-x.html

While you are creating your custom ESXi image for your NUC or SuperMicro I would recommend you remove the native vmw-ahci driver vib from your image. This will force your storage controller to use the newer sata-ahci drivers. ESXi 6.5 contains many new drivers, however the standard image still contains both the native drivers and a newer version. VMware don’t chose to default to the use of the newer version of the drivers because they may not be 100% feature comparable. In this case, if you review the storage controller support on the VMware compatibility list, it clearly states that it is supported with the sata-ahci drivers. The native vmw-achi drivers do not perform well and you will recognise a massive performance improvement by using the new drivers. Anthony Spiteri has done an excellent job detailing the issues and resolution here: http://anthonyspiteri.net/homelab-supermicro-5020d-tnt4-storage-driver-performance-issues-and-fix/ 

Networking

The SuperMicro has a significantly better networking capability than the Intel NUC. Looking at the Intel NUC you could probably use Thunderbolt to Ethernet adapters or USB to Ethernet adapters and build yourself a redundant networking capability. At the end of the day, the SuperMicro has 2x 10Gbe NICs and 2x 1Gbe NICs. Because I want to run an all flash VSAN configuration, I want to use the 10Gbe networking capability to optimise my VSAN performance.

During my 10Gbe vs 1Gbe networking considerations, I also considered the SuperMicro E300-8D due to its 10Gb SFP configuration (and expansion PCIe slot). It is very hard to find a 10GBase-T network switch at a reasonable price and I ended up having to spend the most money on my 10GBase-T 48 port Dell switch. In hindsight, the SuperMicro E300-8D could have been a viable option because I would have been able to run a VSAN supported storage controller and a 10Gb SFP switch is much easier to find at a reasonable price. Of course there is no comparison to the Intel NUC because it doesn’t have 10Gb NICs, let alone multiple 10Gb NICs. I eventually made the decision that a 10Gb networking capability would not only provide me with the performance but also scalability and not need to be replaced in a couple years time.

Storage

The Intel NUC and SuperMicro both have similar storage capabilities. If you are wanting to run VSAN then you will need to buy an NVMe card for the caching tier and a 2.5″ HDD (or SSD) for the capacity tier. The SuperMicro has more SATA ports on the motherboard but no space to mount any additional drives, so not much point in even considering them.

The one thing that you can seriously consider here is to look at the SuperMicro E300-8D. The E300 has a smaller CPU (XEON D-1518 4 core 2.2GHz) but it is larger in size due to a PCIe x8 slot. This would be a great place to use a VSAN supported storage controller!

IPMI

I love that the SuperMicro has a built in IPMI port. This allows me to view a console screen or mount an ISO over the network. To put it simply, I don’t need to go out to my garage to manage my lab.

Noise

Yes, this is where the Intel NUC wins. The NUC doesn’t have any fans and therefore doesn’t make any noise. You could put these inside your house and you wouldn’t know they’re there. In comparison, the SuperMicro could be considered to be quite loud. This wasn’t an issue for me because my lab is in my garage, plus once you turn on my 10Gb 48 port switch, you can disregard any noise that the SuperMicro might be making. Did I really want to sacrifice my cooling capability and running temperature in order to reduce the noise? No, I want fans pushing as much air through my lab to keep it cool and a bit of noise is worth it. In fact, the SuperMicro comes with 2x 40mm fans and a spare slot for a 3rd fan which I promptly populated straight away.

Take a look at Paul Braren’s blog at TinkerTry where he analyses the noise from the SuperMicro servers – https://tinkertry.com/supermicro-superserver-sys-e200-8d-and-sys-e300-are-here

Power Consumption

One of my most critical requirements was lower power consumption. I have had some pretty high electricity bills in the past while running large rack mount servers. I haven’t measured the actual power consumption from between the Intel NUC and SuperMicro however I would be very confident that the Intel NUC would consume less power. Both units are very low in power consumption compared to a large rack mount server so they both meet my requirement.

Rack Mount

This was a big bonus. The SuperMicro E200-8D has rack mount brackets. Not only does this make my lab neat and tidy, its also easily expandable and I can also build in a hot/cold zone within my rack. Where I live it can get very hot in summer (40 degrees celsius or 100 degrees fahrenheit) so keeping my lab cool is a must. By using rack mount panels I have been able to separate the front fan intake on the SuperMicros to the rear hot outlets. I can then duct cold air into the front of the rack and keep my lab operating temperature at a respectable level. If I were to use the Intel NUCs then I would have no way of keeping them cool during Summer.

The below diagrams show the rack mount configuration and part numbers for both the E200-8D (MCP-290-10110-0B) and the E300-8D (MCP-290-30002-0B). Although I could not find these listed for sale anywhere, Eric at MITXPC was able to source the rack mount brackets for me.

Video Port

This might seem simple, however do you have a HDMI capable monitor in your home lab? I don’t. I’m using a fairly old monitor with VGA and DVI ports. The Intel NUC may be 4k capable and offer a HDMI port, which would be great for a media PC but why would you need this in your home lab? If the Intel NUC also had a VGA port then it may be comparable but it only offers a HDMI port. The SuperMicro VGA port also comes in handy when you turn up at a customer site and they don’t have a HDMI port.

USB Capacity

This is a down side for the SuperMicro as it only has 2x USB ports. Because I am running VSAN, both my internal NVMe and SSD can’t be used to boot ESXi, so I use a small USB drive which runs ESXi. While installing ESXi to the SuperMicros I found myself with a lack of USB ports. One is required for a bootable USB drive to install ESXi to, the ESXi install media on another USB drive and a USB keyboard to click through the install. 3x USB ports would have been nice but I mounted the ESXi image over the IPMI connection and clicked through the install process from the comfort of my lounge room, using the IPMI console screen.

This is where the Intel NUC does provide a Micro SDXC slot which you could very well utilise as the ESXi install location. The NUC also has an additional 4x USB ports.

SATADOM

The Supermicro SATA DOM (Disk on Module), is a small SATA3 (6Gb/s) flash memory module designed to be inserted into a SATA connector and provides high performance solid state storage capacity that simulates a hard disk drive (HDD). The SATADOM device could be used as a boot disk for the ESXi installation rather than a bootable USB and are available in 128GB, 64GB, 32GB, and 16GB sizes.

Screen Shot 2017-04-05 at 8.50.09 AM.png

CPU and RAM

The biggest consideration here is performance, capacity and availability, all of which the SuperMicro exceeds the Intel NUC in leaps and bounds. This makes the cost of the SuperMicro look cheap when compared to the Intel NUC based on the numbers, this is detailed below. At a high level the SuperMicro can use either ECC or non-ECC RAM, it uses full sized RAM slots rather than SODIMM, the RAM capacity is 4x larger at 128GB and the CPU capacity is nearly doubled. This makes the cost of the SuperMicro a lot cheaper than the Intel NUC once you start to consider purchasing more than a single unit.

The RAM capacity is a massive point in favour of the SuperMicro’s. This is incredibly important for the consolidation ratio of VMs to Hosts, especially when running all flash VSAN. You must remember to take into consideration that VSAN will consume a large chunk of your RAM. For ease of calculations I will use 10GB as my VSAN memory consumption, however the actual number was 10.5GB. Details of how to calculate your VSAN memory requirements can be found here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113954

Let’s assume you opt for the Intel NUC with a maximum of 32GB RAM. You instantly lose 10GB to VSAN, so you’re left with 22GB RAM to use in your environment. Each Intel NUC will provide you with 7GHz of CPU processing power and 22GB of RAM. This leaves you with 3.14GB RAM for each 1Ghz of CPU used. From my previous analysis of my home lab my VMs average between 200MHz and 500MHz CPU usage. I will use 500Mhz (0.5GHz) for my calculations and a conservative estimate of my consolidation ratio.

7GHz / 0.5GHz = 14 VMs per NUC

I have approximately 65 VMs in my home lab and this would mean that I require 5x Intel NUCs just to meet my current capacity requirements with a consolidation ratio of 14 VMs per host. What’s worse is that I would be highly unlikely to actually get 14 VMs on each Intel NUC because I only have 22GB RAM available to use. Each of the 14 VMs would have approximately 1.5GB RAM allocated.

22GB RAM / 14 VMs = 1.57GB RAM per VM

Based on the above calculations the RAM is a massive constraint on the use of Intel NUCs in a home lab environment. Realistically, based on RAM consumption I would need 9 Intel NUCs in my lab. I have used an estimated 3GB RAM required per VM for the below calculations.

(3GB RAM per VM x 65 VMs) / 22GB RAM per NUC = 8.86 (9) NUCs

Each SuperMicro E200-8D has 11.4GHz CPU processing power and 128GB RAM (less 10GB for VSAN). Applying the same calculations as above.

11.4GHz / 0.5GHz = 22.8 VMs per SuperMicro

118GB RAM / 22.8 VMs = 5.18GB RAM per VM per SuperMicro

As you can see from the above calculation, with 128GB RAM in the SuperMicro the CPU becomes the constraining factor. Leaving 5.18GB of RAM for each VM using 0.5GHz of CPU. This is fairly consistent in a VMware environment where RAM is more heaving utilised than CPU, so you would be better off ensuring you have more RAM than CPU.

Let’s work out what my consolidation ratio will be based on my actual RAM requirements of 3GB RAM per VM.

(3GB RAM per VM x 65 VMs) / 118GB RAM per SuperMicro = 1.65 (2) SuperMicros

The calculations make it very obvious from anyones perspective. To suit my needs I need 2x SuperMicros or 9x Intel NUCs. I could have stuck with the 2x SuperMicros and setup a 2 node VSAN configuration utilising the virtual witness appliance as the 3rd node, however I want to make this enterprise ready so I opted to meet the minimum of 3 nodes for VSAN.

If you factor in the costs for 128GB of ECC RAM then it can get more expensive. Because I was going to buy 3x SuperMicro servers anyway (for VSAN) then why not be more price conscious and use 64GB non-ECC RAM per server. This meant that my lab was more realistically sized with 3 servers at 64GB RAM in each = 192GB RAM and the cost was $1499 per SuperMicro E200-8D (including disks).

Price

From all of the above details if it isn’t already obvious why I opted to build my lab with SuperMicro E200 servers, do the math on the cost.

9x Intel NUCs x $630 each = $5,670 USD

3x SuperMicros x $799 each = $2,397 USD

The costs I have used above are estimates based on a quick search, you may find cheaper prices if you look harder. I haven’t factored in the cost of the RAM, SSD or NVMe as these would be similar additional costs regardless of choosing the NUC or SuperMicro. There are other considerations that may affect the cost comparison of each unit, just to list a couple:

  • The supported SSD and NVMe cards could warrant a difference in price.
  • The RAM costs could vary between choosing to use SODIMM or ECC RAM.
  • The rack mount brackets on the SuperMicro servers are an additional cost.
  • The SuperMicros could likely consume more power during daily use.

If you have read all the way to the end, you are obviously just as interested in getting the “right” configuration for your home lab as I was. Seriously though, if at the end of this article you’re still leaning towards the NUC then just do it. You’re not going to be disappointed.

I purchased my SuperMicro servers from MITXPC as they specialise in micro systems. I found them via Amazon, however the prices on Amazon are significantly more expensive than if you go direct. If you’re interested, ask to speak with Eric Yui as he has been very helpful for me and will look after you.

Screen Shot 2017-04-02 at 10.52.35 AMScreen Shot 2017-04-02 at 10.52.49 AM

Home Lab Build Series

IntroductionHome Lab Build – From the Start

Part 1SuperMicro vs Intel NUC

Part 2 – SuperMicro Build – The Components

Part 3SuperMicro Build – The Installation

Part 4SuperMicro Build – BIOS and IPMI

Part 5 – Networking Configuration

Part 6 – VVD – Automated Deployment Toolkit – vSphere, VSAN, NSX and vDP

Part 7 –  VVD – Automated Deployment Toolkit – vRA, vRO, vROps and Log Insight


PreviousHome Lab Build – From the StartNextSuperMicro Build – The Components

28 thoughts on “SuperMicro vs Intel NUC

    • Hi Rodrigo,
      I had a look back at my original invoice and the following were the costs.

      SuperMicro E200-8D – $799
      Additional Fan – $10
      64GB UDIMM RAM – $330
      128GB M.2 SSD – $60
      1TB SSD – $300

      TOTAL = $1499 per unit

      Like

      • I have to agree with you, the 10Gb networking costs can be very expensive. The E300-8D is an excellent option for cheaper 10Gb networking using SFP. 10Gbase-T switches can be found cheaply, however the there are only between 8 and 12 port models generally available. My switch was very expensive, however for a home lab setup I would assign at least US$1,000 to a new switch and a used one can be found for around US$500. My switch was over $2,000 (used) but I have other use cases for this specific switch that I wanted to accomodate.

        Like

  1. Hi Tai
    Just a note from a E300-8D owner. Skip the usb as boot device and buy a SATADOM. Then you have two usb ports free and you will use one of the sata ports. SuperMicros SATADOM comes in capacity up 128GB and has good speed and ware-out. Which is good choise in a vSAN configuration.

    Best regards
    Michael Ryom

    Like

  2. Pingback: Newsletter: April 8, 2017 | Notes from MWhite

  3. Pingback: Projects, Distractions and Constructive Learning – Vraxx — The Long Road

  4. Pingback: Home Lab build – From the start. | Lab-Rat

  5. Hi Tai,

    Terrific article.

    Have newbie question regarding “34.2GHz / 0.5GHz = 68.4 VMs per SuperMicro” CPU speed. Assuming total speed is derived from 3 Node Cluster? Is this correct?

    Thanks Much!

    Michael

    Like

    • Thanks Michael, you’ve picked up a mistake there. That figure was based on 3x servers and was meant to be based on 1x (11.4 GHz) . It actually makes the SuperMicro look even better because you’ll now get 3x more RAM per each CPU GHz!
      That calculation isn’t important in the rest of the article because RAM is the constraining factor.

      I’ll update the article to reflect the update. Thanks for the pick-up!

      Like

  6. Hiya,

    I am not sure how you calculated it based on the cost of the NUCs to be honest – the cost seems ‘off’ and therefore potentially gives a wrong result.

    32GB of RAM in the NUC are approx. $200 … A single 32GB dimm costs twice as much – Fully loaded the Supermicro costs $1600 on RAM alone, plus the Supermicro is approx. $850 (E200-8d) ..

    So for 128GB of RAM you spend $2400 for a single server…

    The NUC costs $250 plus $200 RAM = $450 for 32GB of RAM – So for the same price you get 5 NUCs and you have $150 spare .. And that is 5x32GB = 160GB of RAM .. prices are approx. of course – but you get the idea …for an additional $300 you get 6 NUCs and 192GB of RAM

    Granted – the CPU on the cheaper NUC are i3 compared to the Xeon ones .. but in all deployments in my 3-NUC lab, with NSX, vRA, vCD and stuff, I always run out of Ram or IO way before i run out of CPU.

    So just by using the expensive part of the lab – the RAM – I think you get more bang for your buck .. if CPU is an issue – then yea, maybe not – but it still won’t be cheaper to use SM I don’t think.

    Like

    • First of all, I will note that you are using prices for the i3 NUC and not the i7, so of course its going to be cheaper. If you want to start reducing the capability of the servers to the lower models then SuperMicro also offer lower spec’d units from $350. For the purpose of your comment, let’s compare the costs of the low spec NUC 7th Gen i3 and the high spec E200-8D.

      My SuperMicro’s cost me $1,499 in total per server with 64GB RAM. The cheapest NUC I can find with 32GB RAM and a similar configuration with NVMe and 2.5″ SSD, is $870 (7th Gen i3, 128GB NVme, 1TB SSD and 32GB RAM). I bought my SuperMicro from MITXPC and they appear to have raised slightly in cost from when I bought them… but that is to be expected. I can’t keep updating the article every time a price changes 🙂 The new price is reflected in the calculation below at $1589 per unit.

      If I need 190GB of RAM in my lab then I can use 3x SuperMicro’s (with 64GB RAM in each) at $1589 per unit = $4767
      http://www.mitxpc.com/proddetail.php?prod=SYS-E200-8D

      For the same capacity I will need 6x NUCs (with 32GB RAM in each) at $870 per unit = $5220
      http://www.mitxpc.com/proddetail.php?prod=BOXNUC7i3BNH

      Using the calculations based on your scenario, the SuperMicro’s are still a cheaper option by over $400. Now, consider using VSAN. Don’t forget about the loss of 10GB RAM to VSAN. How many additional NUCs will you need if you can only use 22GB RAM? Look at my calculations above, instead of using 6 NUCs you’ll need 9. Even with 64GB RAM in the SuperMicro’s you’ll only need 4 of them. No matter how you look at it, as you scale out the SuperMicro’s get cheaper and cheaper.

      You can calculate the costs however you like, the NUC’s are cheaper. Always have been and always will be. The RAM is just one factor in the comparison. In my opinion, even if the NUCs could serve the same amount of RAM and were the same total cost as the SuperMicro, they are still out-gunned in just about every other way that I’ve outlined above.

      The NUCs are still a great option. There’s nothing wrong with them at all. I highly recommend that everyone who is looking to build their home lab, to first start with writing down exactly what they’re requirements and priorities are. You may end up choosing the NUCs for other reasons.

      Like

    • One of my requirements was to be portable and low power consumption. The rack mount brackets take minutes to remove and means that I can take a small server with me to customers and conventions for demonstration purposes. I couldn’t possibly drag 3x 1RU servers around with me. I also couldn’t find any 1RU rack mount servers that use a 60w power supply.
      Most people who want to run a home lab would prefer a small form factor server, not everyone has a rack to mount their hardware, but it is a great option if you need it. A small 19″ network cabinet is a great option to mount hardware like this. If you buy a small network cabinet to mount your hardware with rack mounts then you just won’t fit a 1RU full size server in it because of the length. Therefore, anyone who runs a 1RU server will need to also use a full size rack that is both 19″ wide and nearly 1m deep.

      If you only run a single node home lab and you need less than 32GB RAM, then I would 100% recommend the cheaper NUC. For anything larger, the SuperMicro’s become a lot more appealing and cost effective.

      Like

  7. Pingback: The NUC killer: SuperMicro SuperServer E300-8D - Michael Ryom

  8. Tai,
    When will be released the part 2 (Home Lab Design), I am eager to see as would like to build up the homelab using Supermicro SYS-5028D-TN4T Xeon D-1541 Mini-tower.

    Please keep up

    Like

    • I have all the pictures and configurations documented, it’s nearly ready. Unfortunately I’m travelling for work for a couple weeks so I don’t have much spare time. I plan on getting it all finished in the next few days, but no promises.

      Like

  9. Hello Tai

    Nicely written article !!

    I havent seen any reference that the Supermicro servers are Intel Optane ready . Do you have some link and test run for this ?

    Thanks

    Like

    • I have a Dell S4048T 10GbaseT switch. I’ll go into more detail on the networking side in another article. It’s an excellent switch that has all the capabilities I need to simulate an enterprise leaf and spine network. That makes it much easier when I’ll be deploying the VMWare Validated Design with NSX.

      Like

  10. Pingback: SuperMicro Build – The Hardware | Lab-Rat

  11. Pingback: SuperMicro Build – Installation | Lab-Rat

  12. Pingback: SuperMicro Build – BIOS and IPMI | Lab-Rat

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s