Last Modified: Sun 13 Jan, 2019 09:50 AM
Most people have a modem of some sort that offers an integrated access point. Everything from there on out is wireless; at best they have an old desktop PC hard-wired into it. Let's just say that's not how I roll...
My home network is pretty big for a home LAN. Lots of devices - most of which are wired - and lots of data sloshing around for various things. Every room has at least two ethernet drops, generally with 1 of them populated. Most everything is at least fast ethernet (though I do have something connected at 10mbit via an AUI adapter still), with gigabit being pretty typical. It all terminates in my mechanical room where I have a server cabinet that's almost full.
For the longest time I had been relying on two smallish switches, a Dell PowerConnect 2816, and a Linksys SRW2008. These aren't bad switches - fully managed, line rate gigabit, and most importantly passively cooled - but everytime I wanted to plug in something (or light up a jack in a room that I hadn't been using before) I had to find something to unplug. So I was faced with a problem where I needed more ports.
Another problem is that I could pretty easily saturate gigabit in a couple circumstances. I won't claim to have been doing this regularly, but it did happen - and well, you can see where this is going: why not get some 10gig?
At first, I looked into getting at least another 24 ports of managed fast ethernet - but in 2016 (when the search began) I found that the market for quality fast ethernet just didn't make any sense. Either it cost too much, or it didn't have the features that I wanted. So I started to look at (new or used) gigabit switches with at least 1 or 2 10gbe uplinks. They're out there, but finding the right combination of things I cared about proved to be hard. I eventually settled on these requirements:
I looked and looked and couldn't find anything that really fit my needs. That is, until I saw this thread on the ServeTheHome forum covering the Netgear GSM7328S v2. I mulled it over a few days, and decided that I'd start watching eBay for deals. A few weeks later I got notified of a deal on such a switch with a Buy It Now option, and a few clicks later PayPal was facilitating the payment.
I eagerly awaited the arrival, but was disappointed when it arrived. While the auction pictured and described a GSM7328S v2 switch, what arrived was a v1 - still a decent switch, but not what I wanted as it didn't include the two integrated SFP+ ports, instead having 4 bays for optional proprietary modules.
To my pleasure though, when I notified the seller they simply refunded me my cash and told me to keep the switch. Apparently they didn't want to deal with it. I rewarded them with positive feedback. So I had a free switch - but looking around, the SFP+ option modules (AX743) were stupid expensive. Like, they sell used for around $300 - more than the cost of the switch! I'll note that this switch did come with two of the stacking modules (AX742) populated but with nothing to plug them into. So would I be happy with my free switch?...
Well, of course not - and when another deal on a switch (even cheaper this time) came by, I again jumped on it. This time it was for the switch that I ordered. So I did what any rational network engineer would do: I pulled one of the stacking modules out of the v1 and put it into the v2 and linked them up for a high bandwidth uplink - only that didn't work. My heart sunk. At this point, I turned to Google to save me though, and it led me to the products documentation and manual, where I learned that these two switches, though sharing the same model name, are actually quite different internally, and while I found hints that at one point they could be stacked, that it wasn't possible any longer due to the software diverging.
It was pretty important to me to run the latest code possible for each of them, but the v1 hadn't had an update in a few years, having gone out of support much earlier. The first thing I had done to both of them was update the firmware - so I knew that they were running different versions.
But then I noticed in the manual that when configuring the stacking ports you could specify an alternative mode for them. Rather than configuring them as stacking I could configure them as ethernet. Sure, this meant that I'd need to manage each switch individually and the speed went from 12gbps to 10gbps, but that sounded fine to me if it meant I could use the interfaces - and sure enough, after making that change I soon had a 10gbe link up between the two switches.
So now I had two 24 port gig switches with a high speed link between them, but I wasn't about to just toss them in the rack and start using them because I wasn't happy with how loud they were. While some had reported they were quiet they weren't quiet enough. Each of them had 5 20mm fans blowing as hard as they could, and they were honestly pretty distracting. Maybe in a datacenter they'd be on the quiet side, but they weren't going to work in my house. I figured out how to monitor the temperatures (show hardware on the v1, show environment on v2 - or snmpwalk against .1.3.6.1.4.1.4526.10.43.1.8.1) and set about determining whether the fans were strictly necessary and figured out pretty quickly that they are. Without the fans running they'd climb to 70° C pretty quickly - not sure how far they'd go, whereas with the fans they'd settle in at around 50° or less.
I also made use of my IR thermometer to probe around for hot spots and determined which parts really needed cooling. I found on each where the air had to flow, and then looked at how I could position bigger fans to move enough air in a quieter manner. Through a combination of installing quieter 20mm fans (but that admittedly move much less air), and cutting holes into the top of the case at a few strategic places and mounting 120mm fans, I was able to dramatically reduce the noise, while also making them run cooler. The only downside now is that I had to sacrifice the rack unit above each switch, but I was doing that anyways for cable management, so it ended up working out well enough. They don't really even look that bad to be honest...
Well, there was one other downside in that the slower fans constantly cause the devices to issue alerts about fan failures (even though they're working fine). This is annoying, but I configured the devices to not send traps for those alarms.
Oh, and one more note here: Netgear uses a nonstandard pinout for their fans. While they accept a standard 3 pin 12V fan, you have to change the order of the pins. If you go to do this, pay attention to the order of the wires on the stock fans (they'll have red for hot, black for ground, and another for sensor data to convey how fast they're spinning. The colors on your new fan should have a red and a black, though the third might differ - but get the red and black right and then use process of elimination to figure out where the last one goes. It took me a sec to figure out how to remove the pins from the connector - so don't just tug on them. Instead, use a needle to push the contacts in one at a time which should allow them to pull out easily and reinsert in the right order without mangling anything.
If you want to utilize the SFP+ interfaces on a v2, you'll need to make sure that you get compatible optics. From what I've read, Netgear does the (hated) industry standard action of checking the manufacturer of any SFP+ modules that are used and refusing to work if they're not on their white list. Well, The Fiberstore offers affordable (but I assume counterfeit) transceivers that work fine. I don't know if there are any direct-attach cables that work - if you do, please let me know!
I've got the two SFP+ interfaces on the v2 switch wired up to a pair of servers each with Mellanox ConnectX-2 adapters that are available pretty cheaply these days. They're fast!
I also realized when I found out how to configure the stacking ports as ethernet that these are really almost certainly just CX4 interfaces. Now, I haven't actually plugged them into a PC with a CX4 10gbe card, but I'm 99% certain that they'll work, so if you find yourself with one of these and need another 10gbe port or two and don't want to pay for the SFP+ option (or just have a v1 switch), you can still probably make it work if you get a CX4 NIC. The stacking ports are generally a lot cheaper than the SFP+ options. I'll probably be doing just that at some point.
Also, I discovered the existence of the Dell PowerConnect 5500 series switches after I had already gotten my stuff. These apparently allow you to stack them using HDMI cables - which as messed up as that sounds, apparently works! They don't seem to be too expensive either - and without needing to buy modules for stacking should save a few bucks, though it does mean you can't use the stacking ports as regular interfaces should you want to do that.
I also found a better solution (but more expensive) in the tp-link T2700G-28TQ. While even less reputable than Netgear, these are apparently passively cooled and so silent, and offer a similar feature set to what I have now. Since they're new though, they cost more - but had I known about this when I started out I might have gone this route anyways.
Last Modified: Sun 13 Jan, 2019 09:50 AM