Category: Internet

TCP Offloading

I try not to get too techy in my blog here most of the time, so I apologize in advance for the density of the material.

I've got a few aging production servers that I'm responsible for (Windows 2008-era). Over the past year and a half, I've been having intermittent connectivity problems. Tons of exceptions are being thrown that indicate the server is just...dropping connections randomly. Dropping connections to the client, the SQL Server, or really anything larger than a tiny networking blip. It's not constant enough to warrant a "drop everything and figure this out" response, so it's just been this constantly irritating background noise to more pressing software development concerns.

To add, the error message read "The client disconnected. Invalid viewstate." We're not even using .Net's viewstate that much, so I was understandably confused. I made a few half-hearted attempts to track down the issue, but kept coming up empty.

After finally getting tired of the error and reading a several articles on the subject (also glossing over quite a bit of poking at the code), there seemed to be an issue with Broadcom gigabit NIC cards utilizing the TCP Offload Engine. This nifty feature is supposed to offload checksum calculation and some other TCP stack heavy lifting to the operating system in order to improve networking performance. The recommendation was to disable it entirely.

disable-tcp-offloadAll our production servers have Broadcom ethernet cards, so after disabling it on all our production systems, our dropped connection issues stopped completely. Fast-forward to several months later, and I'm debugging a problem wherein an application crashes when trying to download a large block of data from a remote SQL Server system, and getting the error: "A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired." Pretty obscure.

This system in particular is a virtual machine running in a Hyper-V server with a physical Intel card, so I dismissed that possibility and focused on the code. After losing a day running down that rabbit hole, I discovered from the "Troubleshooting Common Hyper-V Errors Part 3" article that you can, indeed, be using a TCP offloading engine without a custom driver.

The article recommends disabling only Large Send Offload, but I'm really sick of this issue. Rather than taking a stepwise approach to solving the problem, I've just gone ahead and disabled offloading for all servers that are pre-2012 since Microsoft disabled the feature in 2008 R2 onward for gigabit adapters (presumably because it's so problematic). If networking speed is compromised as a result, I'll upgrade the hardware in question.

I hope I can save someone else the time figuring this issue out.

Comments

Referrer Spam Trick

Just a quick post here, since there's not too much to say about this one.

I received the comment below on one of the blogs I administer:

Hi, I left you a DOFOLLOW backlink on my website. This isnt a spam message, i actually did leave you a backlink on my site. If you check the top of the page you will see “Sites we like” and there will be a link to this site. Would you be kind enough to leave me a backlink? If so my website is http://[SCREW YOU, SPAMMER].com please use the anchor text “[I AM A STUPID SPAMMER]” for the link and add it to a post or as a widget. Then please send me a email at backlink@[NOT A CHANCE].com – If you want me to change your links anchor text let me know. Thanks

Since I'm the curious type, I decided to visit the site. Imagine my surprise when the backlink is actually there!

Wow, really?!

Being that the top of the home page is an extremely unlikely place for a backlink, I entered the URL directly. As expected, the link disappeared.

So, it's checking the referrer, and displaying the link to one of the sites spammed when the correct referrer is found. Not a new trick, but pretty clever.

Comments