For a long time, I have been using Mellanox 10G SFP+ 10G networking cards between my main PC and my NAS and they have been flawless. The problem is that it restricts my video card to 8x mode. The performance drop is incredibly minor, but it is something that has always bothered me. I recently updated my PC, and an AQC108 10G ethernet port is included on-board. Hooray! No more 8x limitation!
I proceeded with obtaining an Aquantia AQN-107 card, since they are the most reasonably priced option out there. My PC is running Windows 10, while my NAS runs Linux. Unfortunately, I was seeing some very odd behavior. Copying files to my NAS seemed to work great. However, copying files from my NAS would lead to corruption. How this is even possible, I’m not sure — one would think there would be some sort of error checking in the SMB protocol. This is a direct connection and there isn’t a switch involved.
I ran through various tests, using the Intel I219-V (which have always been flawless) and sure enough, there are no issues using 1G networking.
Obviously, I could go and get Intel 10G NICs for 3x the price and probably solve the issue, but it would bring me back to the whole 8x problem. I may be able to install an Intel card on my NAS (using mixed Aquantia-Intel) and solve the problem that way.
It was my thought, that there may be an incompatibility with the drivers and their features though. The Linux driver seems to be quite a ways behind the ones available for Windows in feature set. I proceeded to turn off all options for the Windows card and do a compare. I was using fc to compare the original with what was saved on the NAS. Sure enough, no problems arose! Thankfully, the corruption appears in a few seconds, so testing each option wasn’t a very lengthy procedure.
I eventually came across the culprit: Recv Segment Coalescing
Disabling this option and leaving all the other features (such as checksum offloading, etc) seems to have solved the issue. I copied and did a compare of almost 7 TB of data and no issues were found. I plan on sending a note to Aquantia to see if this is a known issue and I will update when I hear back from them.
I hope to save someone else a lot of time and aggravation with these findings and thankfully the corruption was one-way. It could have been a lot worse since most of my data flow is also copying data to the NAS, and not the other way around. Still, silent corruption is unacceptable as far as Windows is concerned. I really wish it would have raised an error instead of reporting success.
I’ve tried the newest drivers from Aquantia (2.1.17, 03/19/2019) and the same problem still occurs. I have installed an Intel 10G X550-T1 NIC on my NAS and that did not solve the issue either.