TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

FTP Server, File Server, DDNS, SAMBA, AFP, NFS
HiTach
Starting out
Posts: 31
Joined: Fri Feb 15, 2019 6:03 am

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by HiTach » Fri Feb 07, 2020 2:28 am

I received a welcome proactive message today from Qnap support.
It's nice to know they are thinking of you...

"We don't have an internal fix to the Thunderbolt issue yet - again, because it seems to be a Windows build problem, but there seems to be a new "Windows 10 Insider Preview 1954" version that has come out, and so far in our internal testing, the behavior with Thunderbolt has seemed good.

It's not officially fixed yet, but if you're still having the same issue, please go ahead and update to this version, and let me know if you're still having issues with the Thunderbolt issues. Thanks."

When I get time I'll try this out. If you get there before me, please post results.
Good luck!
Workstation DAS/NAS: TVS-872X(T) *Thunderbolt card removed due to Windows10 1909 bug
8x10TB WD Gold RAID6 54TB static volume
2x1TB Samsung 970 EVO plus R/W RAID1 cache w/EK WB M.2 heatsinks + 2x NB 12-P fans
1x40Gbe Mellanox DAS network connection, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
Backup NAS: TVS-872XT
8x10TB WD Gold RAID6 54TB Thick volume w/snapshots
1x40Gbe Qnap Thunderbolt 3 card, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
LAN switch 10Gbe(2)/1Gbe(8) NETGEAR Nighthawk SX10 (GS810EMX)

doodlewalker
New here
Posts: 9
Joined: Tue Sep 17, 2019 2:52 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by doodlewalker » Fri Feb 21, 2020 4:41 pm

I have briefly tried the latest insider preview on my laptop (Dell XPS 15), does not seem to be fixed and could actually be worse. I could only get 60MBs from the QNAP to the PC (previously ~2GB). PC to QNAP same issue i.e does not transfer at all.

I have noticed in my testing on 1909 that I can transfer from QNAP to PC at expected speeds but as soon as I try to transfer from PC to QNAP it fails as documented. From then on I can no longer get expected speeds on the QNAP to PC transfer, it just sits at ~60MBs until I restart the QNAP, which is a bit worrying.

metaldave
Starting out
Posts: 40
Joined: Wed Sep 19, 2018 7:20 am

Re: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by metaldave » Sat Feb 22, 2020 12:45 am

doodlewalker wrote:
Fri Feb 21, 2020 4:41 pm
I have briefly tried the latest insider preview on my laptop (Dell XPS 15), does not seem to be fixed and could actually be worse. I could only get 60MBs from the QNAP to the PC (previously ~2GB). PC to QNAP same issue i.e does not transfer at all.

I have noticed in my testing on 1909 that I can transfer from QNAP to PC at expected speeds but as soon as I try to transfer from PC to QNAP it fails as documented. From then on I can no longer get expected speeds on the QNAP to PC transfer, it just sits at ~60MBs until I restart the QNAP, which is a bit worrying.
I suspect that Thunderbolt Networking is a TCP/IP-based protocol and, as such, is beholden to packet size "windowing" (wherein the packet size will grow with every packet confirmed transferred between source and destination). To point, I experience a variety of transfer throughput rates depending upon the number of files transferred, size of each file, etc. Also, the processor of the unit makes a significant difference as to what the peak rate may be. Lesser units (like my TS-453BT3) will run at far less than the theoretical line rate of TB3 because of the platform specifications, but higher end processors will do the job better.

In my experience, my download throughput (from QNAP to client) starts around 400 MB/s (3.2 Gbps) for large, single file transfers, but it slows and fluctuates around the area of 160 MB/s (1.2 Gbps). The upload throughput does just about the same, and it fluctuates quite a bit as well. I suspect the combination of drive configuration (SSD versus mechanical, SSD caching, etc.), filesystem (thin/thick/static), and processor would impact. My guess is the interface is so fast (TB3 is a fast-lane connection) that packets arrive out of sequence. When this becomes untenable for the buffer, TCP protocol will throttle back (reduce) the packet size and request retransmission (thus, reducing the throughput). I feel like this could be improved with proper queue buffers, but that's a topic for a different thread.

I have also done some side-by-side comparisons with transfers via the actual Ethernet network and the TB3 Thunderbolt Networking connection. I have to say, the Ethernet performs much more consistently on the 1 Gbps network. This seems limited to about ~95 MB/s (760 Mbps) on download and ~110 MB/s (880 Mbps) on upload, but, strikingly, it's consistent and does not fluctuate. My unit has a 10 Gbps Ethernet interface as well, and I suspect it would perform consistently as well. Unfortunately, I do not have a 10 Gbps Ethernet client to test this hypothesis.

All of that said, this is way better than not transferring data at all (which is what the later Windows 10 builds seemed to do to our Thunderbolt Networking uploads to the QNAP). Once we fail there, it seems like the downloads don't work well either until reboot (almost as though the machine becomes biased toward that connection). I'm sure there's queues and buffers on both sides of the transaction, and I suspect Windows 10 is bursting the bucket upon initiation of the transfer and doesn't know what to do from there.

I'm curious to know if the Linux file transfers have the same issue with fluctuation via Thunderbolt Networking or if it's steady-on like the Ethernet file transfers.

JAKEMON
New here
Posts: 3
Joined: Sat Jan 25, 2020 11:48 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by JAKEMON » Sun Mar 01, 2020 7:53 am

Like many of you. I purchased the QNAP TVS-672XT NAS with all the bells and whistles in December 2019. 32G Ram + (2) 1TB SAMSUNG NVME SSD 970 EVO, Etc. I spent many hours familiarizing, configuration, and testing of this device. I was using a MSI GS65 Gaming Laptop and everything seemed like I was entering the 10G Era. after running a battery of test to verify real world operation I ran into this Issue. I created a Support Ticket with QNAP and the support was not helpful at all!!!! I did not realize until after the purchase that the Updated Versions of Windows 10 were having issues with Thunderbolt 3 Networking particularly with Writing to the NAS; (Read Works Fantastic).. QNAP Claims they have reached out to Microsoft about the issue but Microsoft has not fixed the Issue. I ended up returning the Unit due to the Lack of Support.. Sadly I purchased a NETGEAR READYNAS 626X that is not even close in terms of functionality.
Perhaps Microsoft is in agreement with Apple to get more users to purchase MACBOOK PRO's :mrgreen:

QNAP Please fix Issue and I will return the NETGEAR !

savethebologna
New here
Posts: 3
Joined: Wed Nov 13, 2019 9:15 am

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by savethebologna » Wed Mar 04, 2020 2:41 am

This weekend, our video editing rig forced its way off 1809 and up to 1903 - and after another rollback, doesn't seem to want to let me stay on 1809. This was a huge blow to our productivity since I wasn't in the office. I'll be contacting Qnap and Microsoft this week, and I've instructed my team not to restart the computer, but this is ridiculous.

This is mostly blowing off some steam, but also a little warning/update that it seems we can no longer prevent that update and it may not be much longer for you :(

I'll post an update if I hear anything that hasn't already been said.

doodlewalker
New here
Posts: 9
Joined: Tue Sep 17, 2019 2:52 pm

Re: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by doodlewalker » Thu Mar 05, 2020 7:03 pm

metaldave wrote:
Sat Feb 22, 2020 12:45 am
doodlewalker wrote:
Fri Feb 21, 2020 4:41 pm
I have briefly tried the latest insider preview on my laptop (Dell XPS 15), does not seem to be fixed and could actually be worse. I could only get 60MBs from the QNAP to the PC (previously ~2GB). PC to QNAP same issue i.e does not transfer at all.

I have noticed in my testing on 1909 that I can transfer from QNAP to PC at expected speeds but as soon as I try to transfer from PC to QNAP it fails as documented. From then on I can no longer get expected speeds on the QNAP to PC transfer, it just sits at ~60MBs until I restart the QNAP, which is a bit worrying.
I suspect that Thunderbolt Networking is a TCP/IP-based protocol and, as such, is beholden to packet size "windowing" (wherein the packet size will grow with every packet confirmed transferred between source and destination). To point, I experience a variety of transfer throughput rates depending upon the number of files transferred, size of each file, etc. Also, the processor of the unit makes a significant difference as to what the peak rate may be. Lesser units (like my TS-453BT3) will run at far less than the theoretical line rate of TB3 because of the platform specifications, but higher end processors will do the job better.

In my experience, my download throughput (from QNAP to client) starts around 400 MB/s (3.2 Gbps) for large, single file transfers, but it slows and fluctuates around the area of 160 MB/s (1.2 Gbps). The upload throughput does just about the same, and it fluctuates quite a bit as well. I suspect the combination of drive configuration (SSD versus mechanical, SSD caching, etc.), filesystem (thin/thick/static), and processor would impact. My guess is the interface is so fast (TB3 is a fast-lane connection) that packets arrive out of sequence. When this becomes untenable for the buffer, TCP protocol will throttle back (reduce) the packet size and request retransmission (thus, reducing the throughput). I feel like this could be improved with proper queue buffers, but that's a topic for a different thread.

I have also done some side-by-side comparisons with transfers via the actual Ethernet network and the TB3 Thunderbolt Networking connection. I have to say, the Ethernet performs much more consistently on the 1 Gbps network. This seems limited to about ~95 MB/s (760 Mbps) on download and ~110 MB/s (880 Mbps) on upload, but, strikingly, it's consistent and does not fluctuate. My unit has a 10 Gbps Ethernet interface as well, and I suspect it would perform consistently as well. Unfortunately, I do not have a 10 Gbps Ethernet client to test this hypothesis.

All of that said, this is way better than not transferring data at all (which is what the later Windows 10 builds seemed to do to our Thunderbolt Networking uploads to the QNAP). Once we fail there, it seems like the downloads don't work well either until reboot (almost as though the machine becomes biased toward that connection). I'm sure there's queues and buffers on both sides of the transaction, and I suspect Windows 10 is bursting the bucket upon initiation of the transfer and doesn't know what to do from there.

I'm curious to know if the Linux file transfers have the same issue with fluctuation via Thunderbolt Networking or if it's steady-on like the Ethernet file transfers.
Hi metaldave,

Apologies my results post wasn't clear, I'm running a 472xt with 2GB NVME used as testing (so no real drive bottlenecks) and get following results transferring a single 4GB iso file

Thunderbolt 3 tests to high spec Dell XPS 15 with fast NVME SSD.
Windows 1909 - Fails to copy to QNAP, but copy from QNAP initially fast ~2GBs, but any subsequent copies seems to sit around 60MBs until I restart the QNAP, then I get a fast QNAP to PC (2GBs) again for the first transfer.
Windows Insider Preview - Exactly the same as 1909, except I never get the initial "fast" speeds from the QNAP to the PC, just get 60MBs. So the insider preview in my tests is actually worse.

I also have tried this on a NUC8 with same results (1909). That NUC now runs proxmox (linux) and I get pretty consistent speeds of 1.8-2GBs with single 4GB file transfer in both directions.

My main workstation PC has a direct connection to the Qnap over 10Gb, and I get about 1.1GB when transferring large file. I also have the 2x 1Gbs Ethernet's teamed, and from any 1Gbs device I get around ~100MBs as you'd expect.

So in my tests I don't see what QNAP is claiming that the Insider preview seems to sort the issue.

savethebologna
New here
Posts: 3
Joined: Wed Nov 13, 2019 9:15 am

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by savethebologna » Thu Mar 05, 2020 11:42 pm

Minor update: chatted with Microsoft yesterday and got back that the issue is known and that an update to 1909 should be available "in a month" with the fix. I'm not super optimistic though if the Insiders should have the fix and don't.

JAKEMON
New here
Posts: 3
Joined: Sat Jan 25, 2020 11:48 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by JAKEMON » Sun Mar 22, 2020 4:29 am

JAKEMON wrote:
Sun Mar 01, 2020 7:53 am
Like many of you. I purchased the QNAP TVS-672XT NAS with all the bells and whistles in December 2019. 32G Ram + (2) 1TB SAMSUNG NVME SSD 970 EVO, Etc. I spent many hours familiarizing, configuration, and testing of this device. I was using a MSI GS65 Gaming Laptop and everything seemed like I was entering the 10G Era. after running a battery of test to verify real world operation I ran into this Issue. I created a Support Ticket with QNAP and the support was not helpful at all!!!! I did not realize until after the purchase that the Updated Versions of Windows 10 were having issues with Thunderbolt 3 Networking particularly with Writing to the NAS; (Read Works Fantastic).. QNAP Claims they have reached out to Microsoft about the issue but Microsoft has not fixed the Issue. I ended up returning the Unit due to the Lack of Support.. Sadly I purchased a NETGEAR READYNAS 626X that is not even close in terms of functionality.
Perhaps Microsoft is in agreement with Apple to get more users to purchase MACBOOK PRO's :mrgreen:

QNAP Please fix Issue and I will return the NETGEAR !
Update:
The Issue appears to have been a Hardware Problem with my MSI GS65 Laptop that has Plagued me with other issues! I purchased a LENOVO X1 CARBON WINDOWS 10 PRO 64 1903 and ran the same testing process with the NETGEAR 626X and did not have any Read/Write Issues. The MSI Laptop exhibited the Failure and the LENOVO X1 did not. It is sad that I bought a different NAS and Computer and the problem may have been primarily the MSI Computer. :shock:

HiTach
Starting out
Posts: 31
Joined: Fri Feb 15, 2019 6:03 am

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by HiTach » Sun Mar 22, 2020 4:34 am

Can you check your Windows version(s) against this chart?
https://en.wikipedia.org/wiki/Windows_1 ... on_history
Workstation DAS/NAS: TVS-872X(T) *Thunderbolt card removed due to Windows10 1909 bug
8x10TB WD Gold RAID6 54TB static volume
2x1TB Samsung 970 EVO plus R/W RAID1 cache w/EK WB M.2 heatsinks + 2x NB 12-P fans
1x40Gbe Mellanox DAS network connection, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
Backup NAS: TVS-872XT
8x10TB WD Gold RAID6 54TB Thick volume w/snapshots
1x40Gbe Qnap Thunderbolt 3 card, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
LAN switch 10Gbe(2)/1Gbe(8) NETGEAR Nighthawk SX10 (GS810EMX)

JAKEMON
New here
Posts: 3
Joined: Sat Jan 25, 2020 11:48 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by JAKEMON » Sun Mar 22, 2020 5:25 am

Yes .\ My Version was Windows 10 Pro 64 1903.

metaldave
Starting out
Posts: 40
Joined: Wed Sep 19, 2018 7:20 am

Re: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by metaldave » Mon Mar 23, 2020 6:50 am

Just checking-in with the thread. I have a QNAP support ticket I re-open about once a month to see if they're taking any action on their side. The answer is pretty much the same: it's a known issue with Windows 10, and their developers are trying to work with Microsoft on the issue. Unfortunately, that's just about as good as here-say as they cannot offer a case ID or offer anything to show for the effort. They also encourage the end users (us) to open cases with Microsoft, but, without being an enterprise customer, I don't think that's a great solution either.

I appreciate the report from @savethebologna, but that doesn't seem to definitive either. Users that want to leverage Thunderbolt Networking are, definitely, in the minority, so it seems the priority is such.

That said, the NAS works fine over Ethernet and, as @doodlewalker mentioned, you could get plenty of throughput with 10 Gbps Ethernet (if one were inclined to pick up a Thunderbolt 10 Gbps Ethernet adapter). At least those connections are reliable. If I had a critical application, I might just install Linux and go that route. My primary workflow was encoding videos, and I could, readily, do that via Linux.

Thanks for the input all!

doodlewalker
New here
Posts: 9
Joined: Tue Sep 17, 2019 2:52 pm

Re: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by doodlewalker » Sun Apr 05, 2020 6:03 pm

doodlewalker wrote:
Thu Mar 05, 2020 7:03 pm
metaldave wrote:
Sat Feb 22, 2020 12:45 am
doodlewalker wrote:
Fri Feb 21, 2020 4:41 pm
I have briefly tried the latest insider preview on my laptop (Dell XPS 15), does not seem to be fixed and could actually be worse. I could only get 60MBs from the QNAP to the PC (previously ~2GB). PC to QNAP same issue i.e does not transfer at all.

I have noticed in my testing on 1909 that I can transfer from QNAP to PC at expected speeds but as soon as I try to transfer from PC to QNAP it fails as documented. From then on I can no longer get expected speeds on the QNAP to PC transfer, it just sits at ~60MBs until I restart the QNAP, which is a bit worrying.
I suspect that Thunderbolt Networking is a TCP/IP-based protocol and, as such, is beholden to packet size "windowing" (wherein the packet size will grow with every packet confirmed transferred between source and destination). To point, I experience a variety of transfer throughput rates depending upon the number of files transferred, size of each file, etc. Also, the processor of the unit makes a significant difference as to what the peak rate may be. Lesser units (like my TS-453BT3) will run at far less than the theoretical line rate of TB3 because of the platform specifications, but higher end processors will do the job better.

In my experience, my download throughput (from QNAP to client) starts around 400 MB/s (3.2 Gbps) for large, single file transfers, but it slows and fluctuates around the area of 160 MB/s (1.2 Gbps). The upload throughput does just about the same, and it fluctuates quite a bit as well. I suspect the combination of drive configuration (SSD versus mechanical, SSD caching, etc.), filesystem (thin/thick/static), and processor would impact. My guess is the interface is so fast (TB3 is a fast-lane connection) that packets arrive out of sequence. When this becomes untenable for the buffer, TCP protocol will throttle back (reduce) the packet size and request retransmission (thus, reducing the throughput). I feel like this could be improved with proper queue buffers, but that's a topic for a different thread.

I have also done some side-by-side comparisons with transfers via the actual Ethernet network and the TB3 Thunderbolt Networking connection. I have to say, the Ethernet performs much more consistently on the 1 Gbps network. This seems limited to about ~95 MB/s (760 Mbps) on download and ~110 MB/s (880 Mbps) on upload, but, strikingly, it's consistent and does not fluctuate. My unit has a 10 Gbps Ethernet interface as well, and I suspect it would perform consistently as well. Unfortunately, I do not have a 10 Gbps Ethernet client to test this hypothesis.

All of that said, this is way better than not transferring data at all (which is what the later Windows 10 builds seemed to do to our Thunderbolt Networking uploads to the QNAP). Once we fail there, it seems like the downloads don't work well either until reboot (almost as though the machine becomes biased toward that connection). I'm sure there's queues and buffers on both sides of the transaction, and I suspect Windows 10 is bursting the bucket upon initiation of the transfer and doesn't know what to do from there.

I'm curious to know if the Linux file transfers have the same issue with fluctuation via Thunderbolt Networking or if it's steady-on like the Ethernet file transfers.
Hi metaldave,

Apologies my results post wasn't clear, I'm running a 472xt with 2GB NVME used as testing (so no real drive bottlenecks) and get following results transferring a single 4GB iso file

Thunderbolt 3 tests to high spec Dell XPS 15 with fast NVME SSD.
Windows 1909 - Fails to copy to QNAP, but copy from QNAP initially fast ~2GBs, but any subsequent copies seems to sit around 60MBs until I restart the QNAP, then I get a fast QNAP to PC (2GBs) again for the first transfer.
Windows Insider Preview - Exactly the same as 1909, except I never get the initial "fast" speeds from the QNAP to the PC, just get 60MBs. So the insider preview in my tests is actually worse.

I also have tried this on a NUC8 with same results (1909). That NUC now runs proxmox (linux) and I get pretty consistent speeds of 1.8-2GBs with single 4GB file transfer in both directions.

My main workstation PC has a direct connection to the Qnap over 10Gb, and I get about 1.1GB when transferring large file. I also have the 2x 1Gbs Ethernet's teamed, and from any 1Gbs device I get around ~100MBs as you'd expect.

So in my tests I don't see what QNAP is claiming that the Insider preview seems to sort the issue.
A quick update, I have a fresh install of the current slow ring Windows 10 2004 release build 19041.172, which will be the new 2004 release and due to MS extended preview period this is basically what we're going to get as a release. From a clean install, I added the NAS over TB3 and got the same disappointing results. Copying a single 2GB file from the NAS to the PC gave a "slow" 75MB/s not reaching anywhere near the speeds I got from 1909 (from NAS to PC). The copy from the PC to the NAS is same as 1909 and just fails completely.

So while MS might be working on a fix, it's not in the latest 2004 release just yet. So it's pretty much a year of this function not working on the current windows 10 releases. QNAP Still advertising it as a function though

HiTach
Starting out
Posts: 31
Joined: Fri Feb 15, 2019 6:03 am

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by HiTach » Mon Apr 06, 2020 9:53 am

Thanks for the effort and the update.
I removed the TB3 card from my main 872XT and use both a 40Gbe NIC and a secondary 10Gbe NIC.
For my secondary I had left the TB3 card hoping that this issue would be fixed, but I am just so disappointed with QNAP/Microsoft and TB3 on this device.
I have the QNAP QNA-UC5G1T USB 3.2 Gen 1 to 5GbE which works great from my laptop, a very solid reliable transfer rate, so I shall use this, avoid the headaches and enjoy the reliability.
Still hopeful for this to get fixed somewhere, someday.
Workstation DAS/NAS: TVS-872X(T) *Thunderbolt card removed due to Windows10 1909 bug
8x10TB WD Gold RAID6 54TB static volume
2x1TB Samsung 970 EVO plus R/W RAID1 cache w/EK WB M.2 heatsinks + 2x NB 12-P fans
1x40Gbe Mellanox DAS network connection, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
Backup NAS: TVS-872XT
8x10TB WD Gold RAID6 54TB Thick volume w/snapshots
1x40Gbe Qnap Thunderbolt 3 card, 1x10Gbe QXG-10G1T for local 10gbe virtual switch
LAN switch 10Gbe(2)/1Gbe(8) NETGEAR Nighthawk SX10 (GS810EMX)

themannyram
First post
Posts: 1
Joined: Thu Apr 09, 2020 2:28 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by themannyram » Sat Apr 18, 2020 9:52 am

I was having the same problem but I did something that fixed the problem.

If you go to: Network and Sharing center --> Change adapter Settings --> Double click on your Thunderbolt Networking Device --> and then click on properties --> and then go to the Sharing Tab--> and under "Internet connection sharing", check "Allow other network users to connect through this computer's Internet connection" --> Select Your home Network and check "Allow other network users ect...."

After i finally was able to transfer files.

Make sure to assign folder permissions to your computer in your qnap under "Microsoft Networking host access" setting in your Qnap.

Not sure if this works for others but thats what solved it for me under Windows 10 Pro 1909.

lieszkol
New here
Posts: 3
Joined: Mon Feb 03, 2014 5:32 pm

Re: TVS-882T: Thunderbolt networking is broken on Windows 10 Pro 1903

Post by lieszkol » Mon Apr 20, 2020 1:11 am

^^^ THIS! Worked for me. What a genius! Thanks @themannyram so much for sharing. 4 hours of my life wasted thanks to Microsoft.

P.s. checking or unchecking "Allow other network users..." made no difference to me, but sharing the connection fixed it. Un-sharing it then again stopped the ability to upload.

After sharing the connection my other adapter (used for internet) somehow got a fixed IP address (thanks Windows). After setting that back to DHCP everything started working again!

My serious concerns about the quality of software engineers working at Microsoft has not abated since the release of Windows 10.

Post Reply

Return to “File Sharing”