Poor performance listing files

[schumaku]

@doktornotor: Keep in mind there are real-worls applicaitons doing silly things like that, off user or admin control. Most have an index pointing to some real or synthetic file names - so from the application view, things are organized.

[doktornotor]

Well, technical support using Windows Explorer to browse for a file on network which is lost among tens and tens of thousands of others hardly sounds like a reasonable use case.

I have several shared folders for my users, organised by department, e.g. sales, support, projects, development, admin etc. I have approx 20 users running Windows 7 to these shares. I also have two Linux users in the dev team. Network is 1GB throughout.

The issue :-

If a user in the support team uses Windows Explorer to look at a shared folder on the NAS for a particular file, the browsing or traversing of that folder is very slow sometimes if there are a lot of files (in this case 15,000). The user has to wait for the NAS to return the list of files, which can take minutes!

How on earth could you even find something (let alone quickly) in such mess? How on earth can users be productive with such mess?

Also, some reference: http://www.samba.org/samba/docs/man/Sam ... efile.html
While the above is dated, this confirms that looking into the case sensitive switch is still a good idea, otherwise with the default "auto" setting, Samba negotiates with the client on a per-packet bases. Anyway, the folders should not be a huge dumpspace for zillions of files unless noone ever deals with that in an interactive way.

EDIT: Also, back to previous debate - you really should go iSCSI unless you can get your users organized - and deploy Windows Search service on the iSCSI target volume on the Window Server and add the iSCSI share location to the libraries on users's workstations - so that the files can be indexed by the server and quickly searched for on Windows workstations, instead of "browsing".

Windows Search service feature is available in Server 2008 and above, in case the performance would not be sufficient for your purposes, there are some full-featured products provided by MS.

[storageman]

storageman
in other word it is a SAMBA bug related to buffering. With SAMBA it takes for ever to browse a folder of 86,000 files whereas it is fine via SMB (windows server). Is that correct?
thanks
stanislas

I just think the NAS using uLinux to present SMB to the outside world doesn't work as well as Windows presenting it more "natively" (plus there are different releases of SMB).

It's also a combination of the uLinux OS. For example browsing files in DOS is quicker than via the Windows GUI. So it's down to how efficient the OS works. More RAM helps in any OS to cache operations like browsing for speed. So more RAM makes a difference.

Unless someone's got a better explanation.

[schumaku]

Well, technical support using Windows Explorer to browse for a file on network which is lost among tens and tens of thousands of others

That's QNAP support security vulnerability complaint in-folder

[doktornotor]

That's QNAP support security vulnerability complaint in-folder

i-lold001.jpg

[GrahamL]

Having a folder with 86K files suggests you need to put some better file organization in place firstly.

I don't know on what basis you say that. You have no knowledge of the organisation of the folders and files in this case. In fact, they are organised pretty well. These files are grouped by type and produced on a certain day. There are other folders that pertain to other types of file. When it was on a Windows system, everything was fine. Only since moving to this NAS have I had any issue.

There is nothing inherently wrong with having 86,00 files in one folder (are you sugessting perhaps there is?). In fact, the fault exhibits itself on folders smaller than this. I am using this as a particular example.

[quote="doktornotor"} How on earth could you even find something (let alone quickly) in such mess? How on earth can users be productive with such mess?[/quote]

I do agree that in the case of the support in-folders, we are taking steps to classify the files that come in.

However, the case remains that we never saw any issues when the users browsed to the Windows share. Now, when they browse to the share on the NAS and it goes slow, the NAS is slow for everybody across all shares. This has to be an inherent weakness in the way the NAS works.

One of my dev team has already suggested the case-sensitive config of samba. We've tested it on a test share and it spesds things up a but but we lose the case info. Not ideal.

I think I have to go iSCSI but I'm not pleased to do so. This bypasses the functionality of the NAS effectively and means I have to have another Windows box online. It's a good job I have a spare Windows server to try this with.

[doktornotor]

Manyally browsing a folder with 86K files to find something is absolutely crazy, end of story. Whatever, I do not think we are getting anywhere when continuing this debate. You got the suggestions, feel free to try them or to ignore them.

[GrahamL]

Manyally browsing a folder with 86K files to find something is absolutely crazy, end of story. Whatever, I do not think we are getting anywhere when continuing this debate. You got the suggestions, feel free to try them or to ignore them.

Thanks for your input and suggestions. I am persuing workarounds. I'm just disappointed that I have such a poor performing system in this regard- I was expecting much better than this. You're implying this behaviour is 'normal' and to be expected and that I am wrong for having a large bunch of files in a folder. I can't agree with that - it works fine on Windows and not on the NAS/Linux.

[storageman]

Thanks for your input and suggestions.

I hope you'll thank me as well. I'm actually the only one who gave you a workaround!

You can argue till the cows come home whether it should (have to) cope with browsing this number of files in a folder.

[GrahamL]

Thanks for your input and suggestions.

I hope you'll thank me as well. I'm actually the only one who gave you a workaround!

You can argue till the cows come home whether it should (have to) cope with browsing this number of files in a folder.

Of course! Thanks to you and everybody for their contributions.

I still have a fellow at QNAP support who is asking me to do some tests. He thinks it may be to do with network/domain broadcasts. I'm yet to be convinced but I will follow his suggested tests and see what they come up with. Whether it should or shouldn't, my real world case has highlighted a big difference in behaviour from Windows to the NAS that I wasn't expecting and is not documented anywhere up front. Your suggestions are very helpful, cheers!

Will probably go iSCSI after testing and wait for firmware updates if any.

Thanks

Graham

[schaux]

Hi,

are there any new informations in this case? We decided to try a TS-269L with the new 4.1.1 as Domaincontroller and Fileserver for a 5 persons office.
The performance with 500 and above is extreamly poor. Waiting 2-4 seconds for changing folders with new i5 machines and gigabit network.

would be nice if anyone has new ideas...

[azorie]

I also have ran into this after replacing a old dell server. model 2800, with (6drives in raid5) 15k SCSI drive running windows 2003 server and mostly xp and win7 clients. WE are still stuck with some old windows 2003 servers, so SMB is left at 1.
WE have up to 60-65 local users, it varies, but after moving all the same file over to a QNAP 670 raid 6, I got so many complaints we had to abandon it. I redid the box with raid 5 and a SSD cache, this helped allot. Still once burned management would not sign off on trying it again. I learned that unit with SATA drives just are not going to cut it, when you had 15k scsi and a 1Meg caching hardware controller in a file server, versus these Nas boxes. I did test in the lab with 10 clients but had no real software to test it well enough. Sadly we had to buy a new file server and my name is mud for recommending a small NAS unit. Live and learn. sadly the prices for the QNAP scsi unit are more than just a plain file server costs, go figure. a Ts-EC1279U-SAS-Rp costs almost 7k with no drives.
http://www.newegg.com/Product/Product.a ... 6822107144

where a
http://configure.us.dell.com/dellstore/ ... vw=classic

with 12 drives installed was $8400
with windows storage server, much much faster unit. just my 2cents.

[pwilson]

I also have ran into this after replacing a old dell server. model 2800, with (6drives in raid5) 15k SCSI drive running windows 2003 server and mostly xp and win7 clients. WE are still stuck with some old windows 2003 servers, so SMB is left at 1.
WE have up to 60-65 local users, it varies, but after moving all the same file over to a QNAP 670 raid 6, I got so many complaints we had to abandon it. I redid the box with raid 5 and a SSD cache, this helped allot. Still once burned management would not sign off on trying it again. I learned that unit with SATA drives just are not going to cut it, when you had 15k scsi and a 1Meg caching hardware controller in a file server, versus these Nas boxes. I did test in the lab with 10 clients but had no real software to test it well enough. Sadly we had to buy a new file server and my name is mud for recommending a small NAS unit. Live and learn. sadly the prices for the QNAP scsi unit are more than just a plain file server costs, go figure. a Ts-EC1279U-SAS-Rp costs almost 7k with no drives.
http://www.newegg.com/Product/Product.a ... 6822107144

where a
http://configure.us.dell.com/dellstore/ ... vw=classic

with 12 drives installed was $8400
with windows storage server, much much faster unit. just my 2cents.

It would be amusing to see what the performance of a directory with 86K files would perform like on a Windows machine at any price, even one with a "15k scsi and a 1Meg caching hardware controller" in it. (Consider this observation to be my 2¢ worth)

[storageman]

Storage servers are not the future with virtualization, have limited flexibility compared to backend storage.
NAS are not for performance, never have been, and certainly not up to it for ISCSI, even on the Xeon models (and flakey).
If you want high performance buy a dual controller SAN.
Patricks comment on browsing large number of files is relevant, fundamental browser caching issues in Windows GUIs compared to say speed of dir from Windows command prompt.
Sticking tens of thousands of files in a single folder bad idea and to be avoided at all costs.
... just my 2cents.

[schaux]

85.000 files in one folder is really a other problemdimension. I thought Microsoft said 6.000 per folder is the maxium without performance issues.
But this circumstance explains not my Problem with 500 folders and less...