Fix for Qsirch and Spotlight
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
I've had a look and I think everything should be working ok with the version I last uploaded.
If you need to search for a .mov (or any other extension etc...) use:
ext:"mov"
I'm wondering if ampersand characters are not particularly happy characters - I'll test!
If you need to search for a .mov (or any other extension etc...) use:
ext:"mov"
I'm wondering if ampersand characters are not particularly happy characters - I'll test!
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
Ok - so I put a file called "M&M" on my filesystem and did a search through QSirch on the Web Console and it shows the exact same results as QsirchMac.app. It would do as it's the same API it's querying. I have found something completely maddening in that filenames that include an ampersand must have more than one character before the ampersand and more than one different character after the ampersand, So...
M&M no results
M&SM no results
MM&MM no results
MM&SM result
I've no idea why, gotta take that one up with QNAP. They'll probably suggest buying the QSirch Premium or some bilge.
On a different note if anyone has a link to all the different filters you can apply to the query bar I'll try and add some filter building functionality to QsirchMac.app.
Cheers
MB
M&M no results
M&SM no results
MM&MM no results
MM&SM result
I've no idea why, gotta take that one up with QNAP. They'll probably suggest buying the QSirch Premium or some bilge.
On a different note if anyone has a link to all the different filters you can apply to the query bar I'll try and add some filter building functionality to QsirchMac.app.
Cheers
MB
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
Just a quick update to everyone who has been watching this.
I may have found why the Spotlight search through OS X's Finder isn't working for a lot of us. It's something so simple I just don't know if I should be laughing or crying. Or maybe a bit of both.
As far as I can tell QNAP have created a service that sits in between Samba's native Spotlight RPC (available since Samba 4.3) and the Qsirch API. It runs through a kind of service manager daemon they call "Qooba" and uses curl to query the Qsirch API, which returns JSON and is parsed and fed back to Samba in a form it understands.
The Qooba spotlight service appears to only use HTTP to make the query, so for anyone who uses "Force secure connection (HTTPS) only" option in the Control Panel -> General Settings - you're screwed.
For those who are interested they can run the Qooba spotlight service in the foreground and with full debug level. SSH into your QNAP box, and run this command:
Now do a search and behold as it runs through the process in glorious verbosity. Here's a snippet of it:
I'll be leaving my Qsirch app on GitHub just in case this isn't the cause but for now my Qsirch is working.
I hope the developers just upgrade Samba to 4.12 as it supports elasticsearch db queries out of the box meaning we could do away with all the faff, clunk and unnecessary overhead of a HTTP request.
MB
I may have found why the Spotlight search through OS X's Finder isn't working for a lot of us. It's something so simple I just don't know if I should be laughing or crying. Or maybe a bit of both.
As far as I can tell QNAP have created a service that sits in between Samba's native Spotlight RPC (available since Samba 4.3) and the Qsirch API. It runs through a kind of service manager daemon they call "Qooba" and uses curl to query the Qsirch API, which returns JSON and is parsed and fed back to Samba in a form it understands.
The Qooba spotlight service appears to only use HTTP to make the query, so for anyone who uses "Force secure connection (HTTPS) only" option in the Control Panel -> General Settings - you're screwed.
For those who are interested they can run the Qooba spotlight service in the foreground and with full debug level. SSH into your QNAP box, and run this command:
Code: Select all
qooba --service spotlight --debug 10 --foreground
I have made QNAP support aware of this and have asked them to get the developers to look at some kind of debouncing feature that would give the end user time to finish typing before the search is executed. This causes a huge slow down in searching as the spotlight service tries to search for first few letters of whatever you type first and gets rather bogged down by the huge amount of results returned before finally trying the finished search term. I had to implement this in my Qsirch app and it stopped the majority of the erroneous searches.[~] # qooba --service spotlight --debug 10 --foreground
[2021/06/04 13:57:08, 3] spotlight_service: max_result_num:300
[2021/06/04 13:57:08, 3] spotlight_handler: ---------------- start ----------------
[2021/06/04 13:57:08, 3] spotlight_handler: sk: 4, uid: 0, search_path: /share/CACHEDEV1_DATA/Work, query: *=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw, share_path:/share/CACHEDEV1_DATA/Work
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: original qq = '*=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: buffer = '*=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: len_subq = i(16) - head_of_subq(0) + bracket(0) = 16
[2021/06/04 13:57:08, 3] tokenize_and_map: subq = '*=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_resolver: qp = '*=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_mapper_dispatcher: qp: *=="myfile*"cdw, key: *, operator: ==, value: myfile*
[2021/06/04 13:57:08, 3] tokenize_and_map: buffer = '"myfile"||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: len_subq = i(44) - head_of_subq(11) + bracket(0) = 33
[2021/06/04 13:57:08, 3] tokenize_and_map: subq = 'kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_resolver: qp = 'kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_mapper_dispatcher: qp: kMDItemTextContent=="myfile*"cdw, key: kMDItemTextContent, operator: ==, value: myfile*
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: tokenized qq = '"myfile"||content:"myfile"'
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: replaced qq = '"myfile" OR content:"myfile"'
[2021/06/04 13:57:08, 3] get_share_by_path: The number of share is:5, path:/share/CACHEDEV1_DATA/Work
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:0,(s_ptr+i)->section_name:[Share1]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name:Share1
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share1]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:1,(s_ptr+i)->section_name:[Share2]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share2
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share2]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:2,(s_ptr+i)->section_name:[Share3]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share3
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share3]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:3,(s_ptr+i)->section_name:[Share4]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share4
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share4]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:4,(s_ptr+i)->section_name:[Share5]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share5
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share5]
[2021/06/04 13:57:08, 3] get_share_by_path: Found path:/share/CACHEDEV1_DATA/Share5 matches to share: Share5
[2021/06/04 13:57:08, 4] spotlight_handler: After get_share_by_path(), share_name:[Share5]
[2021/06/04 13:57:08, 4] spotlight_handler: === path_with_share_name:[/Share5] ===
[2021/06/04 13:57:08, 4] spotlight_handler: === new_search_path:[/Share5] ===
[2021/06/04 13:57:08, 3] qsirch: search_path: /Share5, query: "myfile" OR content:"myfile", sid: pqknn5k7
[2021/06/04 13:57:08, 3] qsirch: qsirch_query: http://127.0.0.1:8080/qsirch/v1/api/sea ... D%20%28%22 myfile%22%20OR%20content%3A%22A115490%22%29&show_absolute_path=1&limit=300&auth_token=pqknn5k7
[2021/06/04 13:57:09, 3] qsirch: total_results:1, Qsirch response:
[2021/06/04 13:57:09, 3] qsirch: result[0]: '/share/CACHEDEV1_DATA/Share5/myfile.jpg'
[2021/06/04 13:57:09, 1] spotlight_handler: send: 1
[2021/06/04 13:57:09, 3] spotlight_handler: successfully send the response to Samba
I'll be leaving my Qsirch app on GitHub just in case this isn't the cause but for now my Qsirch is working.
I hope the developers just upgrade Samba to 4.12 as it supports elasticsearch db queries out of the box meaning we could do away with all the faff, clunk and unnecessary overhead of a HTTP request.
MB
-
- Starting out
- Posts: 29
- Joined: Sat Nov 11, 2017 9:38 pm
Re: Fix for Qsirch and Spotlight
Crying. For sure. Why did QNAP not document this anywhere???
Thanks!!
Not sure because that would require that Samba is able to query the database as populated by qsirch. Some kind of interface will probably always be required.
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
There is a mappings file for elasticsearch backend included with samba 4.12+ - it’s actually very very easy.
QNAP support have my feature request open for this now. Just got to keep our fingers crossed they pay attention. They apparently are aware of the issue with searching individual characters as they are typed but haven’t done anything to fix (denounce?) it.
Chin up guys - we’re nearly there
QNAP support have my feature request open for this now. Just got to keep our fingers crossed they pay attention. They apparently are aware of the issue with searching individual characters as they are typed but haven’t done anything to fix (denounce?) it.
Chin up guys - we’re nearly there
-
- Starting out
- Posts: 29
- Joined: Sat Nov 11, 2017 9:38 pm
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
It beggars belief they actually went to the hassle of writing a connector service for samba that uses curl to query the Qsirch API instead of the elasticsearch database directly?! Makes me think the QNAP devs aren’t the sharpest tools in the box .
I disassembled the Qooba binary and Qooba shared library and found it quite interesting.
I disassembled the Qooba binary and Qooba shared library and found it quite interesting.
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
This is the way. Good work around for now.
Even pasting the text in isn’t fast enough and can still lag.
-
- Starting out
- Posts: 29
- Joined: Sat Nov 11, 2017 9:38 pm
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
No, it is essentially using shed loads more network and cpu/memory resources. You can try query the ES database directly (it’s on a none standard port - 5280 or something, can’t remember will advise when I’m back in work), it’s much faster/lighter.
The good thing about QNAP’s integration of the elasticsearch database and samba is that it doesn’t use periodical crawls like FSCrawler would (if you had rolled your own NAS from Linux and used the samba docs this is what you would get). QNAP actually watches the file system for changes so files are searchable straight away. I think it is fswatch? Again, will investigate further when I’m back in the office
The good thing about QNAP’s integration of the elasticsearch database and samba is that it doesn’t use periodical crawls like FSCrawler would (if you had rolled your own NAS from Linux and used the samba docs this is what you would get). QNAP actually watches the file system for changes so files are searchable straight away. I think it is fswatch? Again, will investigate further when I’m back in the office
-
- Starting out
- Posts: 29
- Joined: Sat Nov 11, 2017 9:38 pm
Re: Fix for Qsirch and Spotlight
Yes, I tried samba/fscrawler with an lxc image. It's quite straightforward to set up, but you end up duplicating search index and running another memory hog in addition to qsirch.
-
- Getting the hang of things
- Posts: 50
- Joined: Sat Jan 05, 2019 5:27 pm
Re: Fix for Qsirch and Spotlight
Wow. So turning off the "force HTTPS" does indeed work! It might be obvious, but make sure you connect via SAMBA and not AFS for this Qsirch to be accessable.
I activated Qsirch, allowed it to index (overnight) and then did a reboot just to make sure. Connecting via SSH to the QNAP allows you to see the Qooba spotlight service working, but you do have to kill it first... eg:
I found debug levels below 3 didnt really give any feedback. Level 3 lets you see the qooba > qsirch interraction. These are the first 3 lines when I type the word "movies" into the Finder search. The first thing it does is drown in results for "m*"...
then returning 200+ results for "mo*"
then returning 300 results for "mov*" (by name and contentType)
then 10 results for "movi*"
then "movie*"
and finally for "movies*"
last result sent to samba:
Not really a quick way of searching, 1 minute 48 seconds for the full round. The same search in the qsirch web portal takes about 6 seconds.
I activated Qsirch, allowed it to index (overnight) and then did a reboot just to make sure. Connecting via SSH to the QNAP allows you to see the Qooba spotlight service working, but you do have to kill it first... eg:
Code: Select all
# qooba --pidof spotlight
12345
# kill 12345
# qooba --service spotlight --debug 3 --foreground
Code: Select all
[~] # qooba --service spotlight --debug 3 --foreground
[2021/06/05 12:02:34, 3] spotlight_service: max_result_num:300
[2021/06/05 12:02:34, 3] spotlight_handler: ---------------- start ----------------
[2021/06/05 12:02:34, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="m*"
Code: Select all
[2021/06/05 12:03:01, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="mo*"cdw||kMDItemTextContent=="mo*"
Code: Select all
[2021/06/05 12:03:20, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: kMDItemContentType=="com.apple.quicktime-movie"||kMDItemSubject=="mov*"
Code: Select all
[2021/06/05 12:03:48, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="movi*"cdw||kMDItemTextContent=="movi*"
Code: Select all
[2021/06/05 12:03:50, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: _kMDItemGroupId=="7"||kMDItemSubject=="movie*"
Code: Select all
[2021/06/05 12:03:51, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: _kMDItemGroupId=="7"||kMDItemSubject=="movies*"
Code: Select all
[2021/06/05 12:04:20, 3] spotlight_handler: successfully send the response to Samba
-
- Starting out
- Posts: 39
- Joined: Wed Feb 12, 2020 10:39 pm
Re: Fix for Qsirch and Spotlight
The reason it takes so long for your typed search to come back is because the search is not “debounced” as you type. It wastes 2 http requests almost every keystroke. The qsirch web app has a JavaScript debounce function that keeps things in check.
As mseewald said start the search locally and then click “shared” in the search filter. It’s a decent enough workaround and will ensure snappy results.
As mseewald said start the search locally and then click “shared” in the search filter. It’s a decent enough workaround and will ensure snappy results.
-
- Getting the hang of things
- Posts: 50
- Joined: Sat Jan 05, 2019 5:27 pm
Re: Fix for Qsirch and Spotlight
...and dont forget to restart the spotlight service before killing the SSH session!mousebat wrote: ↑Sat Jun 05, 2021 9:39 pm The reason it takes so long for your typed search to come back is because the search is not “debounced” as you type. It wastes 2 http requests almost every keystroke. The qsirch web app has a JavaScript debounce function that keeps things in check.
As mseewald said start the search locally and then click “shared” in the search filter. It’s a decent enough workaround and will ensure snappy results.
Code: Select all
qooba --service spotlight
-
- Getting the hang of things
- Posts: 50
- Joined: Sat Jan 05, 2019 5:27 pm
Re: Fix for Qsirch and Spotlight
I see that QTS 5.0.0 is bringing Samba 4.13 with it, so can we hope for faster Spotlight seaching? Has anyone taken the plunge and upgraded yet?