Fix for Qsirch and Spotlight

mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

I've had a look and I think everything should be working ok with the version I last uploaded.

If you need to search for a .mov (or any other extension etc...) use:
ext:"mov"

I'm wondering if ampersand characters are not particularly happy characters - I'll test!
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

Ok - so I put a file called "M&M" on my filesystem and did a search through QSirch on the Web Console and it shows the exact same results as QsirchMac.app. It would do as it's the same API it's querying. I have found something completely maddening in that filenames that include an ampersand must have more than one character before the ampersand and more than one different character after the ampersand, So...

M&M no results
M&SM no results
MM&MM no results
MM&SM result

I've no idea why, gotta take that one up with QNAP. They'll probably suggest buying the QSirch Premium or some bilge.

On a different note if anyone has a link to all the different filters you can apply to the query bar I'll try and add some filter building functionality to QsirchMac.app.

Cheers
MB
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

Just a quick update to everyone who has been watching this.

I may have found why the Spotlight search through OS X's Finder isn't working for a lot of us. It's something so simple I just don't know if I should be laughing or crying. Or maybe a bit of both.

As far as I can tell QNAP have created a service that sits in between Samba's native Spotlight RPC (available since Samba 4.3) and the Qsirch API. It runs through a kind of service manager daemon they call "Qooba" and uses curl to query the Qsirch API, which returns JSON and is parsed and fed back to Samba in a form it understands.

The Qooba spotlight service appears to only use HTTP to make the query, so for anyone who uses "Force secure connection (HTTPS) only" option in the Control Panel -> General Settings - you're screwed.

For those who are interested they can run the Qooba spotlight service in the foreground and with full debug level. SSH into your QNAP box, and run this command:

Code: Select all

qooba --service spotlight --debug 10 --foreground
Now do a search and behold as it runs through the process in glorious verbosity. Here's a snippet of it:
[~] # qooba --service spotlight --debug 10 --foreground
[2021/06/04 13:57:08, 3] spotlight_service: max_result_num:300
[2021/06/04 13:57:08, 3] spotlight_handler: ---------------- start ----------------
[2021/06/04 13:57:08, 3] spotlight_handler: sk: 4, uid: 0, search_path: /share/CACHEDEV1_DATA/Work, query: *=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw, share_path:/share/CACHEDEV1_DATA/Work
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: original qq = '*=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: buffer = '*=="myfile*"cdw||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: len_subq = i(16) - head_of_subq(0) + bracket(0) = 16
[2021/06/04 13:57:08, 3] tokenize_and_map: subq = '*=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_resolver: qp = '*=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_mapper_dispatcher: qp: *=="myfile*"cdw, key: *, operator: ==, value: myfile*
[2021/06/04 13:57:08, 3] tokenize_and_map: buffer = '"myfile"||kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] tokenize_and_map: len_subq = i(44) - head_of_subq(11) + bracket(0) = 33
[2021/06/04 13:57:08, 3] tokenize_and_map: subq = 'kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_resolver: qp = 'kMDItemTextContent=="myfile*"cdw'
[2021/06/04 13:57:08, 3] spotlight_mapper_dispatcher: qp: kMDItemTextContent=="myfile*"cdw, key: kMDItemTextContent, operator: ==, value: myfile*
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: tokenized qq = '"myfile"||content:"myfile"'
[2021/06/04 13:57:08, 3] map_spotlight_to_qsirch: replaced qq = '"myfile" OR content:"myfile"'
[2021/06/04 13:57:08, 3] get_share_by_path: The number of share is:5, path:/share/CACHEDEV1_DATA/Work
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:0,(s_ptr+i)->section_name:[Share1]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name:Share1
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share1]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:1,(s_ptr+i)->section_name:[Share2]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share2
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share2]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:2,(s_ptr+i)->section_name:[Share3]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share3
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share3]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:3,(s_ptr+i)->section_name:[Share4]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share4
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share4]
[2021/06/04 13:57:08, 4] get_share_by_path: for-loop, i:4,(s_ptr+i)->section_name:[Share5]
[2021/06/04 13:57:08, 4] get_share_by_path: share_info.share_name: Share5
[2021/06/04 13:57:08, 4] get_share_by_path: Get_NAS_Share_Info_Ex() return:0, share_info.path:[/share/CACHEDEV1_DATA/Share5]
[2021/06/04 13:57:08, 3] get_share_by_path: Found path:/share/CACHEDEV1_DATA/Share5 matches to share: Share5
[2021/06/04 13:57:08, 4] spotlight_handler: After get_share_by_path(), share_name:[Share5]
[2021/06/04 13:57:08, 4] spotlight_handler: === path_with_share_name:[/Share5] ===
[2021/06/04 13:57:08, 4] spotlight_handler: === new_search_path:[/Share5] ===
[2021/06/04 13:57:08, 3] qsirch: search_path: /Share5, query: "myfile" OR content:"myfile", sid: pqknn5k7
[2021/06/04 13:57:08, 3] qsirch: qsirch_query: http://127.0.0.1:8080/qsirch/v1/api/sea ... D%20%28%22 myfile%22%20OR%20content%3A%22A115490%22%29&show_absolute_path=1&limit=300&auth_token=pqknn5k7
[2021/06/04 13:57:09, 3] qsirch: total_results:1, Qsirch response:
[2021/06/04 13:57:09, 3] qsirch: result[0]: '/share/CACHEDEV1_DATA/Share5/myfile.jpg'
[2021/06/04 13:57:09, 1] spotlight_handler: send: 1
[2021/06/04 13:57:09, 3] spotlight_handler: successfully send the response to Samba
I have made QNAP support aware of this and have asked them to get the developers to look at some kind of debouncing feature that would give the end user time to finish typing before the search is executed. This causes a huge slow down in searching as the spotlight service tries to search for first few letters of whatever you type first and gets rather bogged down by the huge amount of results returned before finally trying the finished search term. I had to implement this in my Qsirch app and it stopped the majority of the erroneous searches.

I'll be leaving my Qsirch app on GitHub just in case this isn't the cause but for now my Qsirch is working.

I hope the developers just upgrade Samba to 4.12 as it supports elasticsearch db queries out of the box meaning we could do away with all the faff, clunk and unnecessary overhead of a HTTP request.

MB
mseewald
Starting out
Posts: 29
Joined: Sat Nov 11, 2017 9:38 pm

Re: Fix for Qsirch and Spotlight

Post by mseewald »

mousebat wrote: Fri Jun 04, 2021 9:36 pm .. I just don't know if I should be laughing or crying. Or maybe a bit of both.
Crying. For sure. Why did QNAP not document this anywhere???
mousebat wrote: Fri Jun 04, 2021 9:36 pm For those who are interested they can run the Qooba spotlight service in the foreground and with full debug level
Thanks!!
mousebat wrote: Fri Jun 04, 2021 9:36 pm I hope the developers just upgrade Samba to 4.12 as it supports elasticsearch db queries
Not sure because that would require that Samba is able to query the database as populated by qsirch. Some kind of interface will probably always be required.
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

There is a mappings file for elasticsearch backend included with samba 4.12+ - it’s actually very very easy.

QNAP support have my feature request open for this now. Just got to keep our fingers crossed they pay attention. They apparently are aware of the issue with searching individual characters as they are typed but haven’t done anything to fix (denounce?) it.

Chin up guys - we’re nearly there 😂
mseewald
Starting out
Posts: 29
Joined: Sat Nov 11, 2017 9:38 pm

Re: Fix for Qsirch and Spotlight

Post by mseewald »

mousebat wrote: Sat Jun 05, 2021 6:00 pm issue with searching individual characters as they are typed
I don't seem to have that issue. In my Finder the default search scope is "This Mac", so I can type the search string first, then select the respective folder on the shared drive.
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

It beggars belief they actually went to the hassle of writing a connector service for samba that uses curl to query the Qsirch API instead of the elasticsearch database directly?! Makes me think the QNAP devs aren’t the sharpest tools in the box 😂.

I disassembled the Qooba binary and Qooba shared library and found it quite interesting.
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

mseewald wrote: Sat Jun 05, 2021 6:04 pm
mousebat wrote: Sat Jun 05, 2021 6:00 pm issue with searching individual characters as they are typed
I don't seem to have that issue. In my Finder the default search scope is "This Mac", so I can type the search string first, then select the respective folder on the shared drive.
This is the way. Good work around for now.

Even pasting the text in isn’t fast enough and can still lag.
mseewald
Starting out
Posts: 29
Joined: Sat Nov 11, 2017 9:38 pm

Re: Fix for Qsirch and Spotlight

Post by mseewald »

mousebat wrote: Sat Jun 05, 2021 6:10 pm hassle of writing a connector service for samba that uses curl to query the Qsirch API instead of the elasticsearch database directly
Indeed. Does this have any advantages? Maybe with their bigger QNAP installations?
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

No, it is essentially using shed loads more network and cpu/memory resources. You can try query the ES database directly (it’s on a none standard port - 5280 or something, can’t remember will advise when I’m back in work), it’s much faster/lighter.

The good thing about QNAP’s integration of the elasticsearch database and samba is that it doesn’t use periodical crawls like FSCrawler would (if you had rolled your own NAS from Linux and used the samba docs this is what you would get). QNAP actually watches the file system for changes so files are searchable straight away. I think it is fswatch? Again, will investigate further when I’m back in the office ;)
mseewald
Starting out
Posts: 29
Joined: Sat Nov 11, 2017 9:38 pm

Re: Fix for Qsirch and Spotlight

Post by mseewald »

mousebat wrote: Sat Jun 05, 2021 6:30 pm The good thing about QNAP’s integration of the elasticsearch database and samba is that it doesn’t use periodical crawls like FSCrawler would
Yes, I tried samba/fscrawler with an lxc image. It's quite straightforward to set up, but you end up duplicating search index and running another memory hog in addition to qsirch.
chrisonnas
Getting the hang of things
Posts: 50
Joined: Sat Jan 05, 2019 5:27 pm

Re: Fix for Qsirch and Spotlight

Post by chrisonnas »

Wow. So turning off the "force HTTPS" does indeed work! It might be obvious, but make sure you connect via SAMBA and not AFS for this Qsirch to be accessable.

I activated Qsirch, allowed it to index (overnight) and then did a reboot just to make sure. Connecting via SSH to the QNAP allows you to see the Qooba spotlight service working, but you do have to kill it first... eg:

Code: Select all

# qooba --pidof spotlight
12345
# kill 12345
# qooba --service spotlight --debug 3 --foreground
I found debug levels below 3 didnt really give any feedback. Level 3 lets you see the qooba > qsirch interraction. These are the first 3 lines when I type the word "movies" into the Finder search. The first thing it does is drown in results for "m*"...

Code: Select all

[~] # qooba --service spotlight --debug 3 --foreground
[2021/06/05 12:02:34, 3] spotlight_service: max_result_num:300
[2021/06/05 12:02:34, 3] spotlight_handler: ---------------- start ----------------
[2021/06/05 12:02:34, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="m*"
then returning 200+ results for "mo*"

Code: Select all

[2021/06/05 12:03:01, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="mo*"cdw||kMDItemTextContent=="mo*"
then returning 300 results for "mov*" (by name and contentType)

Code: Select all

[2021/06/05 12:03:20, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: kMDItemContentType=="com.apple.quicktime-movie"||kMDItemSubject=="mov*"
then 10 results for "movi*"

Code: Select all

[2021/06/05 12:03:48, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: *=="movi*"cdw||kMDItemTextContent=="movi*"
then "movie*"

Code: Select all

[2021/06/05 12:03:50, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: _kMDItemGroupId=="7"||kMDItemSubject=="movie*"
and finally for "movies*"

Code: Select all

[2021/06/05 12:03:51, 3] spotlight_handler: sk: 4, uid: 1001, search_path: /share/CACHEDEV1_DATA/Photography, query: _kMDItemGroupId=="7"||kMDItemSubject=="movies*"
last result sent to samba:

Code: Select all

[2021/06/05 12:04:20, 3] spotlight_handler: successfully send the response to Samba
Not really a quick way of searching, 1 minute 48 seconds for the full round. The same search in the qsirch web portal takes about 6 seconds.
mousebat
Starting out
Posts: 39
Joined: Wed Feb 12, 2020 10:39 pm

Re: Fix for Qsirch and Spotlight

Post by mousebat »

The reason it takes so long for your typed search to come back is because the search is not “debounced” as you type. It wastes 2 http requests almost every keystroke. The qsirch web app has a JavaScript debounce function that keeps things in check.

As mseewald said start the search locally and then click “shared” in the search filter. It’s a decent enough workaround and will ensure snappy results.

👍🏻
chrisonnas
Getting the hang of things
Posts: 50
Joined: Sat Jan 05, 2019 5:27 pm

Re: Fix for Qsirch and Spotlight

Post by chrisonnas »

mousebat wrote: Sat Jun 05, 2021 9:39 pm The reason it takes so long for your typed search to come back is because the search is not “debounced” as you type. It wastes 2 http requests almost every keystroke. The qsirch web app has a JavaScript debounce function that keeps things in check.

As mseewald said start the search locally and then click “shared” in the search filter. It’s a decent enough workaround and will ensure snappy results.

👍🏻
...and dont forget to restart the spotlight service before killing the SSH session!

Code: Select all

qooba --service spotlight
chrisonnas
Getting the hang of things
Posts: 50
Joined: Sat Jan 05, 2019 5:27 pm

Re: Fix for Qsirch and Spotlight

Post by chrisonnas »

I see that QTS 5.0.0 is bringing Samba 4.13 with it, so can we hope for faster Spotlight seaching? Has anyone taken the plunge and upgraded yet?
Post Reply

Return to “Qsirch & Qfiling”