QSirch reindexing in loop
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
QSirch reindexing in loop
Hi
since first installation (just presented) QSirch had always issues, for me.
I have also reinitialize the NAS and reinstall QSirch, but
1-if reboot the NAS or update the firmware, the indexing restart from begin as the first time (taking 3-5 days to terminate). This each time !!
2-the CPU load during the indexing is about 50% only for QSirch process, than low to 20-30% when it terminate.
My NAS is a TVS-463 with 16GB of RAM.
I opened also a ticket but the help desk told me that is difficult to replicate my issue, and since months I'm still rest in the same situation.
Can anybody help me?
thanks
bye
since first installation (just presented) QSirch had always issues, for me.
I have also reinitialize the NAS and reinstall QSirch, but
1-if reboot the NAS or update the firmware, the indexing restart from begin as the first time (taking 3-5 days to terminate). This each time !!
2-the CPU load during the indexing is about 50% only for QSirch process, than low to 20-30% when it terminate.
My NAS is a TVS-463 with 16GB of RAM.
I opened also a ticket but the help desk told me that is difficult to replicate my issue, and since months I'm still rest in the same situation.
Can anybody help me?
thanks
bye
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
I assume that ES and/or Tika run out of memory. For me the ES max java heap memory is set to 256 MB which may be sufficient for 50.000-100.000 documents, depending on the type and metadata.
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - MAX_ES_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.
'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.
Edited: -ES_MAX_HEAP +MAX_ES_HEAP
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - MAX_ES_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.
'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.
Edited: -ES_MAX_HEAP +MAX_ES_HEAP
Last edited by iam@nas on Sat Feb 02, 2019 7:00 pm, edited 1 time in total.
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Hi iam@nas,
thank you for your reply.
I've checked the Tika process and I see that some times it uses a lot of CPU.
My TVS-463 has 16 GB of RAM.
I don't think that my problem is an out of memory, but I'm not an expert.
Please, look at the screenshot in attachment...
What can you tell me?
Thanks
Bye
thank you for your reply.
I've checked the Tika process and I see that some times it uses a lot of CPU.
My TVS-463 has 16 GB of RAM.
I don't think that my problem is an out of memory, but I'm not an expert.
Please, look at the screenshot in attachment...
What can you tell me?
Thanks
Bye
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
Open a shell and locate the 'tika_server.log' file. It should be in the '/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/' folder.
If it returns 'Exception in thread "Thread-1234" java.lang.OutOfMemoryError: Java heap space' there was a memory problem. There are some time stamps in the log which are not related to this error. So one can try to guess when the problem occured. It may be a good idea to stop Qsrich in the web UI, rename the log file
and start it again to get a clean log file.
I have no extractors running at the moment. So many running extractors allocating all CPU resources seems to be bad.
Code: Select all
cd /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/
grep "OutOfMem" tika_server.log
Code: Select all
mv tika_server.log tika_server.log.old
I have no extractors running at the moment. So many running extractors allocating all CPU resources seems to be bad.
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Hi iam@nas,
thanks for you reply.
I tried to do as you suggest, but I receive
Binary file tika_server.log matches
Please look at the attachment.
However, before to rename the log, I wait in this days the check from RD of helpdesk.
Moreover to complete the indexing it needs some days.
Why the grep don't have worked for me?
thanks
bye
thanks for you reply.
I tried to do as you suggest, but I receive
Binary file tika_server.log matches
Please look at the attachment.
However, before to rename the log, I wait in this days the check from RD of helpdesk.
Moreover to complete the indexing it needs some days.
Why the grep don't have worked for me?
thanks
bye
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
It seems that your log file contains also some binary data. Or your grep / busybox has some issues with the file.
In any case the grep command matched. Your log file contains "OutOfMem" so there were some Tika memory issues some time ago.
I installed Entware-ng and then most of the standard unix tools (procps*, coreutils*). QTS also contains most of these tools within busybox and thus with a very limited functionality.
In any case the grep command matched. Your log file contains "OutOfMem" so there were some Tika memory issues some time ago.
I installed Entware-ng and then most of the standard unix tools (procps*, coreutils*). QTS also contains most of these tools within busybox and thus with a very limited functionality.
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Hi iam@nas,
thank you again.
After intervention of helpdesk, I will delete the log.
Me too, I've installed Entware to add some tool, but I didn't know of procps*, coreutils* ...
thank you for the precious infos !!
bye
thank you again.
After intervention of helpdesk, I will delete the log.
Me too, I've installed Entware to add some tool, but I didn't know of procps*, coreutils* ...
thank you for the precious infos !!
bye
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Hi iam@nas
I tried your suggestion. I done all
modified to 1024 both
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256
restarted Qsirch
but when i do 'ps -ef|grep elastic' I don't receive ES process with '-Xmx1024M' but
Note that my Qsirch, now, is v.4 version.
Please can you help me again?
thanks
bye
I tried your suggestion. I done all
iam@nas wrote: ↑Sun Jan 27, 2019 3:13 am I assume that ES and/or Tika run out of memory. For me the ES max java heap memory is set to 256 MB which may be sufficient for 50.000-100.000 documents, depending on the type and metadata.
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.
'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.
modified to 1024 both
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256
restarted Qsirch
but when i do 'ps -ef|grep elastic' I don't receive ES process with '-Xmx1024M' but
Code: Select all
9437 admin 2440 S grep elastic
22202 admin 379020 S < /share/CACHEDEV1_DATA/.qpkg/JRE/jre/bin/java -Xms256M -Xmx512M -Xss256k -Djava.awt.headless=true -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch -cp :/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/elasticsearch-1.4.2.jar:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/*:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/sigar/* -Des.logger.level=INFO -Des.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -Des.http.port=5028 -Des.network.host=127.0.0.1 -Des.transport.tcp.port=5029 org.elasticsearch.bootstrap.Elasticsearch
Please can you help me again?
thanks
bye
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
Looks like I made a typo. Did you modify "MAX_ES_HEAP=256" to "MAX_ES_HEAP=1024" in qusion.sh?
Or did you add ES_MAX_HEAP=256 ? If so please remove it as it has no effect.
Or did you add ES_MAX_HEAP=256 ? If so please remove it as it has no effect.
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
hi,
I think I did everything correctly, please also look at the attached screenshot
thanks
I think I did everything correctly, please also look at the attached screenshot
thanks
You do not have the required permissions to view the files attached to this post.
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
Was the file Qsirch/supervisord.conf re-created during startup?
[...]
To get more details please add 3 lines at the beginning pf of qusion.sh so it starts like this:
Restarting Qsirch will create the file /tmp/qsirch.log
"grep ES_ /tmp/qsirch.log" should output
Code: Select all
$ ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should show the current / last day as date.
$ grep "m:ela" -A 5 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
should output:
[program:elasticsearch]
command=[...]
environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=512M,ES_MAX_MEM=1024M
directory=[...]
To get more details please add 3 lines at the beginning pf of qusion.sh so it starts like this:
Code: Select all
#!/bin/sh
set -x
exec >/tmp/qsirch.log
exec 2>&1
# Get java path ... all other code remains as is
"grep ES_ /tmp/qsirch.log" should output
Code: Select all
+ MAX_ES_HEAP=1024
+ SYS_ALLOW_MAX_ES_HEAP=2048
...
+ /sbin/setcfg Global ES_Mem_Max 1024 -f /etc/config/qusion.conf
+ /sbin/setcfg Global Sys_Allow_ES_Mem_Max 2048 -f /etc/config/qusion.conf
...
+ ES_MIN_MEM=512M
+ ES_MAX_MEM=1024M
...
+ /bin/sed -i 's#((ES_MIN_MEM))#512M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MAX_MEM))#1024M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
...
WARNING 2019-02-02 12:59:58,882 - es_sync : should_enlarge_heap_memory 97 - CURRENT_ES_MEM_MAX = 1024, SYS_ALLOW_ES_MEM_MAX = 2048
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Hi
something don't worked because I don't received your output after grep of supervisors.conf, but this:
ES_MIN_MEM=256M,ES_MAX_MEM=512M
the half values
something don't worked because I don't received your output after grep of supervisors.conf, but this:
ES_MIN_MEM=256M,ES_MAX_MEM=512M
the half values
Code: Select all
[program:elasticsearch]
command=/share/CACHEDEV1_DATA/.qpkg/Qsirch/bin/python -u -c 'import os; os.nice(-5); cmd = "/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin/elasticsearch.nas -D es.logger.level=INFO -D es.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -D es.http.port=5028 -D es.network.host=127.0.0.1 -D es.transport.tcp.port=5029".split(); os.execl("/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin/elasticsearch.nas", *cmd)'
environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=256M,ES_MAX_MEM=512M
directory=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/bin
autostart=false
exitcodes=0
thanksiam@nas wrote: ↑Sat Feb 02, 2019 8:22 pm Was the file Qsirch/supervisord.conf re-created during startup?Code: Select all
$ ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf should show the current / last day as date. $ grep "m:ela" -A 5 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf should output: [program:elasticsearch] command=[...] environment=JAVA_HOME=/share/CACHEDEV1_DATA/.qpkg/JRE/jre,ES_MIN_MEM=512M,ES_MAX_MEM=1024M directory=[...]
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
Code: Select all
ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
-rw-r--r-- 1 admin administrators 11456 2019-02-02 11:59 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
-
- Getting the hang of things
- Posts: 66
- Joined: Sun Aug 16, 2015 7:04 pm
Re: QSirch reindexing in loop
Sorry, I don't sure I understood correctly.
The hour and date are correct. I've already stop and restart in App Center, many times.
What do you suggest me?
Thanks
The hour and date are correct. I've already stop and restart in App Center, many times.
Code: Select all
ls -l /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
-rw-r--r-- 1 admin administrators 11455 2019-02-02 12:21 /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
Thanks
TVS-463 - 16GB RAM - QTS 4.3.6
-
- Easy as a breeze
- Posts: 267
- Joined: Wed Jun 15, 2016 2:49 am
Re: QSirch reindexing in loop
In viewtopic.php?f=361&t=145738#p704700 I posted the 'set -x' settings for qusion.sh to create /tmp/qsirch.log after startup. 'set -x' runs the script in verbose mode and in the created output one should see what's going on. It would be great if you could post it.
Last edited by iam@nas on Sat Feb 02, 2019 9:34 pm, edited 1 time in total.