QSirch reindexing in loop

nikonvulcano
Getting the hang of things
Posts: 66
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano »

Here the output

Code: Select all

 grep ES_ /tmp/qsirch.log
+ MAX_ES_HEAP=1024
+ SYS_ALLOW_MAX_ES_HEAP=2048
+ export MAX_ES_HEAP
+ export SYS_ALLOW_MAX_ES_HEAP
+ /sbin/setcfg Global ES_Mem_Max 1024 -f /etc/config/qusion.conf
+ /sbin/setcfg Global Sys_Allow_ES_Mem_Max 2048 -f /etc/config/qusion.conf
++ /sbin/getcfg Global ES_Mem_Max_Override -d 1024 -f /share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/settings.conf
+ MAX_ES_HEAP_OV=512
+ ES_BIND_ADDR=127.0.0.1
+ ES_LOGGER_LEVEL=INFO
+ /sbin/setcfg Global ES_Bind_Address 127.0.0.1 -f /etc/config/qusion.conf
+ ES_MIN_MEM=256M
+ ES_MAX_MEM=512M
+ /bin/sed -i 's#((ES_BIND_ADDR))#127.0.0.1#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_LOGGER_LEVEL))#INFO#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MIN_MEM))#256M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MAX_MEM))#512M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
WARNING 2019-02-02 14:29:05,270 - es_sync        : should_enlarge_heap_memory 97   - CURRENT_ES_MEM_MAX = 512, SYS_ALLOW_ES_MEM_MAX = 2048
+ INIT_ES_RESULT=0
+ case $INIT_ES_RESULT in
bye
TVS-463 - 16GB RAM - QTS 4.3.6
iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas »

Do you have in /share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/settings.conf a line which sets "ES_Mem_Max_Override = 512"? I don't have one and you may need to delete it.

For me

Code: Select all

/sbin/getcfg Global ES_Mem_Max_Override -d 1024 -f /share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/settings.conf
returns "1024" while in your log I see that 512 is returned (+ MAX_ES_HEAP_OV=512). You may also run it on the command line to make sure that 1024 is returned.
nikonvulcano
Getting the hang of things
Posts: 66
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano »

ok,
I was 512 as you say. I modified the value and set it to 1024. Or better to delete the line?

the new outputs, after stop and restart

Code: Select all

ps -ef|grep elastic
19258 admin    458344 S < /share/CACHEDEV1_DATA/.qpkg/JRE/jre/bin/java -Xms512M -Xmx1024M -Xss256k -Djava.awt.headless=true -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch -cp :/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/elasticsearch-1.4.2.jar:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/*:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/sigar/* -Des.logger.level=INFO -Des.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -Des.http.port=5028 -Des.network.host=127.0.0.1 -Des.transport.tcp.port=5029 org.elasticsearch.bootstrap.Elasticsearch
26757 admin      2388 S   grep elastic

Code: Select all

 grep ES_ /tmp/qsirch.log
+ MAX_ES_HEAP=1024
+ SYS_ALLOW_MAX_ES_HEAP=2048
+ export MAX_ES_HEAP
+ export SYS_ALLOW_MAX_ES_HEAP
+ /sbin/setcfg Global ES_Mem_Max 1024 -f /etc/config/qusion.conf
+ /sbin/setcfg Global Sys_Allow_ES_Mem_Max 2048 -f /etc/config/qusion.conf
++ /sbin/getcfg Global ES_Mem_Max_Override -d 1024 -f /share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/settings.conf
+ MAX_ES_HEAP_OV=1024
+ ES_BIND_ADDR=127.0.0.1
+ ES_LOGGER_LEVEL=INFO
+ /sbin/setcfg Global ES_Bind_Address 127.0.0.1 -f /etc/config/qusion.conf
+ ES_MIN_MEM=512M
+ ES_MAX_MEM=1024M
+ /bin/sed -i 's#((ES_BIND_ADDR))#127.0.0.1#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_LOGGER_LEVEL))#INFO#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MIN_MEM))#512M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MAX_MEM))#1024M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
WARNING 2019-02-02 14:57:47,327 - es_sync        : should_enlarge_heap_memory 97   - CURRENT_ES_MEM_MAX = 1024, SYS_ALLOW_ES_MEM_MAX = 2048
+ INIT_ES_RESULT=0
+ case $INIT_ES_RESULT in
bye
TVS-463 - 16GB RAM - QTS 4.3.6
iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: QSirch reindexing in loop

Post by iam@nas »

I would delete the line. I wonder when it was set - maybe by a previous version.
nikonvulcano
Getting the hang of things
Posts: 66
Joined: Sun Aug 16, 2015 7:04 pm

Re: QSirch reindexing in loop

Post by nikonvulcano »

iam@nas,
may be that now is ok, what do you think?

Code: Select all

ps -ef|grep elastic
19048 admin      2284 S   grep elastic
19258 admin    532852 S < /share/CACHEDEV1_DATA/.qpkg/JRE/jre/bin/java -Xms512M -Xmx1024M -Xss256k -Djava.awt.headless=true -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch -cp :/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/elasticsearch-1.4.2.jar:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/*:/share/CACHEDEV1_DATA/.qpkg/Qsirch/elasticsearch/lib/sigar/* -Des.logger.level=INFO -Des.path.logs=/share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/es_logs -Des.http.port=5028 -Des.network.host=127.0.0.1 -Des.transport.tcp.port=5029 org.elasticsearch.bootstrap.Elasticsearch

Code: Select all

grep ES_ /tmp/qsirch.log
+ MAX_ES_HEAP=1024
+ SYS_ALLOW_MAX_ES_HEAP=2048
+ export MAX_ES_HEAP
+ export SYS_ALLOW_MAX_ES_HEAP
+ /sbin/setcfg Global ES_Mem_Max 1024 -f /etc/config/qusion.conf
+ /sbin/setcfg Global Sys_Allow_ES_Mem_Max 2048 -f /etc/config/qusion.conf
++ /sbin/getcfg Global ES_Mem_Max_Override -d 1024 -f /share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/settings.conf
+ MAX_ES_HEAP_OV=1024
+ ES_BIND_ADDR=127.0.0.1
+ ES_LOGGER_LEVEL=INFO
+ /sbin/setcfg Global ES_Bind_Address 127.0.0.1 -f /etc/config/qusion.conf
+ ES_MIN_MEM=512M
+ ES_MAX_MEM=1024M
+ /bin/sed -i 's#((ES_BIND_ADDR))#127.0.0.1#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_LOGGER_LEVEL))#INFO#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MIN_MEM))#512M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_MAX_MEM))#1024M#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
+ /bin/sed -i 's#((ES_NICE))#-5#g' /share/CACHEDEV1_DATA/.qpkg/Qsirch/supervisord.conf
WARNING 2019-02-02 15:44:20,441 - es_sync        : should_enlarge_heap_memory 97   - CURRENT_ES_MEM_MAX = 1024, SYS_ALLOW_ES_MEM_MAX = 2048
+ INIT_ES_RESULT=0
+ case $INIT_ES_RESULT in
bye
TVS-463 - 16GB RAM - QTS 4.3.6
User avatar
jcmbowman
Starting out
Posts: 29
Joined: Tue May 24, 2016 10:17 am

Re: QSirch reindexing in loop

Post by jcmbowman »

I made the changes suggested here:
iam@nas wrote: Sun Jan 27, 2019 3:13 am I assume that ES and/or Tika run out of memory. For me the ES max java heap memory is set to 256 MB which may be sufficient for 50.000-100.000 documents, depending on the type and metadata.
You could try to edit these configuration files and change 256 to 1024 and restart then Qsirch.
/share/CACHEDEV1_DATA/.qpkg/Qsirch/conf/qusion.conf - ES_Mem_Max = 256 - Tika_Mem_Max = 256
/share/CACHEDEV1_DATA/.qpkg/Qsirch/qusion.sh - ES_MAX_HEAP=256
'ps -ef|grep elastic' should then list the ES process with '-Xmx1024M'. If you modified also Tika_Mem_Max it could fix also Tika.

'grep "java.lang.OutOfMemoryError" /share/CACHEDEV1_DATA/.qpkg/Qsirch/tmp/tika_server.log' should show Tika problems.
...and after adding a few exclusions to the index path for my PLEX server and a few other regularly updated working folders I was able to get QSirch to complete an index. Now that the index is complete my CPU usage regularly idles at or below 10% - previously it was sticking to a baseline of roughly 50% CPU because of QSirch.

I have upwards of 1,000,00 files on the TS-451+ and about 10TB of data currently.

thanks for pointing me in the right direction!
TVS-h1288X - 48 GB RAM
2x Samsung SSD 850 EVO 1TB - RAID1 783GB Usable (Apps)
4x WDC WD120EFBX 12TB - RAID5 32.7TB Usable (Data)
2x Samsung SSD 850 EVO 1TB - RAID0 1.8GB Usable (Cache Acceleration for Data)
buggy82
Getting the hang of things
Posts: 86
Joined: Thu Mar 20, 2008 11:54 pm

Re: QSirch reindexing in loop

Post by buggy82 »

Thanks alot. Following this advice and increasing the Java Memory to 1024 as stated has resolved my QSirch issues.

The issues were: Even with QSirch indexing disabled, it was still hammering the CPU with 50%+ load, which in part made other services of the NAS not responding or very slow responding.
I found the OutOfMemory entries in the log files as stated and increased the Heap Memory as instructed. Restarted QSirch and let it index overnight. Now everything is fine.

Thanks a lot iam@nas !

buggy82
TS 409 Pro - decommissioned
TS 559 Pro+ - degraded to an iSCSI target, disabled all services
TS 651 + UX 800P 3xRAID5 (as Storage Pool 1), actually RAID50 + 1 Single Drive (test VMs/temp data, QTS apps)
Post Reply

Return to “Qsirch & Qfiling”