Qumagie slow with big fileset

[nasphotographer]

Qumagie is painfully slow with a big data set for me. granted im on a ts 431+ with a coral ai attachment.
when i say slow, im mainly concerned with viewing my data. the slowness with recognizing photos is not an issue now with the ai accelerator, and 5hat it works in the background means its a non issue for me.
when i log into qumagie, it takes about a minute to fully load my 90000 image library. this is just to view the list of albums it has. when i view specific albums, for example a specific person, its slow to load it, and slow to get to the photo i want to view. its always loading the entire timeline of ALL of the photos in the album. i suspect this is why its slow.
it would be great if there was a pagination system to speed up loading. the natural breakdown by days is a good way to paginate. by every X number of photos is fine too.
the main page album page needs to speed up too. i dont see the reason for the slowness other than poor architectural design of the software.
i love the functionality of qumagie , but this slowness to view the content just makes it almost useless.

[Guapo81]

I use Qumagie also on my main NAS TS-h886 and I am experiencing the exact same thing. In my case my library is 140K pictures.
Allthough the speed of Qumagie on QuTS-hero has improved a bit somewhere over the last months, but still it's like you said slow.

From starting the app till the moment all the pictures load in the home screen of Qumagie takes rougly 20 sec. in my case.
Also loading the smart albums takes rougly 20 sec. Not nearly as bad as 1 min. but still this should be able to load faster with the right paging / indexing system.

[Theliel]

It's been a while since I researched all the possible slowdowns in QuMagie, to the point of fully understanding how it works. And I'm afraid it's not that simple. As an example, my collection is greater than 150k. In my case I use facial detection, but I DO NOT USE QUMAGIE to tag them, I tag EVERYTHING externally, that way the metadata is written in the file itself, so if I reindex everything again, I really don't lose anything at all. I have about 120k faces tagged from about 600 different persons.

Oddly enough, QuMagie is quite "fast". The main problem in the waiting time is the SQL queries that must be made to receive the data from the DB and return it to the browser. This time will vary greatly depending on 5 variables mainly:

-Hardware: Type of storage where the DB is located (HDD/SSD/M2), CPU and RAM
-Your Collection: Number of indexed files (Images/Videos/Music) and their Metadata.

When a file is "indexed", depending on the type of file it is, relative information is passed to the database, information such as file type, duration, title, album, dates... resolution, relevant exif information (GPS , camera, lens...), the facial regions detected, the similarity thresholds in the facial identification system... When QuMagie is used, depending on what "action" is performed, it performs different SQL queries to obtain the information that it needs to display the content on the screen, with the important addition that it also has to respect and take into account all the permissions applied to each of these files, and as a final step, also read the generated thumbnails from disk.

QuMagie over the last 6 months has improved considerably in speed, especially since the last major update. These improvements have come thanks to "simplifying" SQL queries as much as possible, and at the same time caching them for reuse. In my case, the only thing that "slows down" QuMagie right now is accessing "Albums", which takes 10-20 sec. Doesn't seem like much to me considering I have about 120k tags only in faces. It is true that the AI ​​system I use is external, but the DB reads all the tags equally, the only difference is that instead of appearing in "people" it appears in "tags".

That's not to say that QuMagie can't still improve quite a bit. It still has quite weak points, which give room for improvement:

-Store photo thumbnails and face thumbnails in DB, not as separate files
-Make background preloads of the theoretical SQL queries necessary for the other sections, so that while we are on the main screen seeing something, the system can receive the data and cache it in case the user accesses it later.
-Allow in a simpler way (without using tricks), the exclusion of specific folders from the indexed content. That is, to be able to "delete" from the DB all those folders whose content is not very important to us either. Delete the indexing, not the content, of course.
-Simplify the structure of the DB

From a user point of view, it can also help tremendously in improving performance:

1º. Naturally limiting the content. The best way to do this is to not index content that is not important to view/treat in QuMagie. Exclusion of specific folders, which can be done via SSH, becomes extremely useful here.
2º. Make sure that, in case you have more than one hard disk, the system disk (the main one) is the fastest. If this is not the case, it is not difficult to convert another one that we have into a system disk, preferably an SSD / M2
3º. On many occasions, a large amount of content is duplicated, it is essential to always have a healthy collection.

[mustard]

It's been a while since I researched all the possible slowdowns in QuMagie, to the point of fully understanding how it works. And I'm afraid it's not that simple. As an example, my collection is greater than 150k. In my case I use facial detection, but I DO NOT USE QUMAGIE to tag them, I tag EVERYTHING externally, that way the metadata is written in the file itself, so if I reindex everything again, I really don't lose anything at all. I have about 120k faces tagged from about 600 different persons.

This has really confused me. I'm certain Qumagie used to write tags (not facial IDs) to files. It doesn't appear to do that anymore. However, it reads existing tags, and if an existing tag is removed, it removes it from the file.

[anonymopt]

how can i backup the AI facial detection tags?