I did a little bit of testing....

I found this software useful a few years ago... it has gone way better since.

I was compelled to use it again this month, on a much much bigger scale.

It really takes a while to compute the data from 5000 images, doesn't it?

Here is what I have done to speed the things up significantly.

It all starts with the preparation of a rig that will do the job properly... meaning, it will have all the bottlenecks taken care of even before you install this, in my new experience, brilliant software.

If the one has to use many files as I had to use, the hardware issues can make this job tantalizing.

I did some analysis of the hardware usage while I was building my rig and will share the results here.

I have used HP Z620 in its original form as the base. It was all I could afford... I bought it used for about 1000$, with an old Quadro 600 GPU and 32GB ram and 2 SSD 240GB inside...

I have ran my first test on it  as it was... when it came to alignment RC computed the finishing time to be about 6 days... I do not have that kind of time...

First thing obvious was the HDD usage.

The pictures I have used were all at D: drive, but RC makes a lot of usage of its own files stored in its cache folder RC_CH that was on my C: drive.  When I have ran out of RAM the system started to cache on the same drive and RC recalculating the time has prolonged finishing for 3 more days or about...

RC has to read the input files and write out a rather big file after the analysis and that is about 40% of the whole process in my case. System also has to cache.

First thing I did was putting the RC cache on a new, separate, SSD disk.

That has shorten the predicted finish time by half.

Still, when the RAM was all used, the system started to cache... +2 days again.

As I didn't have resources to buy more RAM, I did solve the issue by reducing the quality by just a little bit. I have noticed that only about 1-2 GB of system cache is actually being used so I went from 8000 features per picture to 7000 features per picture, and that did fix the caching problem.

The alignment took about 48 hours to complete, not counting the manual adjustments and preparation.

Than creating the model had to be done... RC asked me which GPU to use... it will not start without some CUDA cores at all. Quadro 600 is a very old card and calculated finish time was 2.5 days with tendency of growing or standing still. I have stopped the yob 2 hours latter with only few percent of it done.

I took a rather new but pretty cheap GeForce GTX 1660 SUPER and restarted the job.

New finish time prediction was only 7.5 hours...

Getting the job done 6 days sooner for only 500$ I have felt much better... Now I have decided to let the job finish and continue to watch for the bottle necks.

The first thing that is obvious is the R/W on the SSD disks. RC is squeezing those and CPU and GPU are not as loaded as they should be. As this phase takes a lot of time to finish I have decided to make a mighty fast RAID for my next run.

I will put as many SSD disks in 2 separate Raid 0 fields. The one for the pictures to be read from and the second for  RC cache folder RC_CH. That should speed things up even more for this phase of the job.

Will continue this after the job I am currently running is finished.

The job still runs. RC is reading my JPGs and writing to cache as crazy. It has used 73,3 GB of cache.

I can see that it is still going trough the reading of my files even after they were read ones before. I cannot explain the reason it is reading some files twice, but the process of analysis is still going on... Coloring data perhaps...

3h and 49 minutes latter, 8 hours to finish... reading of JPGs has stopped. RC is writing some new cache filed reading the old ones, about the same size as before 15 to 40 MB... CPU and GPU usage went up from 3%GPU and 13% CPU average to 17%GPU and 35% CPU, signifying a slightly larger times to process the data, but still a long time to write to cache folder on SSD. Still a HDD speed hungry process...

The finish time has gone up for a few hours again because RC is using the same SSD to read and write now. New finish time is in 9h30min, 4h since the beginning. GPU and CPU load is about the same, around 20% both, indicating the HDD bottle neck still being the major reason for slow speeds.