CUDA error : 2 : out of Memory
Answered
specs
Processor: AMD FX-9590 Eight-Core
Memory: 16.0GB
Graphics Cards
- Geforce GTX 1080 Ti
- Geforce GTX 970
OS: Windows 10 64bit
-----------------------------------------------------------------------------
The Error : CUDA error : 2 : out of Memory
-----------------------------------------------------------------------------
This happens every time I go to reconstruct anything other than the preview reconstruction.
Things I have tried
- https://support.capturingreality.com/hc/en-us/community/posts/360006253079-I-keep-getting-CUDA-error-number-2-WHY-?page=1#community_comment_360000847519
- I have followed the instructions from people on this post and have tried every archived version of nvidia drivers available from 441.28 - 436.02
- I made sure that my cache was set to a drive with plenty of space
- I have restarted the process and created a new scene several times to see if it was a bad file
- in settings I have messed with vertex distance, decimation threshold and force single part mode
- I have changed the reconstruction area anywhere from the whole model to just bits and pieces
Things to note about the setup
- processing 335 images
- total number of points: 1,230,123
- image features range from 60,000 - 90-000
- lens model: EF40mm f/2.8 STM
Reconstruction Settings
- have set both GPUs to use and every combination inbetween
- Normal Model Image downscale: 5
- Mesh calculations: anywhere between 0 - 10
- I have left macimal vertex count per part at: 5,000,000
- Detail decimation factor: anywhere between 0 - 100
I have not changed or messed with anything in the Advanced settings.
I hope this give you enough information about the issue and what I've done to try and make a reconstruction happen. If there is anyone with more information on what I should try next or has a setting I should change I would love to hear from you!
-
Exactly there is no difference if you will use one or two GPU units and support only advice o use one graphic card. Besides all my scans was done with high detail on two GPU units until installation of new version of reality capture :( so definitely it's a software issue in new version
-
Hello,
This is our official statement for this issue.
Two immediate temporary solutions:
1. Physically remove all GPU cards, except one - single card configuration.
or
2. Install old GPU driver. The version 419.67 is working for us.
Reason:
The issue is with the CUDA memory de-allocation function, that has stopped working properly with latest NVIDIA GPU drivers.
More specifically the function CUDAFreeHost() resulted with success code, but the memory was not de-allocated and therefore after some time, the GPU pinned memory was filled up and the SW ended up with the message "CUDA error : 2 : Out of memory".
It seems that this bug causes problems also to another CUDA developers:
https://devtalk.nvidia.com/…/cudafreehost-not-clearing-all…/
We have contacted NVIDIA and we hope that there will be a new driver with fix very soon.
Meanwhile, we are also trying to fix the problem by some workaround in our code and we will try to release hotfix ASAP.
-
Hi, I have this CUDA error popping up once in a while without any clear logic - is sometimes appears, sometimes not -no matter if the project has 40 or 900 images - but when it does, it is usually during the texturing process and it keeps reappearing repeatedly even after canceling the error window many times - so as the result the texturing process could not be finished...
I have tried the newest Nvidia drivers, and the older one recommended above (419.67) but it still keeps appearing, can you help me solve this, please???
PS I do not have SLI - just one older GeForce GTX 750 card...
Please sign in to leave a comment.
Comments
14 comments