Computer Crash during "Calculate Model"

Answered

Comments

29 comments

  • Avatar
    Wishgranter
    Hi Michael Agronin
    If you are experienced, then look at the EVENT VIEWER for CRITICAL ERRORS and put the info here... So that we can pinpoint the issue..
    0
    Comment actions Permalink
  • Avatar
    Diego Torrijos
    Today I have exactly the same problem, change video card from a GTX 780, worked well, but now upgrade to a GTX Titan X card and every time I give calculate model the computer without warning goes off, and probe with different programs to verify the card and all work fine, the problem is only with RC. Every time I try to create a model using, My computer crashes after a few seconds. No warning, no blue screen, just off.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    In your case it looks like a PSU issue...

    Fill out this http://www.coolermaster.com/power-supply-calculator/ and post the results here..
    + ucp.php?i=profile&mode=signature put you PC config in your signature

    How old is your PSU ?
    0
    Comment actions Permalink
  • Avatar
    Diego Torrijos
    Hi Wishgranter,

    Thanks for your quick response.

    Power-supply-calculator:

    Load wattage=805 W ; Recommended PSU wattage=855 W

    My PSU Dell 1100W Power Supply.

    Perform stress tests of the GPU and all good, perform extraction points in ultra resolution with the evaluation version of Agisoft and works well, perhaps RC requires more energy, the problem is only with RC, and just seconds after start calculation model.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    How old is the PSU ? In principle, RC uses the CPU and GPU much more efficiently. even when you see 100 % utilization in agisoft it does not mean your CPU - GPU is working at 100 %

    The problem with PSUs is, even though they say 1000W, in reality you get much lower wattage, as it generates 12-5-3V and so you cannot use the full 1k Watt. If it says 855W, then you need at least 1200 Watt PSU.. I am using 1300 W PSU but I keep the load under 800W.

    Look at the 12V rails amperage ( Volts + Amperes = Va = Watt ), highly probably it is a PSU-related issue..
    Try to remove some HDDs or GPUs, if possible, and test it again, it should be able to work without crashes..
    Look at the EVENT VIEWER for critical errors , if it is Err41 power outage, then it is 100 % sure the PSU
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    Wishgranter - Thanks for the reply.

    I've looked through the event log - your suspicion is correct, all the critical events are Error 41.

    I went through the PSU calculator and came up with 1075W with all three video cards in, and 744 with just one. With the 1300W PSU in there, it should be fine with either situation, but crashes with both. I've contacted Dell, and unfortunately its a proprietary supply and 1300W is the biggest they make that fits.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Michael Agronin
    with the PSUs it is a pain :D

    The main problem is your PSU probably cannot deliver enough watts to the CPU or GPUs per power line ( from PSU )
    Go to RECONSTRUCTION ->SETTINGS -> GPUs to CHOOSE button and deselect ONE or TWO GPUs and try to reconstruct it.

    So what you can do if it is a PSU ? you could eventually use TWO PSUs in your PC ( is it possible ? ) or get more powerful 1500w PSU, or change the 980 for the new 1080 GPUs, as they need much lower wattage ( and you will even get significant speed-up )
    0
    Comment actions Permalink
  • Avatar
    Diego Torrijos
    Maybe you should look into the RC code, my computer is a Dell Precision also, but all applications work fine, except RC, is the only one that crashes.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    No need to look in the code, we know it’s working properly, but, as you can see, more users have problems with their PSUs.

    The thing is we use CUDA sw that is very EFFICIENT, so it uses the GPU quite heavily -> a lot of energy is therefore used...
    In games and much sw you can observe 100 % GPU utilization but in reality it works at just 50-60 % of the possible performance, so you can‘t get power issues there.. RC has been made with max performance in mind so it uses all resources and it‘s very effective..

    I highly recommend you to read a bit about this stuff here:
    http://www.overclock.net/t/1383408/gtx- ... -you-using
    http://www.guru3d.com/articles_pages/ge ... iew,4.html

    GeForce GTX Titan 3x SLI - on your average system the cards require you to have a 1000~1200 Watt power supply unit as a minimum.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    I have a question as I see you are using OLD Xeons the X5690, is the PSU the same age ? I mean is it 3-5 years old ?
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    Wishgranter - I pulled 2 of the GPUs out of the machine, running with a single GTX980 to test. Same exact crash. You're telling me that I'm overloading a 1300W PSU with a single GPU? This does not seem reasonable. According to coolermaster, it should be drawing ~750W peak with 1 GTX980 installed.

    2 PSUs is not an option with this machine, and due to the proprietary nature of the PSU design, I cannot even upgrade beyond 1300W (not that it should be necessary).

    1080s are in our future for this machine, no question. Just another couple weeks...
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Michael Agronin
    When it crashes even with SINGLE PSU, then you have some real problems with the PSU.
    This cannot happen with a new properly configured PSU...

    Can you send me PRECISE info on your PC + the PSU ?

    It looks like a not properly "connected" CPU + GPU to PSU, but I can say more just when I see the detailed PC + PSU info...
    0
    Comment actions Permalink
  • Avatar
    Diego Torrijos
    Not talking about games. I speak equivalent testing programs, Agisoft PhotoScan and Pix4D, all perform under maximum load, but RC crashes.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    I’m quite experienced with all the mentioned sw and KNOW how they work on the GPU side of processing data. do you know that Nvidia is BLOCKING performance under OpenCL in drivers by 50 % ? ( from 286.xx version of their drivers ) so even though it says it’s using the GPU at 100 %, in reality it’s lower than 50% ? so it can’t crash under the load you mention. It’s NOT possible. Look for good GPU monitoring sw, like MSI afterburner, and have a look at it at least there..

    Download this http://www.ozone3d.net/benchmarks/fur/ and stress test you PC config... This one can get to GPU use like RC can do...
    and send me the PC config + detailed PSU info please...
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    Doing some testing this morning. With all 3 GPUs enabled, I crashed RC running high detail. I've a Killawatt PS (1800W max) hooked up to the PSU. It reports real time power consumption and records max levels. Peaked at 680W. Tried again with just 1 GPU enabled, Peaked at 530W.

    It's a Dell Precision T7610 with 1300W PSU. Unfortunately, I don't think there's much more I can provide in terms of info on it - it's proprietary, and Dell is pretty useless (I was talking to them last week about this issue, and they told me several times it was standard ATX PSU, despite the fact its an odd form factor with a hot swap blade connector).

    I will run the ozone3D benchmark and see how that goes...
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Diego Torrijos
    And I have asked you how OLD is your PSU because the PSUs are aging and losing 5-10 % of their capacity per year.. so if you have an OLD PSU, it can be at the end of its life, which is common these days even for pricey models :-(
    Look how it is connected to the PSU RAILS whether the rails have ENOUGH capacity to feed the GPUs ( CPUs )
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    Minor update: This machine also is actively being used with Agisoft Photoscan. Just glancing at the Killawatt - it peaked at 830W running PS without crashing.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Michael Agronin
    if you can take a look at the RAILS connection and their amperage, that can be the issue. if you overload rails, then it can get in this issues as it can’t deliver the required power and it shuts down..
    I’m speaking from my own ( 20-year experience ) with PC assembling. Even my Dual Xeon (overclocked ) is a custom-made PC
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    How do you suggest checking the AMPERAGE of a RAILS? Generally you need to have an ammeter in line to get a reading on that.

    Are you suggesting that CR only works on custom made PCs? Is there an inherent problem using DELL computers with this software?
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Michael Agronin
    As for the amperage on rails, that was the info I wanted to get from you :D So if it’s possible, let me know the precise DELL PC model so that I can dig into this more....

    It’s not a problem with particular PC mark or so, like you think, it’s about PROPER PC config that is capable of stable working...
    So for now look for the precise PC config and send it to my email milos.lukac@capturingreality.com
    0
    Comment actions Permalink
  • Avatar
    Michael Agronin
    It's a Dell T7610 with a 1300W PSU. They don't publish (or at least I haven't been able to find) any detailed specs of the PSU.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Michael Agronin

    here is a bit better zoom in the PSU details :D

    HTB1YhZnKXXXXXcuXXXXq6xXFXXXy.jpg


    It uses 18A per RAIL. but a very unusual connector :D
    https://dl.dropboxusercontent.com/u/150 ... _en-us.pdf

    make a photo from the interior of the PC and send it to milos.lukac@capturingreality.com
    0
    Comment actions Permalink
  • Avatar
    Sander Diels
    Hey all,

    I have had the similar problems but after long searching I have found something that worked for me.
    I'm guessing there's a good chance it will also do the trick for you!

    The problem of the crashes is most likely overheating of your CPU.
    This is because RC runs a heavy process and takes maximum capability of your processor.

    If your computer settings are set in a wrong way it will cause CPU overheating and crashes (because it can not handle the heavy demand)

    Go to your POWER SETTINGS and change the POWER PLAN maximum state to 70-80% and the minimum state to 0%

    This will most likely do the trick and prevent RC from crashing in the future!
    Good luck!

    Full video tutorial -> https://www.youtube.com/watch?v=y5G6MAHAh9I
    (only try part 1 of the video)
    0
    Comment actions Permalink
  • Avatar
    yoz
    Hi folks,

    I have a similar problem, my PC crashes last night during an alignment of 1200 imgs. And i think it's not a RC specific problem because i have same crashes with Pix4D when i push it on big computation.

    I have always the same symptoms : PC reboot and error in windows events log : error 41 Kernal-Power

    I have changed recently my PSU, so it's a new one.

    Here is my config :

    - PSU : Corsair RM850x (850 Watt)
    - Mother board : MSI X99A Gaming 7
    - CPU : Intel Core i7-5930K
    - RAM : 64Go Crucial Ballistix Sport
    - GPU : 2 x MSI GeForce GTX 970
    - SSD : Samsung 850 EVO - SSD 500 Go
    - HDD : 2 x Seagate Barracuda 7200 3To

    So it's very frustrating, because i cannot finish none of my projects.

    Thanks in advance for help !

    Regards.

    Yoz

    ps : here is my bluescreen log :

    bluescreenview.jpg
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Yoz
    We will post a solution once we finish talking about this....
    0
    Comment actions Permalink
  • Avatar
    chris
    I think I've manged to solve my issue with crashes on large scale scenes. at least I know what i can tweak now.

    I've had lots of issue with it crashing on high, while working on large scenes.

    I think the issue that i had was with the model generation algorithm. when it ran out ram, it just went off into nowhere, and lost what it was doing. this was happening about a week starting them.

    while this error was happening i would see page file usage go over 1.4tb. which is pretty crazy. but even giving it enough swap space, the processing just dropped of and did nothing.

    anyway I've now started using minimal distance between two vertices. and playing around with this setting, I've manage to get the jobs to start working again.

    but i did go through a couple of months without getting any models to work before i found this.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Chris
    one thing is a hardware issue, software + driver issues another.
    Many PCs are not designed very well, mostly regarding proper PSU capacity or cabling-connection to PSU etc.
    0
    Comment actions Permalink
  • Avatar
    Jesus Bibian Jr
    Testing out the demo and I'm experiencing this exact problem when calculating the model. No warning, PC simply shuts down. I have a pretty high end machine running high end software on a regular basis, so this seems odd.

    I think users should have the option to set how many GPUs and CPU cores to use within the software like it is in ******. Perhaps this would eliminate these crashes.

    I'll try the suggested workaround and see if that works for me.
    0
    Comment actions Permalink
  • Avatar
    Wishgranter
    Hi Jesus Bibian Jr

    You can play with CPU allocation -> affinity in the TASK MANAGER, depending on the Windows version, but look at the DETAILS and look for AFFINITY, then disable 1+ cores.

    The main problem is, and I have observed it over a long time, the PC crashes are mostly related to not optimal setup, CPU is overheating, the PSU is NOT designed to deliver proper amperage to CPU-GPU ( running on 95-100 % all the time will KILL it very soon, that is why we have 30+ % headroom for PSU ). Few times, like with YOZ, it was highly likely because of drivers and Windows updates + BIOS. So it first needs to be pinpointed where is the actual problem. If you have a good PC, then the software cannot restart it. Speaking from experience with PC building since 1996, I had the first Dual xeon setup in 1998 in our country... so a LOT of experience with PC that restarted for no reasons.. so based on all this I can say a lot about PROPER PC configurations...
    0
    Comment actions Permalink

Please sign in to leave a comment.