I don;t know much about Tensor Flow workloads, but I saw something similar on an Nvidia forum. The guy there ran cuda-memtest and sanity_check.sh. They also had him reinstall the driver. NVRM errors appear to be errors with the Nvidia kernel module, so I can see how those would be items to look at, potentially the BIOS, too. BIOS 2.7.0: [URL]
Hi Guys, So I have two 2080 TIs , I have a 1100 (redundant) PSU on that server, and the gpu kit, I put one of the cards today. One worked, then I put the other one (work project) and the systems complains: PCIE error in Slot 4.  I removed one card and leaved the one that worked, no problems, from Linux I can un tensor flow from python, etc, it works. Now, if I remove that card and put the other one the system doesn't boot. it complains of that error and doesn't boot. I was thinking that the card is broken but that is not the case. I put the card in one Dell Precision T series that we have and it works, no problem. We tried both PCIe slot in the server to no avail. So me and my co-worker were tshooting this trying to alternate the slots and cards, one card doesn't work with the server.   What do you think guys, can this be a firmware issue in the server? Previously this server had a Tesla K40c and no problem at all.    
2080 TI - Dell Poweredge R720