I have turned myself into an incompetent researcher. I have screwed up my career, now I would like some advice how I can recover? I hope I am not at the end of the road. I had flunked advanced engineering mathematics and applied mechanics during my undergrad second year. Though I scored decently in the reexamination during the subsequent semester, I never recovered from the blow to the self-esteem from flunking first and last time in my education.
I liked applied mathematics and mechanics, though I was never good at it. I was very good at experimental mechanics but I hated the manual process of sample preparation, lack of theoretical studies. When it came to choose a research field during master's degree, I chose computational mechanics. I figured, though I might not be good at this, I can always learn and not repeat my undergrad debacle.
My error was, not taking enough courses. My grad school courses were not mathematically rigorous enough and were quite superficial. I did not take the harder courses as I feared failure and repeat of my undergrad flunking.
My PhD advisor, committee did not force me to take the rigorous and advanced courses as I had a PhD topic that did not involve much rigorous mathematical framework development. I did not ask help and guidance from professors due to my social ineptitude, worrying about what they would think about me.
I should have taken the 3 graduate level courses when I had the opportunity. I believe, I would have had more confidence, had done better PhD research.
Now, I am doing postdoc in same topic as my PhD. I am searching for postdoc positions and industry Jobs, I find requirement for candidates with rigorous research experience and graduate level coursework. I have none. I have published papers in non-rigorous journals. I am still afraid of mathematics and numerical methods.
I have screwed up my career, now I would like some advice how I can recover? I should have done things differently 5 years ago. I am old now, and inept at my field of study. I have been trying to self study for the past few years, I still suck big time. I should have stayed in experimental mechanics research. I was natural in it. Computational mechanics is not my cup of tea. I bit off more than I can chew.
It is likely that only someone familiar with your field and your portfolio can really help you assess your options. But since you asked for advice, I will offer some observations:

You spent about half your post looking backwards (all the way to undergrad). This is not productive. Instead, look forwards. Where do you want to go? What is the next step?
When considering &quot;where do you want to go,&quot; I recommend you consider all your options. Many people who have never left the university tend to make only superficial inquiries into career options outside of the university.
For a research career, the biggest concrete problem is your poor track record in research. Note, I am assuming that your research record is truly poor as you say, though self-assessment can be very difficult. The academic job market is brutal even for well-qualified candidates, so a poor research record may close some doors, at least temporarily.
I'm not sure what you mean by &quot;I find requirements for...graduate-level coursework. I have none.&quot; I'm very skeptical. In my experience (granted, I'm in the US), it is very unusual for anyone to care about your coursework once you have a PhD. Second, you do have a graduate-level coursework, though you don't think the courses were sufficiently rigorous.
A PhD certifies that you can do research without supervision -- this includes learning new skills. So, your self-studying seems like a good option. Another good option would be to redirect your focus onto areas which are a better match for your interests and skills. Which brings us back to the bolded questions.