You need to have a think about like humans do you want them to be, socially and sexually?
Humans are very bad at getting pregnant. Studies done on American newly weds who were actively trying to have a child (lots of sex, no contraception), showed that it took on average 6 months for them to conceive. Now that's an average, so some of them would have got lucky on their wedding night. But others would have taken much longer than 6 months. So the probability of each mating resulting in a baby is very low if your males and females are physiologically and biochemically identical to humans.
However, there are plenty animals which are much better at conceiving. The females come into season, they mate a few times (perhaps over a day or two), and get pregnant.
In mammals, the ultimate success in getting pregnant is the mountain hare, Lepus timidus. The female can mate once and get pregnant twice! She stores some of the sperm from a spring mating in her reproductive tract, and can use it to conceive a late summer litter. There is also a species of lizard (Uta stansburiana) where a particular male fathered 3 clutches of eggs: once from mating, and then 2 more after he had died, because the female stored his sperm.
The ultimate version of this is of course ants, wasps and bees: the queen mates once, and stores the sperm to continually lay fertilised eggs for the rest of her life. These eggs are all female and are workers and new queens. She also lays unfertilised eggs which develop into drones (males). Look up Haplodiploidy for more information.
So if you make your humanoids very efficient at mating, and give the women the ability to store sperm, a one night stand with one male can produce an entire lifetime's worth of babies for your woman.
The quirk with the mammals and reptiles which do this, is that they may still need to mate for the 2nd and subsequent pregnancies to start. Mating causes the female to ovulate. This is called Reflex or Induced ovulation. The female hare will mate with any old male for her second litter, and will have both his sperm and the first male's sperm present in her reproductive tract. The first male's sperm has a 'head start' in getting to the eggs.
It's not the sperm, its the sex which kicks the process off. So if you want, your women could have female lovers to trigger the second pregnancy and the ones after that.
Alternatively, real world humans are spontaneous ovulators: we have a menstrual cycle, and release an egg once every 21 to 35 days. So your women could have sex once, store the sperm and use a bit of it every time they ovulate.
The genetic bottleneck may not be as bad as you think. There are plenty species where the males fight and/or display for the right to mate. For instance elephant seals, red deer, birds of paradise, peacocks. The winners are 5% of the male population and they father 95% of the offspring. 15% of the males father the other 5% of the offspring. 80% of the males never, ever mate.
Therefore you can have a small number of males keeping your population going. In the situation above, 80% of male genetic diversity is lost every generation, but there is no sign of deer and elephant seals suffering from inbreeding.