Q:

24 VAC source unable to drive servo motor

Here's my power supply (schematic except without the transformer since my source is already at 24 V AC and using a 100 uF electrolytic as my smoothing capacitor):
24 VAC rectifier -> 100 uF smoothing cap -> LM317 linear regulator (typical application on page 10 of LM317, running at 3.3V)
And my servo motor is an MG90S. I am feeding it 5V from a boost converter using an AAT1217. 
With the servo motor disconnected, I see the servo motor source voltage at 5V and the signal line is giving the pulses required to drive the servo motor. With the servo motor connected, there is a periodic voltage drop to ~1V, causing my MCU to brownout. I hear the servo motor twitching, so it seems like the circuit is trying.
Other stuff I've tried:

Bypassing the 24 VAC and using a 3.3V DC power supply, the servo rotates fine, drawing about 200 mA from the power supply.
Using a 24 VDC source, the servo rotates fine
Using a larger smoothing capacitor (1mF) with 24 VAC, the servo still doesn't work. The voltage drop still occurs and the LM317 gets really hot.

I was thinking that the smoothing capacitance was not large enough so there wasn't enough current going into the servo but (3) disproves that. In addition, using smoothing capacitance calculations (C = (I * t) / dV), it seems like I should not need more than 100 uF. Any ideas what else could be wrong?
So, just to defend the LM317 choice a little,

The servo motor is on no more than 10s per day and not more than a few hundred ms each time.
With that, I was hoping that using a simple linear voltage regulator would be ok in terms of both cost and complexity.

A:

24V AC when rectified and smoothed produces a DC level of about 32V. You are using the LM317 to produce 3.3V and then it seems that you are stepping this up to produce 5V for the motor. The regulator you are using is capable of producing over 1A.
You say your stepper motor consumes 200mA and this will require the LM317 to supply about 300mA into the booster.
300mA thru the LM317 whilst dropping about 29V gives a heat power dissipation of about 8.6 watts and you'll need a heatsink or the LM317 will just shut-down.
With a 24V DC supply the problem will be less but still the power dissipated by the LM317 could be as high as 6 watts.

