Near Bani Walid, Libya (CNN) -- Frustration mounts in the scrubby desert outside Bani Walid, one of the last strongholds of Moammar Gadhafi's regime.

Anti-Gadhafi fighters who have been pushed back from an attempted assault on the town earlier Saturday kick their heels in the shade under a handful of trees by the road.

Pick-up trucks mounted with anti-aircraft guns, civilian cars and an ambulance are parked up alongside. The roar of unseen jets overhead and thud of rockets in the near distance break the silence.

The fighters had gone to the edge of Bani Walid, southwest of Tripoli and one of three major towns in Libya that remain loyal to Gadhafi, but were apparently driven back.

They had encountered resistance from loyalist fighters who were more numerous, better equipped and better trained than they had expected, a National Transitional Council (NTC) official, Abdallah Kenshil, said.

The fighters' frustration at not getting closer to the prize spilled out in gunfire, as they pulled back to a position about 10 kilometers (6 miles) outside Bani Walid. The men shot everything they had into the air -- from AK-47s to anti-aircraft guns -- in a hail of fire more normally let off in celebration but here signalling a darker mood.

After a while, some rounds, perhaps Grad rockets, came in from the air and bullets whizzed overhead -- and the fighters retreated two miles further up the road to regroup.

Saturday's setback reveals some of the weaknesses of the opposition, who lack a unified chain of command to pull together forces made up of groups of pick-up trucks, two or three from one town and two or three from another.

And an argument between a military commander and an official representing the interim government -- one wanting the fighters to go in, the other wanting them to stay back -- exposes a heightened tension in the leadership over how to proceed.

The big prizes, like Tripoli and Benghazi, have been taken and now victory over the final hold-outs remains tantalisingly out of reach. Fighters display more hesitation, perhaps unwilling to get themselves killed in the final stretch of a months-long battle.

Many of the men are from Bani Walid itself -- impatient to go in to the town and fight, they are held back by their commanders.

On this front at least, the men are equipped with AK-47s, anti-aircraft guns, machine guns and small rocket launchers, but they don't appear to have the kind of larger weaponry seen elsewhere in Libya, like tanks and armoured vehicles.

And at times, the journalists have outnumbered them, with as many as 150 -- probably the biggest media throng seen here -- gathered with the fighters outside Bani Walid earlier Saturday.

Sometimes the anti-Gadhafi fighters' frustration breaks out in heated exchanges with the journalists, as they try to tell them what they can and can't film, their relations with the media a barometer of how well the battle is going.

The planes heard overhead are most likely NATO jets, perhaps targeting loyalist sites inside Bani Walid.

Opposition forces don't know how many Gadhafi loyalists are in the town, though they believe they are heavily armed with machine guns and rockets, said Abdulrahman Busin, an NTC spokesman. Two of Gadhafi's sons -- Saif al-Islam and Mutassim -- have been previously reported to be there.

On the back foot for the moment, the ragtag anti-Gadhafi forces are reconsidering their options and perhaps hoping that NATO will somehow clear the way for them and bring this long conflict to an end.

As one of the cars drove away from Bani Walid, a fighter shouted out: "Where are you, NATO?"