The clash erupted on Friday in Tanduo village near Lahad Datu, a town in Sabah state, when Malaysian security forces who had already locked down the area moved to tighten a cordon around the clan.
"The intruders fired at us, then we returned fire," AFP quoted Sabah police chief Hamza Taib as saying in reference to the Friday clashes.
"As a result of the fire, two of my men died, three were injured and... 12 intruders died," he added.
The Filipinos are followers of the sultan of Sulu, Jamalul Kiram III, and have been holed up a remote area on Borneo island since February 12, demanding recognition as the rightful owners of the land.
They have also demanded the terms of an old lease be renegotiated with Malaysia.
Sabah became part of Malaysian territory in 1963, and the country still pays a small amount of rent to the Sulu Sultanate each year.
In the wake of the violence, Malaysian Prime Minister Najib Razak said that he has authorized police and armed forces to take all necessary measures to end the embarrassing security breach.
Philippine President Benigno Aquino's spokesman also said the incident left a "small window of opportunity to arrive at a peaceful conclusion."
However, the Filipinos blamed the violence on the Malaysian side, saying the clash broke out after Malaysian snipers killed 10 of the sultan's men and wounded four others, warning there was more to come.
Philippine Foreign Department spokesman Raul Hernandez said Manila had formally demanded a full account of the incident.