although those who did speak a kind of truth about the gray caps and ambergris ' past were condemned to the fringes , they did have their own organization : the ambergrisians for the original inhabitants society .