https://huggingface.co/tiiuae/falcon-40b/blob/561820f7eef0cc56a31ea38af15ca1acb07fab5d/modelling_RW.py#L199
This should be the hidden size divided by the query heads.
· Sign up or log in to comment
