Q:

In Gen. 4:7, why is the participle רֹבֵץ declined in the masculine gender?

In Gen. 4:7, it is written,

הֲלוֹא אִם תֵּיטִיב שְׂאֵת וְאִם לֹא תֵיטִיב לַפֶּתַח חַטָּאת רֹבֵץ וְאֵלֶיךָ תְּשׁוּקָתוֹ וְאַתָּה תִּמְשָׁל בּוֹ

My question concerns the word רֹבֵץ, a participle declined in the masculine gender. But, the word it modifies, or so it seems, חַטָּאת ("sin"), is a noun possessing a grammatically feminine gender. In Hebrew, like many other languages, the participle modifying a noun must agree in gender (among other things) with the noun. 
For example, in Basics of Biblical Hebrew Grammar, it is written,

Source: Pratico, Gary Davis; Van Pelt, Miles V. Basics of Biblical Hebrew Grammar. 2nd ed. Grand Rapids: Zondervan, 2001.
Why then is רֹבֵץ declined in the masculine gender?

A:

The most common explanation for this text and its perplexing syntax -- without recourse to emendation -- is that רֹבֵץ is said to be a "nominalized participle", and thus not subject to the gender agreement of a typical participle or adjective. So the solution is not to be found in some nuanced understanding of חַטָּאת, which is where I would first have thought to look. The idea would then be something like this:

At the door, sin is a croucher (msc), and its (msc, the "croucher") desire is for you...

(Similarly, G. Wenham, Genesis 1-15 (Word, 1986), p. 94.) I don't know how old this explanation is, but it goes back at least to George Spurrell, Notes on the Hebrew text of the book of Genesis (Oxford, 1887), p. 50, who in turn cross-references Gesenius, the equivalent of GKC §145u.

BDB make a special point of saying that these words ... are nothing special:

HALOT offers the following under רבץ:, which seems to assume the same explanation: 

—4. to lie, lurk: חַטָּאת רֹבֵץ Gn 47 literally sin is a lurker, meaning sin lurks, רֹבֵץ a nominalised participle, Gesenius-K. §145u; Gesenius-B.; Westermann BK 1/1:384, 385; F.W. Golka Fschr. Westermann 63...

A variation on this theme is to understand רֹבֵץ not as a participle, but as reflecting Akkadian rābiṣu (see Chicago Assyrian Dictionary, R, pp. 20ff.):

(an official representative of
  and  commissioned  by  a  higher  authority,
  attorney),   
(a  demon  and  protective genius);...

The possibilities this connection offers are explored by a number of commentators, and at greatest length by E.A. Speiser in Genesis commentary (pp. 32-33; in 1964, one of the earliest in the Anchor Bible series which morphed considerably as it progressed).
However, this explanation appears not to be known to Barry Bandstra, who offers quite a different discussion in his Genesis 1-11: A Handbook on the Hebrew Text (Baylor, 2008), p. 240:

Here the discussion is much more about whether חַטָּאת can somehow be regarded as masculine. But, if cogent, the older "nominalized participle" explanation seems much more simple. It might, perhaps, have an analogue in the ad sensum construction in Ecclesiastes 12:9 (see Waltke-O'Connor, Introduction to Biblical Hebrew Syntax (Eisenbrauns, 1999), p. 109 = §6.6b)

הָיָ֥ה קֹהֶ֖לֶת חָכָ֑ם
  The teacher (f) was wise (msc)

That is if קהלת is to be regarded as fem; some take it as a msc. So that may or may not be a true parallel. It is, however, noted by Hamilton as a possible basis/precendent on which to understand Gen 4:7's חטאת as masculine! (V.P. Hamilton, The Book of Genesis, Chapters 1 - 17 (NICOT; Eerdmans, 1995), p. 225.)

