1. A method for profiling network traffic of a network, comprising: obtaining a signature library comprising a plurality of first network data layer signatures each representing first data characteristics associated with a corresponding application executing in the network, wherein the plurality of first network data layer signatures are generated by analyzing a first network data layer using a payload based signature generation algorithm; generating, by a processor of a computer system using a statistical feature based signature generation algorithm, a second network data layer group behavioral model associated with the signature library, wherein the second network data layer group behavioral model represents a common behavior in a second network data layer of a plurality of historical flows identified from the network traffic, wherein the first network data layer and the second network data layer are identified by a hierarchical network data model of the network traffic, wherein each of the plurality of first network data layer signatures correlates to a subset of the plurality of historical flows; generating a target flow set based on the second network data layer group behavioral model applied to the second network data layer, comprising: selecting, by the processor, a flow in the network traffic for including in a target flow set, wherein the second network data layer of the flow matches the second network data layer group behavioral model, wherein the first network data layer of the flow is not correlated to any of the plurality of first network data layer signatures; and expanding, in response to generating the target flow set based on the second network data layer group behavioral model applied to the second network data layer, the signature library based on the payload based signature generation algorithm applied to the first network data layer, comprising: analyzing, by the processor using the payload based signature generation algorithm, the target flow set to generate a new first network data layer signature; and adding the new first network data layer signature to the signature library.