We prove the transformation formula of Donaldson-Thomas (DT) invariants counting two dimensional torsion sheaves on Calabi-Yau 3-folds under flops. The error term is described by the Dedekind eta function and the Jacobi theta function, and our result gives evidence of a 3-fold version of Vafa-Witten's S-duality conjecture. As an application, we prove a blow-up formula of DT type invariants on the total spaces of canonical line bundles on smooth projective surfaces. It gives an analogue of the similar blow-up formula in the original S-duality conjecture by Yoshioka, Li-Qin and G "ottsche.
Motivated by S-duality modularity conjectures in string theory, we define new invariants counting a restricted class of two-dimensional torsion sheaves, enumerating pairs (Z H ) in a Calabi–Yau threefold (X ). Here (H ) is a member of a sufficiently positive linear system and (Z ) is a one-dimensional subscheme of it. The associated sheaf is the ideal sheaf of (Z H ), pushed forward to (X ) and considered as a certain Joyce–Song pair in the derived category of (X ). We express these invariants in terms of the MNOP invariants of (X ). The modified elliptic genus for an M5-brane wrapped on a four-cycle of a Calabi-Yau threefold encodes the degeneracies of an infinite set of BPS states in four dimensions. By holomorphy and modular invariance, it can be determined completely from the knowledge of a finite set of such BPS states. We show the feasibility of such a computation and determine the exact modified elliptic genus for an M5-brane wrapping a hyperplane section of the quintic threefold. We study the wall-crossing phenomena of D4-D2-D0 bound states with two units of D4-branechargeontheresolvedconifold. We identify the walls of marginal stability and evaluate the discrete changes of the BPS indices by using the Kontsevich-Soibelman wall-crossing formula. In particular, we find that the field theories on D4-branes in two large radius limits are properly connected by the wall-crossings involving the flop transition of the conifold. We also find that in one of the large radius limits there are stable bound states of two D4-D2-D0 fragments. Motivated by the S-duality conjecture, we study the Donaldson-Thomas invariants of the 2 dimensional Gieseker stable sheaves on a threefold. These sheaves are supported on the fibers of a nonsingular threefold X fibered over a nonsingular curve. In the case where X is a K3 fibration, we express these invariants in terms of the Euler characteristic of the Hilbert scheme of points on the K3 fiber and the Noether-Lefschetz numbers of the fibration. We prove that a certain generating function of these invariants is a vector modular form of weight -3 2 as predicted in S-duality.   We determine the modified elliptic genus of an M5-brane wrapped on various one modulus Calabi-Yau spaces, using modular invariance together with some known Gopakumar-Vafa invariants of small degrees. As a bonus, we find nontrivial relations among Gopakumar-Vafa invariants of different degrees and genera from modular invariance.    We define new invariants of 3d Calabi-Yau categories endowed with a stability structure. Intuitively, they count the number of semistable objects with fixed class in the K-theory of the category ("number of BPS states with given charge" in physics language). Formally, our motivic DT-invariants are elements of quantum tori over a version of the Grothendieck ring of varieties over the ground field. Via the quasi-classical limit "as the motive of affine line approaches to 1" we obtain numerical DT-invariants which are closely related to those introduced by Behrend. We study some properties of both motivic and numerical DT-invariants including the wall-crossing formulas and integrality. We discuss the relationship with the mathematical works (in the non-triangulated case) of Joyce, Bridgeland and Toledano-Laredo, as well as with works of physicists on Seiberg-Witten model (string junctions), classification of N=2 supersymmetric theories (Cecotti-Vafa) and structure of the moduli space of vector multiplets. Relating the theory of 3d Calabi-Yau categories with distinguished set of generators (called cluster collection) with the theory of quivers with potential we found the connection with cluster transformations and cluster varieties (both classical and quantum). Let X be the total space of the canonical bundle of P^2. We study the generalized Donaldson-Thomas invariants, defined in the work of Joyce-Song, of the moduli spaces of the 2-dimensional Gieseker semistable sheaves on X with first Chern class equal to k times the class of the zero section of X. When k=1, 2 or 3, and semistability implies stability, we express the invariants in terms of known modular forms. We prove a combinatorial formula for the invariants when k=2 in the presence of the strictly semistable sheaves, and verify the BPS integrality conjecture of Joyce-Song in some cases. We discuss the wall-crossing of the BPS bound states of a non-compact holomorphic D4-brane with D2 and D0-branes on the conifold. We use the Kontsevich-Soibelman wall-crossing formula and analyze the BPS degeneracy in various chambers. In particular we obtain a relation between BPS degeneracies in two limiting attractor chambers related by a flop transition. Our result is consistent with known results and predicts BPS degeneracies in all chambers. We prove a comparison formula for the Donaldson-Thomas curve-counting invariants of two smooth and projective Calabi-Yau threefolds related by a flop. By results of Bridgeland any two such varieties are derived equivalent. Furthermore there exist pairs of categories of perverse coherent sheaves on both sides which are swapped by this equivalence. Using the theory developed by Joyce we construct the motivic Hall algebras of these categories. These algebras provide a bridge relating the invariants on both sides of the flop.