electron structure calculations based on solving the kohn - sham density functional theory ( ksdft )  play an important role in the analysis of electronic , structural and optical properties of molecules , solids and other nano structures .the kohn - sham equations define a nonlinear eigenvalue problem \psi_{i } = \varepsilon_{i } \psi_{i},\\  & \rho({{\mathbf{r } } } ) = \sum_{i } f_{i } { \lvert\psi_{i}({{\mathbf{r}}})\rvert}^2 , \quad \int  \psi^{*}_{i}({{\mathbf{r } } } ) \psi_{j}({{\mathbf{r } } } ) { \,\mathrm{d}}{{\mathbf{r}}}= \delta_{ij } ,  \end{split }  \label{eqn : ks}\ ] ] where are the kohn - sham eigenvalues ( or quasi - particle energies ) and are called the kohn - sham wavefunctions or orbitals .these eigenfunctions define the electron density , which in turn defines the kohn - sham hamiltonian = -\frac12 \delta + { \mathcal{v}}[\rho ] + { v_{\mathrm{ion } } } , \label{eqn : ksham}\ ] ] where is the laplacian operator , is a nonlinear function of , and is a potential function that is independent of .the parameters that appear in the definition of , which are often referred to as the occupation number , are defined by where is proportional to the inverse of the temperature and is called the chemical potential chosen to ensure that s satisfy for a system that contains electrons .the right hand side of ( [ eqn : fermidirac ] ) is known as the fermi - dirac function evaluated at .when is sufficiently large , the fermi - dirac function behaves like a step function that drops from 1 to 0 at ( which lies between and ) .spin degeneracy is omitted here for simplicity .  in this paper, we assume the kohn - sham system ( [ eqn : ks ] ) is defined within the domain ^ 3 ] by and represent the operator by .it follows from the spectral theory that the charge density defined in ( [ eqn : ks ] ) can be written as -\mu i)}\right]^{-1}({{\mathbf{r}}},{{\mathbf{r } } } ) .\label{eqn : rhofix}\ ] ] here ({{\mathbf{r}}},{{\mathbf{r}}}) ] , where ({{\mathbf{r } } } ) = \int    \frac{\rho({{\mathbf{r}}}')}{|{{\mathbf{r}}}-{{\mathbf{r}}}'|}{\,\mathrm{d}}{{\mathbf{r } } } '  + v_{{\mathrm{xc}}}[\rho]({{\mathbf{r } } } ) ,  \label{eqn : vform}\ ] ] where the first term corresponds to the electron - electron repulsion , and  ] in  requires solving a poisson equation , but the computation of requires either diagonalizing the hamiltonian  ] directly  .therefore , computing is much more costly than computing  ] defined in ( [ eqn : vfix ] ) at .if we define , subtracting from both sides of ( [ eqn : fixiter ] ) and approximating  ] with respect to evaluated at .it follows from the chain rule that taking the functional derivative of  ] with respect to .we will denote this jacobian by .its value at will be denoted by . in the physics literature , this jacobian is often referred to as a _ dielectric _ operator  , and denoted by . furthermore ,when is positive definite ,   only has real eigenvalues because it can be symmetrized through a similarity transformation where the symmetrized dielectric operator is hermitian and has real eigenvalues .we remark that the assumption that is positive definite may not always hold , especially when the material contains low electron density regions in which the exchange - correlation kernel contains large negative entries .however , because in general the product of and is much smaller in magnitude than , it is reasonable to expect that the eigenvalues of are close to those of , which are real .this type of approximation is also used in section [ sec : elliptic ] where we discuss how to construct an effective preconditioner for accelerating the fixed point iteration .it follows from ( [ eqn : errorsimplemixing ] ) that simple mixing will lead to convergence if if is an eigenvalue of , then the condition given in  implies that consequently , must satisfy note that ( [ eqn : alfabond1 ] ) is only meaningful when holds .therefore , is often referred to as the _ stability condition _ of a material  . furthermore ,when is bounded , it is always possible to find a parameter to ensure the convergence of the modified fixed point iteration even though the convergence may be slow .we should comment that the stability condition holds in most cases because is typically much smaller in magnitude than .note that is positive definite , and is negative semidefinite .when the stability condition fails , phase transition may occur , such as the transition from uniform electron gas to wigner crystals in the presence of low electron density  .such a case is beyond the scope of the current study .nonetheless , can become very large in practice even when the stability condition holds , especially for metallic systems of large sizes , as we will show in section [ sec : convergence ] .a large requires to be set to a small value to ensure convergence . even though convergence can be achieved, it may be extremely slow .the simple mixing scheme selects as a scalar in ( [ eqn : simplemixingupdate ] ) .if we replace the scalar by the inverse of the jacobian matrix of the function  ] or its inverse can not be easily evaluated .however , we may apply a jacobian - free krylov newton technique  to obtain newton s update  ,\ ] ] by solving the linear system iteratively using , for example , the gmres algorithm  .the matrix vector multiplication of the form , which is required in each gmres iteration , can be approximated by finite difference -{\mathcal{v}}[f(v_k)]}{\epsilon},\ ] ] for an appropriately chosen scalar .the finite difference calculation requires one additional function evaluation of  ] within a subspace that contains previous approximations to . in pulay s original work  , the optimal approximation to from is expressed as , where ( ) are previous approximations to , and the coefficients chosen to satisfy the constraint .when s are all sufficiently close to the solution of ( [ eqn : vfix ] ) , \approx \alpha_j { \mathcal{v}}[f(v_j)] ] .note that ( [ eqn : pulaymin ] ) can be reformulated as an unconstrained minimization problem if is required to take the form , where can be any unconstrained real number .again , if we assume  ] with respect to yields , where is the same as that defined in ( [ syk ] ) .  in , pulay s method for updating is defined as ) , \label{eqn : pulay}\ ] ] where is an initial approximation to the inverse of the jacobian ( at the solution ) .substituting into ( [ eqn : pulay ] ) yields exactly anderson s updating formula .if is the solution to ( [ eqn : vfix ] ) , then subtracting it from both sides of the quasi - newton updating formula \right)\ ] ] yields where is the jacobian of the function  ] into two subdomains :  ] .the value is set to in the first subdomain and in the second subdomain ..test cases and scf parameters used for the 1d model .[ cols="^,^,^,^,^",options="header " , ]   for all three cases , we apply anderson s method , anderson s method combined with the kerker preconditioner , and anderson s method combined with the elliptic preconditioner to the scf iteration .the parameter used in the anderson scheme is set to in all tests .the parameter is set to for the insulating and metallic cases , and for the hybrid case .the converged electron density associated with the three 1d test cases as well as the 74 smallest eigenvalues associated with the hamiltonian defined by the converged are shown in figure [ fig : rhoandeig1d ] .the first eigenvalues correspond to occupied states , and the rest correspond to the first unoccupied states . for the insulator case, the electron density fluctuates between and .there is a finite gap between the highest occupied eigenvalue ( ) and the lowest unoccupied eigenvalue ( ) .the band gap is a.u .. the electron density associated with the metallic case is relatively uniform in the entire domain .the corresponding eigenvalues lie on a parabola ( which is the correct distribution for uniform electron gas . ) in this case , there is no gap between the occupied eigenvalues and the unoccupied eigenvalues . for the hybrid case ,the electron density is uniformly close to a constant in the metallic region ( except at the boundary ) , and fluctuates in the insulating region .there is no gap between the occupied and unoccupied states .+  +  in figure [ fig : converge1d ] , we show the convergence behavior of all three acceleration schemes for three test cases by plotting the relative self - consistency error in potential against the iteration number . in each one of the subfigures , the blue line with circles , the red line with triangles and the black line with triangles correspond to tests performed on a 32-atom , 64-atom and 128-atom system respectively .we observe that the combination of anderson s method and the elliptic preconditioner gives the best performance in all test cases . in particular , the number of scf iterations required to reach convergence is more or less independent from the type of system and system size .we can clearly see that the use of the kerker preconditioner leads to deterioration in convergence speed when the system size increases for insulating and hybrid systems .on the other hand , anderson s method alone is not sufficient to guarantee the convergence of scf iteration for metallic and hybrid systems .all these observed behaviors are consistent with the analysis we presented in the previous section .+  +   in this subsection , we compare the performance of different preconditioning techniques discussed in section [ sec : advance ] when they are applied to a 3d problem constructed in kssolv  , a matlab toolbox for solving kohn - sham problems for molecules and solids .we have chosen to use the kssolv toolbox because of its ease of use , especially for prototyping new algorithms .the results presented here can be reproduced by other more advanced dft software packages such as quantum espresso  , with some additional programming effort .the model problem we construct consists of a chain of sodium atoms placed in a vacuum region that extends on both ends of the chain .the sodium chain contains a number of body - centered cubic ( bcc ) unit cells .the dimension of the unit cell along each direction is a.u .. each unit cell contains two sodium atoms . to examine the size dependency of the preconditioning techniques, we tested both a 16-unit cell ( 32-atoms ) model and a larger 32-unit cell ( 64 atoms ) model .the converged electron density on the plane ( or the [ 100 ] plane in crystallography terminology ) associated with the 32-atom model is shown in figure [ fig : naconfig ] .slice of the electron density of the 32-atom sodium system with large vacuum regions at both ends . , title="fig:",scaledwidth=80.0% ] +  figure [ fig : convna ] shows how anderson s method , the combination of anderson s method and the kerker preconditioner and the combination of anderson s method and the elliptic preconditioner behave for both the 32-atom and the 64-atom sodium systems .for the 32-atom problem , the parameter for the anderson s method is set to .the parameter required in both the kerker preconditioner and the elliptic preconditioner is set to 0.05 . for the 64-atom problem ,the parameter is set to . for simplicity the function required in the elliptic preconditioneris set to a constant function .the function ( shown in figure [ fig : naellipticb ] for the 32-atom problem ) is constructed by convolving a square wave function with a value of in the sodium region and in the vacuum region with a gaussian kernel .the scf iteration is declared to be converged when the relative self - consistency error in the potential is less than .slices of the function used in the elliptic preconditioner for a 32-atom sodium system with large vacuum regions at both ends.,title="fig:",scaledwidth=80.0% ]  as we can clearly see from figure [ fig : convna ] , the use of the anderson s method with the elliptic preconditioner leads to rapid convergence . furthermore , the number of iterations ( around 30 ) required to reach convergence does not change significantly as we move from the 32-atom problem to the 64-atom problem .using anderson s method alone enables us to reach convergence in 60 iterations for the 32-atom problem .however , it fails to reach convergence within 100 iterations for the 64-atom case .when anderson s method is combined with the kerker preconditioner , the scf iteration converges very slowly for both the 32-atom and the 64-atom problems .na atoms ( a ) and na atoms ( b).,title="fig:",scaledwidth=35.0% ]  na atoms ( a ) and na atoms ( b).,title="fig:",scaledwidth=35.0% ]we discussed techniques for accelerating the convergence of the self - consistent iteration for solving the kohn - sham problem .these techniques make use of the spectral properties of the jacobian operator associated with the kohn - sham fixed point map . they can also be viewed as preconditioners for a fixed point iteration .we pointed out the crucial difference between insulating and metallic systems and different strategies for constructing preconditioners for these two types of systems .a desirable property of the preconditioner is that the number of fixed point iterations is independent of the size of the system .we showed how this property can be maintained for both insulators and metals .furthermore , we proposed a new preconditioner that treats insulating and metallic systems in a unified way .this preconditioner , which we refer to as an elliptic preconditioner , is constructed by solving an elliptic pde with spatially dependent variable coefficients .constructing preconditioners for insulating and metallic systems simply amounts to setting these coefficients to appropriate functions .the real advantage of this type of preconditioner is that it allows us to tackle more difficult problems that contain both insulating and metallic components at low temperature .we showed by simple numerical examples that this is indeed the case .although the size of the systems used in our examples are relatively small because we are limited by the use of matlab , we can already see the benefit of an elliptic preconditioner in terms of keeping the number of scf iterations relatively constant even as the system size gets larger . to fully test whether the preconditioner can achieve the goal of keeping the scf iterations system size independent , we should implement the elliptic preconditioner in a standard electronic structure calculation software packages such as quantum espresso  , abinit  and siesta  etc .that are properly parallelized , which we plan to do in the near future .this work was supported by the laboratory directed research and development program of lawrence berkeley national laboratory under the u.s .department of energy contract number de - ac02 - 05ch11231 ( l. l. and c. y. ) .we would like to thank eric cancs , roberto car , weinan e , weiguo gao , jianfeng lu , lin - wang wang and lexing ying for helpful discussion .

Abstract: we discuss techniques for accelerating the self consistent field ( scf ) iteration for solving the kohn - sham equations . these techniques are all based on constructing approximations to the inverse of the jacobian associated with a fixed point map satisfied by the total potential . they can be viewed as preconditioners for a fixed point iteration . we point out different requirements for constructing preconditioners for insulating and metallic systems respectively , and discuss how to construct preconditioners to keep the convergence rate of the fixed point iteration independent of the size of the atomistic system . we propose a new preconditioner that can treat insulating and metallic system in a unified way . the new preconditioner , which we call an elliptic preconditioner , is constructed by solving an elliptic partial differential equation . the elliptic preconditioner is shown to be more effective in accelerating the convergence of a fixed point iteration than the existing approaches for large inhomogeneous systems at low temperature .   kohn - sham density functional theory , self consistent field iteration , fixed point iteration , elliptic preconditioner  65f08 , 65j15 , 65z05


synthetic adhesives are usually soft , essentially incompressible materials that do not flow .they are used in the form of thin films of typically 100 m m in thickness .their adhesive properties are often tested in a controlled test that mimics the typical situation of two solid bodies linked by an adhesive . in the so - called probe - tack geometry , introduced by zosel ( 1985 ) , a flat , solid punch , called the probe ,is brought into contact with an adhesive film deposited on a rigid substratum . after a certain contact time , the force is recorded while the probe is being pulled away ( figure 1a ) . because the adhesive material is almost incompressible , pulling the solid bodies apart causes a strong convergent deformation and thus a strong resistance to separation , i.e. to good adhesion .furthermore , because the resistance of the adhesive is so important , the deformability of the entire system , and in particular the compliance of the machine itself may play a role .elastic energy is stored for some time , until the adhesive gives way and causes a sudden separation , as discussed in a specific geometry by francis et al .sudden separation is possible when instabilities develop in the bulk of the adhesive material so as to relieve much of the stress .such instabilities have been observed directly during separation in a modified version of the probe - tack test ( lakrout et al . , 1999 ) and shown to fall into two main categories , viz .fingering instabilities and cavitation .both types of instabilities are driven by the need for relieving the stress .when air fingers protrude from the edge of the sample towards the center , they bring atmospheric pressure well into the sample and thus relieve the negative pressure that has developed near the center due to the applied traction . in the case of purely viscous liquids , this is the well - known saffman- taylor instability ( saffman et al . , 1958 ) .cavitation also relieves stress since the growing bubbles provide some of the extra volume required by the plate separation . sinceadhesive materials are usually more elastic than viscous , they are most often subject to cavitation , which involves strong deformations on the scale of the sample thickness .they almost never undergo fingering , which would involve deformations on the scale of the entire sample . when a very viscous liquid is used in this geometry instead of proper adhesive material , fingering is observed at low separation rates while cavitation occurs at higher rates ( poivet et al . , unpublished observations ) , which further illustrates the fact that both mechanisms compete in relieving the applied tensile stress .both fingering and cavitation involve strong deformations in the sample during separation and thus cause a large energy dissipation , ranging typically from to j / m for good adhesives , which is much higher than typical surface energies in the j / m range .a number of properties are associated with an adhesive .it is usually a soft material that does not flow and made of polymers whose molecular architecture may vary ( cross - linked polymers , block - copolymers ). it can accommodate large defor- mations in order to dissipate a large amount of energy , yet it is essentially incompressible .the need for relieving the applied stress is so strong in the thin film geometry that if the adhesive material is too resistant , e.g. if it is very elastic , fingering and cavitation from the edge of the sample may occur at the interface , as shown by ghatak ( 2000 ) and shull ( 2000 ) .this highlights the fact that not only the bulk of the adhesive film , but also its interfaces with the solid bodies set up during the bonding process , must resist separation efficiently .the quality of the contact between the adhesive and the substrata results from the interplay of inter- actions at different length scales . on the molecular scale, it results generically from van der waals interactions ( israelachvili , 1992 ) .surface chemical bonds ( gent et al . , 1972 ) , macromolecular interdigita- tion ( raphal et al . , 1992 ) or macromolecule elongation ( lake et al . , 1967) , however , may enhance significantly the strength of the interface for specific substratum - adhesive pairs .surface treatments and cleaning are also essential . on the micrometre scale , solid substrates usually display some degree of surface roughness ( greenwood et al . , 1966 ) that may reduce the degree of intimacy of the contact with the adhesive if the adhesive material is not very soft ( dahlquist , 1969 ; fuller et al . , 1975 ; creton et al . , 1996 ; crevoisier et al . , 1999 ) . for very soft adhesives , however , the surface roughness of the substratum may paradoxically enhance the strength of the interface , as small air bubbles trapped at the interface may generate suction effects upon traction ( gay & leibler , 1999b ) .  when seeking a bad adhesion even with good adhesives , the main strategy is to weaken the contact at the body surface , and in particular to use non - wetting surfaces , i.e. surfaces such that the gain in surface energy upon making contact with the adhesive is small .when spread on such surfaces , most liquids do not spread spontaneously but gather into droplets . on very weakly wetting surfaces , in particular on lotus leaves and on other plants and on some insect wings ( wagner et al ., 1996 ; barthlott et al . , 1997 ; neinhuis et al . ,1997 ) , millimetric drops are almost complete spheres and can easily roll .solid surfaces were successfully designed to mimic this `` lotus effect '' and the surface design , an array of micrometer - sized poles , enhances the weak wettability achieved through chemical treatment ( bico et al . , 1999 ) .the mechanisms described above , by which the adhesive material undergoes heterogeneous defor- mations upon traction , involve the bulk of the adhesive film .however , as seen from a distance , the adhesive joint appears to be the interface between both solid bodies , and its deformations due to traction can be seen as a fracture propagating at this interface . in this perspective, adhesion phenomena are connected to the field of fracture mechanics .a small notch cut on one side of an elastic band ( figure 2a ) illustrates a number of points .if the band is pulled gently , it stretches homogeneously except in the vicinity of the notch where it is less stretched ( figure 2b ) .in other words , elastic energy is stored in the material but the notch relieves some of this energy . if the band was cut further , the notch would be longer and relieve more stress .this will not happen spontaneously unless it is pulled more strongly .it will generally happen when the elastic energy released by the notch propagation is sufficient to break further bonds at the interface and propagate the fracture .this is a central concept in fracture mechanics , called griffith s criterion ( griffith , 1920 ) .since the energy stored depends on the elastic properties of the entire system , not only does the elasticity of the testing apparatus play a role if it is soft enough , but very non - intuitive behaviors may arise .the fracture may propagate within the adhesive film i.e. its cohesion affected , or it may take place at the interface with one of the substrates . more generally , when considering a stack of several layers from different materials ( figure 2c ) , the system may break upon traction or peeling .the system may choose between different fracture mechanisms ( interfacial or cohesive , with further choice in a layered system ) .at first , it usually chooses the weakest mechanism in terms of force since it triggers separation first and thereby relieves the stress on the other possible mechan- isms . in the long run , however , fracture propagation is driven by the elastic energy stored in the system under tension which must exceed the energy required by the fracture .the system thus tends to choose the separation mechanism that dissipates the smallest amount of energy per unit surface area since the fracture will then be able to propagate faster .but , when a sheet of paper is torn out of note - pad , depending on how exactly the paper is pulled and how the note - pad is held , the sheet may detach neatly or be torn . due to the mismatch in the elastic properties of the note - pad and the paper sheet ,the fracture may propagate along a path that does not minimize the energy dissipation .more generally , in a layered system , the elastic properties of the various layers may prevent the fracture from migrating towards its energetically optimal location .as discussed above , separating both substrata in the probe - tack geometry is difficult because the lateral extension of the adhesive joint is much greater than its thickness ( thin geometry ) .it is also essential that both substrata are rigid ( figure 1a ) . indeed , when an adhesive tape is considered , the sticky side is the adhesive film , while the non - sticky side is the ( very flexible ) backing .thus , the backing plays the role of one of the substrata . butpeeling a tape off a table is quite easy because the backing is flexible and only a restricted region of the adhesive is under tension at any given time during peeling ( figure 1c ) .peeling is , in general , so much easier that it is avoided as much as possible in practical situations .for instance , adhesive hooks for bathroom or kitchen utensiles are designed in such a way that the hook deformations induced by a hanging weight cause the adhesive layer to be locally compressed rather than peeled , otherwise the adhesive joint would soon be ruined ( figure 1e , f ) .although similar to peeling at first sight , detach- ment by tilting is difficult if both the substrata are rigid . indeed , tilting one object ( for instance , the probe in the probe - tack geometry ) involves com- pression of the adhesive film along the vicinity of one edge , and traction on the other edge and on most of the region of contact with the object ( figure 1b ) .hence , tilting the object is very similar to pulling it .tilting becomes similar to peeling only at the point where the adhesive joint is not longer thin , i.e. if its lateral dimensions are comparable to its thickness , it is then easy .this feature is used by tropical lizards ( geckoes ) , which can climb walls at considerable velocities .their adhesive feet consist of a hierarchy of structures .the smallest structure is the spatula ( 0.2 m in width ) .adherence to the wall is important , even if it is rough , because spatulae can achieve contact with it somewhat independently of one another .however , foot removal is obviously achievable without too much effort . because spatulae are independent, they can be tilted simultaneously and peeled away from the wall . due to their small lateral dimension ,this can be done with little effort ( autumn et al . , 2000 ) . in short , geckos use peeling on the lowest scale to lift their feet .a round object can roll on a sticky surface , whereby a new contact with the adhesive is made at the front , while separation , similar to peeling , is achieved at the rear ( figure 1d ) .because the separation dissipates more energy than is provided by the contact , a sufficient sideward force must be applied for the object to move .experiments conducted on glass cylinders in contact with rubber ( charmet et al . , 1995 and references therein ) show that such rolling can occur also if the applied force has a normal component .for example , cylinders can roll both on and under an inclined rubber sheet .similarly , marine organisms attached to a boat s hull undergo a sideways force as well as a lift force ( because the flow is usually in the inertial rather than viscous regime ). they may thus roll if the water velocity is moderate , and they may even detach if the lift component of the force is high .if the organisms are rigid , gregarious settlement behavior hinders flow - driven detachment , since when attached together , they constitute a solid with a large lateral extension , which turns easy rolling into more demanding tilting ( figure 3 ) .testing fouling release properties of coatings some- times involves real - size experiments on ships , but more usually laboratory scale characterization .hydrodynamic effects scale with dimension . in particular ,boundary layers are much smaller in laboratory tests than under ship - operating con- ditions ( schultz et al . , 2003 ) ,thus attached bodies should be scaled down accordingly ( figure 4a , b ) .the interpretation of laboratory tests therefore needs to combine information from different types of experiments .for example , small ` toy ' particles on small ships can be used in laboratory tests to mimic organisms to measure what lateral and normal drag forces and torques they undergo at various locations on the ship and at various equivalent ship speeds ( figure 4b ) .after proper upscaling of these forces and torques , the laboratory or small boat tests with live organisms should be performed under con- ditions where forces and torques can be reproduced ( figure 4d ) .these parameters can be measured if the organisms are located on small , instrumented patches inside larger panels ( figure 4e ) .alterna- tively , if the organism under study is rigid , the forces and torques can be applied to it mechanically , in the absence of any flow ( figure 4c ) .if the organism is much more rigid than the coating , or , more generally , if its mechanical properties are known , its detach- ment can be mimicked with more convenient real- size ` toy ' objects with an experimental set - up that controls the forces and torques .the mechanics of adhesion briefly described in the previous sections suggest the following direction for fouling release coatings . if an aggregate of rigid organisms on the surface is considered on a classical thin coating , the only detachment mode is tilting ( figure 3 ) .but , if the coating is flexible in some sense , then an easier detachment mode , similar to peeling , may appear .it is suggested that a coating consisting in an inextensible , flexible , very thin upper layer and a much softer lower layer , may provide enough local compliance to accompany a tilting object and peeling away from it , thus easing its detachment ( figure 5 ) .such a coating may be capable of easily releasing rigid organisms .synthetic adhesives are efficient because they involve several mechanisms for dissipating a large amount of energy upon detachment ( see reviews by kinloch , 1996 , gay & leibler , 1999a , and creton & fabre , 2002 ) .some of these mechanisms , and the associated mechanical aspects , have been reviewed and may enhance understanding of the settlement and release of fouling organisms and thereby lead to improvements in the design of fouling release and non - adhesive coatings .i acknowledge fruitful discussions and collabo- rations on adhesion phenomena over the past few years , with arnaud chiche , ioulia chikina , costantino creton , guillaume de crevoisier , pascale fabre , gwendal josse , ludwik leibler , frdric nallet , sylwia poivet , elie raphal , didier roux .i gratefully thank the organizers of the conference for the invitation to give a presentation .autumn k , chang w - p , fearing r , hsieh t , kenny t , liang l , zesch w , full r j ( 2000 ) adhesive force of a single gecko foot- hair ._ nature ( lond ) _ * 405 * : 681685    charmet j - c , verjus c , barquins m ( 1995 ) sur la dimension du contact et la cintique de roulement dun cylindre long et rigide sous la surface plane et lisse dun massif de caoutchouc souple ._ c r acad sci srie ii _ * 321 * : 443450

Abstract: various adhesion mechanisms that have been understood in the field of synthetic adhesives are described and these are linked with situations relevant to fouling issues . the review mainly deals with mechanical aspects of adhesion phenomena , with an emphasis on the role of the elasticity of the bodies , called substrata , attached by adhesive . the consequences of thin film geometry of the adhesive material are described , such as various heterogeneous deformations upon traction . the importance of the bonding process is discussed , as well as some examples of non - wetting surfaces . some basic ideas of fracture mechanics are provided and in particular , the behavior of layered systems is discussed . rolling sticky objects and peeled ( flexible ) adhesive tapes display similar mechan- isms and it is shown how they differ from the normal separation of rigid bodies . some issues directly related to fouling issues are also discussed , such as forces and torques acting on shells , the advantages of gregarious settlement behavior and concepts for fouling release and antifouling .   * keywords : * adhesion mechanisms ; synthetic adhesives ; the bonding process ; non - wetting surfaces ; fracture mechanics ; fouling issues .


with the proliferation of text data available online , text classification has attracted a lot of interest .traditionally , -grams are considered as document features and are subsequently fed to a classifier such as support vector machines ( svms ) .one - hot - encoding representations , although prominent in the literature , have two significant drawbacks : ( i ) they result in a very high dimensional and sparse feature space , and ( ii ) they do not encode similarity between words . lately , a lot of research has been devoted to the direction of distributed representations . distributed representations of words , are continuous , low dimensional , dense vectors that characterize the meaning and the semantic content of words .each dimension of the embedding represents a latent feature of the word , hopefully capturing useful syntactic and semantic properties . as a result ,semantically similar words , such as `` strong '' and `` powerful '' , will be close in the output vectorial space .  in this work , we present a focused contribution as part of an interesting classification application .we investigate the performance of word embeddings learned using the skip - gram model in the context of large scale , multi - label document classification .we report results on real - world classification problems with up to 10,000 classes and we demonstrate a straightforward way to combine document - level embeddings with one - hot - encoding representations .the contributions of our work are twofold : we , first , propose an efficient way to achieve satisfactory classification performance using only embedding representations and , we further improve it by applying a naturally parallelizable fusion mechanism between the distributed representations and one - hot - encoding representations .several methods have been proposed for obtaining distributed word - level representations .we cite for instance : .extensions of those methods have been proposed to cope with larger portions of text such as sentences or paragraphs : . however , generalizing from words to larger text spans can be inefficient , since for every unseen span new passes over a neural network are required . hence , we focus on methods that given a dictionary of word representations apply composition functions to produce document representations .  although distributed embeddings have been applied in a plethora of tasks from analogies evaluation to extractive summarization , their application on large scale text classification has not been investigated . what is more , most of the work in this direction deals with short text spans , such as sentences or tweets , and the size of the investigated problems with regard to the number of classes is small .recently , for instance , investigated the problem of short text similarity with applications to classification with a limited number of classes , like binary sentiment analysis and ternary sentiment analysis .more frequently , word embeddings for text classification are used for initializing architectures such as convolutional and recurrent networks .the works of and for instance , are in this line but , again , they limit their study at sentence - length spans .the latter is , also , due to hardware limitations of the gpus used : their limited memory capacity in conjunction with the vocabulary size of large scale classification problems require many data transfer operations which results in significant overhead . in this work we place ourselves at the large scalesetting ( number of classes in the order of ) where compositional methods based on neural networks are difficult to be applied .the remainder of this paper is organised as follows : in section 2 we propose and evaluate composition functions for obtaining document from word representations , and in section 3 we discuss their integration with one - hot - encoding schemes .finally , section 4 concludes with remarks to our future work .lccc2cmc1.8 cm & instances & vocabulary & avg .labels per instance & avg . doc .size ( words ) + train data & 12.5 m & 868,219 & - & 215 + pubmed & 225,000&289,789&1.42&239 + pubmed & 225,000&302,802&3.23&233 + pubmed & 225,000&303,637&6.64&230 +  we explore three functions to compose document representations : _ min _ , _ average _ and _ max _ which have been used as simple yet effective methods for compositionality learning in vector - based semantics , to obtain a document s representation .we use the output of each composition function as document features and we evaluate both their classification performance as well as the performance of their concatenation : \ ] ] where is the representation of a document and is the result of applying the composition element - wise , in the distributed vectors of the words of the documents . for instance , assuming we want the representation of `` harsh winter '' , the composition function requires the vector representations of the words `` harsh '' and `` winter '' . then , applying the _ max _ function will produce a vector of the same dimensionality with the original representations where each element will have the maximum value of the corresponding elements of the original word representations . in this process , we assume access to a vocabulary of distributed representations , where each word is associated with a -dimensional vector .this vocabulary , which we assume readily available , may have been generated beforehand .table [ table : data ] shows the data we use throughout the paper .they are abstracts of biomedical texts from pubmed released by the bioasq challenge organisers as well as texts from wikipedia . to obtain the dictionary of word embeddings we used the skipgram model of ` word2vec ` tool .we have kept the tool s default parameters for skip - gram apart from the number of iterations that we have set to 15 . for training the representations , we used 10 m pubmed abstracts and we added 2.5 m wikipedia documents for better generalization ( `` train data '' of table [ table : data ] ) . in the preprocessing steps we applied lower - casing , we space - padded the punctuation symbols and we filtered the words with less than 5 occurrences . for classification purposes we used the remaining pubmed abstracts from the released data .we created three classification datasets : `` pubmed '' with being the most common classes in the data .we performed a stratified split in train - test ( 200k-25k instances ) parts . in the experiments hereafter , we use svms with linear kernel which have been widely used for text classification . the value that controls the importance of the regularization term in the optimization problemwas selected using 5-fold cross validation on the training data of each experiment and for each representation .the classification problem is multi - label : each instance is associated with several classes as shown in table [ table : data ] .we cope with the multi - label problem using a binary relevance approach .for svms , tackling the multi - label problem with binary relevance results in predicting for each instance every label with positive distance from the separating hyperplanes of the one - vs - rest binary problems .if the classifier does not return any label for an instance , the most common label of the training data is assigned . for our implementations , we have used python s scikit - learn .we now present results for classification experiments on the pubmed and pubmed datasets when the described composition functions are used .figure [ fig : class1 ] shows the scores for the measure obtained on the test data , when varying the size of the training data . from the figure , first note that the _ avg _ function performs better compared to both _ min _ and_ max_. the latter two , perform equally but they are not competitive .however , the best results are consistently obtained with the concatenation ( _ conc _ ) of the outputs of the three composition functions .interestingly , adding the _ min _ and _ max _ representations creates a richer representation that benefits the performance . to this direction , note the steep increase in the performance of _ conc _ representations with the availability of training data : being richer , those representations have bigger discriminative power . depending on the dataset and the dimension of the representations , the achieved improvements using _ conc _ vary from - 5 points which is important for such problems .we believe that the _ avg _ function does not retain enough information for large documents , given that they consist on average of more than 200 words ( table [ table : data ] ) . to this end ,table [ table : docdimensionimpact ] reports the micro measure for the pubmed dataset , with respect to the document length in words .note that the best performance is achieved for smaller documents independently of the embedding dimension ..the impact of the document length on the classification performance ( micro measure ) when using _ avg _ representations and the embeddings dimension is .the representations perform better on smaller documents , which is in line with the outcome that the _ avg _ representations do not capture enough information for large documents .[ cols="<,^,^,^,^,^ " , ]   another observation from figure [ fig : class1 ] and table [ table : docdimensionimpact ] concerns the effect of the dimension of word representations in the classification performance .representations of bigger dimensions benefit performance .in fact , increasing the dimension from 50 to 400 , improves the measure for _ conc _ for pubmed ( resp .pubmed ) by ( resp . ) points .summarizing , we highlight here two of the advantages of the proposed approach : ( i ) although simple , our composition functions have yielded significant performance improvements and , ( ii ) their application on large datasets is naturally parallelizable , hence , they can be easily applied on real , large scale problems .lccccccccc & & & + ( lr)2 - 10 _ hash _ & & & + _ tf - idf _ & & & + ( lr)1 - 10  & d=100&d=200&d=400&d=100&d=200&d=400&d=100&d=200&d=400 + ( lr)2 - 4 ( lr)5 - 7(lr)8 - 10  _ x_conc_&0.592 & 0.614&0.626&0.362&0.41&0.436&0.386&0.434&0.454 + _ hash+x_conc_&0.651 & 0.654&0.646&0.464&0.476&0.473&0.488&0.495&0.491 + _ tfidf+x_conc_&**0.66** & * * 0.66** & 0.656&0.484 & 0.486 & * * 0.487** & 0.507 & 0.507 & * * 0.512** +in the previous section we have shown that the concatenated representations obtained by the output of the composition functions achieved the best classification performance .we , now , compare them with the traditional one - hot - encoding representations .we focus on two ways of representing text : ( i ) by using _ tf - idf _ representations of unigrams , and ( ii ) by employing a _ hash _function . forthe former , to generate the tf - idf representations we used sub - linear term frequency counts multiplied by their respective smoothed inverse document frequency , i.e. , for a term we have .for the latter , the hash function , given a text string , transforms it on a numerical value in a pre - specified space , that is used as the index to generate a vector representation .increasing the output dimension of the hash functions reduces the probability of collisions , i.e. different words mapped on the same vector indices , but also increases the output vector size .we investigate this trade - off in figure [ fig : hashdim ] : we present the effect of the size of the hash representations with respect to classification performance for pubmed and pubmed which have the biggest vocabulary size .we also report exact timings of each scenario , executed on 10 cores of an intel(r ) xeon(r ) cpu e5 - 2640 v3 @ 2.60ghz . from the figure , note that after 70k features the classification performance does not improve when increasing the size of the feature space .on the other hand , increasing the representation s dimension , the training time also increases . for reference , _ tf - idf _ representations in the same computational setting need 602 sec . and 1203 sec .for pubmed and pubmed respectively . as a result , we set the hash dimension to 70,000 features .note the significant dimensionality reduction achieved , given the vocabulary sizes of our problems reported in table [ table : data ] .table [ table : classification ] details the scores achieved with regard to the representations used .we first discuss the performance when single representations are used : _ x_conc _, _ tf - idf _ and _ hash_. notice that _ tf - idf _ performs better than both _x_conc _ and _ hash _ , with the latter achieving the lowest performance .the performance of the three representations on pubmed is comparable , but in the bigger classification problems _ tf - idf _ and performs considerably better .we , thus , consider the _ tf - idf _ representations as our baseline model , and we examine how the models with concatenated representations behave compared to it .we investigate now whether the fusion of distributed document representations with the one - hot - encoding representations benefits the classification performance .the last two lines of table [ table : classification ] present the performance of the fusion , by concatenation , of the embedding representations with with _ hash _ and _ tf - idf_. in the experiments , both _ hash _ and_ tf - idf _ consistently achieve better performance when combined with the distributed representations .for instance , for pubmed and the _ tf - idf __ hash _ ) representations improve in absolute numbers by 2 ( resp .3.5 ) points .we have performed two - sided student s t - tests ( ) to compare whether the improvements obtained for each classification problem are statistically significant compared to using _ tf - idf _ representations .those results , indicated by in table [ table : classification ] , reveal that the concatenation of _ tf - idf _ with distributed representations improves the classification performance in a statistically significant way .in addition to the important improvements , note than in the fused representations , the effect of diminishes .for instance in pubmed , one can obtain the optimal performance using embedding dimensions with and similar observations can be made in the rest of the datasets .   andin this work we have restricted ourselves in representations learned in the word level .this is advantageous in terms of speed since the dictionary of representations can be generated offline .then , applying composition functions is naturally parallelizable and fast for prediction. however , this poses the challenge of having robust composition functions , which if carefully selected , can result in performance gains such as those reported above .also , similarly to the bag - of - words paradigm , it does not take into account the word order and the words grouping in coherent segments like sentences or phrases .  in this line, it would be interesting to further investigate how more complex embeddings such as paragraph vectors perform .even if such approaches are computationally expensive in the document level , previous research has shown their effectiveness on the sentence level .hence , a direct extension of this work is to test the investigated composition functions with sentence level representations . in terms of applications ,those sentence representations can be directly used to evaluate the effectiveness of the embeddings and the composition functions .importantly , their can be evaluated simultaneously in different levels of text granularity from sentences to documents in the framework of passage retrieval and document classification from instance .another interesting line of research concerns the memory efficiency of such dense representations .recent research efforts have investigated ways of compressing the learned ( or composed ) representations using either linear ( e.g. pca ) or non - linear ( e.g. auto - encoders ) approaches to decrease the memory requirements or the dimension of the representations .such approaches , apart from having a positive effect on the memory footprint , also have a positive effect on the required computational requirements for training .in this work we have evaluated different composition functions for obtaining document - level representations using distributed embeddings of words . summarizing our findings , adding the concatenated vector of word - level skip - gram derived features to tf - idf unigrams performs better than tf - idf features alone .also , the result seems to be more pronounced as the representations cardinality increases and as the output label space increases . given the obtained improvements ,we have also outlined promising future research directions .w. blacoe and m. lapata . a comparison of vector - based representations for semantic composition . in _ proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning_ , pages 546556 .association for computational linguistics , 2012 .g. forman and e. kirshenbaum .extremely fast text feature extraction for classification and indexing . in _ proceedings of the 17th acm conference on information and knowledge management _ , pages 12211230 .acm , 2008 .m. kgebck , o. mogren , n. tahmasebi , and d. dubhashi .extractive summarization using continuous vector space models . in _ proceedings of the 2nd workshop on continuous vector space models and their compositionality ( cvsc)@ eacl _ ,pages 3139 , 2014 .j. kim , f. rousseau , and m. vazirgiannis .convolutional sentence kernel from word embeddings for short text categorization . in _ proceedings of the 2015 conference on empirical methods in natural language processing . in emnlp _ , volume 15 , 2015 .t. mikolov , i. sutskever , k. chen , g. s. corrado , and j. dean . distributed representations of words and phrases and their compositionality . in _ advances in neural information processing systems _ ,pages 31113119 , 2013 .b. pang and l. lee .seeing stars : exploiting class relationships for sentiment categorization with respect to rating scales . in _ proceedings of the 43rd annual meeting on association for computational linguistics _ , pages 115124 .association for computational linguistics , 2005 .f. pedregosa , g. varoquaux , a. gramfort , v. michel , b. thirion , o. grisel , m. blondel , p. prettenhofer , r. weiss , v. dubourg , j. vanderplas , a. passos , d. cournapeau , m. brucher , m. perrot , and e. duchesnay .scikit - learn : machine learning in python ., 12:28252830 , 2011 .r. socher , e. h. huang , j. pennin , c. d. manning , and a. y. ng .dynamic pooling and unfolding recursive autoencoders for paraphrase detection . in _ advances in neural information processing systems _ , pages 801809 , 2011 .g. tsatsaronis , g. balikas , p. malakasiotis , i. partalas , m. zschunke , m. r. alvers , d. weissenborn , a. krithara , s. petridis , d. polychronopoulos , et al .an overview of the bioasq large - scale biomedical semantic indexing and question answering competition ., 16(1):1 , 2015 .

Abstract: we investigate the integration of word embeddings as classification features in the setting of large scale text classification . such representations have been used in a plethora of tasks , however their application in classification scenarios with thousands of classes has not been extensively researched , partially due to hardware limitations . in this work , we examine efficient composition functions to obtain document - level from word - level embeddings and we subsequently investigate their combination with the traditional one - hot - encoding representations . by presenting empirical evidence on large , multi - class , multi - label classification problems , we demonstrate the efficiency and the performance benefits of this combination .


clustering plays a basic role in many parts of data engineering , pattern recognition and image analysis .one of the most important is gaussian mixture models .it is hard to overestimate the role of gmm in computer science , including object detection , object tracking , learning and modelling , feature selection , classification or statistical background subtraction .gmm accommodates data of varied structure , e.g. the component distributions can concentrate around surfaces of lower dimension obtained by principal components ( pca ) .however , it often happens that clusters are concentrated around lower dimensional manifolds which are not linear .since one non - gaussian component can often be approximated by several gaussian ones , these clusters are in practice represented by introducing more gaussian components which can be seen as a form of piecewise linear approximation , see fig .[ fig : acacec ] . due to the intrinsic linearity of the gaussian model , when there are nonlinear manifolds in the data cloud , it is natural that many components are required and the fitting error is large .consequently , the constructed model does not reflect optimally the internal structure of the data .a similar result gives cross entropy clustering approach , compare fig [ fig : letterb1 ] and [ fig : letterb2 ] .there are several methods attempting to solve the problem of fitting nonlinear manifolds , e.g. principal curves and principal surfaces .principal curves / surfaces algorithms are typically capable of expressing a single complex manifold . in authors present an adaptation of the gaussian mixture model called active curve axis gaussian mixture models ( acagmm ) , which uses a nonlinear curved gaussian probability model in clustering . in its basic version it works with data on the plane andadapts to the quadratic curves . in other wordsacagmm uses a wider class then typical gaussians namely gaussians which are curved over parabolas .since our paper aims at solving the same task as acagmm , let us first explain the method and present the typical steps behind it .first , using an additional tool , the authors find the `` right '' number of clusters ( one of the possible methods is given in , however , one can also use ) .then for each cluster the pca algorithm is applied to determine the reasonable basis , and a gaussian curved along the optimal parabola is used .the coordinate system is nonlinear , see fig .[ fig : letterb3 ] ( the coordinate is chosen as a distance from the parabola , and is the length on the parabola from the projected point to the parabola s vertex ) .acagmm has found applications in particularly in human hand motion recognition .it can also be fuzzified .[ fig : letterb1 ]  [ fig : letterb2 ]  [ fig : letterb3 ]  [ fig : letterb4 ]   acagmm works well in practice , however , it has some limitations .the model is naturally restricted to quadratic functions as the nonlinear coordinate system requires the projection onto the graph and length of the curve .the use of the method in higher dimensional case , although possible , is practically rather limited .moreover , acagmm is not a theoretically based density model ( see appendix for the detailed explanation ) , and therefore it is not in fact formally em based , but only uses its optimization algorithm .consequently , contrary to the classical em , the mle cost function does not necessarily decrease with iterations .let us recall that in general em aims at finding , and gaussian densities ( where is given beforehand and denotes the number of densities which convex combination builds the desired density model ) such that the convex combination optimally approximates the scatter of our data with respect to mle cost function the em procedure consists of the expectation and maximization steps .while the expectation step is relatively simple , the maximization usually needs complicated numerical optimization even for relatively simple gaussian models .     in this paperwe propose the method which is based on the cec model , instead of the expectation maximization ( em ) and gaussian density model in a curvilinear coordinate system . a goal of cec is to minimize the cost function , which is a minor modification of that given in by substituting sum with maximum : instead of focusing on the density estimation as its main task , cec aims itself directly to the clustering problem .it occurs that at the small cost of minimally worse density approximation we gain speed in implementation and the ease of using more complicated density models .roughly speaking , the advantage is obtained because models do not mix with each other , since we take the maximum instead of sum .consequently , we are able to construct an algorithm which is easy to adapt to the higher dimensional case . the results of and acagmm are similar on the plane , compare fig . [fig : letterb3 ] and fig .[ fig : letterb4 ] .the effect of our algorithm in on a shark - type set is shown in fig .[ fig:3d_1 ] .the method is able to reduce unnecessary clusters . in fig .[ fig : dog ] we present a convergence process of with initial number of clusters , which is reduced to .  , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ] , which is reduced to .,title="fig:",width=81 ]  this paper is arranged as follows . in the next section the theoretical background of the density model will be presented .since acagmm works in only for parabolas we start with a similar situation .then we describe a general model for data in . in the third chapter we present the theoretical background of the method . in particular , we prove that the cost function decreases in every iteration , see theorem [ the : ener ] .the last chapter presents numerical experiments . in appendixwe include details of the description of the acagmm model .in this section , the -adapted gaussian distribution , where is a continuous function , will be presented .the goal of this approach is to transform a normal distribution ( which assumes the intrinsic linearity of the model ) to the case of curves ( or more generally to manifolds ) , which are given by the graph of the function  .the above model will be used in the method .since acagmm works in the two - dimensional case ( in higher dimensional ones the authors use pca to reduce problems to 2d ) with parabolas ( for ) , we start from comparison acagmm and our model in such a case .let for be given .the two dimensional gaussian density for  ] be given .the acagmm approach uses the orthogonal projection of the point onto the parabola which is denoted by and the arc length between and which is denoted by . consequently the acagmm function is given by   this approach is very intuitive but it causes two basic problems .it is very hard ( or even impossible ) to give explicit formulas for orthogonal projection and arc length for more complicated curves in higher dimensional spaces .calculations are complicated ( from the numerical point of view ) , consequently the field of possible generalizations of acagmm is limited . moreover , the function which was used in acagmm , see formula ( [ acagmm_fun ] ) , is not a density .the jacobian of the respective transformation was not included ( see appendix ) .in our paper we use a simpler approach , which is based on the euclidean norm and the following formula for the density function : ) = n(m_1,\sigma_1 ^ 2)(x_1)\cdot n(m_2,\sigma_2 ^ 2)(x_2-f(x_1)).\ ] ] since we do not use orthogonal projection and arc length , it is easy to calculate the parameters of our generalized gaussian distribution , see fig .[ fig : el ] .the practical difference in between acagmm and our approach is quite small instead since our method does not apply the change of coordinates given by pca .] , see fig .[ fig : one_claster ] . nevertheless , our model is more flexible , as we can use an arbitrary class of functions for which least squares methods work .[ fig : fu1 ]   + [ fig : fu2 ]    in this subsection , the general notion of -adapted gaussian will be presented .let us recall that the standard gaussian density in is defined by where denotes the mean , is the covariance matrix and is the square of the mahalanobis norm .  in our workwe use a multidimensional gaussian density in a curvilinear coordinate system which is spread along the function ( -adapted gaussian density ) .we treat one of the variables ( for simplicity , the last one ) separately . in such a casewe consider only those ( where denotes the set of -dimensional square matrices ) which have the diagonal block matrix form where and . for and we will use the notation for , we denote the set containing vectors from with removed coordinate , and for a function , we denote   let , , , be given . the -adapted gaussian density for , and is defined as follows   level sets for -adapted gaussian distributions with different types of functions are presented in fig . [fig : e ] .the -adapted gaussian function , where ) , , , is a density .let be a -dimensional gaussian density such , that where , , .let us consider a substitution in such a case , the jacobian is equal to consequently , is a density .we will use the family of all -dimensional gaussian densities .moreover , for , we will consider family of -adapted gaussian functions for the family , we define we show that if contains all linear transformations , then .let us start with simple lemma .[ lem : fam ] let , , and be given .then for we have where such , that .let us denote and  ] _ obtain new _ clustering by matching elements to the cluster such that ^t , \sigma_{\hat i } , \sigma_{i } ) ) \right ) ]in this section we present a comparison of the method with acagmm , gmm and cec . since acagmm is not a density model , the log - likelihood function is not well - defined . nevertheless , by the input the jacobian of acagmm transformation , we obtain a valid probability distribution , see appendix .  [fig : cir1 ]  [ fig : cir2 ]  [ fig : cir3 ]  [ fig : cir4 ]  + [ fig : cir5 ]  [ fig : cir6 ]  [ fig : cir7 ]   to compare the results we use the standard bayesian information criterion ( bic ) and akaike information criterion ( aic ) where is a number of parameters in the model , is a number of points , and is a maximized value of the log - likelihood function .consequently , we need a number of parameters which are used in each model . in a case of , acagmm uses two scalars for mean , three scalars for covariance matrix , two scalars for parabola and one for local coordinate system ( obtained by pca ) . on the other hand , in wedo not need scalar for the local coordinate system .consequently , uses two scalars for mean , three scalars for covariance matrix and two scalars for parabola .let us start from a synthetic data set .first , we report the results of afcec , acagmm , cec and gmm in the case of a circle - type set , see fig . [fig : cir ] . fig .[ fig : cir5 ] shows how the log - likelihood function changes when the number of clusters increases from to .similar relation , in respect to number of parameters , is presented in fig .[ fig : cir6 ] . for a similar values of log - likelihood function, we need 2 clusters in and acagmm and 4 in gmm and cec , see fig .[ fig : cir ] .in such a case , the bic criterion shows that algorithms which use curved densities model better fit data with using smaller number of parameters .[ fig : sp1 ]  [ fig : sp2 ]  [ fig : sp3 ]  [ fig : sp3 ]   [ fig : sp5 ]  [ fig : sp6 ]  [ fig : sp7 ]   similar situation can be observed in a more complex case of spiral - type set , see fig .[ fig : sp ] . in table[ tab : real_datasets_ll ] , the mean and maximum value of log likelihood for 100 initializations of algorithms are shown .as we see for a similar values of log - likelihood function , we have to use 9 clusters for and acagmm and 14 for gmm and cec . the comparison of algorithms by using bic and aic with similar values of log - likelihood function we present in tab .[ tab : spirala_datasets ] .algorithms which are able to adapt to curve type structures ( acagmm , ) better fit data .more precisely , the log - likelihood function takes a larger value with the same number of parameters , see fig .[ fig : cir6 ] and fig .[ fig : sp6 ] . since log - likelihood increases with growing of the number of classes , we use bic criterion which takes into account the number of parameters . in the case of acagmm and afcec, we obtain optimal value of bic after about 4 - 6 iterations . in conclusion , acagmm and better fit data ( yield a higher value of log - likelihood function ) while require lower number of parameters .algorithms acagmm and give a comparable value of log - likelihood , see fig .[ fig : cir5 ] and fig .[ fig : sp5 ]. nevertheless , uses less parameters , see fig .[ fig : cir6 ] and fig .[ fig : sp6 ] .moreover , strong theoretical background of the method guarantees that the cost function decreases in each iteration .consequently , we obtain a simple stop condition for our method ..comparison of afcec , acagmm , cec and gmm in the case of spiral - type set , see fig .[ fig : sp ] . [ cols="<,^,^,^,^,^ " , ]   chinese characters mainly consist of straight - line strokes ( horizontal , vertical ) and curve strokes ( slash , backslash and many types of hooks ) .gmm has already been employed for structure analysis of chinese characters , and achieves commendable performance . however , some lines extracted by gmm may be too short and is quite difficult to join these short lines to form semantic strokes due to the ambiguity of joining .this problem becomes more serious when analyzing handwritten characters by gmm , and this was the motivation to use acagmm to represent chinese characters . in tab .[ tab : real_datasets ] , we present a comparison of afcec , acagmm , gmm and cec on chinese and latin characters :  utf8gbsn ( dog ) , ( beg ) , ( father ) , ( mother ) , ( fire ) , ( master ) , b , r , s. the number of clusters has been determined so as to obtain a similar value of log - likelihood function .as it was previously mentioned , acagmm does not use densities .more precisely , the jacobian of the transformation was not taken into consideration .however , the em procedure , which was used in acagmm , works with probability distributions .therefore , from the theoretical point of view the above procedure is incorrect .moreover , if we want to compare our method by using of the log - likelihood function we need densities .let us start from numerical integration of the original acagmm function and of the model rescaled by jacobian correction .the simpson method , on the square \times [ -5,5]$ ] with segments was used .the integral in the case of acagmm is equal to .after correction we obtain ( with a precision of ) .let us consider situation of the acagmm model .suppose and are zero mean independent gaussian distributions with variances : moreover , let where .let represent the jacobian of the original transformation in such a case , we have   let us consider the function expressed as parametric equation ( in the case of acagmm it is a parabola ) . using the formula from (* table 1 . )we obtain the orthogonal projection of point on curve : {r + \sqrt{d } } + \sqrt[3 ] { r - \sqrt{d } } & d>0  \\ 0 , \ 0 , \ 0 & d = 0 , q = r=0 \\ 2\sqrt{-q } , \-\sqrt{-q } , \ -\sqrt{-q } & d = 0 , q \neq 0 , r \neq 0 \\ 2\sqrt{-q } \cos(\frac{\phi+2i\pi}{3 } ) , i=0,1,2 , & d<0 \\\mbox { where } \phi = acos \left(\frac{r}{\sqrt{-q^3}}\right ) &  \end{array } \right.\ ] ] where  and   on the other hand , the arc length of between zero and ( * ? ? ?* formula ( 10 ) ) is given by consequently , we have  where .( 1.5,1 ) ( 1.5,0 ) ( 2.5,0 ) ; plot ( , ( 1*(-1.5)^2 ) ;  ( 1.5 + 0.4,0.6 ) circle [ radius=0.02 ] ; at ( 1.5 + 0.4,0.6+.1 ) ; ( 1.5 + 0.6416396472194656,0.6416396472194656 ^ 2 ) circle [ radius=0.02 ] ; at ( 1.8 + 0.6416396472194656,0.6416396472194656 ^ 2 ) ; ( 1.5 + 0.6416396472194656,0.6416396472194656 ^ 2 ) ( 1.5 + 0.4,0.6 ) ; at ( 1.98,0.45 ) ; at ( 1.8,0.2 ) ;  ( 0,1 ) ( 1,0 ) ; plot ( , ( 0 ) ;  ( 0.7889957113463548,0.3063430560334737 ) circle [ radius=0.02 ] ; at ( ( 0.7889957113463548,0.3063430560334737+.1 ) ; ( 0.7889957113463548,0 ) circle [ radius=0.02 ] ; ( 0.7889957113463548,0.3063430560334737 ) ( 0.7889957113463548,0 ) ; at ( .74,0.15 ) ; at ( .4,0.1 ) ; ( 0.7,1.1)(2.1,1.1 ) ; at ( ( 1.4,1.2 ) ;  our goal is to determine the jacobian of our transformation , see fig .[ fig : acagmmfix ] .let us consider an arbitrary small neighborhood of .in such a case , the local curvature of at is the same as the curvature of the osculating circle at .the radius of curvature in the case of parametric form of curve is given by consequently , our goal is to determinate how a set is changing under the influence of the transformation , see fig . [fig : acagmmfix1 ] .a small square neighborhood of the point is mapped to a trapezoid ( asymptotically when a size of square converges to zero ) .this operation is showed in fig .[ fig : acagmmfix1 ] .it is easy to see that the square area changes linearly depending on the distance .if we consider the situation where , we obtain that our square is collapsed to a point . consequently , for points above the curve jacobianis asymptotically proportional to in a natural way , if a point is under the curve , the square area is increasing under the influence of the transformation .therefore , the jacobian is asymptotically equal to   plot ( , ( 1*(-1.5)^2 ) ; ( 2,0.5 ^ 2 ) ( 1.5 - 1.05666,1.7351 ) ; ( 2.28,0.78 ^ 2 ) ( 1.5 - 1.05666,1.7351 ) ;  ( 1.5 + 0.4,0.6 ) circle [ radius=0.02 ] ; at ( 1.7,0.8 ) ; at ( 1.5,1 ) ; at ( 2,0.4 ) ; ( 1.5 + 0.6416396472194656,0.6416396472194656 ^ 2 ) circle [ radius=0.02 ] ; ( 1.5 + 0.6416396472194656,0.6416396472194656 ^ 2 ) ( 1.5 + 0.4,0.6 ) ; ( 1.5 + 0.6416396472194656,0.6416396472194656 ^ 2 ) ( 1.5 - 1.05666,1.7351 ) ; plot ( , ( -sqrt(2.15305 ^ 2-(-(1.5 - 1.05666))*(-(1.5 - 1.05666 ) ) ) + 1.7351 ) ; plot ( , ( -sqrt(1.945 ^ 2-(-(1.5 - 1.05666))*(-(1.5 - 1.05666 ) ) ) + 1.7351 ) ; plot ( , ( -sqrt(1.945 ^ 2-(-(1.5 - 1.05666))*(-(1.5 - 1.05666 ) ) ) + 1.7351 ) ;  plot ( , ( -sqrt(1.745 ^ 2-(-(1.5 - 1.05666))*(-(1.5 - 1.05666 ) ) ) + 1.7351 ) ; plot ( , ( -sqrt(1.745 ^ 2-(-(1.5 - 1.05666))*(-(1.5 - 1.05666 ) ) ) + 1.7351 ) ;  ( 1.7,0.524385 ) ( 1.85,0.391846 ) ; ( 1.935,0.829575 ) ( 2.1,0.716031 ) ; plot ( , ( 0 ) ; ( -0.8 - 0.5,0 ) ( 0.4 - 0.5,0 ) ; ( -0.4 - 0.5,0 ) ( -0.4 - 0.5,2.15305 ) ; ( 0 - 0.5,0 ) ( 0 - 0.5,2.15305 ) ;  ( -0.4 - 0.5,0.3063430560334737 + 0.11 ) ( 0 - 0.5,0.3063430560334737 + 0.11 ) ;( -0.4 - 0.5,0.3063430560334737 - 0.11 ) ( 0 - 0.5,0.3063430560334737 - 0.11 ) ; ( -0.4 - 0.5,0.3063430560334737 + 0.11 ) ( -0.4 - 0.5,0.3063430560334737 - 0.11 ) ; ( 0 - 0.5,0.3063430560334737 - 0.11 ) ( 0 - 0.5,0.3063430560334737 + 0.11 ) ;  ( -0.8 - 0.5,0.3063430560334737 + 0.11 ) ( 0.4 - 0.5,0.3063430560334737 + 0.11 ) ; ( -0.8 - 0.5,0.3063430560334737 - 0.11 ) ( 0.4 - 0.5,0.3063430560334737 - 0.11 ) ; at ( -0.8,0.1 ) ; at ( -0.8,1.2 ) ; at ( -0.7,0.5 ) ; ( 0.7889957113463548 - 1.5,0.3063430560334737 ) circle [ radius=0.02 ] ; ( 0.7889957113463548 - 1.5,0 ) circle [ radius=0.02 ] ; ( 0.7889957113463548 - 1.5,0 ) ( 0.7889957113463548 - 1.5,0.3063430560334737 ) ; ( 0.7889957113463548 - 1.5,2.15305 ) ( 0.7889957113463548 - 1.5,0.3063430560334737 ) ;  ( 0,9 ) ( 0,0 ) ( 5,0 ) ; ( 0,-1 ) ( 0,0 ) ( -5,0 ) ; plot ( , ( ) ^2 ) ;  ( 4,4 ) circle [ radius=0.1 ] ; at ( 4,4.5 ) ;  ( -1.459261,2.129444 ) circle [ radius=0.1 ] ; ( -1.459261,2.129444 ) ( 4,4 ) ;  ( -0.6498321,0.4222817 ) circle [ radius=0.1 ] ; ( -0.6498321,0.4222817) ( 4,4 ) ;  ( 2.109093,4.448275 ) circle [ radius=0.1 ] ; ( 2.109093,4.448275) ( 4,4 ) ;  ( 0,6 ) circle [ radius=0.1 ] ; at ( 0,6.5 ) ;  ( 2.345208,5.5 ) circle [ radius=0.1 ] ; ( 2.345208,5.5 ) ( 0,6 ) ;  ( -2.345208,5.5 ) circle [ radius=0.1 ] ; ( -2.345208,5.5 ) ( 0,6 ) ;  ( 0,0 ) circle [ radius=0.1 ] ; ( 0,0 ) ( 0,6 ) ;  ( -4,1 ) circle [ radius=0.1 ] ; at ( -5,1.5 ) ;  ( -1.391769,1.93702 ) circle [ radius=0.1 ] ; ( -1.391769,1.93702 ) ( -4,1 ) ;  now we have the formula for the jacobian of acagmm transformation , but it depends on the relation between a point and its orthogonal projection . more precisely , we have to verify which formula should be used ( or equivalently on which side of parabola a point is found ) , see fig . [fig : acagmmfix2 ] .we can easily verify where the point is in relation to the orthogonal projection by checking the orientation of a basis containing the normal vector and the tangent vector at a point .consequently , we have to verify the sign of the determinant study is cofounded by the european union from resources of the european social fund. project po kl `` information technologies : research and their interdisciplinary applications '' , agreement uda - pokl.04.01.01 - 00 - 051/10 - 00 .  45 natexlab#1#1[2]#2 , , , ., , , , . , , , , ( ) ., , , volume , , ., , , , ( ) . , , , in : , volume , , pp . ., , , , , ( ) . , , , ( ) ., , in : , volume , , pp . ., , , ( ) ., , , , ( ) . , , , , , in : , , p. ., , , ( ) . , , , , ( ) ., , , in : , volume , , pp . ., , , , , ( ) . , , , , , , , ( ) ., , , in : , volume , ., , , in : , , pp . . , , , , in : , volume , , pp . ., , , ( ) . , , ph.d .thesis , citeseer , ., , , ( ) ., , , ( ) . , , , , ( ) ., , , , ( ) . , , , ( ) . , , , ( ) ., , , ( ) . , , , volume , , . , , , , ( ) ., , , , ( ) . , , , ( ) ., , , ( ) . , , , , , ., , , , ( ) . , , , ., , , , volume , , .

Abstract: gaussian mixture models ( gmm ) have found many applications in density estimation and data clustering . however , the model does not adapt well to curved and strongly nonlinear data . recently there appeared an improvement called acagmm ( active curve axis gaussian mixture model ) , which fits gaussians along curves using an em - like ( expectation maximization ) approach .  using the ideas standing behind acagmm , we build an alternative active function model of clustering , which has some advantages over acagmm . in particular it is naturally defined in arbitrary dimensions and enables an easy adaptation to clustering of complicated datasets along the predefined family of functions . moreover , it does not need external methods to determine the number of clusters as it automatically reduces the number of groups on - line .   clustering , gaussian mixture models , expectation maximization , cross - entropy clustering , active curve axis gaussian mixture model .


one reason why network science has recently attracted significant research attention is that network structure efficiently encodes the complexity of a large variety of systems , from the brain to different techno - social infrastructures . in the last fifteen years orso there has been significant progress in characterizing not only universal properties of complex networks , e.g. , scale - free degree distributions or small - world properties , but also their specific features that distinguish one network from another degree correlations , community structure , or motif distributions .more recently , considerable effort has focused on quantifying network complexity using new network entropy measures borrowed from information theory , statistical mechanics , and quantum information .the entropy of a network ensemble evaluates the total number of networks belonging to the ensemble .the more complex , sophisticated , and unique the network structure , the smaller the number of networks in the ensemble having these peculiar properties , the smaller the entropy .the entropy measures have proven useful for solving inference problems involving real - world networks .the statistical mechanics treatment of network ensembles can be used to characterize the likelihood that a real dataset is generated by a model .some real networks have been shown to belong with high likelihood to ensembles of random geometric graphs in hyperbolic spaces , modelling trade - offs between popularity and similarity in network evolution , and casting preferential attachment as an emergent phenomenon .more recently , the entropy of multiplex ensembles has been proposed to characterize the complexity of multilayer networks .  by definition ,a network ensemble is a set of graphs with probability measure .it is important to make a distinction between microcanonical and canonical ensembles . in microcanonical ensembles , some structural network propertiesare fixed to given values .for example , the total number of links in graphs of size can be fixed to , or the degrees of all nodes can be fixed to degree sequence , . in this case , the ensemble consists of all graphs of size , and the probability measure is uniform : if the number of graphs satisfying the constraints is , then for all such graphs , , and for all other graphs that do not satisfy the constraints . in the canonical counterparts of these ensembles ,the same structural constraints are fixed only on average the resulting ensembles are maximum - entropy ensembles under the constraints that the expected values of the number of edges or node degrees in the ensemble are or . the probability measure in this case is not uniform the closer the to satisfying the constraints , the larger the . in random graph ensembles with a fixed exact or expected number of links, the canonical distribution converges to the microcanonical distribution in the thermodynamic limit . however , as soon as the number of imposed constraints is extensive , the canonical and microcanonical network ensembles are not equal even in the thermodynamic limit , and neither are their entropies .for example , the entropy of the microcanonical ensemble with a fixed degree sequence is not equal to the entropy of the canonical ensemble where only the expected degree sequence is fixed .  in network theorythere is usually no question of how precisely we know the node degrees : given a graph , its degree sequence is uniquely defined .however when a network practitioner works with real network data , she is typically given a collection of network measurements .does she have to treat the measured degrees of nodes as precisely defined as well , given that measurements are always imprecise and ever - changing , and so is the network itself ?the answer is usually ` no'the relevant information is not the exact degree sequence , but its statistical properties , such as the distribution of these degrees .the ensembles of networks with a given exact or even expected degree sequence do not account for possible statistical fluctuations of node degrees in a given dataset , motivating us to consider here ensembles of random networks whose exact or expected degree sequences are independently sampled from a given distribution .this approach is a way to explore only the statistical properties of networks , and not their specific linking diagrams that might be affected by false or missing links , almost always present in real data .  specifically , we study the distribution of entropy in network ensembles with a given exact or expected degree distribution .we find that in both cases ( hard / exact and soft / expected ) , if the network ensemble is sparse , the average entropy is well - defined and self - averaging .we also evaluate the probability that ensemble entropy is equal to a particular value , conditioned on the total number of links in the network , and show that this conditional entropy distribution is always well - behaved . characterizing large entropy deviations in the ensemble with a given degree distribution and average degree , we observe that a condensation phenomena can occur in the network .this phenomena occurs only if the average degree in the network exceeds the degree distribution average . for is a symmetry under permutation of the labels of the nodes of the network , meaning that if we fix the average degree of the network , then the degrees of all nodes are , where is the network size . instead ,if , we observe a spontaneous breaking of this symmetry , with a single node having an degree .these results hold in both hard and soft ensembles , i.e. , ensembles with fixed exact or expected degree distributions .we begin with reviewing in section ii what is known about the entropy of network ensembles with a fixed expected or exact degree sequence .we then move to section iii and iv where we analyze some properties of the entropy distributions in the ensemble with a given expected and exact degree distribution , respectively .final remarks are in section v.a network ensemble is specified once probability is assigned to every network of size .we denote nodes by .the set of simple undirected unweighted labeled networks of size is bijective to the set of symmetric boolean adjacency matrices having zeroes on the diagonal . depending on whether nodes and are connected or not in network , element of s adjacency matrix is either or .we next impose the constraint that the degree of each node is fixed to some .we can treat this constraint as hard or soft .if it is hard , we impose it exactly .the resulting ensemble is a microcanonical network ensemble with given degree sequence , known as the configuration model . in the soft case, we relax the constraint , and demand that the degree of each node , averaged over all networks in the ensemble , is , which no longer has to be integer but can be any non - negative real number .the resulting ensemble is a canonical network ensemble with a given expected degree sequence , belonging to the class of random graphs known as exponential random graphs . in what followswe denote by the probability of in the canonical ( ) or micro - canonical ( ) ensembles .the entropy of these ensemble evaluates the typical number of networks in the ensemble and is given by s=-_gp_e(g|\{k_i})p_e(g|\{k_i } ) , [ entropy0 ] where the sum is performed over all networks in the ensemble .the probability distribution in the canonical ensemble is defined as the distribution that maximizes entropy s(\{k_i})=-_gp_c(g|\{k_i})p_c(g|\{k_i } ) , [ entropy ] subject to the following constraints : by summing over all networks , we sum over their adjacency matrices . introducing lagrangian multipliers to enforce the conditions in eq . , and lagrangian multiplier to normalize the probability measure , we solve the system of equations =0\end{aligned}\ ] ] to find the expression for distribution : \label{pc}\ ] ] where the normalization constant z_c = e^-=_gis called the `` partition function . ''since the probability distribution in eq . has an exponential form , this ensemble is called _ exponential random graphs_.  in this ensemble , we can relate entropy in eq . to partition function : \nonumber \\ & = & \sum_{i=1}^n \lambda_{i } k_i+\log{z_c}. \label{sc}\end{aligned}\ ] ] we call the entropy of the canonical ensemble the _ shannon entropy_.  the probability of a link between node and node in the ensemble is given by p_ij=&= & + & = & , [ pijc ] + where are called `` hidden variables '' . upon this change of variables , the constraints in eq . translate to k_i=_jip_ij=_ji .[ kpij ] this system of equations can be solved for yielding the values of lagrangian multipliers .using in eq . , a simpler expression for distribution reads p_c(g|\{k_i})=_ijp_ij^a_ij(1-p_ij)^1-a_ij .[ eq : p_c(pij ) ] therefore probability is actually the probability to generate network with hidden variables by connecting node pairs and with probability given by eq .( [ pijc ] ) , and not connecting them with probability . using these link existence probabilities , the entropy of the ensemble in eq . can be written as s(\{k_i})=-_i <j[p_ijp_ij+(1-p_ij)(1-p_ij ) ] .[ s ] by inserting the explicit dependence of probabilities on hidden variables , we can extract the leading term of the entropy that depends only on the average degree in the network , and the subleading term that increases linearly with : s(\{k_i})=s(\{k_i})-n(\{k_i } ) , [ s2 ] where and in any sparse network are given by ( \{k_i})&=&nn , + n(\{k_i})&=&_i < j[p_ij(n p_ij)+(1-p_ij)(1-p_ij ) ] + & = & .[ sigma ] if all expected degrees , where is the average expected degree , then hidden variables are proportional to expected degrees , , and we can approximate probabilities in eq . by p_ij==. which corresponds to the case where links in the network are uncorrelated .  in this case the expression for the extensive entropy term in eq . simplifies to ( \{k_i})&=&_i k_i k_i-[1 + ] .   in the microcanonical ensemble ,all networks satisfy the hard constraint that the degree sequence is exactly .we assume here that the degree sequence is graphical , meaning that it can be realized by at least one network .this condition is obviously satisfied if the degree sequence is read off from a real network .the probability distribution in the ensemble is uniform all networks satisfying this constraint have the same probability \label{pm}\ ] ] where  ] ._ + for large and  ] .the saddle point eqs .( [ sp ] ) have a solution only if the average degree is equal to or less than the expected degree of the degree distribution , i.e. only if m = ndh dh ( h)(h ) p(h , h ) .in fact the lagrangian multipliers must be real and greater than zero to guarantee that given by eq .( [ sp ] ) is well defined .following , to explore large deviation properties of a fat - tailed distribution , we use the following ansatz : n(h)=(n-1)_c(h)+(h , h_c ) .this ansatz accounts for a spontaneous breaking of permutation symmetry between the hidden variables in the ensemble , and reflects our expectation to detect condensation in some large - deviation realization in the ensemble . with this ansatz ,probability becomes p(s|)=ddd(h ) e^g(_c , h_c , , , ) , [ ps1b ] with & & g(_c , h_c , , , ) = -ndh _ c(h ) + & & - + & & -i n^2dh dh _c(h ) [ s(h , h)+p(h , h ) ] + & & -i 2 n^2 dh _c(h ) [ s(h_c , h)+p(h_c , h ] ) + & & -in dh + in+is , [ gb ] where functions and are as in eqs .( [ shhphh ] ) .the problem of minimizing function with respect to all its parameters has a non - trivial solution only if , in which case we have n = mn+2n^2dh _ c(h ) p(h_c , h ) with , and for any . in figure we show the phase diagram , where is the exponent of the hidden variable distribution , and  ] and .,title="fig:",width=297 ]the results presented in the previous section concerning the soft ensembles remain qualitatively unchanged if we consider the hard ensembles of networks with a given degree distribution of exact degrees .note that we are treating here always quenched disorder .in fact here we consider ensembles of networks of fixed degree sequence , where each degree sequence is drawn randomly from a given degree distribution . as in the soft case , in this hard casewe assume that the disorder is quenched , and that the exact degree sequence is fixed but unknown and drawn from degree distribution .the probability of degree sequence is thus , and for each we consider the microcanonical ensemble of networks with the fixed sequence of exact degrees . in this ensemble ,network has probability defined in eq .( [ pm ] ) , so that the probability distribution in the ensemble is p_m(g)= p(\{k_i } ) p_m(g|\{k_i } ) .given degree sequence , and using eq .( [ sso ] ) and eq .( [ s2 ] ) , the gibbs entropy is the sum of three contributions , n(\{k_i})&=&s(\{k_i})-n(\{k_i } ) + & = & s(\{k_i})-n(\{k_i})-n(\{k_i } , where the leading term of is . in what followswe analyze the entropy distribution in the ensemble p(n)= p(\{k_i } ) .if we can not compute the full distribution exactly , we may still characterize its average , variance , and relative error & = & p(\{k_i } ) n(\{k_i } ) + & = & _ i=1^ndk_i p(\{k_i } ) ^2 , + & = & .we first consider the probability distribution of the leading term of entropy , defined as p(s)= _i=1^n p(k_i ) , where depends only on the average degree in the network .according to the generalized central limit theorem , and similarly to the soft case , we have the following two cases :  * _ case _ : + the distribution converges to a gaussian distribution and the relative error on the average of is vanishing in the large network limit .in fact we find that ( \{k_i})n^-1/2 . *_ case  ] between distributions and .the saddle point eqs .( [ spb ] ) have a solution only if the average degree is equal to or less than the expected degree over the degree distribution , < m = ndh dh ( h)(h ) p(h , h ) .in fact the lagrangian multipliers must be real and greater than zero to guarantee that given by eq .( [ spb ] ) is well defined .therefore , following the same logic as in the soft case , we use the following ansatz : n(h)=(n-1)_c(h)+(h , h_c ) , with this this ansatz , probability becomes p(n|)=ddd(h ) e^g(_c , h_c , , , ) , with & & g(_c , h_c , , , ) = -ndh _ c(h ) + & & - + & & -i n^2dh dh _c(h ) [ s(h , h)+p(h , h ) ] + & & + i n dh _c(h ) ( ) + & & -i 2 n^2 dh _c(h ) [ s(h_c , h)+p(h_c , h ] ) + & & + i n ( ) + & & -indh + in+in , [ gbb ] where functions , , and are as in eqs .( [ shhphhb ] ) .the problem of minimizing function with respect to all its parameters , has a non - trivial solution only if , in which case we have n = mn+2n^2dh _ c(h ) p(h_c , h ) with , and for any .the phase diagram of with and for is the same as in the soft case shown in figure [ fig1 ] .if , we can observe condensation a single node in the network can acquire a degree of the order of .motivated by the observation that in modeling real networks , the degree distribution is a more reasonable and realistic constraint than the degree sequence , we have studied the entropy distribution in the ensembles of random networks with a given degree distribution .we found that entropy is self - averaging , thanks to the structural degree cutoff at .the fluctuations of entropy are mainly determined by the fluctuations of the average degree in the ensembles .networks with average degree exceeding a certain threshold , , exhibit large deviation or condensation effects a single node can attract links .interestingly , this condensation is different from the bose - einstein condensation in complex networks in that the condensation considered here corresponds only to some `` large deviation configurations . ''it is not typically expected in the ensemble .we thank m. marsili and m. ostilli for interesting discussions .this work was supported by nsf grants no .cns-1344289 , cns-1039646 , and cns-0964236 ; darpa grant no. hr0011 - 12 - 1 - 0012 ; and by cisco systems .    , physics reports * 486 * , 75(2010 ) .r. milo , s. shen - orr , s. itzkovitz , n. kashtan , d. chklovskii , u. alon , science * 298 * , 824 ( 2002 ) .g. bianconi and a. capocci , phys .90 , 078701 ( 2003 ) .g. bianconi and m. marsili , epl * 74 * , 740 ( 2006 ) .m. ostilli , phys .e * 89 * , 022807 ( 2014 ) . , 28005 ( 2008 ) .m. molloy and b. reed , random structures algorithms , * 6 * , 161 ( 1995 ) .f. chung and l. lu , internet mathematics , * 1 * , 91 ( 2004 ) .g. caldarelli , a. capocci , p. de los rios , m. a. munz , phys .89 * , 258702 , ( 2002 ) .m. boguna , and r. pastor - satorras , phys .lett . * 68*,036112 ( 2003 ) .  ,e10012 ( 2010 ) . , 178701 ( 2011 ) . .bouchaud and m. potters , _ theory of financial risk and derivative pricing : from statistical physics to risk management _ , ( cambridge university press , cambridge , 2003 ) .g. bianconi and a .-barabasi , phys .lett . * 86 * , 5632 ( 2001 ) .

Abstract: the entropy of network ensembles characterizes the amount of information encoded in the network structure , and can be used to quantify network complexity , and the relevance of given structural properties observed in real network datasets with respect to a random hypothesis . in many real networks the degrees of individual nodes are not fixed but change in time , while their statistical properties , such as the degree distribution , are preserved . here we characterize the distribution of entropy of random networks with given degree sequences , where each degree sequence is drawn randomly from a given degree distribution . we show that the leading term of the entropy of scale - free network ensembles depends only on the network size and average degree , and that entropy is self - averaging , meaning that its relative variance vanishes in the thermodynamic limit . we also characterize large fluctuations of entropy that are fully determined by the average degree in the network . finally , above a certain threshold , large fluctuations of the average degree in the ensemble can lead to condensation , meaning that a single node in a network of size  can attract links .